No
BOOLEAN
1712.05608
Introduction </s> Deep Ġneural Ġnetworks Ġ( D NN s ), Ġin Ġparticular Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks , Ġwith Ġhuge Ġarchitectures Ġhave Ġbeen Ġproven Ġsuccessful Ġin Ġwide Ġrange Ġof Ġtasks Ġincluding Ġaudio Ġprocessing Ġsuch Ġas Ġspeech Ġto Ġtext Ġ[ 1 Ġ- Ġ4 ], Ġemotion Ġrecognition Ġ[ 5 Ġ- Ġ8 ], Ġspeech / non - speech Ġ( e . g ., Ġof Ġnon - speech Ġinclude Ġnoise , Ġmusic , Ġetc ., ) Ġclassification Ġ[ 9 Ġ- Ġ12 ], Ġetc . </s> Training Ġthese Ġdeep Ġarchitectures Ġrequire Ġlarge Ġamount Ġof Ġannot ated Ġdata , Ġas Ġa Ġresult , Ġthey Ġcannot Ġbe Ġused Ġin Ġlow Ġdata Ġresource Ġscenarios Ġwhich Ġis Ġcommon Ġin Ġspeech - based Ġapplications Ġ[ 13 Ġ- Ġ15 ]. ĠApart Ġfrom Ġcollecting Ġlarge Ġdata Ġcorpus , Ġannot ating Ġthe Ġdata Ġis Ġalso Ġvery Ġdifficult , Ġand Ġrequires Ġmanual Ġsupervision Ġand Ġefforts . ĠEspecially , Ġannotation Ġof Ġspeech Ġfor Ġtasks Ġlike Ġemotion Ġrecognition Ġalso Ġsuffer Ġfrom Ġlack Ġof Ġagreement Ġamong Ġthe Ġannot ators Ġ[ 16 ]. ĠHence , Ġthere Ġis Ġa Ġneed Ġto Ġbuild Ġreliable Ġsystems Ġthat Ġcan Ġwork Ġin Ġlow Ġresource Ġscenario . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġnovel Ġapproach Ġto Ġaddress Ġthe Ġtask Ġof Ġclassification Ġin Ġlow Ġdata Ġresource Ġscenarios . ĠOur Ġapproach Ġinvolves Ġsimultaneously Ġconsidering Ġmore Ġthan Ġone Ġsample Ġ( in Ġthis Ġwork , Ġtwo Ġsamples Ġare Ġconsidered ) Ġto Ġtrain Ġthe Ġclass ifier . ĠWe Ġcall Ġthis Ġapproach Ġas Ġsimultaneous Ġtwo Ġsample Ġlearning Ġ( s 2 s L ). ĠThe Ġproposed Ġapproach Ġis Ġalso Ġapplicable Ġto Ġlow Ġresource Ġdata Ġsuffering Ġwith Ġdata Ġimbalance . ĠThe Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> Prop osed Ġapproach </s> The Ġs 2 s L Ġapproach Ġproposed Ġto Ġaddress Ġlow Ġdata Ġresource Ġproblem Ġis Ġexplained Ġin Ġthis ĠSection . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠML P Ġ( modified Ġto Ġhandle Ġour Ġdata Ġrepresentation ) Ġas Ġthe Ġbase Ġclass ifier . ĠHere , Ġwe Ġexplain Ġthe Ġs 2 s L Ġapproach Ġby Ġconsidering Ġtwo - class Ġclassification Ġtask . </s> Data Ġrepresentation </s> Consider Ġa Ġtwo - class Ġclassification Ġtask Ġwith ĠIN LINE FORM 0 Ġden oting Ġthe Ġset Ġof Ġclass Ġlabels , Ġand Ġlet ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġbe Ġthe Ġnumber Ġof Ġsamples Ġcorresponding Ġto ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠIn Ġgeneral , Ġto Ġtrain Ġa Ġclass ifier , Ġthe Ġsamples Ġin Ġthe Ġtrain Ġset Ġare Ġprovided Ġas Ġinput - output Ġpairs Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġ- dimensional Ġfeature Ġvector Ġrepresenting Ġthe ĠIN LINE FORM 2 Ġsample Ġcorresponding Ġto ĠIN LINE FORM 3 Ġclass Ġlabel , Ġand ĠIN LINE FORM 4 Ġ, Ġrefers Ġto Ġoutput Ġlabel Ġof ĠIN LINE FORM 5 Ġclass . ĠIn Ġthe Ġproposed Ġdata Ġrepresentation Ġformat , Ġcalled Ġsimultaneous Ġtwo Ġsample Ġ( s 2 s ) Ġrepresentation , Ġwe Ġwill Ġsimultaneously Ġconsider Ġtwo Ġsamples Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġare Ġthe ĠIN LINE FORM 2 Ġ- dimensional Ġfeature Ġvectors Ġrepresenting Ġthe ĠIN LINE FORM 3 Ġsample Ġin ĠIN LINE FORM 4 Ġclass Ġand ĠIN LINE FORM 5 Ġsample Ġin Ġthe ĠIN LINE FORM 6 Ġclass , Ġrespectively ; Ġand ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġrefers Ġto Ġthe Ġoutput Ġlabels Ġof ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġclass , Ġrespectively . </s> H ence , Ġin Ġs 2 s Ġdata Ġrepresentation , Ġwe Ġwill Ġhave Ġan Ġinput Ġfeature Ġvector Ġof Ġlength ĠIN LINE FORM 0 Ġi . e ., ĠIN LINE FORM 1 Ġ, Ġand Ġoutput Ġclass Ġlabels Ġas Ġeither ĠIN LINE FORM 2 Ġor ĠIN LINE FORM 3 Ġ. ĠEach Ġsample Ġcan Ġbe Ġcombined Ġwith Ġall Ġthe Ġsamples Ġ( i . e ., Ġwith Ġ( ĠIN LINE FORM 4 Ġ) Ġsamples ) Ġin Ġthe Ġdataset . ĠTherefore , Ġby Ġrepresenting Ġthe Ġdata Ġin Ġthe Ġs 2 s Ġformat , Ġthe Ġnumber Ġof Ġsamples Ġin Ġthe Ġtrain Ġset Ġincreases Ġto ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġsamples . ĠWe Ġhypothes ize Ġthat Ġthe Ġs 2 s Ġformat Ġis Ġexpected Ġto Ġhelp Ġthe Ġnetwork Ġnot Ġonly Ġto Ġlearn Ġthe Ġcharacteristics Ġof Ġthe Ġtwo Ġclasses Ġseparately , Ġbut Ġalso Ġthe Ġdifference Ġand Ġsimilarity Ġin Ġcharacteristics Ġof Ġthe Ġtwo Ġclasses . </s> Class ifier Ġtraining </s> ML P , Ġthe Ġmost Ġcommonly Ġused Ġfeed Ġforward Ġneural Ġnetwork , Ġis Ġconsidered Ġas Ġthe Ġbase Ġclass ifier Ġto Ġvalidate Ġour Ġproposed Ġs 2 s Ġframework . ĠGenerally , ĠML Ps Ġare Ġtrained Ġusing Ġthe Ġdata Ġformat Ġgiven Ġby Ġeq . ĠIN LINE FORM 0 Ġ. ĠBut Ġto Ġtrain Ġthe ĠML P Ġon Ġour Ġs 2 s Ġbased Ġdata Ġrepresentation Ġ( as Ġin Ġeq . ĠIN LINE FORM 1 Ġ), Ġthe Ġfollowing Ġmodifications Ġare Ġmade Ġto Ġthe ĠML P Ġarchitecture Ġ( re fer Ġto ĠFigure ĠFIG REF 4 Ġ). </s> We Ġhave ĠIN LINE FORM 0 Ġunits Ġ( instead Ġof ĠIN LINE FORM 1 Ġunits ) Ġin Ġthe Ġinput Ġlayer Ġto Ġaccept Ġthe Ġtwo Ġsamples Ġi . e ., ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġsimultaneously . </s> The Ġstructure Ġof Ġthe Ġhidden Ġlayer Ġin Ġthis Ġapproach Ġis Ġsimilar Ġto Ġthat Ġof Ġa Ġregular ĠML P . ĠThe Ġnumber Ġof Ġhidden Ġlayers Ġand Ġhidden Ġunits Ġcan Ġbe Ġvaried Ġdepending Ġupon Ġthe Ġcomplexity Ġof Ġthe Ġproblem . ĠThe Ġnumber Ġof Ġunits Ġin Ġthe Ġhidden Ġlayer Ġis Ġselected Ġempir ically Ġby Ġvarying Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfrom Ġ2 Ġto Ġtwice Ġthe Ġlength Ġof Ġthe Ġinput Ġlayer Ġ( i . e ., Ġ2 Ġto ĠIN LINE FORM 0 Ġ) Ġand Ġthe Ġunit Ġat Ġwhich Ġthe Ġhighest Ġperformance Ġis Ġobtained Ġare Ġselected . ĠIn Ġthis Ġpaper , Ġwe Ġconsidered Ġonly Ġa Ġsingle Ġhidden Ġlayer . ĠRect ified Ġlinear Ġunits Ġ( Re LU ) Ġare Ġused Ġfor Ġhidden Ġlayer . </s> The Ġoutput Ġlayer Ġwill Ġconsist Ġof Ġunits Ġequal Ġto Ġtwice Ġthe Ġconsidered Ġnumber Ġof Ġclasses Ġin Ġthe Ġclassification Ġtask Ġi . e , Ġthe Ġoutput Ġlayer Ġwill Ġhave Ġfour Ġunits Ġfor Ġtwo - class Ġclassification Ġtask . ĠThe Ġs igm oid Ġactivation Ġfunction Ġ( not Ġsoft max ) Ġis Ġused Ġon Ġthe Ġoutput Ġlayer Ġunits . ĠUnlike Ġregular ĠML P , Ġwe Ġuse Ġs igm oid Ġactivation Ġunits Ġin Ġthe Ġoutput Ġlayer , Ġwith Ġbinary Ġcross - ent ropy Ġas Ġthe Ġcost Ġfunction , Ġbecause Ġthe Ġoutput Ġlabels Ġin Ġthe Ġproposed Ġs 2 s Ġbased Ġdata Ġrepresentation Ġwill Ġhave Ġmore Ġthan Ġone Ġunit Ġactive Ġat Ġa Ġtime Ġ( not Ġone - hot Ġencoded Ġoutput ) Ġand Ġthis Ġcondition Ġcannot Ġbe Ġhandled Ġusing Ġsoft max Ġfunction . </s> As Ġcan Ġbe Ġseen Ġfrom ĠFigure ĠFIG REF 4 Ġ, Ġthe Ġoutput Ġlayer Ġin Ġour Ġproposed Ġmethod Ġhas Ġoutputs ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġwhich Ġcorresponds Ġto Ġthe Ġoutputs Ġassociated Ġwith Ġthe Ġinput Ġfeature Ġvectors ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġrespectively . ĠFor Ġa Ġtwo - class Ġclassification Ġproblem , Ġthere Ġwill Ġbe Ġfour Ġunits Ġin Ġthe Ġoutput Ġlayer Ġand Ġthe Ġpossible Ġoutput Ġlabels Ġare ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġcorresponding Ġto Ġthe Ġclass Ġlabels ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠThis Ġarchitecture Ġis Ġreferred Ġto Ġas Ġs 2 s - ML P . ĠIn Ġs 2 s L , Ġs 2 s - ML P Ġis Ġtrained Ġusing Ġthe Ġs 2 s Ġdata Ġrepresentation Ġformat . ĠFurther , Ġthe Ġs 2 s - ML P Ġis Ġtrained Ġusing Ġadam Ġoptim izer . </s> Class ifier Ġtesting </s> Generally , Ġthe Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġis Ġprovided Ġas Ġinput Ġto Ġthe Ġtrained ĠML P Ġin Ġthe Ġtesting Ġphase Ġand Ġthe Ġclass Ġlabel Ġis Ġdecided Ġbased Ġon Ġthe Ġobtained Ġoutput . </s> However , Ġin Ġs 2 s L Ġmethod , Ġthe Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġshould Ġalso Ġbe Ġconverted Ġto Ġthe Ġs 2 s Ġdata Ġrepresentation Ġformat Ġto Ġtest Ġthe Ġtrained Ġs 2 s - ML P . ĠWe Ġpropose Ġa Ġtesting Ġapproach , Ġwhere Ġthe Ġgiven Ġtest Ġsample Ġis Ġcombined Ġwith Ġa Ġset Ġof Ġpre selected Ġreference Ġsamples , Ġwhose Ġclass Ġlabel Ġis Ġknown Ġa Ġprior i , Ġto Ġgenerate Ġmultiple Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġrefer Ġto Ġthe ĠIN LINE FORM 2 Ġ- dimensional Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġand Ġthe ĠIN LINE FORM 3 Ġreference Ġsample , Ġrespectively . ĠIN LINE FORM 4 Ġrefers Ġto Ġthe Ġconsidered Ġnumber Ġof Ġreference Ġsamples . ĠThese Ġreference Ġsamples Ġcan Ġbe Ġselected Ġfrom Ġany Ġof Ġthe Ġtwo Ġclasses . </s> For Ġtesting Ġthe Ġs 2 s - ML P Ġ( as Ġshown Ġin ĠFigure ĠFIG REF 4 Ġ), Ġeach Ġtest Ġsample ĠIN LINE FORM 0 Ġ( same Ġas Ġ' ĠIN LINE FORM 1 Ġ' Ġin Ġ( 3 )) Ġis Ġcombined Ġwith Ġall Ġthe ĠIN LINE FORM 2 Ġreference Ġsamples Ġ( ĠIN LINE FORM 3 Ġ) Ġto Ġform ĠIN LINE FORM 4 Ġdifferent Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample ĠIN LINE FORM 5 Ġ. ĠThe Ġcorresponding Ġoutputs Ġ( ĠIN LINE FORM 6 Ġ) Ġobtained Ġfrom Ġs 2 s - ML P Ġfor Ġthe ĠIN LINE FORM 7 Ġgenerated Ġinstances Ġof ĠIN LINE FORM 8 Ġare Ġcombined Ġby Ġvoting - based Ġdecision Ġapproach Ġto Ġobtain Ġthe Ġfinal Ġdecision ĠIN LINE FORM 9 Ġ. ĠThe Ġclass Ġlabel Ġthat Ġgets Ġmaximum Ġvotes Ġis Ġconsidered Ġas Ġthe Ġpredicted Ġoutput Ġlabel . </s> Exper iments </s> We Ġvalidate Ġthe Ġperformance Ġof Ġthe Ġproposed Ġs 2 s L Ġby Ġproviding Ġthe Ġpreliminary Ġresults Ġobtained Ġon Ġtwo Ġdifferent Ġtasks Ġnamely , ĠSpeech / Music Ġdiscrimination Ġand Ġemotion Ġclassification . ĠWe Ġconsidered Ġthe ĠGT Z AN ĠMusic - Spe ech Ġdataset Ġ[ 17 ], Ġconsisting Ġof Ġ120 Ġaudio Ġfiles Ġ( 60 Ġspeech Ġand Ġ60 Ġmusic ), Ġfor Ġtask Ġof Ġclass ifying Ġspeech Ġand Ġmusic . ĠEach Ġaudio Ġfile Ġ( of Ġ2 Ġseconds Ġduration ) Ġis Ġrepresented Ġusing Ġa Ġ13 - dimensional Ġmel - frequency Ġc ep st ral Ġcoefficient Ġ( MF CC ) Ġvector , Ġwhere Ġeach ĠMF CC Ġvector Ġis Ġthe Ġaverage Ġof Ġall Ġthe Ġframe Ġlevel Ġ( frame Ġsize Ġof Ġ30 Ġmsec Ġand Ġan Ġoverlap Ġof Ġ10 Ġmsec ) ĠMF CC Ġvectors . ĠIt Ġis Ġto Ġbe Ġnoted Ġthat Ġour Ġmain Ġintention Ġfor Ġthis Ġtask Ġis Ġnot Ġbetter Ġfeature Ġselection , Ġbut Ġto Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġapproach , Ġin Ġparticular Ġfor Ġlow Ġdata Ġscenarios . </s> The Ġstandard ĠBerlin Ġspeech Ġemotion Ġdatabase Ġ( EM O - DB ) Ġ[ 18 ] Ġconsisting Ġof Ġ5 35 Ġutter ances Ġcorresponding Ġto Ġ7 Ġdifferent Ġemotions Ġis Ġconsidered Ġfor Ġthe Ġtask Ġof Ġemotion Ġclassification . ĠEach Ġutter ance Ġis Ġrepresented Ġby Ġa Ġ19 - dimensional Ġfeature Ġvector Ġobtained Ġby Ġusing Ġthe Ġfeature Ġselection Ġalgorithm Ġfrom ĠWE KA Ġtool kit Ġ[ 19 ] Ġon Ġthe Ġ384 - dimensional Ġutter ance Ġlevel Ġfeature Ġvector Ġobtained Ġusing Ġopen S MI LE Ġtool kit Ġ[ 20 ]. ĠFor Ġtwo Ġclass Ġclassification , Ġwe Ġconsidered Ġthe Ġtwo Ġmost Ġconfusing Ġemotion Ġpairs Ġi . e ., Ġ( Ne utral , Sad ) Ġand Ġ( Ang er , ĠHappy ). ĠData Ġcorresponding Ġto ĠSpeech / Music Ġclassification Ġ( 60 Ġspeech Ġand Ġ60 Ġmusic Ġsamples ) Ġand ĠNeutral / Sad Ġclassification Ġ( 79 Ġneutral Ġand Ġ62 Ġsad Ġutter ances ) Ġis Ġbalanced Ġwhereas ĠAnger / Happy Ġclassification Ġtask Ġhas Ġdata Ġimbalance , Ġwith Ġanger Ġforming Ġthe Ġmajority Ġclass Ġ( 127 Ġsamples ) Ġand Ġhappy Ġforming Ġthe Ġminority Ġclass Ġ( 71 Ġsamples ). ĠTherefore , Ġwe Ġshow Ġthe Ġperformance Ġof Ġs 2 s L Ġon Ġboth , Ġbalanced Ġand Ġim balanced Ġdatasets . </s> All Ġexperimental Ġresults Ġare Ġvalidated Ġusing Ġ5 - fold Ġcross Ġvalidation Ġ( 80 % Ġof Ġdata Ġfor Ġtraining Ġand Ġ20 % Ġfor Ġtesting Ġin Ġeach Ġfold ). ĠFurther , Ġto Ġanalyze Ġthe Ġeffectiveness Ġof Ġs 2 s L Ġin Ġlow Ġresource Ġscenarios , Ġdifferent Ġproportions Ġof Ġtraining Ġdata , Ġwithin Ġeach Ġfold , Ġare Ġconsidered Ġto Ġtrain Ġthe Ġsystem . ĠFor Ġthis Ġanalysis , Ġwe Ġconsidered Ġ4 Ġdifferent Ġproportions Ġi . e ., ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġtraining Ġdata Ġto Ġtrain Ġthe Ġclass ifier . ĠFor Ġinstance , ĠIN LINE FORM 4 Ġmeans Ġconsidering Ġonly Ġhalf Ġof Ġthe Ġoriginal Ġtraining Ġdata Ġto Ġtrain Ġthe Ġclass ifier , Ġand ĠIN LINE FORM 5 Ġmeans Ġconsidering Ġthe Ġcomplete Ġtraining Ġdata . Ġ5 - fold Ġcross Ġvalidation Ġis Ġconsidered Ġfor Ġall Ġdata Ġproportions . ĠAccuracy Ġ( in Ġ% ) Ġis Ġused Ġas Ġa Ġperformance Ġmeasure Ġfor Ġbalanced Ġdata Ġclassification Ġtasks Ġ( i . e ., ĠSpeech / Music Ġclassification Ġand ĠNeutral / Sad Ġemotion Ġclassification ), Ġwhereas Ġthe Ġmore Ġpreferred ĠIN LINE FORM 6 Ġmeasure Ġ[ 21 ] Ġis Ġused Ġas Ġa Ġmeasure Ġfor Ġim balanced Ġdata Ġclassification Ġtask Ġ( i . e ., ĠAnger / Happy Ġemotion Ġclassification ). </s> Table ĠT AB REF 14 Ġshow Ġthe Ġresults Ġobtained Ġfor Ġproposed Ġs 2 s L Ġapproach Ġin Ġcomparison Ġto Ġthat Ġof ĠML P Ġfor Ġthe Ġtasks Ġof ĠSpeech / Music Ġand ĠNeutral / Sad Ġclassification , Ġby Ġconsidering Ġdifferent Ġproportions Ġof Ġtraining Ġdata . ĠThe Ġvalues Ġin ĠTable ĠT AB REF 14 Ġare Ġmean Ġaccur acies Ġ( in Ġ% ) Ġobtained Ġby Ġ5 - fold Ġcross Ġvalidation . ĠIt Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 14 Ġthat Ġfor Ġboth Ġtasks , Ġs 2 s L Ġmethod Ġoutper forms ĠML P , Ġespecially Ġat Ġlow Ġresource Ġconditions . Ġs 2 s L Ġshows Ġan Ġabsolute Ġimprovement Ġin Ġaccuracy Ġof ĠIN LINE FORM 0 Ġ% Ġand ĠIN LINE FORM 1 Ġ% Ġover ĠML P Ġfor ĠSpeech / Music Ġand ĠNeutral / Sad Ġclassification Ġtasks , Ġrespectively , Ġwhen ĠIN LINE FORM 2 Ġof Ġthe Ġoriginal Ġtraining Ġdata Ġis Ġused Ġin Ġexperiments . </s> Table ĠT AB REF 14 Ġshow Ġthe Ġresults Ġ( in Ġterms Ġof ĠIN LINE FORM 0 Ġvalues ) Ġobtained Ġfor Ġproposed Ġs 2 s L Ġapproach Ġin Ġcomparison Ġto Ġthat Ġof ĠML P Ġfor ĠAnger / Happy Ġclassification Ġ( data Ġimbalance Ġproblem ). ĠHere , Ġstate - of - the - art Ġmethods Ġi . e ., ĠE us boost Ġ[ 22 ] Ġand ĠM WM OTE Ġ[ 23 ] Ġare Ġalso Ġconsidered Ġfor Ġcomparison . ĠIt Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 14 Ġthat Ġthe Ġs 2 s L Ġmethod Ġoutper forms ĠML P , Ġand Ġalso Ġperforms Ġbetter Ġthan ĠE us boost Ġand ĠM WM OTE Ġtechniques Ġon Ġim balanced Ġdata Ġ( around Ġ3 Ġ% Ġabsolute Ġimprovement Ġin ĠIN LINE FORM 1 Ġvalue Ġfor Ġs 2 s L Ġcompared Ġto ĠM WM OTE , Ġwhen ĠIN LINE FORM 2 Ġof Ġthe Ġtraining Ġdata Ġis Ġconsidered ). ĠIn Ġparticular , Ġat Ġlower Ġamounts Ġof Ġtraining Ġdata , Ġs 2 s L Ġoutper forms Ġall Ġthe Ġother Ġmethods , Ġillustrating Ġits Ġeffectiveness Ġeven Ġfor Ġlow Ġres ourced Ġdata Ġimbalance Ġproblems . Ġs 2 s L Ġmethod Ġshows Ġan Ġabsolute Ġimprovement Ġof Ġ6 % Ġ( ĠIN LINE FORM 3 Ġ) Ġin ĠIN LINE FORM 4 Ġvalue Ġover Ġthe Ġsecond Ġbest Ġ( ĠIN LINE FORM 5 Ġfor ĠM WM OTE ), Ġwhen Ġonly ĠIN LINE FORM 6 Ġof Ġthe Ġtraining Ġdata Ġis Ġused . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġintroduced Ġa Ġnovel Ġs 2 s Ġframework Ġto Ġeffectively Ġlearn Ġthe Ġclass Ġdiscrim in ative Ġcharacteristics , Ġeven Ġfrom Ġlow Ġdata Ġresources . ĠIn Ġthis Ġframework , Ġmore Ġthan Ġone Ġsample Ġ( here , Ġtwo Ġsamples ) Ġare Ġsimultaneously Ġconsidered Ġto Ġtrain Ġthe Ġclass ifier . ĠFurther , Ġthis Ġframework Ġallows Ġto Ġgenerate Ġmultiple Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample , Ġby Ġconsidering Ġpre selected Ġreference Ġsamples , Ġto Ġachieve Ġa Ġmore Ġprofound Ġdecision Ġmaking . ĠWe Ġillustrated Ġthe Ġsignificance Ġof Ġour Ġapproach Ġby Ġproviding Ġthe Ġexperimental Ġresults Ġfor Ġtwo Ġdifferent Ġtasks Ġnamely , Ġspeech / music Ġdiscrimination Ġand Ġemotion Ġclassification . ĠFurther , Ġwe Ġshowed Ġthat Ġthe Ġs 2 s Ġframework Ġcan Ġalso Ġhandle Ġthe Ġlow Ġres ourced Ġdata Ġimbalance Ġproblem . </s> References </s> [ 1 ] ĠH inton , ĠG ., ĠDeng , ĠL ., ĠYu , ĠD ., ĠDahl , ĠG . ĠE ., ĠMohamed , ĠA . ĠR ., ĠJ ait ly , ĠN ., ĠSenior , ĠA ., ĠVan hou c ke , ĠV ., ĠNguyen , ĠP ., ĠS ain ath , ĠT . ĠN . Ġ& ĠKings bury , ĠB . Ġ( 2012 ) ĠDeep Ġneural Ġnetworks Ġfor Ġacoustic Ġmodeling Ġin Ġspeech Ġrecognition : ĠThe Ġshared Ġviews Ġof Ġfour Ġresearch Ġgroups . ĠIEEE ĠSignal ĠProcessing ĠMagazine , Ġpp . Ġ82 âĢĵ 97 . </s> [ 2 ] ĠV iny als , ĠO ., ĠRav uri , ĠS . ĠV . Ġ& ĠP ove y , ĠD . Ġ( 2012 ) ĠRevis iting Ġrecurrent Ġneural Ġnetworks Ġfor Ġrobust ĠAS R . ĠIn ĠProc . ĠIEEE ĠInternational ĠConference Ġon ĠAc oust ics , ĠSpeech Ġand ĠSignal ĠProcessing Ġ( IC AS SP ), Ġpp . Ġ40 85 âĢĵ 40 88 . </s> [ 3 ] ĠLu , ĠL ., ĠZhang , ĠX ., ĠCho ĠK . Ġ& ĠRen als , ĠS . Ġ( 2015 ) ĠA Ġstudy Ġof Ġthe Ġrecurrent Ġneural Ġnetwork Ġenc oder - dec oder Ġfor Ġlarge Ġvocabulary Ġspeech Ġrecognition . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ3 249 âĢĵ 325 3 . </s> [ 4 ] ĠZhang , ĠY ., ĠPe z esh ki , ĠM ., ĠBra kel , ĠP ., ĠZhang , ĠS ., ĠBeng io , ĠC . L . Y . Ġ& ĠCour ville , ĠA . Ġ( 2017 ) ĠTowards Ġend - to - end Ġspeech Ġrecognition Ġwith Ġdeep Ġconv olution al Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 17 01 . 027 20 . </s> [ 5 ] ĠHan , ĠK ., ĠYu , ĠD . Ġ& ĠTas he v , ĠI . Ġ( 2014 ) ĠSpeech Ġemotion Ġrecognition Ġusing Ġdeep Ġneural Ġnetwork Ġand Ġextreme Ġlearning Ġmachine . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ223 âĢĵ 227 . </s> [ 6 ] ĠTr ige org is , ĠG ., ĠRing eval , ĠF ., ĠBr ue ck ner , ĠR ., ĠMarch i , ĠE ., ĠNicola ou , ĠM . ĠA ., ĠSch ull er , ĠB ., Ġ& ĠZ afe iri ou , ĠS . Ġ( 2016 ) ĠAd ieu Ġfeatures ? ĠEnd - to - end Ġspeech Ġemotion Ġrecognition Ġusing Ġa Ġdeep Ġconv olution al Ġrecurrent Ġnetwork . ĠIn ĠProc . ĠIC AS SP , Ġpp . Ġ5 200 âĢĵ 5 204 ). </s> [ 7 ] ĠHuang , ĠChe - We i , Ġ& ĠSh rik anth ĠS . ĠNar ayan an . Ġ( 2016 ) ĠAttention ĠAss isted ĠDiscovery Ġof ĠSub - Ut ter ance ĠStructure Ġin ĠSpeech ĠEm otion ĠRecogn ition . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ13 87 âĢĵ 13 91 . </s> [ 8 ] ĠZhang , ĠZ ., ĠRing eval , ĠF ., ĠHan , ĠJ ., ĠDeng , ĠJ ., ĠMarch i , ĠE . Ġ& ĠSch ull er , ĠB . Ġ( 2016 ) ĠF acing Ġrealism Ġin Ġspontaneous Ġemotion Ġrecognition Ġfrom Ġspeech : ĠFeature Ġenhancement Ġby Ġaut oen c oder Ġwith ĠL ST M Ġneural Ġnetworks . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ359 3 âĢĵ 35 97 . </s> [ 9 ] ĠSche ire r , ĠE . ĠD . Ġ& ĠSl aney , ĠM . Ġ( 2003 ) ĠMulti - feature Ġspeech / music Ġdiscrimination Ġsystem . ĠU . S . ĠPatent Ġ6 , 570 , 99 1 . </s> [ 10 ] ĠPik rak is , ĠA . Ġ& ĠThe odor idis ĠS . Ġ( 2014 ) ĠSpeech - music Ġdiscrimination : ĠA Ġdeep Ġlearning Ġperspective . ĠIn ĠProc . ĠEuropean ĠSignal ĠProcessing ĠConference Ġ( E US IP CO ), Ġpp . Ġ6 16 âĢĵ 620 . </s> [ 11 ] ĠChoi , ĠK ., ĠF az ek as , ĠG . Ġ& ĠSand ler , ĠM . Ġ( 2016 ) ĠAutomatic Ġtagging Ġusing Ġdeep Ġconv olution al Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 16 06 . 00 298 . </s> [ 12 ] ĠZ azo ĠCand il , ĠR ., ĠS ain ath , ĠT . N ., ĠSim ko , ĠG . Ġ& ĠPar ada , ĠC . Ġ( 2016 ) ĠFeature Ġlearning Ġwith Ġraw - wave form ĠCL D NN s Ġfor ĠVoice ĠActivity ĠDetection . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ3 668 âĢĵ 36 72 . </s> [ 13 ] ĠTian , ĠL ., ĠMoore , ĠJD . Ġ& ĠL ai , ĠC . Ġ( 2015 ) ĠEm otion Ġrecognition Ġin Ġspontaneous Ġand Ġacted Ġdialog ues . ĠIn ĠProc . ĠInternational ĠConference Ġon ĠAffect ive ĠComputing Ġand ĠIntelligent ĠInter action Ġ( AC II ), Ġpp . Ġ698 âĢĵ 704 . </s> [ 14 ] ĠD ump ala , ĠS . H ., ĠChak rab ort y , ĠR . Ġ& ĠK opp ar ap u , ĠS . K . Ġ( 2017 ) Ġk - FF NN : ĠA Ġprior i Ġknowledge Ġinfused Ġfeed - forward Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 17 04 . 070 55 . </s> [ 15 ] ĠD ump ala , ĠS . H ., ĠChak rab ort y , ĠR . Ġ& ĠK opp ar ap u , ĠS . K . Ġ( 2018 ) ĠKnowledge - driven Ġfeed - forward Ġneural Ġnetwork Ġfor Ġaudio Ġaffect ive Ġcontent Ġanalysis . ĠTo Ġappear Ġin ĠAAA I - 2018 Ġworkshop Ġon ĠAffect ive ĠContent ĠAnalysis . </s> [ 16 ] ĠDeng , ĠJ ., ĠZhang , ĠZ ., ĠMarch i , ĠE . Ġ& ĠSch ull er , ĠB . Ġ( 2013 ) ĠSp arse Ġaut oen c oder - based Ġfeature Ġtransfer Ġlearning Ġfor Ġspeech Ġemotion Ġrecognition . ĠIn ĠProc . ĠHum aine ĠAssociation ĠConference Ġon ĠAffect ive ĠComputing Ġand ĠIntelligent ĠInter action Ġ( AC II ), Ġpp . Ġ5 11 âĢĵ 516 . </s> [ 17 ] ĠGeorge ĠT zan et akis , ĠG t zan Ġmus ics pe ech . ĠAv ail abe Ġon - line Ġat Ġhttp :// m ars y as . info / download / dat as ets . </s> [ 18 ] ĠBurk hardt , ĠF ., ĠPa es ch ke , ĠA ., ĠR olf es , ĠM ., ĠSend l meier , ĠW . F . Ġ& ĠWeiss , ĠB . Ġ( 2005 ) ĠA Ġdatabase Ġof Ġg erman Ġemotional Ġspeech . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ15 17 âĢĵ 15 20 . </s> [ 19 ] ĠHall , ĠM ., ĠFrank , ĠE ., ĠHolmes , ĠG ., ĠPf ah ring er , ĠB ., ĠRe ut em ann , ĠP . Ġ& ĠW itten , ĠI . H . Ġ( 2009 ) ĠThe ĠWE KA Ġdata Ġmining Ġsoftware : Ġan Ġupdate . ĠAC M ĠSIG K DD Ġexplor ations Ġnewsletter , Ġ11 ( 1 ), Ġpp . Ġ10 âĢĵ 18 . </s> [ 20 ] ĠEy ben , ĠF ., ĠW ening er , ĠF ., ĠGross , ĠF . Ġ& ĠSch ull er , ĠB . Ġ( 2013 ) ĠRecent Ġdevelopments Ġin Ġopens mile , Ġthe Ġmun ich Ġopen - source Ġmultimedia Ġfeature Ġextract or . ĠIn ĠProc . ĠAC M Ġinternational Ġconference Ġon ĠMult imedia , Ġpp . Ġ8 35 âĢĵ 8 38 . </s> [ 21 ] ĠMar ate a , ĠA ., ĠPet ros ino , ĠA . Ġ& ĠMan zo , ĠM . Ġ( 2014 ) ĠAdjusted Ġf - me asure Ġand Ġkernel Ġscaling Ġfor Ġim balanced Ġdata Ġlearning . ĠInformation ĠSciences Ġ257 : 331 âĢĵ 341 . </s> [ 22 ] ĠGal ar , ĠM ., ĠFernandez , ĠA ., ĠBar rene che a , ĠE . Ġ& ĠHerrera , ĠF . Ġ( 2013 ) ĠE us boost : ĠEnh ancing Ġen semb les Ġfor Ġhighly Ġim balanced Ġdata - sets Ġby Ġevolutionary Ġunders am pling . ĠPattern ĠRecogn ition Ġ46 ( 12 ): 34 60 âĢĵ 34 71 . </s> [ 23 ] ĠBar ua , ĠS ., ĠIslam , ĠM . ĠM ., ĠYao , ĠX . Ġ& ĠMur ase , ĠK . Ġ( 2014 ) ĠM wm ote âĢĵ majority Ġweighted Ġminority Ġovers am pling Ġtechnique Ġfor Ġim balanced Ġdata Ġset Ġlearning . ĠIEEE ĠTransactions Ġon ĠKnowledge Ġand ĠData ĠEngineering Ġ26 ( 2 ): 405 âĢĵ 425 .
Introduction Deep neural networks (DNNs), in particular convolutional and recurrent neural networks, with huge architectures have been proven successful in wide range of tasks including audio processing such as speech to text [1 - 4], emotion recognition [5 - 8], speech/non-speech (e.g., of non-speech include noise, music, etc.,) classification [9 - 12], etc. Training these deep architectures require large amount of annotated data, as a result, they cannot be used in low data resource scenarios which is common in speech-based applications [13 - 15]. Apart from collecting large data corpus, annotating the data is also very difficult, and requires manual supervision and efforts. Especially, annotation of speech for tasks like emotion recognition also suffer from lack of agreement among the annotators [16]. Hence, there is a need to build reliable systems that can work in low resource scenario. In this work, we propose a novel approach to address the task of classification in low data resource scenarios. Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier. We call this approach as simultaneous two sample learning (s2sL). The proposed approach is also applicable to low resource data suffering with data imbalance. The contributions of this paper are: Proposed approach The s2sL approach proposed to address low data resource problem is explained in this Section. In this work, we use MLP (modified to handle our data representation) as the base classifier. Here, we explain the s2sL approach by considering two-class classification task. Data representation Consider a two-class classification task with INLINEFORM0 denoting the set of class labels, and let INLINEFORM1 and INLINEFORM2 be the number of samples corresponding to INLINEFORM3 and INLINEFORM4 , respectively. In general, to train a classifier, the samples in the train set are provided as input-output pairs as follows. DISPLAYFORM0 where INLINEFORM0 refers to the INLINEFORM1 -dimensional feature vector representing the INLINEFORM2 sample corresponding to INLINEFORM3 class label, and INLINEFORM4 , refers to output label of INLINEFORM5 class. In the proposed data representation format, called simultaneous two sample (s2s) representation, we will simultaneously consider two samples as follows. DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 are the INLINEFORM2 -dimensional feature vectors representing the INLINEFORM3 sample in INLINEFORM4 class and INLINEFORM5 sample in the INLINEFORM6 class, respectively; and INLINEFORM7 , INLINEFORM8 refers to the output labels of INLINEFORM9 and INLINEFORM10 class, respectively. Hence, in s2s data representation, we will have an input feature vector of length INLINEFORM0 i.e., INLINEFORM1 , and output class labels as either INLINEFORM2 or INLINEFORM3 . Each sample can be combined with all the samples (i.e., with ( INLINEFORM4 ) samples) in the dataset. Therefore, by representing the data in the s2s format, the number of samples in the train set increases to INLINEFORM5 from INLINEFORM6 samples. We hypothesize that the s2s format is expected to help the network not only to learn the characteristics of the two classes separately, but also the difference and similarity in characteristics of the two classes. Classifier training MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework. Generally, MLPs are trained using the data format given by eq. INLINEFORM0 . But to train the MLP on our s2s based data representation (as in eq. INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ). We have INLINEFORM0 units (instead of INLINEFORM1 units) in the input layer to accept the two samples i.e., INLINEFORM2 and INLINEFORM3 , simultaneously. The structure of the hidden layer in this approach is similar to that of a regular MLP. The number of hidden layers and hidden units can be varied depending upon the complexity of the problem. The number of units in the hidden layer is selected empirically by varying the number of hidden units from 2 to twice the length of the input layer (i.e., 2 to INLINEFORM0 ) and the unit at which the highest performance is obtained are selected. In this paper, we considered only a single hidden layer. Rectified linear units (ReLU) are used for hidden layer. The output layer will consist of units equal to twice the considered number of classes in the classification task i.e, the output layer will have four units for two-class classification task. The sigmoid activation function (not softmax) is used on the output layer units. Unlike regular MLP, we use sigmoid activation units in the output layer, with binary cross-entropy as the cost function, because the output labels in the proposed s2s based data representation will have more than one unit active at a time (not one-hot encoded output) and this condition cannot be handled using softmax function. As can be seen from Figure FIGREF4 , the output layer in our proposed method has outputs INLINEFORM0 and INLINEFORM1 which corresponds to the outputs associated with the input feature vectors INLINEFORM2 and INLINEFORM3 , respectively. For a two-class classification problem, there will be four units in the output layer and the possible output labels are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 corresponding to the class labels INLINEFORM8 and INLINEFORM9 , respectively. This architecture is referred to as s2s-MLP. In s2sL, s2s-MLP is trained using the s2s data representation format. Further, the s2s-MLP is trained using adam optimizer. Classifier testing Generally, the feature vector corresponding to the test sample is provided as input to the trained MLP in the testing phase and the class label is decided based on the obtained output. However, in s2sL method, the feature vector corresponding to the test sample should also be converted to the s2s data representation format to test the trained s2s-MLP. We propose a testing approach, where the given test sample is combined with a set of preselected reference samples, whose class label is known a priori, to generate multiple instances of the same test sample as follows. DISPLAYFORM0 where INLINEFORM0 , INLINEFORM1 refer to the INLINEFORM2 -dimensional feature vector corresponding to the test sample and the INLINEFORM3 reference sample, respectively. INLINEFORM4 refers to the considered number of reference samples. These reference samples can be selected from any of the two classes. For testing the s2s-MLP (as shown in Figure FIGREF4 ), each test sample INLINEFORM0 (same as ' INLINEFORM1 ' in (3)) is combined with all the INLINEFORM2 reference samples ( INLINEFORM3 ) to form INLINEFORM4 different instances of the same test sample INLINEFORM5 . The corresponding outputs ( INLINEFORM6 ) obtained from s2s-MLP for the INLINEFORM7 generated instances of INLINEFORM8 are combined by voting-based decision approach to obtain the final decision INLINEFORM9 . The class label that gets maximum votes is considered as the predicted output label. Experiments We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification. We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music. Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors. It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios. The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification. Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]. For two class classification, we considered the two most confusing emotion pairs i.e., (Neutral,Sad) and (Anger, Happy). Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples). Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets. All experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold). Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system. For this analysis, we considered 4 different proportions i.e., INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier. For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data. 5-fold cross validation is considered for all data proportions. Accuracy (in %) is used as a performance measure for balanced data classification tasks (i.e., Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i.e., Anger/Happy emotion classification). Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data. The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation. It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions. s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments. Table TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem). Here, state-of-the-art methods i.e., Eusboost [22] and MWMOTE [23] are also considered for comparison. It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered). In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems. s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used. Conclusions In this paper, we introduced a novel s2s framework to effectively learn the class discriminative characteristics, even from low data resources. In this framework, more than one sample (here, two samples) are simultaneously considered to train the classifier. Further, this framework allows to generate multiple instances of the same test sample, by considering preselected reference samples, to achieve a more profound decision making. We illustrated the significance of our approach by providing the experimental results for two different tasks namely, speech/music discrimination and emotion classification. Further, we showed that the s2s framework can also handle the low resourced data imbalance problem. References [1] Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N. & Kingsbury, B. (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, pp. 82–97. [2] Vinyals, O., Ravuri, S. V. & Povey, D. (2012) Revisiting recurrent neural networks for robust ASR. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4085–4088. [3] Lu, L., Zhang, X., Cho K. & Renals, S. (2015) A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition. In Proc. INTERSPEECH, pp. 3249–3253. [4] Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Bengio, C.L.Y. & Courville, A. (2017) Towards end-to-end speech recognition with deep convolutional neural networks. arXiv preprint, arXiv:1701.02720. [5] Han, K., Yu, D. & Tashev, I. (2014) Speech emotion recognition using deep neural network and extreme learning machine. In Proc. Interspeech, pp. 223–227. [6] Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. In Proc. ICASSP, pp. 5200–5204). [7] Huang, Che-Wei, & Shrikanth S. Narayanan. (2016) Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition. In Proc. INTERSPEECH, pp. 1387–1391. [8] Zhang, Z., Ringeval, F., Han, J., Deng, J., Marchi, E. & Schuller, B. (2016) Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks. In Proc. INTERSPEECH, pp. 3593–3597. [9] Scheirer, E. D. & Slaney, M. (2003) Multi-feature speech/music discrimination system. U.S. Patent 6,570,991. [10] Pikrakis, A. & Theodoridis S. (2014) Speech-music discrimination: A deep learning perspective. In Proc. European Signal Processing Conference (EUSIPCO), pp. 616–620. [11] Choi, K., Fazekas, G. & Sandler, M. (2016) Automatic tagging using deep convolutional neural networks. arXiv preprint, arXiv:1606.00298. [12] Zazo Candil, R., Sainath, T.N., Simko, G. & Parada, C. (2016) Feature learning with raw-waveform CLDNNs for Voice Activity Detection. In Proc. Interspeech, pp. 3668–3672. [13] Tian, L., Moore, JD. & Lai, C. (2015) Emotion recognition in spontaneous and acted dialogues. In Proc. International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 698–704. [14] Dumpala, S.H., Chakraborty, R. & Kopparapu, S.K. (2017) k-FFNN: A priori knowledge infused feed-forward neural networks. arXiv preprint, arXiv:1704.07055. [15] Dumpala, S.H., Chakraborty, R. & Kopparapu, S.K. (2018) Knowledge-driven feed-forward neural network for audio affective content analysis. To appear in AAAI-2018 workshop on Affective Content Analysis. [16] Deng, J., Zhang, Z., Marchi, E. & Schuller, B. (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition. In Proc. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 511–516. [17] George Tzanetakis, Gtzan musicspeech. Availabe on-line at http://marsyas.info/download/datasets. [18] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F. & Weiss, B. (2005) A database of german emotional speech. In Proc. Interspeech, pp. 1517–1520. [19] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. & Witten, I.H. (2009) The WEKA data mining software: an update. ACM SIGKDD explorations newsletter, 11(1), pp. 10–18. [20] Eyben, F., Weninger, F., Gross, F. & Schuller, B. (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proc. ACM international conference on Multimedia, pp. 835–838. [21] Maratea, A., Petrosino, A. & Manzo, M. (2014) Adjusted f-measure and kernel scaling for imbalanced data learning. Information Sciences 257:331–341. [22] Galar, M., Fernandez, A., Barrenechea, E. & Herrera, F. (2013) Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling. Pattern Recognition 46(12):3460–3471. [23] Barua, S., Islam, M. M., Yao, X. & Murase, K. (2014) Mwmote–majority weighted minority oversampling technique for imbalanced data set learning. IEEE Transactions on Knowledge and Data Engineering 26(2):405–425.
Do they use pretrained models?
d667731ea20605580c398a1224a0094d1155ebbb
Do Ġthey Ġuse Ġpret rained Ġmodels ?
No
<s> Do Ġthey Ġuse Ġpret rained Ġmodels ? </s> Introduction </s> Deep Ġneural Ġnetworks Ġ( D NN s ), Ġin Ġparticular Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks , Ġwith Ġhuge Ġarchitectures Ġhave Ġbeen Ġproven Ġsuccessful Ġin Ġwide Ġrange Ġof Ġtasks Ġincluding Ġaudio Ġprocessing Ġsuch Ġas Ġspeech Ġto Ġtext Ġ[ 1 Ġ- Ġ4 ], Ġemotion Ġrecognition Ġ[ 5 Ġ- Ġ8 ], Ġspeech / non - speech Ġ( e . g ., Ġof Ġnon - speech Ġinclude Ġnoise , Ġmusic , Ġetc ., ) Ġclassification Ġ[ 9 Ġ- Ġ12 ], Ġetc . </s> Training Ġthese Ġdeep Ġarchitectures Ġrequire Ġlarge Ġamount Ġof Ġannot ated Ġdata , Ġas Ġa Ġresult , Ġthey Ġcannot Ġbe Ġused Ġin Ġlow Ġdata Ġresource Ġscenarios Ġwhich Ġis Ġcommon Ġin Ġspeech - based Ġapplications Ġ[ 13 Ġ- Ġ15 ]. ĠApart Ġfrom Ġcollecting Ġlarge Ġdata Ġcorpus , Ġannot ating Ġthe Ġdata Ġis Ġalso Ġvery Ġdifficult , Ġand Ġrequires Ġmanual Ġsupervision Ġand Ġefforts . ĠEspecially , Ġannotation Ġof Ġspeech Ġfor Ġtasks Ġlike Ġemotion Ġrecognition Ġalso Ġsuffer Ġfrom Ġlack Ġof Ġagreement Ġamong Ġthe Ġannot ators Ġ[ 16 ]. ĠHence , Ġthere Ġis Ġa Ġneed Ġto Ġbuild Ġreliable Ġsystems Ġthat Ġcan Ġwork Ġin Ġlow Ġresource Ġscenario . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġnovel Ġapproach Ġto Ġaddress Ġthe Ġtask Ġof Ġclassification Ġin Ġlow Ġdata Ġresource Ġscenarios . ĠOur Ġapproach Ġinvolves Ġsimultaneously Ġconsidering Ġmore Ġthan Ġone Ġsample Ġ( in Ġthis Ġwork , Ġtwo Ġsamples Ġare Ġconsidered ) Ġto Ġtrain Ġthe Ġclass ifier . ĠWe Ġcall Ġthis Ġapproach Ġas Ġsimultaneous Ġtwo Ġsample Ġlearning Ġ( s 2 s L ). ĠThe Ġproposed Ġapproach Ġis Ġalso Ġapplicable Ġto Ġlow Ġresource Ġdata Ġsuffering Ġwith Ġdata Ġimbalance . ĠThe Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> Prop osed Ġapproach </s> The Ġs 2 s L Ġapproach Ġproposed Ġto Ġaddress Ġlow Ġdata Ġresource Ġproblem Ġis Ġexplained Ġin Ġthis ĠSection . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠML P Ġ( modified Ġto Ġhandle Ġour Ġdata Ġrepresentation ) Ġas Ġthe Ġbase Ġclass ifier . ĠHere , Ġwe Ġexplain Ġthe Ġs 2 s L Ġapproach Ġby Ġconsidering Ġtwo - class Ġclassification Ġtask . </s> Data Ġrepresentation </s> Consider Ġa Ġtwo - class Ġclassification Ġtask Ġwith ĠIN LINE FORM 0 Ġden oting Ġthe Ġset Ġof Ġclass Ġlabels , Ġand Ġlet ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġbe Ġthe Ġnumber Ġof Ġsamples Ġcorresponding Ġto ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠIn Ġgeneral , Ġto Ġtrain Ġa Ġclass ifier , Ġthe Ġsamples Ġin Ġthe Ġtrain Ġset Ġare Ġprovided Ġas Ġinput - output Ġpairs Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġ- dimensional Ġfeature Ġvector Ġrepresenting Ġthe ĠIN LINE FORM 2 Ġsample Ġcorresponding Ġto ĠIN LINE FORM 3 Ġclass Ġlabel , Ġand ĠIN LINE FORM 4 Ġ, Ġrefers Ġto Ġoutput Ġlabel Ġof ĠIN LINE FORM 5 Ġclass . ĠIn Ġthe Ġproposed Ġdata Ġrepresentation Ġformat , Ġcalled Ġsimultaneous Ġtwo Ġsample Ġ( s 2 s ) Ġrepresentation , Ġwe Ġwill Ġsimultaneously Ġconsider Ġtwo Ġsamples Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġare Ġthe ĠIN LINE FORM 2 Ġ- dimensional Ġfeature Ġvectors Ġrepresenting Ġthe ĠIN LINE FORM 3 Ġsample Ġin ĠIN LINE FORM 4 Ġclass Ġand ĠIN LINE FORM 5 Ġsample Ġin Ġthe ĠIN LINE FORM 6 Ġclass , Ġrespectively ; Ġand ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġrefers Ġto Ġthe Ġoutput Ġlabels Ġof ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġclass , Ġrespectively . </s> H ence , Ġin Ġs 2 s Ġdata Ġrepresentation , Ġwe Ġwill Ġhave Ġan Ġinput Ġfeature Ġvector Ġof Ġlength ĠIN LINE FORM 0 Ġi . e ., ĠIN LINE FORM 1 Ġ, Ġand Ġoutput Ġclass Ġlabels Ġas Ġeither ĠIN LINE FORM 2 Ġor ĠIN LINE FORM 3 Ġ. ĠEach Ġsample Ġcan Ġbe Ġcombined Ġwith Ġall Ġthe Ġsamples Ġ( i . e ., Ġwith Ġ( ĠIN LINE FORM 4 Ġ) Ġsamples ) Ġin Ġthe Ġdataset . ĠTherefore , Ġby Ġrepresenting Ġthe Ġdata Ġin Ġthe Ġs 2 s Ġformat , Ġthe Ġnumber Ġof Ġsamples Ġin Ġthe Ġtrain Ġset Ġincreases Ġto ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġsamples . ĠWe Ġhypothes ize Ġthat Ġthe Ġs 2 s Ġformat Ġis Ġexpected Ġto Ġhelp Ġthe Ġnetwork Ġnot Ġonly Ġto Ġlearn Ġthe Ġcharacteristics Ġof Ġthe Ġtwo Ġclasses Ġseparately , Ġbut Ġalso Ġthe Ġdifference Ġand Ġsimilarity Ġin Ġcharacteristics Ġof Ġthe Ġtwo Ġclasses . </s> Class ifier Ġtraining </s> ML P , Ġthe Ġmost Ġcommonly Ġused Ġfeed Ġforward Ġneural Ġnetwork , Ġis Ġconsidered Ġas Ġthe Ġbase Ġclass ifier Ġto Ġvalidate Ġour Ġproposed Ġs 2 s Ġframework . ĠGenerally , ĠML Ps Ġare Ġtrained Ġusing Ġthe Ġdata Ġformat Ġgiven Ġby Ġeq . ĠIN LINE FORM 0 Ġ. ĠBut Ġto Ġtrain Ġthe ĠML P Ġon Ġour Ġs 2 s Ġbased Ġdata Ġrepresentation Ġ( as Ġin Ġeq . ĠIN LINE FORM 1 Ġ), Ġthe Ġfollowing Ġmodifications Ġare Ġmade Ġto Ġthe ĠML P Ġarchitecture Ġ( re fer Ġto ĠFigure ĠFIG REF 4 Ġ). </s> We Ġhave ĠIN LINE FORM 0 Ġunits Ġ( instead Ġof ĠIN LINE FORM 1 Ġunits ) Ġin Ġthe Ġinput Ġlayer Ġto Ġaccept Ġthe Ġtwo Ġsamples Ġi . e ., ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġsimultaneously . </s> The Ġstructure Ġof Ġthe Ġhidden Ġlayer Ġin Ġthis Ġapproach Ġis Ġsimilar Ġto Ġthat Ġof Ġa Ġregular ĠML P . ĠThe Ġnumber Ġof Ġhidden Ġlayers Ġand Ġhidden Ġunits Ġcan Ġbe Ġvaried Ġdepending Ġupon Ġthe Ġcomplexity Ġof Ġthe Ġproblem . ĠThe Ġnumber Ġof Ġunits Ġin Ġthe Ġhidden Ġlayer Ġis Ġselected Ġempir ically Ġby Ġvarying Ġthe Ġnumber Ġof Ġhidden Ġunits Ġfrom Ġ2 Ġto Ġtwice Ġthe Ġlength Ġof Ġthe Ġinput Ġlayer Ġ( i . e ., Ġ2 Ġto ĠIN LINE FORM 0 Ġ) Ġand Ġthe Ġunit Ġat Ġwhich Ġthe Ġhighest Ġperformance Ġis Ġobtained Ġare Ġselected . ĠIn Ġthis Ġpaper , Ġwe Ġconsidered Ġonly Ġa Ġsingle Ġhidden Ġlayer . ĠRect ified Ġlinear Ġunits Ġ( Re LU ) Ġare Ġused Ġfor Ġhidden Ġlayer . </s> The Ġoutput Ġlayer Ġwill Ġconsist Ġof Ġunits Ġequal Ġto Ġtwice Ġthe Ġconsidered Ġnumber Ġof Ġclasses Ġin Ġthe Ġclassification Ġtask Ġi . e , Ġthe Ġoutput Ġlayer Ġwill Ġhave Ġfour Ġunits Ġfor Ġtwo - class Ġclassification Ġtask . ĠThe Ġs igm oid Ġactivation Ġfunction Ġ( not Ġsoft max ) Ġis Ġused Ġon Ġthe Ġoutput Ġlayer Ġunits . ĠUnlike Ġregular ĠML P , Ġwe Ġuse Ġs igm oid Ġactivation Ġunits Ġin Ġthe Ġoutput Ġlayer , Ġwith Ġbinary Ġcross - ent ropy Ġas Ġthe Ġcost Ġfunction , Ġbecause Ġthe Ġoutput Ġlabels Ġin Ġthe Ġproposed Ġs 2 s Ġbased Ġdata Ġrepresentation Ġwill Ġhave Ġmore Ġthan Ġone Ġunit Ġactive Ġat Ġa Ġtime Ġ( not Ġone - hot Ġencoded Ġoutput ) Ġand Ġthis Ġcondition Ġcannot Ġbe Ġhandled Ġusing Ġsoft max Ġfunction . </s> As Ġcan Ġbe Ġseen Ġfrom ĠFigure ĠFIG REF 4 Ġ, Ġthe Ġoutput Ġlayer Ġin Ġour Ġproposed Ġmethod Ġhas Ġoutputs ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġwhich Ġcorresponds Ġto Ġthe Ġoutputs Ġassociated Ġwith Ġthe Ġinput Ġfeature Ġvectors ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġrespectively . ĠFor Ġa Ġtwo - class Ġclassification Ġproblem , Ġthere Ġwill Ġbe Ġfour Ġunits Ġin Ġthe Ġoutput Ġlayer Ġand Ġthe Ġpossible Ġoutput Ġlabels Ġare ĠIN LINE FORM 4 Ġ, ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġcorresponding Ġto Ġthe Ġclass Ġlabels ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġrespectively . ĠThis Ġarchitecture Ġis Ġreferred Ġto Ġas Ġs 2 s - ML P . ĠIn Ġs 2 s L , Ġs 2 s - ML P Ġis Ġtrained Ġusing Ġthe Ġs 2 s Ġdata Ġrepresentation Ġformat . ĠFurther , Ġthe Ġs 2 s - ML P Ġis Ġtrained Ġusing Ġadam Ġoptim izer . </s> Class ifier Ġtesting </s> Generally , Ġthe Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġis Ġprovided Ġas Ġinput Ġto Ġthe Ġtrained ĠML P Ġin Ġthe Ġtesting Ġphase Ġand Ġthe Ġclass Ġlabel Ġis Ġdecided Ġbased Ġon Ġthe Ġobtained Ġoutput . </s> However , Ġin Ġs 2 s L Ġmethod , Ġthe Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġshould Ġalso Ġbe Ġconverted Ġto Ġthe Ġs 2 s Ġdata Ġrepresentation Ġformat Ġto Ġtest Ġthe Ġtrained Ġs 2 s - ML P . ĠWe Ġpropose Ġa Ġtesting Ġapproach , Ġwhere Ġthe Ġgiven Ġtest Ġsample Ġis Ġcombined Ġwith Ġa Ġset Ġof Ġpre selected Ġreference Ġsamples , Ġwhose Ġclass Ġlabel Ġis Ġknown Ġa Ġprior i , Ġto Ġgenerate Ġmultiple Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample Ġas Ġfollows . ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġrefer Ġto Ġthe ĠIN LINE FORM 2 Ġ- dimensional Ġfeature Ġvector Ġcorresponding Ġto Ġthe Ġtest Ġsample Ġand Ġthe ĠIN LINE FORM 3 Ġreference Ġsample , Ġrespectively . ĠIN LINE FORM 4 Ġrefers Ġto Ġthe Ġconsidered Ġnumber Ġof Ġreference Ġsamples . ĠThese Ġreference Ġsamples Ġcan Ġbe Ġselected Ġfrom Ġany Ġof Ġthe Ġtwo Ġclasses . </s> For Ġtesting Ġthe Ġs 2 s - ML P Ġ( as Ġshown Ġin ĠFigure ĠFIG REF 4 Ġ), Ġeach Ġtest Ġsample ĠIN LINE FORM 0 Ġ( same Ġas Ġ' ĠIN LINE FORM 1 Ġ' Ġin Ġ( 3 )) Ġis Ġcombined Ġwith Ġall Ġthe ĠIN LINE FORM 2 Ġreference Ġsamples Ġ( ĠIN LINE FORM 3 Ġ) Ġto Ġform ĠIN LINE FORM 4 Ġdifferent Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample ĠIN LINE FORM 5 Ġ. ĠThe Ġcorresponding Ġoutputs Ġ( ĠIN LINE FORM 6 Ġ) Ġobtained Ġfrom Ġs 2 s - ML P Ġfor Ġthe ĠIN LINE FORM 7 Ġgenerated Ġinstances Ġof ĠIN LINE FORM 8 Ġare Ġcombined Ġby Ġvoting - based Ġdecision Ġapproach Ġto Ġobtain Ġthe Ġfinal Ġdecision ĠIN LINE FORM 9 Ġ. ĠThe Ġclass Ġlabel Ġthat Ġgets Ġmaximum Ġvotes Ġis Ġconsidered Ġas Ġthe Ġpredicted Ġoutput Ġlabel . </s> Exper iments </s> We Ġvalidate Ġthe Ġperformance Ġof Ġthe Ġproposed Ġs 2 s L Ġby Ġproviding Ġthe Ġpreliminary Ġresults Ġobtained Ġon Ġtwo Ġdifferent Ġtasks Ġnamely , ĠSpeech / Music Ġdiscrimination Ġand Ġemotion Ġclassification . ĠWe Ġconsidered Ġthe ĠGT Z AN ĠMusic - Spe ech Ġdataset Ġ[ 17 ], Ġconsisting Ġof Ġ120 Ġaudio Ġfiles Ġ( 60 Ġspeech Ġand Ġ60 Ġmusic ), Ġfor Ġtask Ġof Ġclass ifying Ġspeech Ġand Ġmusic . ĠEach Ġaudio Ġfile Ġ( of Ġ2 Ġseconds Ġduration ) Ġis Ġrepresented Ġusing Ġa Ġ13 - dimensional Ġmel - frequency Ġc ep st ral Ġcoefficient Ġ( MF CC ) Ġvector , Ġwhere Ġeach ĠMF CC Ġvector Ġis Ġthe Ġaverage Ġof Ġall Ġthe Ġframe Ġlevel Ġ( frame Ġsize Ġof Ġ30 Ġmsec Ġand Ġan Ġoverlap Ġof Ġ10 Ġmsec ) ĠMF CC Ġvectors . ĠIt Ġis Ġto Ġbe Ġnoted Ġthat Ġour Ġmain Ġintention Ġfor Ġthis Ġtask Ġis Ġnot Ġbetter Ġfeature Ġselection , Ġbut Ġto Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġapproach , Ġin Ġparticular Ġfor Ġlow Ġdata Ġscenarios . </s> The Ġstandard ĠBerlin Ġspeech Ġemotion Ġdatabase Ġ( EM O - DB ) Ġ[ 18 ] Ġconsisting Ġof Ġ5 35 Ġutter ances Ġcorresponding Ġto Ġ7 Ġdifferent Ġemotions Ġis Ġconsidered Ġfor Ġthe Ġtask Ġof Ġemotion Ġclassification . ĠEach Ġutter ance Ġis Ġrepresented Ġby Ġa Ġ19 - dimensional Ġfeature Ġvector Ġobtained Ġby Ġusing Ġthe Ġfeature Ġselection Ġalgorithm Ġfrom ĠWE KA Ġtool kit Ġ[ 19 ] Ġon Ġthe Ġ384 - dimensional Ġutter ance Ġlevel Ġfeature Ġvector Ġobtained Ġusing Ġopen S MI LE Ġtool kit Ġ[ 20 ]. ĠFor Ġtwo Ġclass Ġclassification , Ġwe Ġconsidered Ġthe Ġtwo Ġmost Ġconfusing Ġemotion Ġpairs Ġi . e ., Ġ( Ne utral , Sad ) Ġand Ġ( Ang er , ĠHappy ). ĠData Ġcorresponding Ġto ĠSpeech / Music Ġclassification Ġ( 60 Ġspeech Ġand Ġ60 Ġmusic Ġsamples ) Ġand ĠNeutral / Sad Ġclassification Ġ( 79 Ġneutral Ġand Ġ62 Ġsad Ġutter ances ) Ġis Ġbalanced Ġwhereas ĠAnger / Happy Ġclassification Ġtask Ġhas Ġdata Ġimbalance , Ġwith Ġanger Ġforming Ġthe Ġmajority Ġclass Ġ( 127 Ġsamples ) Ġand Ġhappy Ġforming Ġthe Ġminority Ġclass Ġ( 71 Ġsamples ). ĠTherefore , Ġwe Ġshow Ġthe Ġperformance Ġof Ġs 2 s L Ġon Ġboth , Ġbalanced Ġand Ġim balanced Ġdatasets . </s> All Ġexperimental Ġresults Ġare Ġvalidated Ġusing Ġ5 - fold Ġcross Ġvalidation Ġ( 80 % Ġof Ġdata Ġfor Ġtraining Ġand Ġ20 % Ġfor Ġtesting Ġin Ġeach Ġfold ). ĠFurther , Ġto Ġanalyze Ġthe Ġeffectiveness Ġof Ġs 2 s L Ġin Ġlow Ġresource Ġscenarios , Ġdifferent Ġproportions Ġof Ġtraining Ġdata , Ġwithin Ġeach Ġfold , Ġare Ġconsidered Ġto Ġtrain Ġthe Ġsystem . ĠFor Ġthis Ġanalysis , Ġwe Ġconsidered Ġ4 Ġdifferent Ġproportions Ġi . e ., ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġof Ġthe Ġtraining Ġdata Ġto Ġtrain Ġthe Ġclass ifier . ĠFor Ġinstance , ĠIN LINE FORM 4 Ġmeans Ġconsidering Ġonly Ġhalf Ġof Ġthe Ġoriginal Ġtraining Ġdata Ġto Ġtrain Ġthe Ġclass ifier , Ġand ĠIN LINE FORM 5 Ġmeans Ġconsidering Ġthe Ġcomplete Ġtraining Ġdata . Ġ5 - fold Ġcross Ġvalidation Ġis Ġconsidered Ġfor Ġall Ġdata Ġproportions . ĠAccuracy Ġ( in Ġ% ) Ġis Ġused Ġas Ġa Ġperformance Ġmeasure Ġfor Ġbalanced Ġdata Ġclassification Ġtasks Ġ( i . e ., ĠSpeech / Music Ġclassification Ġand ĠNeutral / Sad Ġemotion Ġclassification ), Ġwhereas Ġthe Ġmore Ġpreferred ĠIN LINE FORM 6 Ġmeasure Ġ[ 21 ] Ġis Ġused Ġas Ġa Ġmeasure Ġfor Ġim balanced Ġdata Ġclassification Ġtask Ġ( i . e ., ĠAnger / Happy Ġemotion Ġclassification ). </s> Table ĠT AB REF 14 Ġshow Ġthe Ġresults Ġobtained Ġfor Ġproposed Ġs 2 s L Ġapproach Ġin Ġcomparison Ġto Ġthat Ġof ĠML P Ġfor Ġthe Ġtasks Ġof ĠSpeech / Music Ġand ĠNeutral / Sad Ġclassification , Ġby Ġconsidering Ġdifferent Ġproportions Ġof Ġtraining Ġdata . ĠThe Ġvalues Ġin ĠTable ĠT AB REF 14 Ġare Ġmean Ġaccur acies Ġ( in Ġ% ) Ġobtained Ġby Ġ5 - fold Ġcross Ġvalidation . ĠIt Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 14 Ġthat Ġfor Ġboth Ġtasks , Ġs 2 s L Ġmethod Ġoutper forms ĠML P , Ġespecially Ġat Ġlow Ġresource Ġconditions . Ġs 2 s L Ġshows Ġan Ġabsolute Ġimprovement Ġin Ġaccuracy Ġof ĠIN LINE FORM 0 Ġ% Ġand ĠIN LINE FORM 1 Ġ% Ġover ĠML P Ġfor ĠSpeech / Music Ġand ĠNeutral / Sad Ġclassification Ġtasks , Ġrespectively , Ġwhen ĠIN LINE FORM 2 Ġof Ġthe Ġoriginal Ġtraining Ġdata Ġis Ġused Ġin Ġexperiments . </s> Table ĠT AB REF 14 Ġshow Ġthe Ġresults Ġ( in Ġterms Ġof ĠIN LINE FORM 0 Ġvalues ) Ġobtained Ġfor Ġproposed Ġs 2 s L Ġapproach Ġin Ġcomparison Ġto Ġthat Ġof ĠML P Ġfor ĠAnger / Happy Ġclassification Ġ( data Ġimbalance Ġproblem ). ĠHere , Ġstate - of - the - art Ġmethods Ġi . e ., ĠE us boost Ġ[ 22 ] Ġand ĠM WM OTE Ġ[ 23 ] Ġare Ġalso Ġconsidered Ġfor Ġcomparison . ĠIt Ġcan Ġbe Ġobserved Ġfrom ĠTable ĠT AB REF 14 Ġthat Ġthe Ġs 2 s L Ġmethod Ġoutper forms ĠML P , Ġand Ġalso Ġperforms Ġbetter Ġthan ĠE us boost Ġand ĠM WM OTE Ġtechniques Ġon Ġim balanced Ġdata Ġ( around Ġ3 Ġ% Ġabsolute Ġimprovement Ġin ĠIN LINE FORM 1 Ġvalue Ġfor Ġs 2 s L Ġcompared Ġto ĠM WM OTE , Ġwhen ĠIN LINE FORM 2 Ġof Ġthe Ġtraining Ġdata Ġis Ġconsidered ). ĠIn Ġparticular , Ġat Ġlower Ġamounts Ġof Ġtraining Ġdata , Ġs 2 s L Ġoutper forms Ġall Ġthe Ġother Ġmethods , Ġillustrating Ġits Ġeffectiveness Ġeven Ġfor Ġlow Ġres ourced Ġdata Ġimbalance Ġproblems . Ġs 2 s L Ġmethod Ġshows Ġan Ġabsolute Ġimprovement Ġof Ġ6 % Ġ( ĠIN LINE FORM 3 Ġ) Ġin ĠIN LINE FORM 4 Ġvalue Ġover Ġthe Ġsecond Ġbest Ġ( ĠIN LINE FORM 5 Ġfor ĠM WM OTE ), Ġwhen Ġonly ĠIN LINE FORM 6 Ġof Ġthe Ġtraining Ġdata Ġis Ġused . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġintroduced Ġa Ġnovel Ġs 2 s Ġframework Ġto Ġeffectively Ġlearn Ġthe Ġclass Ġdiscrim in ative Ġcharacteristics , Ġeven Ġfrom Ġlow Ġdata Ġresources . ĠIn Ġthis Ġframework , Ġmore Ġthan Ġone Ġsample Ġ( here , Ġtwo Ġsamples ) Ġare Ġsimultaneously Ġconsidered Ġto Ġtrain Ġthe Ġclass ifier . ĠFurther , Ġthis Ġframework Ġallows Ġto Ġgenerate Ġmultiple Ġinstances Ġof Ġthe Ġsame Ġtest Ġsample , Ġby Ġconsidering Ġpre selected Ġreference Ġsamples , Ġto Ġachieve Ġa Ġmore Ġprofound Ġdecision Ġmaking . ĠWe Ġillustrated Ġthe Ġsignificance Ġof Ġour Ġapproach Ġby Ġproviding Ġthe Ġexperimental Ġresults Ġfor Ġtwo Ġdifferent Ġtasks Ġnamely , Ġspeech / music Ġdiscrimination Ġand Ġemotion Ġclassification . ĠFurther , Ġwe Ġshowed Ġthat Ġthe Ġs 2 s Ġframework Ġcan Ġalso Ġhandle Ġthe Ġlow Ġres ourced Ġdata Ġimbalance Ġproblem . </s> References </s> [ 1 ] ĠH inton , ĠG ., ĠDeng , ĠL ., ĠYu , ĠD ., ĠDahl , ĠG . ĠE ., ĠMohamed , ĠA . ĠR ., ĠJ ait ly , ĠN ., ĠSenior , ĠA ., ĠVan hou c ke , ĠV ., ĠNguyen , ĠP ., ĠS ain ath , ĠT . ĠN . Ġ& ĠKings bury , ĠB . Ġ( 2012 ) ĠDeep Ġneural Ġnetworks Ġfor Ġacoustic Ġmodeling Ġin Ġspeech Ġrecognition : ĠThe Ġshared Ġviews Ġof Ġfour Ġresearch Ġgroups . ĠIEEE ĠSignal ĠProcessing ĠMagazine , Ġpp . Ġ82 âĢĵ 97 . </s> [ 2 ] ĠV iny als , ĠO ., ĠRav uri , ĠS . ĠV . Ġ& ĠP ove y , ĠD . Ġ( 2012 ) ĠRevis iting Ġrecurrent Ġneural Ġnetworks Ġfor Ġrobust ĠAS R . ĠIn ĠProc . ĠIEEE ĠInternational ĠConference Ġon ĠAc oust ics , ĠSpeech Ġand ĠSignal ĠProcessing Ġ( IC AS SP ), Ġpp . Ġ40 85 âĢĵ 40 88 . </s> [ 3 ] ĠLu , ĠL ., ĠZhang , ĠX ., ĠCho ĠK . Ġ& ĠRen als , ĠS . Ġ( 2015 ) ĠA Ġstudy Ġof Ġthe Ġrecurrent Ġneural Ġnetwork Ġenc oder - dec oder Ġfor Ġlarge Ġvocabulary Ġspeech Ġrecognition . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ3 249 âĢĵ 325 3 . </s> [ 4 ] ĠZhang , ĠY ., ĠPe z esh ki , ĠM ., ĠBra kel , ĠP ., ĠZhang , ĠS ., ĠBeng io , ĠC . L . Y . Ġ& ĠCour ville , ĠA . Ġ( 2017 ) ĠTowards Ġend - to - end Ġspeech Ġrecognition Ġwith Ġdeep Ġconv olution al Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 17 01 . 027 20 . </s> [ 5 ] ĠHan , ĠK ., ĠYu , ĠD . Ġ& ĠTas he v , ĠI . Ġ( 2014 ) ĠSpeech Ġemotion Ġrecognition Ġusing Ġdeep Ġneural Ġnetwork Ġand Ġextreme Ġlearning Ġmachine . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ223 âĢĵ 227 . </s> [ 6 ] ĠTr ige org is , ĠG ., ĠRing eval , ĠF ., ĠBr ue ck ner , ĠR ., ĠMarch i , ĠE ., ĠNicola ou , ĠM . ĠA ., ĠSch ull er , ĠB ., Ġ& ĠZ afe iri ou , ĠS . Ġ( 2016 ) ĠAd ieu Ġfeatures ? ĠEnd - to - end Ġspeech Ġemotion Ġrecognition Ġusing Ġa Ġdeep Ġconv olution al Ġrecurrent Ġnetwork . ĠIn ĠProc . ĠIC AS SP , Ġpp . Ġ5 200 âĢĵ 5 204 ). </s> [ 7 ] ĠHuang , ĠChe - We i , Ġ& ĠSh rik anth ĠS . ĠNar ayan an . Ġ( 2016 ) ĠAttention ĠAss isted ĠDiscovery Ġof ĠSub - Ut ter ance ĠStructure Ġin ĠSpeech ĠEm otion ĠRecogn ition . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ13 87 âĢĵ 13 91 . </s> [ 8 ] ĠZhang , ĠZ ., ĠRing eval , ĠF ., ĠHan , ĠJ ., ĠDeng , ĠJ ., ĠMarch i , ĠE . Ġ& ĠSch ull er , ĠB . Ġ( 2016 ) ĠF acing Ġrealism Ġin Ġspontaneous Ġemotion Ġrecognition Ġfrom Ġspeech : ĠFeature Ġenhancement Ġby Ġaut oen c oder Ġwith ĠL ST M Ġneural Ġnetworks . ĠIn ĠProc . ĠINTER SP E ECH , Ġpp . Ġ359 3 âĢĵ 35 97 . </s> [ 9 ] ĠSche ire r , ĠE . ĠD . Ġ& ĠSl aney , ĠM . Ġ( 2003 ) ĠMulti - feature Ġspeech / music Ġdiscrimination Ġsystem . ĠU . S . ĠPatent Ġ6 , 570 , 99 1 . </s> [ 10 ] ĠPik rak is , ĠA . Ġ& ĠThe odor idis ĠS . Ġ( 2014 ) ĠSpeech - music Ġdiscrimination : ĠA Ġdeep Ġlearning Ġperspective . ĠIn ĠProc . ĠEuropean ĠSignal ĠProcessing ĠConference Ġ( E US IP CO ), Ġpp . Ġ6 16 âĢĵ 620 . </s> [ 11 ] ĠChoi , ĠK ., ĠF az ek as , ĠG . Ġ& ĠSand ler , ĠM . Ġ( 2016 ) ĠAutomatic Ġtagging Ġusing Ġdeep Ġconv olution al Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 16 06 . 00 298 . </s> [ 12 ] ĠZ azo ĠCand il , ĠR ., ĠS ain ath , ĠT . N ., ĠSim ko , ĠG . Ġ& ĠPar ada , ĠC . Ġ( 2016 ) ĠFeature Ġlearning Ġwith Ġraw - wave form ĠCL D NN s Ġfor ĠVoice ĠActivity ĠDetection . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ3 668 âĢĵ 36 72 . </s> [ 13 ] ĠTian , ĠL ., ĠMoore , ĠJD . Ġ& ĠL ai , ĠC . Ġ( 2015 ) ĠEm otion Ġrecognition Ġin Ġspontaneous Ġand Ġacted Ġdialog ues . ĠIn ĠProc . ĠInternational ĠConference Ġon ĠAffect ive ĠComputing Ġand ĠIntelligent ĠInter action Ġ( AC II ), Ġpp . Ġ698 âĢĵ 704 . </s> [ 14 ] ĠD ump ala , ĠS . H ., ĠChak rab ort y , ĠR . Ġ& ĠK opp ar ap u , ĠS . K . Ġ( 2017 ) Ġk - FF NN : ĠA Ġprior i Ġknowledge Ġinfused Ġfeed - forward Ġneural Ġnetworks . Ġar X iv Ġpre print , Ġar X iv : 17 04 . 070 55 . </s> [ 15 ] ĠD ump ala , ĠS . H ., ĠChak rab ort y , ĠR . Ġ& ĠK opp ar ap u , ĠS . K . Ġ( 2018 ) ĠKnowledge - driven Ġfeed - forward Ġneural Ġnetwork Ġfor Ġaudio Ġaffect ive Ġcontent Ġanalysis . ĠTo Ġappear Ġin ĠAAA I - 2018 Ġworkshop Ġon ĠAffect ive ĠContent ĠAnalysis . </s> [ 16 ] ĠDeng , ĠJ ., ĠZhang , ĠZ ., ĠMarch i , ĠE . Ġ& ĠSch ull er , ĠB . Ġ( 2013 ) ĠSp arse Ġaut oen c oder - based Ġfeature Ġtransfer Ġlearning Ġfor Ġspeech Ġemotion Ġrecognition . ĠIn ĠProc . ĠHum aine ĠAssociation ĠConference Ġon ĠAffect ive ĠComputing Ġand ĠIntelligent ĠInter action Ġ( AC II ), Ġpp . Ġ5 11 âĢĵ 516 . </s> [ 17 ] ĠGeorge ĠT zan et akis , ĠG t zan Ġmus ics pe ech . ĠAv ail abe Ġon - line Ġat Ġhttp :// m ars y as . info / download / dat as ets . </s> [ 18 ] ĠBurk hardt , ĠF ., ĠPa es ch ke , ĠA ., ĠR olf es , ĠM ., ĠSend l meier , ĠW . F . Ġ& ĠWeiss , ĠB . Ġ( 2005 ) ĠA Ġdatabase Ġof Ġg erman Ġemotional Ġspeech . ĠIn ĠProc . ĠIn ters pe ech , Ġpp . Ġ15 17 âĢĵ 15 20 . </s> [ 19 ] ĠHall , ĠM ., ĠFrank , ĠE ., ĠHolmes , ĠG ., ĠPf ah ring er , ĠB ., ĠRe ut em ann , ĠP . Ġ& ĠW itten , ĠI . H . Ġ( 2009 ) ĠThe ĠWE KA Ġdata Ġmining Ġsoftware : Ġan Ġupdate . ĠAC M ĠSIG K DD Ġexplor ations Ġnewsletter , Ġ11 ( 1 ), Ġpp . Ġ10 âĢĵ 18 . </s> [ 20 ] ĠEy ben , ĠF ., ĠW ening er , ĠF ., ĠGross , ĠF . Ġ& ĠSch ull er , ĠB . Ġ( 2013 ) ĠRecent Ġdevelopments Ġin Ġopens mile , Ġthe Ġmun ich Ġopen - source Ġmultimedia Ġfeature Ġextract or . ĠIn ĠProc . ĠAC M Ġinternational Ġconference Ġon ĠMult imedia , Ġpp . Ġ8 35 âĢĵ 8 38 . </s> [ 21 ] ĠMar ate a , ĠA ., ĠPet ros ino , ĠA . Ġ& ĠMan zo , ĠM . Ġ( 2014 ) ĠAdjusted Ġf - me asure Ġand Ġkernel Ġscaling Ġfor Ġim balanced Ġdata Ġlearning . ĠInformation ĠSciences Ġ257 : 331 âĢĵ 341 . </s> [ 22 ] ĠGal ar , ĠM ., ĠFernandez , ĠA ., ĠBar rene che a , ĠE . Ġ& ĠHerrera , ĠF . Ġ( 2013 ) ĠE us boost : ĠEnh ancing Ġen semb les Ġfor Ġhighly Ġim balanced Ġdata - sets Ġby Ġevolutionary Ġunders am pling . ĠPattern ĠRecogn ition Ġ46 ( 12 ): 34 60 âĢĵ 34 71 . </s> [ 23 ] ĠBar ua , ĠS ., ĠIslam , ĠM . ĠM ., ĠYao , ĠX . Ġ& ĠMur ase , ĠK . Ġ( 2014 ) ĠM wm ote âĢĵ majority Ġweighted Ġminority Ġovers am pling Ġtechnique Ġfor Ġim balanced Ġdata Ġset Ġlearning . ĠIEEE ĠTransactions Ġon ĠKnowledge Ġand ĠData ĠEngineering Ġ26 ( 2 ): 405 âĢĵ 425 .