They use text transcription.
ABSTRACTIVE
both
ABSTRACTIVE
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7 . We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal. To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27 . Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words. The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2 . In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input. Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function.
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree. Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill. The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day.” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input.
1810.04635
Introduction </s> Recently , Ġdeep Ġlearning Ġalgorithms Ġhave Ġsuccessfully Ġaddressed Ġproblems Ġin Ġvarious Ġfields , Ġsuch Ġas Ġimage Ġclassification , Ġmachine Ġtranslation , Ġspeech Ġrecognition , Ġtext - to - speech Ġgeneration Ġand Ġother Ġmachine Ġlearning Ġrelated Ġareas ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠSimilarly , Ġsubstantial Ġimprovements Ġin Ġperformance Ġhave Ġbeen Ġobtained Ġwhen Ġdeep Ġlearning Ġalgorithms Ġhave Ġbeen Ġapplied Ġto Ġstatistical Ġspeech Ġprocessing ĠB IB REF 3 Ġ. ĠThese Ġfundamental Ġimprovements Ġhave Ġled Ġresearchers Ġto Ġinvestigate Ġadditional Ġtopics Ġrelated Ġto Ġhuman Ġnature , Ġwhich Ġhave Ġlong Ġbeen Ġobjects Ġof Ġstudy . ĠOne Ġsuch Ġtopic Ġinvolves Ġunderstanding Ġhuman Ġemotions Ġand Ġreflecting Ġit Ġthrough Ġmachine Ġintelligence , Ġsuch Ġas Ġemotional Ġdialogue Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> In Ġdeveloping Ġemotionally Ġaware Ġintelligence , Ġthe Ġvery Ġfirst Ġstep Ġis Ġbuilding Ġrobust Ġemotion Ġclass ifiers Ġthat Ġdisplay Ġgood Ġperformance Ġregardless Ġof Ġthe Ġapplication ; Ġthis Ġoutcome Ġis Ġconsidered Ġto Ġbe Ġone Ġof Ġthe Ġfundamental Ġresearch Ġgoals Ġin Ġaffect ive Ġcomputing ĠB IB REF 6 Ġ. ĠIn Ġparticular , Ġthe Ġspeech Ġemotion Ġrecognition Ġtask Ġis Ġone Ġof Ġthe Ġmost Ġimportant Ġproblems Ġin Ġthe Ġfield Ġof Ġpar aling u istics . ĠThis Ġfield Ġhas Ġrecently Ġbroad ened Ġits Ġapplications , Ġas Ġit Ġis Ġa Ġcrucial Ġfactor Ġin Ġoptimal Ġhuman - computer Ġinteractions , Ġincluding Ġdialog Ġsystems . ĠThe Ġgoal Ġof Ġspeech Ġemotion Ġrecognition Ġis Ġto Ġpredict Ġthe Ġemotional Ġcontent Ġof Ġspeech Ġand Ġto Ġclassify Ġspeech Ġaccording Ġto Ġone Ġof Ġseveral Ġlabels Ġ( i . e ., Ġhappy , Ġsad , Ġneutral , Ġand Ġangry ). ĠVarious Ġtypes Ġof Ġdeep Ġlearning Ġmethods Ġhave Ġbeen Ġapplied Ġto Ġincrease Ġthe Ġperformance Ġof Ġemotion Ġclass ifiers ; Ġhowever , Ġthis Ġtask Ġis Ġstill Ġconsidered Ġto Ġbe Ġchallenging Ġfor Ġseveral Ġreasons . ĠFirst , Ġinsufficient Ġdata Ġfor Ġtraining Ġcomplex Ġneural Ġnetwork - based Ġmodels Ġare Ġavailable , Ġdue Ġto Ġthe Ġcosts Ġassociated Ġwith Ġhuman Ġinvolvement . ĠSecond , Ġthe Ġcharacteristics Ġof Ġemotions Ġmust Ġbe Ġlearned Ġfrom Ġlow - level Ġspeech Ġsignals . ĠFeature - based Ġmodels Ġdisplay Ġlimited Ġskills Ġwhen Ġapplied Ġto Ġthis Ġproblem . </s> To Ġovercome Ġthese Ġlimitations , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġuses Ġhigh - level Ġtext Ġtranscription , Ġas Ġwell Ġas Ġlow - level Ġaudio Ġsignals , Ġto Ġutilize Ġthe Ġinformation Ġcontained Ġwithin Ġlow - resource Ġdatasets Ġto Ġa Ġgreater Ġdegree . ĠGiven Ġrecent Ġimprovements Ġin Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġtechnology ĠB IB REF 7 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, Ġspeech Ġtranscription Ġcan Ġbe Ġcarried Ġout Ġusing Ġaudio Ġsignals Ġwith Ġconsiderable Ġskill . ĠThe Ġemotional Ġcontent Ġof Ġspeech Ġis Ġclearly Ġindicated Ġby Ġthe Ġemotion Ġwords Ġcontained Ġin Ġa Ġsentence ĠB IB REF 10 Ġ, Ġsuch Ġas ĠâĢ ľ love ly âĢ Ŀ Ġand ĠâĢ ľ aw esome , âĢ Ŀ Ġwhich Ġcarry Ġstrong Ġemotions Ġcompared Ġto Ġgeneric Ġ( non - em otion ) Ġwords , Ġsuch Ġas ĠâĢ ľ person âĢ Ŀ Ġand ĠâĢ ľ day . âĢ Ŀ ĠThus , Ġwe Ġhypothes ize Ġthat Ġthe Ġspeech Ġemotion Ġrecognition Ġmodel Ġwill Ġbe Ġbenefit Ġfrom Ġthe Ġincorporation Ġof Ġhigh - level Ġtextual Ġinput . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġdeep Ġdual Ġrecurrent Ġenc oder Ġmodel Ġthat Ġsimultaneously Ġutilizes Ġaudio Ġand Ġtext Ġdata Ġin Ġrecognizing Ġemotions Ġfrom Ġspeech . ĠExt ensive Ġexperiments Ġare Ġconducted Ġto Ġinvestigate Ġthe Ġefficacy Ġand Ġproperties Ġof Ġthe Ġproposed Ġmodel . ĠOur Ġproposed Ġmodel Ġoutper forms Ġprevious Ġstate - of - the - art Ġmethods Ġby Ġ68 . 8 % Ġto Ġ71 . 8 % Ġwhen Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġwell - stud ied Ġdatasets . ĠBased Ġon Ġan Ġerror Ġanalysis Ġof Ġthe Ġmodels , Ġwe Ġshow Ġthat Ġour Ġproposed Ġmodel Ġaccurately Ġidentifies Ġemotion Ġclasses . ĠMoreover , Ġthe Ġneutral Ġclass Ġmis class ification Ġbias Ġfrequently Ġexhibited Ġby Ġprevious Ġmodels , Ġwhich Ġfocus Ġon Ġaudio Ġfeatures , Ġis Ġless Ġpronounced Ġin Ġour Ġmodel . </s> Related Ġwork </s> Class ical Ġmachine Ġlearning Ġalgorithms , Ġsuch Ġas Ġhidden ĠMark ov Ġmodels Ġ( HM Ms ), Ġsupport Ġvector Ġmachines Ġ( S V Ms ), Ġand Ġdecision Ġtree - based Ġmethods , Ġhave Ġbeen Ġemployed Ġin Ġspeech Ġemotion Ġrecognition Ġproblems ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠRecently , Ġresearchers Ġhave Ġproposed Ġvarious Ġneural Ġnetwork - based Ġarchitectures Ġto Ġimprove Ġthe Ġperformance Ġof Ġspeech Ġemotion Ġrecognition . ĠAn Ġinitial Ġstudy Ġutilized Ġdeep Ġneural Ġnetworks Ġ( D NN s ) Ġto Ġextract Ġhigh - level Ġfeatures Ġfrom Ġraw Ġaudio Ġdata Ġand Ġdemonstrated Ġits Ġeffectiveness Ġin Ġspeech Ġemotion Ġrecognition ĠB IB REF 14 Ġ. ĠWith Ġthe Ġadvancement Ġof Ġdeep Ġlearning Ġmethods , Ġmore Ġcomplex Ġneural - based Ġarchitectures Ġhave Ġbeen Ġproposed . ĠConv olution al Ġneural Ġnetwork Ġ( CNN )- based Ġmodels Ġhave Ġbeen Ġtrained Ġon Ġinformation Ġderived Ġfrom Ġraw Ġaudio Ġsignals Ġusing Ġspect rog rams Ġor Ġaudio Ġfeatures Ġsuch Ġas ĠMel - frequency Ġc ep st ral Ġcoefficients Ġ( MF CC s ) Ġand Ġlow - level Ġdescript ors Ġ( LL Ds ) ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠThese Ġneural Ġnetwork - based Ġmodels Ġare Ġcombined Ġto Ġproduce Ġhigher - complex ity Ġmodels ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, Ġand Ġthese Ġmodels Ġachieved Ġthe Ġbest - recorded Ġperformance Ġwhen Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset . </s> Another Ġline Ġof Ġresearch Ġhas Ġfocused Ġon Ġadopting Ġvariant Ġmachine Ġlearning Ġtechniques Ġcombined Ġwith Ġneural Ġnetwork - based Ġmodels . ĠOne Ġresearcher Ġutilized Ġthe Ġmulti object Ġlearning Ġapproach Ġand Ġused Ġgender Ġand Ġnatural ness Ġas Ġauxiliary Ġtasks Ġso Ġthat Ġthe Ġneural Ġnetwork - based Ġmodel Ġlearned Ġmore Ġfeatures Ġfrom Ġa Ġgiven Ġdataset ĠB IB REF 20 Ġ. ĠAnother Ġresearcher Ġinvestigated Ġtransfer Ġlearning Ġmethods , Ġleveraging Ġexternal Ġdata Ġfrom Ġrelated Ġdomains ĠB IB REF 21 Ġ. </s> As Ġemotional Ġdialogue Ġis Ġcomposed Ġof Ġsound Ġand Ġspoken Ġcontent , Ġresearchers Ġhave Ġalso Ġinvestigated Ġthe Ġcombination Ġof Ġacoustic Ġfeatures Ġand Ġlanguage Ġinformation , Ġbuilt Ġbelief Ġnetwork - based Ġmethods Ġof Ġidentifying Ġemotional Ġkey Ġphrases , Ġand Ġassessed Ġthe Ġemotional Ġsal ience Ġof Ġverbal Ġcues Ġfrom Ġboth Ġphon eme Ġsequences Ġand Ġwords ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠHowever , Ġnone Ġof Ġthese Ġstudies Ġhave Ġutilized Ġinformation Ġfrom Ġspeech Ġsignals Ġand Ġtext Ġsequences Ġsimultaneously Ġin Ġan Ġend - to - end Ġlearning Ġneural Ġnetwork - based Ġmodel Ġto Ġclassify Ġemotions . </s> Model </s> This Ġsection Ġdescribes Ġthe Ġmethod ologies Ġthat Ġare Ġapplied Ġto Ġthe Ġspeech Ġemotion Ġrecognition Ġtask . ĠWe Ġstart Ġby Ġintroducing Ġthe Ġrecurrent Ġenc oder Ġmodel Ġfor Ġthe Ġaudio Ġand Ġtext Ġmod alities Ġindividually . ĠWe Ġthen Ġpropose Ġa Ġmultim od al Ġapproach Ġthat Ġenc odes Ġboth Ġaudio Ġand Ġtextual Ġinformation Ġsimultaneously Ġvia Ġa Ġdual Ġrecurrent Ġenc oder . </s> Audio ĠRec urrent ĠEnc oder Ġ( ARE ) </s> Mot ivated Ġby Ġthe Ġarchitecture Ġused Ġin ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġwe Ġbuild Ġan Ġaudio Ġrecurrent Ġenc oder Ġ( ARE ) Ġto Ġpredict Ġthe Ġclass Ġof Ġa Ġgiven Ġaudio Ġsignal . ĠOnce ĠMF CC Ġfeatures Ġhave Ġbeen Ġextracted Ġfrom Ġan Ġaudio Ġsignal , Ġa Ġsubset Ġof Ġthe Ġsequential Ġfeatures Ġis Ġfed Ġinto Ġthe ĠR NN Ġ( i . e ., Ġg ated Ġrecurrent Ġunits Ġ( GR Us )), Ġwhich Ġleads Ġto Ġthe Ġformation Ġof Ġthe Ġnetwork 's Ġinternal Ġhidden Ġstate ĠIN LINE FORM 0 Ġto Ġmodel Ġthe Ġtime Ġseries Ġpatterns . ĠThis Ġinternal Ġhidden Ġstate Ġis Ġupdated Ġat Ġeach Ġtime Ġstep Ġwith Ġthe Ġinput Ġdata ĠIN LINE FORM 1 Ġand Ġthe Ġhidden Ġstate Ġof Ġthe Ġprevious Ġtime Ġstep ĠIN LINE FORM 2 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠR NN Ġfunction Ġwith Ġweight Ġparameter ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġrepresents Ġthe Ġhidden Ġstate Ġat Ġt - ĠIN LINE FORM 3 Ġtime Ġstep , Ġand ĠIN LINE FORM 4 Ġrepresents Ġthe Ġt - ĠIN LINE FORM 5 ĠMF CC Ġfeatures Ġin ĠIN LINE FORM 6 Ġ. ĠAfter Ġencoding Ġthe Ġaudio Ġsignal ĠIN LINE FORM 7 Ġwith Ġthe ĠR NN , Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe ĠR NN , ĠIN LINE FORM 8 Ġ, Ġis Ġconsidered Ġto Ġbe Ġthe Ġrepresentative Ġvector Ġthat Ġcontains Ġall Ġof Ġthe Ġsequential Ġaudio Ġdata . ĠThis Ġvector Ġis Ġthen Ġconc aten ated Ġwith Ġanother Ġpros odic Ġfeature Ġvector , ĠIN LINE FORM 9 Ġ, Ġto Ġgenerate Ġa Ġmore Ġinformative Ġvector Ġrepresentation Ġof Ġthe Ġsignal , ĠIN LINE FORM 10 Ġ. ĠThe ĠMF CC Ġand Ġthe Ġpros odic Ġfeatures Ġare Ġextracted Ġfrom Ġthe Ġaudio Ġsignal Ġusing Ġthe Ġopen S MI LE Ġtool kit ĠB IB REF 26 Ġ, ĠIN LINE FORM 11 Ġ, Ġrespectively . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġby Ġapplying Ġthe Ġsoft max Ġfunction Ġto Ġthe Ġvector ĠIN LINE FORM 12 Ġ. ĠFor Ġa Ġgiven Ġaudio Ġsample ĠIN LINE FORM 13 Ġ, Ġwe Ġassume Ġthat ĠIN LINE FORM 14 Ġis Ġthe Ġtrue Ġlabel Ġvector , Ġwhich Ġcontains Ġall Ġz eros Ġbut Ġcontains Ġa Ġone Ġat Ġthe Ġcorrect Ġclass , Ġand ĠIN LINE FORM 15 Ġis Ġthe Ġpredicted Ġprobability Ġdistribution Ġfrom Ġthe Ġsoft max Ġlayer . ĠThe Ġtraining Ġobjective Ġthen Ġtakes Ġthe Ġfollowing Ġform : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcalculated Ġrepresentative Ġvector Ġof Ġthe Ġaudio Ġsignal Ġwith Ġdimension ality ĠIN LINE FORM 1 Ġ. ĠThe ĠIN LINE FORM 2 Ġand Ġthe Ġbias ĠIN LINE FORM 3 Ġare Ġlearned Ġmodel Ġparameters . ĠC Ġis Ġthe Ġtotal Ġnumber Ġof Ġclasses , Ġand ĠN Ġis Ġthe Ġtotal Ġnumber Ġof Ġsamples Ġused Ġin Ġtraining . ĠThe Ġupper Ġpart Ġof ĠFigure Ġshows Ġthe Ġarchitecture Ġof Ġthe ĠARE Ġmodel . </s> Text ĠRec urrent ĠEnc oder Ġ( T RE ) </s> We Ġassume Ġthat Ġspeech Ġtranscripts Ġcan Ġbe Ġextracted Ġfrom Ġaudio Ġsignals Ġwith Ġhigh Ġaccuracy , Ġgiven Ġthe Ġadvancement Ġof ĠAS R Ġtechnologies ĠB IB REF 7 Ġ. ĠWe Ġattempt Ġto Ġuse Ġthe Ġprocessed Ġtextual Ġinformation Ġas Ġanother Ġmod ality Ġin Ġpredicting Ġthe Ġemotion Ġclass Ġof Ġa Ġgiven Ġsignal . ĠTo Ġuse Ġtextual Ġinformation , Ġa Ġspeech Ġtranscript Ġis Ġtoken ized Ġand Ġindexed Ġinto Ġa Ġsequence Ġof Ġtokens Ġusing Ġthe ĠNatural ĠLanguage ĠTool kit Ġ( N LT K ) ĠB IB REF 27 Ġ. ĠEach Ġtoken Ġis Ġthen Ġpassed Ġthrough Ġa Ġword - embed ding Ġlayer Ġthat Ġconverts Ġa Ġword Ġindex Ġto Ġa Ġcorresponding Ġ300 - dimensional Ġvector Ġthat Ġcontains Ġadditional Ġcontextual Ġmeaning Ġbetween Ġwords . ĠThe Ġsequence Ġof Ġembedded Ġtokens Ġis Ġfed Ġinto Ġa Ġtext Ġrecurrent Ġenc oder Ġ( T RE ) Ġin Ġsuch Ġa Ġway Ġthat Ġthe Ġaudio ĠMF CC Ġfeatures Ġare Ġencoded Ġusing Ġthe ĠARE Ġrepresented Ġby Ġequation ĠEQ REF 2 Ġ. ĠIn Ġthis Ġcase , ĠIN LINE FORM 0 Ġis Ġthe Ġt - ĠIN LINE FORM 1 Ġembedded Ġtoken Ġfrom Ġthe Ġtext Ġinput . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġfrom Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe Ġtext - R NN Ġusing Ġthe Ġsoft max Ġfunction . </s> We Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġlast Ġhidden Ġstate Ġof Ġthe Ġtext - R NN , ĠIN LINE FORM 1 Ġ, Ġand Ġthe ĠIN LINE FORM 2 Ġand Ġbias ĠIN LINE FORM 3 Ġare Ġlearned Ġmodel Ġparameters . ĠThe Ġlower Ġpart Ġof ĠFigure Ġindicates Ġthe Ġarchitecture Ġof Ġthe ĠTRE Ġmodel . </s> Mult im od al ĠDual ĠRec urrent ĠEnc oder Ġ( MD RE ) </s> We Ġpresent Ġa Ġnovel Ġarchitecture Ġcalled Ġthe Ġmultim od al Ġdual Ġrecurrent Ġenc oder Ġ( MD RE ) Ġto Ġovercome Ġthe Ġlimitations Ġof Ġexisting Ġapproaches . ĠIn Ġthis Ġstudy , Ġwe Ġconsider Ġmultiple Ġmod alities , Ġsuch Ġas ĠMF CC Ġfeatures , Ġpros odic Ġfeatures Ġand Ġtranscripts , Ġwhich Ġcontain Ġsequential Ġaudio Ġinformation , Ġstatistical Ġaudio Ġinformation Ġand Ġtextual Ġinformation , Ġrespectively . ĠThese Ġtypes Ġof Ġdata Ġare Ġthe Ġsame Ġas Ġthose Ġused Ġin Ġthe ĠARE Ġand ĠTRE Ġcases . ĠThe ĠMD RE Ġmodel Ġemploys Ġtwo ĠR NN s Ġto Ġencode Ġdata Ġfrom Ġthe Ġaudio Ġsignal Ġand Ġtextual Ġinputs Ġindependently . ĠThe Ġaudio - R NN Ġenc odes ĠMF CC Ġfeatures Ġfrom Ġthe Ġaudio Ġsignal Ġusing Ġequation ĠEQ REF 2 Ġ. ĠThe Ġlast Ġhidden Ġstate Ġof Ġthe Ġaudio - R NN Ġis Ġconc aten ated Ġwith Ġthe Ġpros odic Ġfeatures Ġto Ġform Ġthe Ġfinal Ġvector Ġrepresentation ĠIN LINE FORM 0 Ġ, Ġand Ġthis Ġvector Ġis Ġthen Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġlayer Ġto Ġform Ġthe Ġaudio Ġencoding Ġvector ĠA . ĠOn Ġthe Ġother Ġhand , Ġthe Ġtext - R NN Ġenc odes Ġthe Ġword Ġsequence Ġof Ġthe Ġtranscript Ġusing Ġequation ĠEQ REF 2 Ġ. ĠThe Ġfinal Ġhidden Ġstates Ġof Ġthe Ġtext - R NN Ġare Ġalso Ġpassed Ġthrough Ġanother Ġfully Ġconnected Ġneural Ġnetwork Ġlayer Ġto Ġform Ġa Ġtextual Ġencoding Ġvector ĠT . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġby Ġapplying Ġthe Ġsoft max Ġfunction Ġto Ġthe Ġconc aten ation Ġof Ġthe Ġvectors ĠA Ġand ĠT . ĠWe Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfeed - forward Ġneural Ġnetwork Ġwith Ġweight Ġparameter ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġare Ġfinal Ġencoding Ġvectors Ġfrom Ġthe Ġaudio - R NN Ġand Ġtext - R NN , Ġrespectively . ĠIN LINE FORM 4 Ġand Ġthe Ġbias ĠIN LINE FORM 5 Ġare Ġlearned Ġmodel Ġparameters . </s> Mult im od al ĠDual ĠRec urrent ĠEnc oder Ġwith ĠAttention Ġ( MD RE A ) </s> Insp ired Ġby Ġthe Ġconcept Ġof Ġthe Ġattention Ġmechanism Ġused Ġin Ġneural Ġmachine Ġtranslation ĠB IB REF 28 Ġ, Ġwe Ġpropose Ġa Ġnovel Ġmultim od al Ġattention Ġmethod Ġto Ġfocus Ġon Ġthe Ġspecific Ġparts Ġof Ġa Ġtranscript Ġthat Ġcontain Ġstrong Ġemotional Ġinformation , Ġconditioning Ġon Ġthe Ġaudio Ġinformation . ĠFigure Ġshows Ġthe Ġarchitecture Ġof Ġthe ĠMD RE A Ġmodel . ĠFirst , Ġthe Ġaudio Ġdata Ġand Ġtext Ġdata Ġare Ġencoded Ġwith Ġthe Ġaudio - R NN Ġand Ġtext - R NN Ġusing Ġequation ĠEQ REF 2 Ġ. ĠWe Ġthen Ġconsider Ġthe Ġfinal Ġaudio Ġencoding Ġvector ĠIN LINE FORM 0 Ġas Ġa Ġcontext Ġvector . ĠAs Ġseen Ġin Ġequation ĠEQ REF 9 Ġ, Ġduring Ġeach Ġtime Ġstep Ġt , Ġthe Ġdot Ġproduct Ġbetween Ġthe Ġcontext Ġvector Ġe Ġand Ġthe Ġhidden Ġstate Ġof Ġthe Ġtext - R NN Ġat Ġeach Ġt - th Ġsequence ĠIN LINE FORM 1 Ġis Ġevaluated Ġto Ġcalculate Ġa Ġsimilarity Ġscore ĠIN LINE FORM 2 Ġ. ĠUsing Ġthis Ġscore ĠIN LINE FORM 3 Ġas Ġa Ġweight Ġparameter , Ġthe Ġweighted Ġsum Ġof Ġthe Ġsequences Ġof Ġthe Ġhidden Ġstate Ġof Ġthe Ġtext - R NN , ĠIN LINE FORM 4 Ġ, Ġis Ġcalculated Ġto Ġgenerate Ġan Ġattention - application Ġvector ĠZ . ĠThis Ġattention - application Ġvector Ġis Ġconc aten ated Ġwith Ġthe Ġfinal Ġencoding Ġvector Ġof Ġthe Ġaudio - R NN ĠIN LINE FORM 5 Ġ( equ ation ĠEQ REF 7 Ġ), Ġwhich Ġwill Ġbe Ġpassed Ġthrough Ġthe Ġsoft max Ġfunction Ġto Ġpredict Ġthe Ġemotion Ġclass . ĠWe Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand Ġthe Ġbias ĠIN LINE FORM 1 Ġare Ġlearned Ġmodel Ġparameters . </s> Dat as et </s> We Ġevaluate Ġour Ġmodel Ġusing Ġthe ĠInteractive ĠEm otional ĠDy adic ĠMotion ĠCapture Ġ( I EM OC AP ) ĠB IB REF 18 Ġdataset . ĠThis Ġdataset Ġwas Ġcollected Ġfollowing Ġtheatrical Ġtheory Ġin Ġorder Ġto Ġsimulate Ġnatural Ġdy adic Ġinteractions Ġbetween Ġactors . ĠWe Ġuse Ġcateg orical Ġevaluations Ġwith Ġmajority Ġagreement . ĠWe Ġuse Ġonly Ġfour Ġemotional Ġcategories Ġhappy , Ġsad , Ġangry , Ġand Ġneutral Ġto Ġcompare Ġthe Ġperformance Ġof Ġour Ġmodel Ġwith Ġother Ġresearch Ġusing Ġthe Ġsame Ġcategories . ĠThe ĠI EM OC AP Ġdataset Ġincludes Ġfive Ġsessions , Ġand Ġeach Ġsession Ġcontains Ġutter ances Ġfrom Ġtwo Ġspeakers Ġ( one Ġmale Ġand Ġone Ġfemale ). ĠThis Ġdata Ġcollection Ġprocess Ġresulted Ġin Ġ10 Ġunique Ġspeakers . ĠFor Ġconsistent Ġcomparison Ġwith Ġprevious Ġwork , Ġwe Ġmerge Ġthe Ġexcitement Ġdataset Ġwith Ġthe Ġhappiness Ġdataset . ĠThe Ġfinal Ġdataset Ġcontains Ġa Ġtotal Ġof Ġ55 31 Ġutter ances Ġ( 16 36 Ġhappy , Ġ10 84 Ġsad , Ġ110 3 Ġangry , Ġ17 08 Ġneutral ). </s> Feature Ġextraction </s> To Ġextract Ġspeech Ġinformation Ġfrom Ġaudio Ġsignals , Ġwe Ġuse ĠMF CC Ġvalues , Ġwhich Ġare Ġwidely Ġused Ġin Ġanalyzing Ġaudio Ġsignals . ĠThe ĠMF CC Ġfeature Ġset Ġcontains Ġa Ġtotal Ġof Ġ39 Ġfeatures , Ġwhich Ġinclude Ġ12 ĠMF CC Ġparameters Ġ( 1 - 12 ) Ġfrom Ġthe Ġ26 ĠMel - frequency Ġbands Ġand Ġlog - energy Ġparameters , Ġ13 Ġdelta Ġand Ġ13 Ġacceleration Ġcoefficients ĠThe Ġframe Ġsize Ġis Ġset Ġto Ġ25 Ġms Ġat Ġa Ġrate Ġof Ġ10 Ġms Ġwith Ġthe ĠHam ming Ġfunction . ĠAccording Ġto Ġthe Ġlength Ġof Ġeach Ġwave Ġfile , Ġthe Ġsequential Ġstep Ġof Ġthe ĠMF CC Ġfeatures Ġis Ġvaried . ĠTo Ġextract Ġadditional Ġinformation Ġfrom Ġthe Ġdata , Ġwe Ġalso Ġuse Ġpros odic Ġfeatures , Ġwhich Ġshow Ġeffectiveness Ġin Ġaffect ive Ġcomputing . ĠThe Ġpros odic Ġfeatures Ġare Ġcomposed Ġof Ġ35 Ġfeatures , Ġwhich Ġinclude Ġthe ĠF 0 Ġfrequency , Ġthe Ġvoicing Ġprobability , Ġand Ġthe Ġloud ness Ġcont ours . ĠAll Ġof Ġthese ĠMF CC Ġand Ġpros odic Ġfeatures Ġare Ġextracted Ġfrom Ġthe Ġdata Ġusing Ġthe ĠOpen S MI LE Ġtool kit ĠB IB REF 26 Ġ. </s> Im plementation Ġdetails </s> Among Ġthe Ġvariants Ġof Ġthe ĠR NN Ġfunction , Ġwe Ġuse ĠGR Us Ġas Ġthey Ġyield Ġcomparable Ġperformance Ġto Ġthat Ġof Ġthe ĠL ST M Ġand Ġinclude Ġa Ġsmaller Ġnumber Ġof Ġweight Ġparameters ĠB IB REF 29 Ġ. ĠWe Ġuse Ġa Ġmax Ġenc oder Ġstep Ġof Ġ750 Ġfor Ġthe Ġaudio Ġinput , Ġbased Ġon Ġthe Ġimplementation Ġchoices Ġpresented Ġin ĠB IB REF 30 Ġand Ġ128 Ġfor Ġthe Ġtext Ġinput Ġbecause Ġit Ġcovers Ġthe Ġmaximum Ġlength Ġof Ġthe Ġtranscripts . ĠThe Ġvocabulary Ġsize Ġof Ġthe Ġdataset Ġis Ġ3 , 747 , Ġincluding Ġthe ĠâĢ ľ _ UN K _ " Ġtoken , Ġwhich Ġrepresents Ġunknown Ġwords , Ġand Ġthe ĠâĢ ľ _ P AD _ " Ġtoken , Ġwhich Ġis Ġused Ġto Ġindicate Ġpadding Ġinformation Ġadded Ġwhile Ġpreparing Ġmini - batch Ġdata . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġand Ġthe Ġnumber Ġof Ġlayers Ġin Ġthe ĠR NN Ġfor Ġeach Ġmodel Ġ( ARE , ĠTRE , ĠMD RE Ġand ĠMD RE A ) Ġare Ġselected Ġbased Ġon Ġextensive Ġhyper param eter Ġsearch Ġexperiments . ĠThe Ġweights Ġof Ġthe Ġhidden Ġunits Ġare Ġinitialized Ġusing Ġorth og onal Ġweights ĠB IB REF 31 Ġ], Ġand Ġthe Ġtext Ġembed ding Ġlayer Ġis Ġinitialized Ġfrom Ġpret rained Ġword - embed ding Ġvectors ĠB IB REF 32 Ġ. </s> In Ġpreparing Ġthe Ġtextual Ġdataset , Ġwe Ġfirst Ġuse Ġthe Ġreleased Ġtranscripts Ġof Ġthe ĠI EM OC AP Ġdataset Ġfor Ġsimplicity . ĠTo Ġinvestigate Ġthe Ġpractical Ġperformance , Ġwe Ġthen Ġprocess Ġall Ġof Ġthe ĠI EM OC AP Ġaudio Ġdata Ġusing Ġan ĠAS R Ġsystem Ġ( the ĠGoogle ĠCloud ĠSpeech ĠAPI ) Ġand Ġretrieve Ġthe Ġtranscripts . ĠThe Ġperformance Ġof Ġthe ĠGoogle ĠAS R Ġsystem Ġis Ġreflected Ġby Ġits Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġ5 . 53 %. </s> Performance Ġevaluation </s> As Ġthe Ġdataset Ġis Ġnot Ġexplicitly Ġsplit Ġbeforehand Ġinto Ġtraining , Ġdevelopment , Ġand Ġtesting Ġsets , Ġwe Ġperform Ġ5 - fold Ġcross Ġvalidation Ġto Ġdetermine Ġthe Ġoverall Ġperformance Ġof Ġthe Ġmodel . ĠThe Ġdata Ġin Ġeach Ġfold Ġare Ġsplit Ġinto Ġtraining , Ġdevelopment , Ġand Ġtesting Ġdatasets Ġ( 8 : 0 . 5 : 1 . 5 , Ġrespectively ). ĠAfter Ġtraining Ġthe Ġmodel , Ġwe Ġmeasure Ġthe Ġweighted Ġaverage Ġprecision Ġ( W AP ) Ġover Ġthe Ġ5 - fold Ġdataset . ĠWe Ġtrain Ġand Ġevaluate Ġthe Ġmodel Ġ10 Ġtimes Ġper Ġfold , Ġand Ġthe Ġmodel Ġperformance Ġis Ġassessed Ġin Ġterms Ġof Ġthe Ġmean Ġscore Ġand Ġstandard Ġdeviation . </s> We Ġexamine Ġthe ĠW AP Ġvalues , Ġwhich Ġare Ġshown Ġin ĠTable Ġ1 . ĠFirst , Ġour ĠARE Ġmodel Ġshows Ġthe Ġbaseline Ġperformance Ġbecause Ġwe Ġuse Ġminimal Ġaudio Ġfeatures , Ġsuch Ġas Ġthe ĠMF CC Ġand Ġpros odic Ġfeatures Ġwith Ġsimple Ġarchitectures . ĠOn Ġthe Ġother Ġhand , Ġthe ĠTRE Ġmodel Ġshows Ġhigher Ġperformance Ġgain Ġcompared Ġto Ġthe ĠARE . ĠFrom Ġthis Ġresult , Ġwe Ġnote Ġthat Ġtextual Ġdata Ġare Ġinformative Ġin Ġemotion Ġprediction Ġtasks , Ġand Ġthe Ġrecurrent Ġenc oder Ġmodel Ġis Ġeffective Ġin Ġunderstanding Ġthese Ġtypes Ġof Ġsequential Ġdata . ĠSecond , Ġthe Ġnewly Ġproposed Ġmodel , ĠMD RE , Ġshows Ġa Ġsubstantial Ġperformance Ġgain . ĠIt Ġthus Ġachieves Ġthe Ġstate - of - the - art Ġperformance Ġwith Ġa ĠW AP Ġvalue Ġof Ġ0 . 718 . ĠThis Ġresult Ġshows Ġthat Ġmultim od al Ġinformation Ġis Ġa Ġkey Ġfactor Ġin Ġaffect ive Ġcomputing . ĠLastly , Ġthe Ġattention Ġmodel , ĠMD RE A , Ġalso Ġoutper forms Ġthe Ġbest Ġexisting Ġresearch Ġresults Ġ( W AP Ġ0 . 690 Ġto Ġ0 . 688 ) ĠB IB REF 19 Ġ. ĠHowever , Ġthe ĠMD RE A Ġmodel Ġdoes Ġnot Ġmatch Ġthe Ġperformance Ġof Ġthe ĠMD RE Ġmodel , Ġeven Ġthough Ġit Ġutilizes Ġa Ġmore Ġcomplex Ġarchitecture . ĠWe Ġbelieve Ġthat Ġthis Ġresult Ġarises Ġbecause Ġinsufficient Ġdata Ġare Ġavailable Ġto Ġproperly Ġdetermine Ġthe Ġcomplex Ġmodel Ġparameters Ġin Ġthe ĠMD RE A Ġmodel . ĠMoreover , Ġwe Ġpresume Ġthat Ġthis Ġmodel Ġwill Ġshow Ġbetter Ġperformance Ġwhen Ġthe Ġaudio Ġsignals Ġare Ġaligned Ġwith Ġthe Ġtextual Ġsequence Ġwhile Ġapplying Ġthe Ġattention Ġmechanism . ĠWe Ġleave Ġthe Ġimplementation Ġof Ġthis Ġpoint Ġas Ġa Ġfuture Ġresearch Ġdirection . </s> To Ġinvestigate Ġthe Ġpractical Ġperformance Ġof Ġthe Ġproposed Ġmodels , Ġwe Ġconduct Ġfurther Ġexperiments Ġwith Ġthe ĠAS R - process ed Ġtranscript Ġdata Ġ( see ĠâĢ ľ - AS R âĢ Ŀ Ġmodels Ġin ĠTable Ġ). ĠThe Ġlabel Ġaccuracy Ġof Ġthe Ġprocessed Ġtranscripts Ġis Ġ5 . 53 % ĠW ER . ĠThe ĠTRE - AS R , ĠMD RE - AS R Ġand ĠMD RE A - AS R Ġmodels Ġreflect Ġdegraded Ġperformance Ġcompared Ġto Ġthat Ġof Ġthe ĠTRE , ĠMD RE Ġand ĠMD RE A Ġmodels . ĠHowever , Ġthe Ġperformance Ġof Ġthese Ġmodels Ġis Ġstill Ġcompetitive ; Ġin Ġparticular , Ġthe ĠMD RE - AS R Ġmodel Ġoutper forms Ġthe Ġprevious Ġbest - performing Ġmodel , Ġ3 CNN - L ST M 10 H Ġ( W AP Ġ0 . 691 Ġto Ġ0 . 688 ). </s> Error Ġanalysis </s> We Ġanalyze Ġthe Ġpredictions Ġof Ġthe ĠARE , ĠTRE , Ġand ĠMD RE Ġmodels . ĠFigure Ġshows Ġthe Ġconfusion Ġmatrix Ġof Ġeach Ġmodel . ĠThe ĠARE Ġmodel Ġ( Fig . Ġ) Ġincorrectly Ġclass ifies Ġmost Ġinstances Ġof Ġhappy Ġas Ġneutral Ġ( 43 . 51 %); Ġthus , Ġit Ġshows Ġreduced Ġaccuracy Ġ( 35 . 15 %) Ġin Ġpredicting Ġthe Ġthe Ġhappy Ġclass . ĠOverall , Ġmost Ġof Ġthe Ġemotion Ġclasses Ġare Ġfrequently Ġconfused Ġwith Ġthe Ġneutral Ġclass . ĠThis Ġobservation Ġis Ġin Ġline Ġwith Ġthe Ġfindings Ġof ĠB IB REF 30 Ġ, Ġwho Ġnoted Ġthat Ġthe Ġneutral Ġclass Ġis Ġlocated Ġin Ġthe Ġcenter Ġof Ġthe Ġactivation - val ence Ġspace , Ġcompl icating Ġits Ġdiscrimination Ġfrom Ġthe Ġother Ġclasses . </s> Interestingly , Ġthe ĠTRE Ġmodel Ġ( Fig . Ġ) Ġshows Ġgreater Ġprediction Ġgains Ġin Ġpredicting Ġthe Ġhappy Ġclass Ġwhen Ġcompared Ġto Ġthe ĠARE Ġmodel Ġ( 35 . 15 % Ġto Ġ75 . 73 %). ĠThis Ġresult Ġseems Ġplausible Ġbecause Ġthe Ġmodel Ġcan Ġbenefit Ġfrom Ġthe Ġdifferences Ġamong Ġthe Ġdistributions Ġof Ġwords Ġin Ġhappy Ġand Ġneutral Ġexpressions , Ġwhich Ġgives Ġmore Ġemotional Ġinformation Ġto Ġthe Ġmodel Ġthan Ġthat Ġof Ġthe Ġaudio Ġsignal Ġdata . ĠOn Ġthe Ġother Ġhand , Ġit Ġis Ġstriking Ġthat Ġthe ĠTRE Ġmodel Ġincorrectly Ġpredicts Ġinstances Ġof Ġthe Ġsad Ġclass Ġas Ġthe Ġhappy Ġclass Ġ16 . 20 % Ġof Ġthe Ġtime , Ġeven Ġthough Ġthese Ġemotional Ġstates Ġare Ġoppos ites Ġof Ġone Ġanother . </s> The ĠMD RE Ġmodel Ġ( Fig . Ġ) Ġcompens ates Ġfor Ġthe Ġweaknesses Ġof Ġthe Ġprevious Ġtwo Ġmodels Ġ( ARE Ġand ĠTRE ) Ġand Ġbenefits Ġfrom Ġtheir Ġstrengths Ġto Ġa Ġsurprising Ġdegree . ĠThe Ġvalues Ġarranged Ġalong Ġthe Ġdiagonal Ġaxis Ġshow Ġthat Ġall Ġof Ġthe Ġaccur acies Ġof Ġthe Ġcorrectly Ġpredicted Ġclass Ġhave Ġincreased . ĠFurthermore , Ġthe Ġoccurrence Ġof Ġthe Ġincorrect ĠâĢ ľ s ad - to - happy " Ġcases Ġin Ġthe ĠTRE Ġmodel Ġis Ġreduced Ġfrom Ġ16 . 20 % Ġto Ġ9 . 15 %. </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġmultim od al Ġdual Ġrecurrent Ġenc oder Ġmodel Ġthat Ġsimultaneously Ġutilizes Ġtext Ġdata , Ġas Ġwell Ġas Ġaudio Ġsignals , Ġto Ġpermit Ġthe Ġbetter Ġunderstanding Ġof Ġspeech Ġdata . ĠOur Ġmodel Ġenc odes Ġthe Ġinformation Ġfrom Ġaudio Ġand Ġtext Ġsequences Ġusing Ġdual ĠR NN s Ġand Ġthen Ġcombines Ġthe Ġinformation Ġfrom Ġthese Ġsources Ġusing Ġa Ġfeed - forward Ġneural Ġmodel Ġto Ġpredict Ġthe Ġemotion Ġclass . ĠExt ensive Ġexperiments Ġshow Ġthat Ġour Ġproposed Ġmodel Ġoutper forms Ġother Ġstate - of - the - art Ġmethods Ġin Ġclass ifying Ġthe Ġfour Ġemotion Ġcategories , Ġand Ġaccur acies Ġranging Ġfrom Ġ68 . 8 % Ġto Ġ71 . 8 % Ġare Ġobtained Ġwhen Ġthe Ġmodel Ġis Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset . ĠIn Ġparticular , Ġit Ġresolves Ġthe Ġissue Ġin Ġwhich Ġpredictions Ġfrequently Ġincorrectly Ġyield Ġthe Ġneutral Ġclass , Ġas Ġoccurs Ġin Ġprevious Ġmodels Ġthat Ġfocus Ġon Ġaudio Ġfeatures . </s> In Ġthe Ġfuture Ġwork , Ġwe Ġaim Ġto Ġextend Ġthe Ġmod alities Ġto Ġaudio , Ġtext Ġand Ġvideo Ġinputs . ĠFurthermore , Ġwe Ġplan Ġto Ġinvestigate Ġthe Ġapplication Ġof Ġthe Ġattention Ġmechanism Ġto Ġdata Ġderived Ġfrom Ġmultiple Ġmod alities . ĠThis Ġapproach Ġseems Ġlikely Ġto Ġuncover Ġenhanced Ġlearning Ġschemes Ġthat Ġwill Ġincrease Ġperformance Ġin Ġboth Ġspeech Ġemotion Ġrecognition Ġand Ġother Ġmultim od al Ġclassification Ġtasks . </s> Acknowled gments </s> K . ĠJung Ġis Ġwith Ġthe ĠDepartment Ġof ĠElectrical Ġand ĠComputer ĠEngineering , ĠAS RI , ĠSeoul ĠNational ĠUniversity , ĠSeoul , ĠKorea . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠMinistry Ġof ĠTrade , ĠIndustry Ġ& ĠEnergy Ġ( M OT IE , ĠKorea ) Ġunder ĠIndustrial ĠTechnology ĠInnovation ĠProgram Ġ( No . 100 73 144 ).
Introduction Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2 . Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3 . These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study. One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5 . In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6 . In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics. This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems. The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i.e., happy, sad, neutral, and angry). Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons. First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement. Second, the characteristics of emotions must be learned from low-level speech signals. Feature-based models display limited skills when applied to this problem. To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree. Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill. The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day.” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input. In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods by 68.8% to 71.8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets. Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes. Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model. Related work Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13 . Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition. An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14 . With the advancement of deep learning methods, more complex neural-based architectures have been proposed. Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17 . These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset. Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models. One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20 . Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21 . As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23 . However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions. Model This section describes the methodologies that are applied to the speech emotion recognition task. We start by introducing the recurrent encoder model for the audio and text modalities individually. We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder. Audio Recurrent Encoder (ARE) Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal. Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i.e., gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns. This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0 where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6 . After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data. This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10 . The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively. Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12 . For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer. The training objective then takes the following form: DISPLAYFORM0 where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1 . The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters. C is the total number of classes, and N is the total number of samples used in training. The upper part of Figure shows the architecture of the ARE model. Text Recurrent Encoder (TRE) We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7 . We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal. To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27 . Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words. The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2 . In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input. Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function. We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0 where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters. The lower part of Figure indicates the architecture of the TRE model. Multimodal Dual Recurrent Encoder (MDRE) We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches. In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively. These types of data are the same as those used in the ARE and TRE cases. The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently. The audio-RNN encodes MFCC features from the audio signal using equation EQREF2 . The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A. On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2 . The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T. Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T. We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0 where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively. INLINEFORM4 and the bias INLINEFORM5 are learned model parameters. Multimodal Dual Recurrent Encoder with Attention (MDREA) Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information. Figure shows the architecture of the MDREA model. First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2 . We then consider the final audio encoding vector INLINEFORM0 as a context vector. As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2 . Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z. This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class. We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0 where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters. Dataset We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset. This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors. We use categorical evaluations with majority agreement. We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories. The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female). This data collection process resulted in 10 unique speakers. For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset. The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral). Feature extraction To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals. The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function. According to the length of each wave file, the sequential step of the MFCC features is varied. To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing. The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours. All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26 . Implementation details Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29 . We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts. The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data. The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments. The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32 . In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity. To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts. The performance of the Google ASR system is reflected by its word error rate (WER) of 5.53%. Performance evaluation As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model. The data in each fold are split into training, development, and testing datasets (8:0.5:1.5, respectively). After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset. We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation. We examine the WAP values, which are shown in Table 1. First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures. On the other hand, the TRE model shows higher performance gain compared to the ARE. From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data. Second, the newly proposed model, MDRE, shows a substantial performance gain. It thus achieves the state-of-the-art performance with a WAP value of 0.718. This result shows that multimodal information is a key factor in affective computing. Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0.690 to 0.688) BIBREF19 . However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture. We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model. Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism. We leave the implementation of this point as a future research direction. To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table ). The label accuracy of the processed transcripts is 5.53% WER. The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models. However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0.691 to 0.688). Error analysis We analyze the predictions of the ARE, TRE, and MDRE models. Figure shows the confusion matrix of each model. The ARE model (Fig. ) incorrectly classifies most instances of happy as neutral (43.51%); thus, it shows reduced accuracy (35.15%) in predicting the the happy class. Overall, most of the emotion classes are frequently confused with the neutral class. This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes. Interestingly, the TRE model (Fig. ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35.15% to 75.73%). This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data. On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16.20% of the time, even though these emotional states are opposites of one another. The MDRE model (Fig. ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree. The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased. Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16.20% to 9.15%. Conclusions In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data. Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class. Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68.8% to 71.8% are obtained when the model is applied to the IEMOCAP dataset. In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features. In the future work, we aim to extend the modalities to audio, text and video inputs. Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities. This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks. Acknowledgments K. Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea. This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No.10073144).
Do they use datasets with transcribed text or do they determine text from the audio?
9de2f73a3db0c695e5e0f5a3d791fdc370b1df6e
Do Ġthey Ġuse Ġdatasets Ġwith Ġtransc ribed Ġtext Ġor Ġdo Ġthey Ġdetermine Ġtext Ġfrom Ġthe Ġaudio ?
They Ġuse Ġtext Ġtranscription .
<s> Do Ġthey Ġuse Ġdatasets Ġwith Ġtransc ribed Ġtext Ġor Ġdo Ġthey Ġdetermine Ġtext Ġfrom Ġthe Ġaudio ? </s> Introduction </s> Recently , Ġdeep Ġlearning Ġalgorithms Ġhave Ġsuccessfully Ġaddressed Ġproblems Ġin Ġvarious Ġfields , Ġsuch Ġas Ġimage Ġclassification , Ġmachine Ġtranslation , Ġspeech Ġrecognition , Ġtext - to - speech Ġgeneration Ġand Ġother Ġmachine Ġlearning Ġrelated Ġareas ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠSimilarly , Ġsubstantial Ġimprovements Ġin Ġperformance Ġhave Ġbeen Ġobtained Ġwhen Ġdeep Ġlearning Ġalgorithms Ġhave Ġbeen Ġapplied Ġto Ġstatistical Ġspeech Ġprocessing ĠB IB REF 3 Ġ. ĠThese Ġfundamental Ġimprovements Ġhave Ġled Ġresearchers Ġto Ġinvestigate Ġadditional Ġtopics Ġrelated Ġto Ġhuman Ġnature , Ġwhich Ġhave Ġlong Ġbeen Ġobjects Ġof Ġstudy . ĠOne Ġsuch Ġtopic Ġinvolves Ġunderstanding Ġhuman Ġemotions Ġand Ġreflecting Ġit Ġthrough Ġmachine Ġintelligence , Ġsuch Ġas Ġemotional Ġdialogue Ġmodels ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> In Ġdeveloping Ġemotionally Ġaware Ġintelligence , Ġthe Ġvery Ġfirst Ġstep Ġis Ġbuilding Ġrobust Ġemotion Ġclass ifiers Ġthat Ġdisplay Ġgood Ġperformance Ġregardless Ġof Ġthe Ġapplication ; Ġthis Ġoutcome Ġis Ġconsidered Ġto Ġbe Ġone Ġof Ġthe Ġfundamental Ġresearch Ġgoals Ġin Ġaffect ive Ġcomputing ĠB IB REF 6 Ġ. ĠIn Ġparticular , Ġthe Ġspeech Ġemotion Ġrecognition Ġtask Ġis Ġone Ġof Ġthe Ġmost Ġimportant Ġproblems Ġin Ġthe Ġfield Ġof Ġpar aling u istics . ĠThis Ġfield Ġhas Ġrecently Ġbroad ened Ġits Ġapplications , Ġas Ġit Ġis Ġa Ġcrucial Ġfactor Ġin Ġoptimal Ġhuman - computer Ġinteractions , Ġincluding Ġdialog Ġsystems . ĠThe Ġgoal Ġof Ġspeech Ġemotion Ġrecognition Ġis Ġto Ġpredict Ġthe Ġemotional Ġcontent Ġof Ġspeech Ġand Ġto Ġclassify Ġspeech Ġaccording Ġto Ġone Ġof Ġseveral Ġlabels Ġ( i . e ., Ġhappy , Ġsad , Ġneutral , Ġand Ġangry ). ĠVarious Ġtypes Ġof Ġdeep Ġlearning Ġmethods Ġhave Ġbeen Ġapplied Ġto Ġincrease Ġthe Ġperformance Ġof Ġemotion Ġclass ifiers ; Ġhowever , Ġthis Ġtask Ġis Ġstill Ġconsidered Ġto Ġbe Ġchallenging Ġfor Ġseveral Ġreasons . ĠFirst , Ġinsufficient Ġdata Ġfor Ġtraining Ġcomplex Ġneural Ġnetwork - based Ġmodels Ġare Ġavailable , Ġdue Ġto Ġthe Ġcosts Ġassociated Ġwith Ġhuman Ġinvolvement . ĠSecond , Ġthe Ġcharacteristics Ġof Ġemotions Ġmust Ġbe Ġlearned Ġfrom Ġlow - level Ġspeech Ġsignals . ĠFeature - based Ġmodels Ġdisplay Ġlimited Ġskills Ġwhen Ġapplied Ġto Ġthis Ġproblem . </s> To Ġovercome Ġthese Ġlimitations , Ġwe Ġpropose Ġa Ġmodel Ġthat Ġuses Ġhigh - level Ġtext Ġtranscription , Ġas Ġwell Ġas Ġlow - level Ġaudio Ġsignals , Ġto Ġutilize Ġthe Ġinformation Ġcontained Ġwithin Ġlow - resource Ġdatasets Ġto Ġa Ġgreater Ġdegree . ĠGiven Ġrecent Ġimprovements Ġin Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġtechnology ĠB IB REF 7 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, Ġspeech Ġtranscription Ġcan Ġbe Ġcarried Ġout Ġusing Ġaudio Ġsignals Ġwith Ġconsiderable Ġskill . ĠThe Ġemotional Ġcontent Ġof Ġspeech Ġis Ġclearly Ġindicated Ġby Ġthe Ġemotion Ġwords Ġcontained Ġin Ġa Ġsentence ĠB IB REF 10 Ġ, Ġsuch Ġas ĠâĢ ľ love ly âĢ Ŀ Ġand ĠâĢ ľ aw esome , âĢ Ŀ Ġwhich Ġcarry Ġstrong Ġemotions Ġcompared Ġto Ġgeneric Ġ( non - em otion ) Ġwords , Ġsuch Ġas ĠâĢ ľ person âĢ Ŀ Ġand ĠâĢ ľ day . âĢ Ŀ ĠThus , Ġwe Ġhypothes ize Ġthat Ġthe Ġspeech Ġemotion Ġrecognition Ġmodel Ġwill Ġbe Ġbenefit Ġfrom Ġthe Ġincorporation Ġof Ġhigh - level Ġtextual Ġinput . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġdeep Ġdual Ġrecurrent Ġenc oder Ġmodel Ġthat Ġsimultaneously Ġutilizes Ġaudio Ġand Ġtext Ġdata Ġin Ġrecognizing Ġemotions Ġfrom Ġspeech . ĠExt ensive Ġexperiments Ġare Ġconducted Ġto Ġinvestigate Ġthe Ġefficacy Ġand Ġproperties Ġof Ġthe Ġproposed Ġmodel . ĠOur Ġproposed Ġmodel Ġoutper forms Ġprevious Ġstate - of - the - art Ġmethods Ġby Ġ68 . 8 % Ġto Ġ71 . 8 % Ġwhen Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset , Ġwhich Ġis Ġone Ġof Ġthe Ġmost Ġwell - stud ied Ġdatasets . ĠBased Ġon Ġan Ġerror Ġanalysis Ġof Ġthe Ġmodels , Ġwe Ġshow Ġthat Ġour Ġproposed Ġmodel Ġaccurately Ġidentifies Ġemotion Ġclasses . ĠMoreover , Ġthe Ġneutral Ġclass Ġmis class ification Ġbias Ġfrequently Ġexhibited Ġby Ġprevious Ġmodels , Ġwhich Ġfocus Ġon Ġaudio Ġfeatures , Ġis Ġless Ġpronounced Ġin Ġour Ġmodel . </s> Related Ġwork </s> Class ical Ġmachine Ġlearning Ġalgorithms , Ġsuch Ġas Ġhidden ĠMark ov Ġmodels Ġ( HM Ms ), Ġsupport Ġvector Ġmachines Ġ( S V Ms ), Ġand Ġdecision Ġtree - based Ġmethods , Ġhave Ġbeen Ġemployed Ġin Ġspeech Ġemotion Ġrecognition Ġproblems ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠRecently , Ġresearchers Ġhave Ġproposed Ġvarious Ġneural Ġnetwork - based Ġarchitectures Ġto Ġimprove Ġthe Ġperformance Ġof Ġspeech Ġemotion Ġrecognition . ĠAn Ġinitial Ġstudy Ġutilized Ġdeep Ġneural Ġnetworks Ġ( D NN s ) Ġto Ġextract Ġhigh - level Ġfeatures Ġfrom Ġraw Ġaudio Ġdata Ġand Ġdemonstrated Ġits Ġeffectiveness Ġin Ġspeech Ġemotion Ġrecognition ĠB IB REF 14 Ġ. ĠWith Ġthe Ġadvancement Ġof Ġdeep Ġlearning Ġmethods , Ġmore Ġcomplex Ġneural - based Ġarchitectures Ġhave Ġbeen Ġproposed . ĠConv olution al Ġneural Ġnetwork Ġ( CNN )- based Ġmodels Ġhave Ġbeen Ġtrained Ġon Ġinformation Ġderived Ġfrom Ġraw Ġaudio Ġsignals Ġusing Ġspect rog rams Ġor Ġaudio Ġfeatures Ġsuch Ġas ĠMel - frequency Ġc ep st ral Ġcoefficients Ġ( MF CC s ) Ġand Ġlow - level Ġdescript ors Ġ( LL Ds ) ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. ĠThese Ġneural Ġnetwork - based Ġmodels Ġare Ġcombined Ġto Ġproduce Ġhigher - complex ity Ġmodels ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, Ġand Ġthese Ġmodels Ġachieved Ġthe Ġbest - recorded Ġperformance Ġwhen Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset . </s> Another Ġline Ġof Ġresearch Ġhas Ġfocused Ġon Ġadopting Ġvariant Ġmachine Ġlearning Ġtechniques Ġcombined Ġwith Ġneural Ġnetwork - based Ġmodels . ĠOne Ġresearcher Ġutilized Ġthe Ġmulti object Ġlearning Ġapproach Ġand Ġused Ġgender Ġand Ġnatural ness Ġas Ġauxiliary Ġtasks Ġso Ġthat Ġthe Ġneural Ġnetwork - based Ġmodel Ġlearned Ġmore Ġfeatures Ġfrom Ġa Ġgiven Ġdataset ĠB IB REF 20 Ġ. ĠAnother Ġresearcher Ġinvestigated Ġtransfer Ġlearning Ġmethods , Ġleveraging Ġexternal Ġdata Ġfrom Ġrelated Ġdomains ĠB IB REF 21 Ġ. </s> As Ġemotional Ġdialogue Ġis Ġcomposed Ġof Ġsound Ġand Ġspoken Ġcontent , Ġresearchers Ġhave Ġalso Ġinvestigated Ġthe Ġcombination Ġof Ġacoustic Ġfeatures Ġand Ġlanguage Ġinformation , Ġbuilt Ġbelief Ġnetwork - based Ġmethods Ġof Ġidentifying Ġemotional Ġkey Ġphrases , Ġand Ġassessed Ġthe Ġemotional Ġsal ience Ġof Ġverbal Ġcues Ġfrom Ġboth Ġphon eme Ġsequences Ġand Ġwords ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠHowever , Ġnone Ġof Ġthese Ġstudies Ġhave Ġutilized Ġinformation Ġfrom Ġspeech Ġsignals Ġand Ġtext Ġsequences Ġsimultaneously Ġin Ġan Ġend - to - end Ġlearning Ġneural Ġnetwork - based Ġmodel Ġto Ġclassify Ġemotions . </s> Model </s> This Ġsection Ġdescribes Ġthe Ġmethod ologies Ġthat Ġare Ġapplied Ġto Ġthe Ġspeech Ġemotion Ġrecognition Ġtask . ĠWe Ġstart Ġby Ġintroducing Ġthe Ġrecurrent Ġenc oder Ġmodel Ġfor Ġthe Ġaudio Ġand Ġtext Ġmod alities Ġindividually . ĠWe Ġthen Ġpropose Ġa Ġmultim od al Ġapproach Ġthat Ġenc odes Ġboth Ġaudio Ġand Ġtextual Ġinformation Ġsimultaneously Ġvia Ġa Ġdual Ġrecurrent Ġenc oder . </s> Audio ĠRec urrent ĠEnc oder Ġ( ARE ) </s> Mot ivated Ġby Ġthe Ġarchitecture Ġused Ġin ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġwe Ġbuild Ġan Ġaudio Ġrecurrent Ġenc oder Ġ( ARE ) Ġto Ġpredict Ġthe Ġclass Ġof Ġa Ġgiven Ġaudio Ġsignal . ĠOnce ĠMF CC Ġfeatures Ġhave Ġbeen Ġextracted Ġfrom Ġan Ġaudio Ġsignal , Ġa Ġsubset Ġof Ġthe Ġsequential Ġfeatures Ġis Ġfed Ġinto Ġthe ĠR NN Ġ( i . e ., Ġg ated Ġrecurrent Ġunits Ġ( GR Us )), Ġwhich Ġleads Ġto Ġthe Ġformation Ġof Ġthe Ġnetwork 's Ġinternal Ġhidden Ġstate ĠIN LINE FORM 0 Ġto Ġmodel Ġthe Ġtime Ġseries Ġpatterns . ĠThis Ġinternal Ġhidden Ġstate Ġis Ġupdated Ġat Ġeach Ġtime Ġstep Ġwith Ġthe Ġinput Ġdata ĠIN LINE FORM 1 Ġand Ġthe Ġhidden Ġstate Ġof Ġthe Ġprevious Ġtime Ġstep ĠIN LINE FORM 2 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠR NN Ġfunction Ġwith Ġweight Ġparameter ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġrepresents Ġthe Ġhidden Ġstate Ġat Ġt - ĠIN LINE FORM 3 Ġtime Ġstep , Ġand ĠIN LINE FORM 4 Ġrepresents Ġthe Ġt - ĠIN LINE FORM 5 ĠMF CC Ġfeatures Ġin ĠIN LINE FORM 6 Ġ. ĠAfter Ġencoding Ġthe Ġaudio Ġsignal ĠIN LINE FORM 7 Ġwith Ġthe ĠR NN , Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe ĠR NN , ĠIN LINE FORM 8 Ġ, Ġis Ġconsidered Ġto Ġbe Ġthe Ġrepresentative Ġvector Ġthat Ġcontains Ġall Ġof Ġthe Ġsequential Ġaudio Ġdata . ĠThis Ġvector Ġis Ġthen Ġconc aten ated Ġwith Ġanother Ġpros odic Ġfeature Ġvector , ĠIN LINE FORM 9 Ġ, Ġto Ġgenerate Ġa Ġmore Ġinformative Ġvector Ġrepresentation Ġof Ġthe Ġsignal , ĠIN LINE FORM 10 Ġ. ĠThe ĠMF CC Ġand Ġthe Ġpros odic Ġfeatures Ġare Ġextracted Ġfrom Ġthe Ġaudio Ġsignal Ġusing Ġthe Ġopen S MI LE Ġtool kit ĠB IB REF 26 Ġ, ĠIN LINE FORM 11 Ġ, Ġrespectively . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġby Ġapplying Ġthe Ġsoft max Ġfunction Ġto Ġthe Ġvector ĠIN LINE FORM 12 Ġ. ĠFor Ġa Ġgiven Ġaudio Ġsample ĠIN LINE FORM 13 Ġ, Ġwe Ġassume Ġthat ĠIN LINE FORM 14 Ġis Ġthe Ġtrue Ġlabel Ġvector , Ġwhich Ġcontains Ġall Ġz eros Ġbut Ġcontains Ġa Ġone Ġat Ġthe Ġcorrect Ġclass , Ġand ĠIN LINE FORM 15 Ġis Ġthe Ġpredicted Ġprobability Ġdistribution Ġfrom Ġthe Ġsoft max Ġlayer . ĠThe Ġtraining Ġobjective Ġthen Ġtakes Ġthe Ġfollowing Ġform : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcalculated Ġrepresentative Ġvector Ġof Ġthe Ġaudio Ġsignal Ġwith Ġdimension ality ĠIN LINE FORM 1 Ġ. ĠThe ĠIN LINE FORM 2 Ġand Ġthe Ġbias ĠIN LINE FORM 3 Ġare Ġlearned Ġmodel Ġparameters . ĠC Ġis Ġthe Ġtotal Ġnumber Ġof Ġclasses , Ġand ĠN Ġis Ġthe Ġtotal Ġnumber Ġof Ġsamples Ġused Ġin Ġtraining . ĠThe Ġupper Ġpart Ġof ĠFigure Ġshows Ġthe Ġarchitecture Ġof Ġthe ĠARE Ġmodel . </s> Text ĠRec urrent ĠEnc oder Ġ( T RE ) </s> We Ġassume Ġthat Ġspeech Ġtranscripts Ġcan Ġbe Ġextracted Ġfrom Ġaudio Ġsignals Ġwith Ġhigh Ġaccuracy , Ġgiven Ġthe Ġadvancement Ġof ĠAS R Ġtechnologies ĠB IB REF 7 Ġ. ĠWe Ġattempt Ġto Ġuse Ġthe Ġprocessed Ġtextual Ġinformation Ġas Ġanother Ġmod ality Ġin Ġpredicting Ġthe Ġemotion Ġclass Ġof Ġa Ġgiven Ġsignal . ĠTo Ġuse Ġtextual Ġinformation , Ġa Ġspeech Ġtranscript Ġis Ġtoken ized Ġand Ġindexed Ġinto Ġa Ġsequence Ġof Ġtokens Ġusing Ġthe ĠNatural ĠLanguage ĠTool kit Ġ( N LT K ) ĠB IB REF 27 Ġ. ĠEach Ġtoken Ġis Ġthen Ġpassed Ġthrough Ġa Ġword - embed ding Ġlayer Ġthat Ġconverts Ġa Ġword Ġindex Ġto Ġa Ġcorresponding Ġ300 - dimensional Ġvector Ġthat Ġcontains Ġadditional Ġcontextual Ġmeaning Ġbetween Ġwords . ĠThe Ġsequence Ġof Ġembedded Ġtokens Ġis Ġfed Ġinto Ġa Ġtext Ġrecurrent Ġenc oder Ġ( T RE ) Ġin Ġsuch Ġa Ġway Ġthat Ġthe Ġaudio ĠMF CC Ġfeatures Ġare Ġencoded Ġusing Ġthe ĠARE Ġrepresented Ġby Ġequation ĠEQ REF 2 Ġ. ĠIn Ġthis Ġcase , ĠIN LINE FORM 0 Ġis Ġthe Ġt - ĠIN LINE FORM 1 Ġembedded Ġtoken Ġfrom Ġthe Ġtext Ġinput . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġfrom Ġthe Ġlast Ġhidden Ġstate Ġof Ġthe Ġtext - R NN Ġusing Ġthe Ġsoft max Ġfunction . </s> We Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġlast Ġhidden Ġstate Ġof Ġthe Ġtext - R NN , ĠIN LINE FORM 1 Ġ, Ġand Ġthe ĠIN LINE FORM 2 Ġand Ġbias ĠIN LINE FORM 3 Ġare Ġlearned Ġmodel Ġparameters . ĠThe Ġlower Ġpart Ġof ĠFigure Ġindicates Ġthe Ġarchitecture Ġof Ġthe ĠTRE Ġmodel . </s> Mult im od al ĠDual ĠRec urrent ĠEnc oder Ġ( MD RE ) </s> We Ġpresent Ġa Ġnovel Ġarchitecture Ġcalled Ġthe Ġmultim od al Ġdual Ġrecurrent Ġenc oder Ġ( MD RE ) Ġto Ġovercome Ġthe Ġlimitations Ġof Ġexisting Ġapproaches . ĠIn Ġthis Ġstudy , Ġwe Ġconsider Ġmultiple Ġmod alities , Ġsuch Ġas ĠMF CC Ġfeatures , Ġpros odic Ġfeatures Ġand Ġtranscripts , Ġwhich Ġcontain Ġsequential Ġaudio Ġinformation , Ġstatistical Ġaudio Ġinformation Ġand Ġtextual Ġinformation , Ġrespectively . ĠThese Ġtypes Ġof Ġdata Ġare Ġthe Ġsame Ġas Ġthose Ġused Ġin Ġthe ĠARE Ġand ĠTRE Ġcases . ĠThe ĠMD RE Ġmodel Ġemploys Ġtwo ĠR NN s Ġto Ġencode Ġdata Ġfrom Ġthe Ġaudio Ġsignal Ġand Ġtextual Ġinputs Ġindependently . ĠThe Ġaudio - R NN Ġenc odes ĠMF CC Ġfeatures Ġfrom Ġthe Ġaudio Ġsignal Ġusing Ġequation ĠEQ REF 2 Ġ. ĠThe Ġlast Ġhidden Ġstate Ġof Ġthe Ġaudio - R NN Ġis Ġconc aten ated Ġwith Ġthe Ġpros odic Ġfeatures Ġto Ġform Ġthe Ġfinal Ġvector Ġrepresentation ĠIN LINE FORM 0 Ġ, Ġand Ġthis Ġvector Ġis Ġthen Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġlayer Ġto Ġform Ġthe Ġaudio Ġencoding Ġvector ĠA . ĠOn Ġthe Ġother Ġhand , Ġthe Ġtext - R NN Ġenc odes Ġthe Ġword Ġsequence Ġof Ġthe Ġtranscript Ġusing Ġequation ĠEQ REF 2 Ġ. ĠThe Ġfinal Ġhidden Ġstates Ġof Ġthe Ġtext - R NN Ġare Ġalso Ġpassed Ġthrough Ġanother Ġfully Ġconnected Ġneural Ġnetwork Ġlayer Ġto Ġform Ġa Ġtextual Ġencoding Ġvector ĠT . ĠFinally , Ġthe Ġemotion Ġclass Ġis Ġpredicted Ġby Ġapplying Ġthe Ġsoft max Ġfunction Ġto Ġthe Ġconc aten ation Ġof Ġthe Ġvectors ĠA Ġand ĠT . ĠWe Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġfeed - forward Ġneural Ġnetwork Ġwith Ġweight Ġparameter ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġare Ġfinal Ġencoding Ġvectors Ġfrom Ġthe Ġaudio - R NN Ġand Ġtext - R NN , Ġrespectively . ĠIN LINE FORM 4 Ġand Ġthe Ġbias ĠIN LINE FORM 5 Ġare Ġlearned Ġmodel Ġparameters . </s> Mult im od al ĠDual ĠRec urrent ĠEnc oder Ġwith ĠAttention Ġ( MD RE A ) </s> Insp ired Ġby Ġthe Ġconcept Ġof Ġthe Ġattention Ġmechanism Ġused Ġin Ġneural Ġmachine Ġtranslation ĠB IB REF 28 Ġ, Ġwe Ġpropose Ġa Ġnovel Ġmultim od al Ġattention Ġmethod Ġto Ġfocus Ġon Ġthe Ġspecific Ġparts Ġof Ġa Ġtranscript Ġthat Ġcontain Ġstrong Ġemotional Ġinformation , Ġconditioning Ġon Ġthe Ġaudio Ġinformation . ĠFigure Ġshows Ġthe Ġarchitecture Ġof Ġthe ĠMD RE A Ġmodel . ĠFirst , Ġthe Ġaudio Ġdata Ġand Ġtext Ġdata Ġare Ġencoded Ġwith Ġthe Ġaudio - R NN Ġand Ġtext - R NN Ġusing Ġequation ĠEQ REF 2 Ġ. ĠWe Ġthen Ġconsider Ġthe Ġfinal Ġaudio Ġencoding Ġvector ĠIN LINE FORM 0 Ġas Ġa Ġcontext Ġvector . ĠAs Ġseen Ġin Ġequation ĠEQ REF 9 Ġ, Ġduring Ġeach Ġtime Ġstep Ġt , Ġthe Ġdot Ġproduct Ġbetween Ġthe Ġcontext Ġvector Ġe Ġand Ġthe Ġhidden Ġstate Ġof Ġthe Ġtext - R NN Ġat Ġeach Ġt - th Ġsequence ĠIN LINE FORM 1 Ġis Ġevaluated Ġto Ġcalculate Ġa Ġsimilarity Ġscore ĠIN LINE FORM 2 Ġ. ĠUsing Ġthis Ġscore ĠIN LINE FORM 3 Ġas Ġa Ġweight Ġparameter , Ġthe Ġweighted Ġsum Ġof Ġthe Ġsequences Ġof Ġthe Ġhidden Ġstate Ġof Ġthe Ġtext - R NN , ĠIN LINE FORM 4 Ġ, Ġis Ġcalculated Ġto Ġgenerate Ġan Ġattention - application Ġvector ĠZ . ĠThis Ġattention - application Ġvector Ġis Ġconc aten ated Ġwith Ġthe Ġfinal Ġencoding Ġvector Ġof Ġthe Ġaudio - R NN ĠIN LINE FORM 5 Ġ( equ ation ĠEQ REF 7 Ġ), Ġwhich Ġwill Ġbe Ġpassed Ġthrough Ġthe Ġsoft max Ġfunction Ġto Ġpredict Ġthe Ġemotion Ġclass . ĠWe Ġuse Ġthe Ġsame Ġtraining Ġobjective Ġas Ġthe ĠARE Ġmodel , Ġand Ġthe Ġpredicted Ġprobability Ġdistribution Ġfor Ġthe Ġtarget Ġclass Ġis Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand Ġthe Ġbias ĠIN LINE FORM 1 Ġare Ġlearned Ġmodel Ġparameters . </s> Dat as et </s> We Ġevaluate Ġour Ġmodel Ġusing Ġthe ĠInteractive ĠEm otional ĠDy adic ĠMotion ĠCapture Ġ( I EM OC AP ) ĠB IB REF 18 Ġdataset . ĠThis Ġdataset Ġwas Ġcollected Ġfollowing Ġtheatrical Ġtheory Ġin Ġorder Ġto Ġsimulate Ġnatural Ġdy adic Ġinteractions Ġbetween Ġactors . ĠWe Ġuse Ġcateg orical Ġevaluations Ġwith Ġmajority Ġagreement . ĠWe Ġuse Ġonly Ġfour Ġemotional Ġcategories Ġhappy , Ġsad , Ġangry , Ġand Ġneutral Ġto Ġcompare Ġthe Ġperformance Ġof Ġour Ġmodel Ġwith Ġother Ġresearch Ġusing Ġthe Ġsame Ġcategories . ĠThe ĠI EM OC AP Ġdataset Ġincludes Ġfive Ġsessions , Ġand Ġeach Ġsession Ġcontains Ġutter ances Ġfrom Ġtwo Ġspeakers Ġ( one Ġmale Ġand Ġone Ġfemale ). ĠThis Ġdata Ġcollection Ġprocess Ġresulted Ġin Ġ10 Ġunique Ġspeakers . ĠFor Ġconsistent Ġcomparison Ġwith Ġprevious Ġwork , Ġwe Ġmerge Ġthe Ġexcitement Ġdataset Ġwith Ġthe Ġhappiness Ġdataset . ĠThe Ġfinal Ġdataset Ġcontains Ġa Ġtotal Ġof Ġ55 31 Ġutter ances Ġ( 16 36 Ġhappy , Ġ10 84 Ġsad , Ġ110 3 Ġangry , Ġ17 08 Ġneutral ). </s> Feature Ġextraction </s> To Ġextract Ġspeech Ġinformation Ġfrom Ġaudio Ġsignals , Ġwe Ġuse ĠMF CC Ġvalues , Ġwhich Ġare Ġwidely Ġused Ġin Ġanalyzing Ġaudio Ġsignals . ĠThe ĠMF CC Ġfeature Ġset Ġcontains Ġa Ġtotal Ġof Ġ39 Ġfeatures , Ġwhich Ġinclude Ġ12 ĠMF CC Ġparameters Ġ( 1 - 12 ) Ġfrom Ġthe Ġ26 ĠMel - frequency Ġbands Ġand Ġlog - energy Ġparameters , Ġ13 Ġdelta Ġand Ġ13 Ġacceleration Ġcoefficients ĠThe Ġframe Ġsize Ġis Ġset Ġto Ġ25 Ġms Ġat Ġa Ġrate Ġof Ġ10 Ġms Ġwith Ġthe ĠHam ming Ġfunction . ĠAccording Ġto Ġthe Ġlength Ġof Ġeach Ġwave Ġfile , Ġthe Ġsequential Ġstep Ġof Ġthe ĠMF CC Ġfeatures Ġis Ġvaried . ĠTo Ġextract Ġadditional Ġinformation Ġfrom Ġthe Ġdata , Ġwe Ġalso Ġuse Ġpros odic Ġfeatures , Ġwhich Ġshow Ġeffectiveness Ġin Ġaffect ive Ġcomputing . ĠThe Ġpros odic Ġfeatures Ġare Ġcomposed Ġof Ġ35 Ġfeatures , Ġwhich Ġinclude Ġthe ĠF 0 Ġfrequency , Ġthe Ġvoicing Ġprobability , Ġand Ġthe Ġloud ness Ġcont ours . ĠAll Ġof Ġthese ĠMF CC Ġand Ġpros odic Ġfeatures Ġare Ġextracted Ġfrom Ġthe Ġdata Ġusing Ġthe ĠOpen S MI LE Ġtool kit ĠB IB REF 26 Ġ. </s> Im plementation Ġdetails </s> Among Ġthe Ġvariants Ġof Ġthe ĠR NN Ġfunction , Ġwe Ġuse ĠGR Us Ġas Ġthey Ġyield Ġcomparable Ġperformance Ġto Ġthat Ġof Ġthe ĠL ST M Ġand Ġinclude Ġa Ġsmaller Ġnumber Ġof Ġweight Ġparameters ĠB IB REF 29 Ġ. ĠWe Ġuse Ġa Ġmax Ġenc oder Ġstep Ġof Ġ750 Ġfor Ġthe Ġaudio Ġinput , Ġbased Ġon Ġthe Ġimplementation Ġchoices Ġpresented Ġin ĠB IB REF 30 Ġand Ġ128 Ġfor Ġthe Ġtext Ġinput Ġbecause Ġit Ġcovers Ġthe Ġmaximum Ġlength Ġof Ġthe Ġtranscripts . ĠThe Ġvocabulary Ġsize Ġof Ġthe Ġdataset Ġis Ġ3 , 747 , Ġincluding Ġthe ĠâĢ ľ _ UN K _ " Ġtoken , Ġwhich Ġrepresents Ġunknown Ġwords , Ġand Ġthe ĠâĢ ľ _ P AD _ " Ġtoken , Ġwhich Ġis Ġused Ġto Ġindicate Ġpadding Ġinformation Ġadded Ġwhile Ġpreparing Ġmini - batch Ġdata . ĠThe Ġnumber Ġof Ġhidden Ġunits Ġand Ġthe Ġnumber Ġof Ġlayers Ġin Ġthe ĠR NN Ġfor Ġeach Ġmodel Ġ( ARE , ĠTRE , ĠMD RE Ġand ĠMD RE A ) Ġare Ġselected Ġbased Ġon Ġextensive Ġhyper param eter Ġsearch Ġexperiments . ĠThe Ġweights Ġof Ġthe Ġhidden Ġunits Ġare Ġinitialized Ġusing Ġorth og onal Ġweights ĠB IB REF 31 Ġ], Ġand Ġthe Ġtext Ġembed ding Ġlayer Ġis Ġinitialized Ġfrom Ġpret rained Ġword - embed ding Ġvectors ĠB IB REF 32 Ġ. </s> In Ġpreparing Ġthe Ġtextual Ġdataset , Ġwe Ġfirst Ġuse Ġthe Ġreleased Ġtranscripts Ġof Ġthe ĠI EM OC AP Ġdataset Ġfor Ġsimplicity . ĠTo Ġinvestigate Ġthe Ġpractical Ġperformance , Ġwe Ġthen Ġprocess Ġall Ġof Ġthe ĠI EM OC AP Ġaudio Ġdata Ġusing Ġan ĠAS R Ġsystem Ġ( the ĠGoogle ĠCloud ĠSpeech ĠAPI ) Ġand Ġretrieve Ġthe Ġtranscripts . ĠThe Ġperformance Ġof Ġthe ĠGoogle ĠAS R Ġsystem Ġis Ġreflected Ġby Ġits Ġword Ġerror Ġrate Ġ( WER ) Ġof Ġ5 . 53 %. </s> Performance Ġevaluation </s> As Ġthe Ġdataset Ġis Ġnot Ġexplicitly Ġsplit Ġbeforehand Ġinto Ġtraining , Ġdevelopment , Ġand Ġtesting Ġsets , Ġwe Ġperform Ġ5 - fold Ġcross Ġvalidation Ġto Ġdetermine Ġthe Ġoverall Ġperformance Ġof Ġthe Ġmodel . ĠThe Ġdata Ġin Ġeach Ġfold Ġare Ġsplit Ġinto Ġtraining , Ġdevelopment , Ġand Ġtesting Ġdatasets Ġ( 8 : 0 . 5 : 1 . 5 , Ġrespectively ). ĠAfter Ġtraining Ġthe Ġmodel , Ġwe Ġmeasure Ġthe Ġweighted Ġaverage Ġprecision Ġ( W AP ) Ġover Ġthe Ġ5 - fold Ġdataset . ĠWe Ġtrain Ġand Ġevaluate Ġthe Ġmodel Ġ10 Ġtimes Ġper Ġfold , Ġand Ġthe Ġmodel Ġperformance Ġis Ġassessed Ġin Ġterms Ġof Ġthe Ġmean Ġscore Ġand Ġstandard Ġdeviation . </s> We Ġexamine Ġthe ĠW AP Ġvalues , Ġwhich Ġare Ġshown Ġin ĠTable Ġ1 . ĠFirst , Ġour ĠARE Ġmodel Ġshows Ġthe Ġbaseline Ġperformance Ġbecause Ġwe Ġuse Ġminimal Ġaudio Ġfeatures , Ġsuch Ġas Ġthe ĠMF CC Ġand Ġpros odic Ġfeatures Ġwith Ġsimple Ġarchitectures . ĠOn Ġthe Ġother Ġhand , Ġthe ĠTRE Ġmodel Ġshows Ġhigher Ġperformance Ġgain Ġcompared Ġto Ġthe ĠARE . ĠFrom Ġthis Ġresult , Ġwe Ġnote Ġthat Ġtextual Ġdata Ġare Ġinformative Ġin Ġemotion Ġprediction Ġtasks , Ġand Ġthe Ġrecurrent Ġenc oder Ġmodel Ġis Ġeffective Ġin Ġunderstanding Ġthese Ġtypes Ġof Ġsequential Ġdata . ĠSecond , Ġthe Ġnewly Ġproposed Ġmodel , ĠMD RE , Ġshows Ġa Ġsubstantial Ġperformance Ġgain . ĠIt Ġthus Ġachieves Ġthe Ġstate - of - the - art Ġperformance Ġwith Ġa ĠW AP Ġvalue Ġof Ġ0 . 718 . ĠThis Ġresult Ġshows Ġthat Ġmultim od al Ġinformation Ġis Ġa Ġkey Ġfactor Ġin Ġaffect ive Ġcomputing . ĠLastly , Ġthe Ġattention Ġmodel , ĠMD RE A , Ġalso Ġoutper forms Ġthe Ġbest Ġexisting Ġresearch Ġresults Ġ( W AP Ġ0 . 690 Ġto Ġ0 . 688 ) ĠB IB REF 19 Ġ. ĠHowever , Ġthe ĠMD RE A Ġmodel Ġdoes Ġnot Ġmatch Ġthe Ġperformance Ġof Ġthe ĠMD RE Ġmodel , Ġeven Ġthough Ġit Ġutilizes Ġa Ġmore Ġcomplex Ġarchitecture . ĠWe Ġbelieve Ġthat Ġthis Ġresult Ġarises Ġbecause Ġinsufficient Ġdata Ġare Ġavailable Ġto Ġproperly Ġdetermine Ġthe Ġcomplex Ġmodel Ġparameters Ġin Ġthe ĠMD RE A Ġmodel . ĠMoreover , Ġwe Ġpresume Ġthat Ġthis Ġmodel Ġwill Ġshow Ġbetter Ġperformance Ġwhen Ġthe Ġaudio Ġsignals Ġare Ġaligned Ġwith Ġthe Ġtextual Ġsequence Ġwhile Ġapplying Ġthe Ġattention Ġmechanism . ĠWe Ġleave Ġthe Ġimplementation Ġof Ġthis Ġpoint Ġas Ġa Ġfuture Ġresearch Ġdirection . </s> To Ġinvestigate Ġthe Ġpractical Ġperformance Ġof Ġthe Ġproposed Ġmodels , Ġwe Ġconduct Ġfurther Ġexperiments Ġwith Ġthe ĠAS R - process ed Ġtranscript Ġdata Ġ( see ĠâĢ ľ - AS R âĢ Ŀ Ġmodels Ġin ĠTable Ġ). ĠThe Ġlabel Ġaccuracy Ġof Ġthe Ġprocessed Ġtranscripts Ġis Ġ5 . 53 % ĠW ER . ĠThe ĠTRE - AS R , ĠMD RE - AS R Ġand ĠMD RE A - AS R Ġmodels Ġreflect Ġdegraded Ġperformance Ġcompared Ġto Ġthat Ġof Ġthe ĠTRE , ĠMD RE Ġand ĠMD RE A Ġmodels . ĠHowever , Ġthe Ġperformance Ġof Ġthese Ġmodels Ġis Ġstill Ġcompetitive ; Ġin Ġparticular , Ġthe ĠMD RE - AS R Ġmodel Ġoutper forms Ġthe Ġprevious Ġbest - performing Ġmodel , Ġ3 CNN - L ST M 10 H Ġ( W AP Ġ0 . 691 Ġto Ġ0 . 688 ). </s> Error Ġanalysis </s> We Ġanalyze Ġthe Ġpredictions Ġof Ġthe ĠARE , ĠTRE , Ġand ĠMD RE Ġmodels . ĠFigure Ġshows Ġthe Ġconfusion Ġmatrix Ġof Ġeach Ġmodel . ĠThe ĠARE Ġmodel Ġ( Fig . Ġ) Ġincorrectly Ġclass ifies Ġmost Ġinstances Ġof Ġhappy Ġas Ġneutral Ġ( 43 . 51 %); Ġthus , Ġit Ġshows Ġreduced Ġaccuracy Ġ( 35 . 15 %) Ġin Ġpredicting Ġthe Ġthe Ġhappy Ġclass . ĠOverall , Ġmost Ġof Ġthe Ġemotion Ġclasses Ġare Ġfrequently Ġconfused Ġwith Ġthe Ġneutral Ġclass . ĠThis Ġobservation Ġis Ġin Ġline Ġwith Ġthe Ġfindings Ġof ĠB IB REF 30 Ġ, Ġwho Ġnoted Ġthat Ġthe Ġneutral Ġclass Ġis Ġlocated Ġin Ġthe Ġcenter Ġof Ġthe Ġactivation - val ence Ġspace , Ġcompl icating Ġits Ġdiscrimination Ġfrom Ġthe Ġother Ġclasses . </s> Interestingly , Ġthe ĠTRE Ġmodel Ġ( Fig . Ġ) Ġshows Ġgreater Ġprediction Ġgains Ġin Ġpredicting Ġthe Ġhappy Ġclass Ġwhen Ġcompared Ġto Ġthe ĠARE Ġmodel Ġ( 35 . 15 % Ġto Ġ75 . 73 %). ĠThis Ġresult Ġseems Ġplausible Ġbecause Ġthe Ġmodel Ġcan Ġbenefit Ġfrom Ġthe Ġdifferences Ġamong Ġthe Ġdistributions Ġof Ġwords Ġin Ġhappy Ġand Ġneutral Ġexpressions , Ġwhich Ġgives Ġmore Ġemotional Ġinformation Ġto Ġthe Ġmodel Ġthan Ġthat Ġof Ġthe Ġaudio Ġsignal Ġdata . ĠOn Ġthe Ġother Ġhand , Ġit Ġis Ġstriking Ġthat Ġthe ĠTRE Ġmodel Ġincorrectly Ġpredicts Ġinstances Ġof Ġthe Ġsad Ġclass Ġas Ġthe Ġhappy Ġclass Ġ16 . 20 % Ġof Ġthe Ġtime , Ġeven Ġthough Ġthese Ġemotional Ġstates Ġare Ġoppos ites Ġof Ġone Ġanother . </s> The ĠMD RE Ġmodel Ġ( Fig . Ġ) Ġcompens ates Ġfor Ġthe Ġweaknesses Ġof Ġthe Ġprevious Ġtwo Ġmodels Ġ( ARE Ġand ĠTRE ) Ġand Ġbenefits Ġfrom Ġtheir Ġstrengths Ġto Ġa Ġsurprising Ġdegree . ĠThe Ġvalues Ġarranged Ġalong Ġthe Ġdiagonal Ġaxis Ġshow Ġthat Ġall Ġof Ġthe Ġaccur acies Ġof Ġthe Ġcorrectly Ġpredicted Ġclass Ġhave Ġincreased . ĠFurthermore , Ġthe Ġoccurrence Ġof Ġthe Ġincorrect ĠâĢ ľ s ad - to - happy " Ġcases Ġin Ġthe ĠTRE Ġmodel Ġis Ġreduced Ġfrom Ġ16 . 20 % Ġto Ġ9 . 15 %. </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġmultim od al Ġdual Ġrecurrent Ġenc oder Ġmodel Ġthat Ġsimultaneously Ġutilizes Ġtext Ġdata , Ġas Ġwell Ġas Ġaudio Ġsignals , Ġto Ġpermit Ġthe Ġbetter Ġunderstanding Ġof Ġspeech Ġdata . ĠOur Ġmodel Ġenc odes Ġthe Ġinformation Ġfrom Ġaudio Ġand Ġtext Ġsequences Ġusing Ġdual ĠR NN s Ġand Ġthen Ġcombines Ġthe Ġinformation Ġfrom Ġthese Ġsources Ġusing Ġa Ġfeed - forward Ġneural Ġmodel Ġto Ġpredict Ġthe Ġemotion Ġclass . ĠExt ensive Ġexperiments Ġshow Ġthat Ġour Ġproposed Ġmodel Ġoutper forms Ġother Ġstate - of - the - art Ġmethods Ġin Ġclass ifying Ġthe Ġfour Ġemotion Ġcategories , Ġand Ġaccur acies Ġranging Ġfrom Ġ68 . 8 % Ġto Ġ71 . 8 % Ġare Ġobtained Ġwhen Ġthe Ġmodel Ġis Ġapplied Ġto Ġthe ĠI EM OC AP Ġdataset . ĠIn Ġparticular , Ġit Ġresolves Ġthe Ġissue Ġin Ġwhich Ġpredictions Ġfrequently Ġincorrectly Ġyield Ġthe Ġneutral Ġclass , Ġas Ġoccurs Ġin Ġprevious Ġmodels Ġthat Ġfocus Ġon Ġaudio Ġfeatures . </s> In Ġthe Ġfuture Ġwork , Ġwe Ġaim Ġto Ġextend Ġthe Ġmod alities Ġto Ġaudio , Ġtext Ġand Ġvideo Ġinputs . ĠFurthermore , Ġwe Ġplan Ġto Ġinvestigate Ġthe Ġapplication Ġof Ġthe Ġattention Ġmechanism Ġto Ġdata Ġderived Ġfrom Ġmultiple Ġmod alities . ĠThis Ġapproach Ġseems Ġlikely Ġto Ġuncover Ġenhanced Ġlearning Ġschemes Ġthat Ġwill Ġincrease Ġperformance Ġin Ġboth Ġspeech Ġemotion Ġrecognition Ġand Ġother Ġmultim od al Ġclassification Ġtasks . </s> Acknowled gments </s> K . ĠJung Ġis Ġwith Ġthe ĠDepartment Ġof ĠElectrical Ġand ĠComputer ĠEngineering , ĠAS RI , ĠSeoul ĠNational ĠUniversity , ĠSeoul , ĠKorea . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠMinistry Ġof ĠTrade , ĠIndustry Ġ& ĠEnergy Ġ( M OT IE , ĠKorea ) Ġunder ĠIndustrial ĠTechnology ĠInnovation ĠProgram Ġ( No . 100 73 144 ).