Friends, EmotionPush
EXTRACTIVE
EmotionLines BIBREF6
EXTRACTIVE
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues. The former comes from the scripts of the Friends TV sitcom. The other is made up of Facebook messenger chats. Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances. All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content. Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral. If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral”.
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues. The former comes from the scripts of the Friends TV sitcom. The other is made up of Facebook messenger chats. Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances. All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content. Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral. If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral”.
1908.06264
Introduction </s> Em otion Ġdetection Ġhas Ġlong Ġbeen Ġa Ġtopic Ġof Ġinterest Ġto Ġscholars Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġdomain . ĠResearchers Ġaim Ġto Ġrecognize Ġthe Ġemotion Ġbehind Ġthe Ġtext Ġand Ġdistribute Ġsimilar Ġones Ġinto Ġthe Ġsame Ġgroup . ĠEst ablish ing Ġan Ġemotion Ġclass ifier Ġcan Ġnot Ġonly Ġunderstand Ġeach Ġuser 's Ġfeeling Ġbut Ġalso Ġbe Ġextended Ġto Ġvarious Ġapplication , Ġfor Ġexample , Ġthe Ġmotivation Ġbehind Ġa Ġuser 's Ġinterests ĠB IB REF 0 . ĠBased Ġon Ġreleasing Ġof Ġlarge Ġtext Ġcorpus Ġon Ġsocial Ġmedia Ġand Ġthe Ġemotion Ġcategories Ġproposed Ġby ĠB IB REF 1 , ĠB IB REF 2 , Ġnumerous Ġmodels Ġhave Ġprovided Ġand Ġachieved Ġfabulous Ġprecision Ġso Ġfar . ĠFor Ġexample , ĠDeep Mo ji ĠB IB REF 3 Ġwhich Ġutilized Ġtransfer Ġlearning Ġconcept Ġto Ġenhance Ġemotions Ġand Ġsarc asm Ġunderstanding Ġbehind Ġthe Ġtarget Ġsentence . ĠCAR ER ĠB IB REF 4 Ġlearned Ġcontextual ized Ġaffect Ġrepresentations Ġto Ġmake Ġitself Ġmore Ġsensitive Ġto Ġrare Ġwords Ġand Ġthe Ġscenario Ġbehind Ġthe Ġtexts . </s> As Ġmethods Ġbecome Ġmature , Ġtext - based Ġemotion Ġdetecting Ġapplications Ġcan Ġbe Ġextended Ġfrom Ġa Ġsingle Ġutter ance Ġto Ġa Ġdialogue Ġcontributed Ġby Ġa Ġseries Ġof Ġutter ances . ĠTable ĠT AB REF 2 Ġillustrates Ġthe Ġdifference Ġbetween Ġsingle Ġutter ance Ġand Ġdialogue Ġemotion Ġrecognition . ĠThe Ġsame Ġutter ances Ġin ĠTable ĠT AB REF 2 , Ġeven Ġthe Ġsame Ġperson Ġsaid Ġthe Ġsame Ġsentence , Ġthe Ġemotion Ġit Ġconvey Ġmay Ġbe Ġvarious , Ġwhich Ġmay Ġdepend Ġon Ġdifferent Ġbackground Ġof Ġthe Ġconversation , Ġtone Ġof Ġspeaking Ġor Ġpersonality . ĠTherefore , Ġfor Ġemotion Ġdetection , Ġthe Ġinformation Ġfrom Ġpreceding Ġutter ances Ġin Ġa Ġconversation Ġis Ġrelatively Ġcritical . </s> In ĠSocial N LP Ġ2019 ĠEm otion X , Ġthe Ġchallenge Ġis Ġto Ġrecognize Ġemotions Ġfor Ġall Ġutter ances Ġin ĠEm otion L ines Ġdataset , Ġa Ġdataset Ġconsists Ġof Ġdialog ues . ĠAccording Ġto Ġthe Ġneeds Ġfor Ġconsidering Ġcontext Ġat Ġthe Ġsame Ġtime , Ġwe Ġdevelop Ġtwo Ġclassification Ġmodels , Ġinspired Ġby Ġbid irection al Ġenc oder Ġrepresentations Ġfrom Ġtransform ers Ġ( BER T ) ĠB IB REF 5 , ĠFriends BER T Ġand ĠChat BER T . ĠIn Ġthis Ġpaper , Ġwe Ġintroduce Ġour Ġapproaches Ġincluding Ġcausal Ġutter ance Ġmodeling , Ġmodel Ġpre - training , Ġand Ġfine - turn ing . </s> Dat as et </s> Em otion L ines ĠB IB REF 6 Ġis Ġa Ġdialogue Ġdataset Ġcomposed Ġof Ġtwo Ġsubs ets , ĠFriends Ġand ĠEm otion Push , Ġaccording Ġto Ġthe Ġsource Ġof Ġthe Ġdialog ues . ĠThe Ġformer Ġcomes Ġfrom Ġthe Ġscripts Ġof Ġthe ĠFriends ĠTV Ġsitcom . ĠThe Ġother Ġis Ġmade Ġup Ġof ĠFacebook Ġmessenger Ġchats . ĠEach Ġsubset Ġincludes Ġ$ 1 , 000 $ ĠEnglish Ġdialog ues , Ġand Ġeach Ġdialogue Ġcan Ġbe Ġfurther Ġdivided Ġinto Ġa Ġfew Ġconsecutive Ġutter ances . ĠAll Ġthe Ġutter ances Ġare Ġannot ated Ġby Ġfive Ġannot ators Ġon Ġa Ġcrowd - s ourcing Ġplatform Ġ( Amazon ĠMechanical ĠTurk ), Ġand Ġthe Ġlabeling Ġwork Ġis Ġonly Ġbased Ġon Ġthe Ġtextual Ġcontent . ĠAn not ator Ġvotes Ġfor Ġone Ġof Ġthe Ġseven Ġemotions , Ġnamely ĠEk man âĢ Ļ s Ġsix Ġbasic Ġemotions ĠB IB REF 1 , Ġplus Ġthe Ġneutral . ĠIf Ġnone Ġof Ġthe Ġemotion Ġgets Ġmore Ġthan Ġthree Ġvotes , Ġthe Ġutter ance Ġwill Ġbe Ġmarked Ġas ĠâĢ ľ non - neutral âĢ Ŀ . </s> For Ġthe Ġdatasets , Ġthere Ġare Ġproperties Ġworth Ġadditional Ġmentioning . ĠAlthough ĠFriends Ġand ĠEm otion Push Ġshare Ġthe Ġsame Ġdata Ġformat , Ġthey Ġare Ġquite Ġdifferent Ġin Ġnature . ĠFriends Ġis Ġa Ġspeech - based Ġdataset Ġwhich Ġis Ġannot ated Ġdialog ues Ġfrom Ġthe ĠTV Ġsitcom . ĠIt Ġmeans Ġmost Ġof Ġthe Ġutter ances Ġare Ġgenerated Ġby Ġthe Ġa Ġfew Ġmain Ġcharacters . ĠThe Ġpersonality Ġof Ġa Ġcharacter Ġoften Ġaffects Ġthe Ġway Ġof Ġspeaking , Ġand Ġtherefore ĠâĢ ľ who Ġis Ġthe Ġspeaker " Ġmight Ġprovide Ġextra Ġclues Ġfor Ġemotion Ġprediction . ĠIn Ġcontrast , ĠEm otion Push Ġdoes Ġnot Ġhave Ġthis Ġtrait Ġdue Ġto Ġthe Ġanonymous Ġmechanism . ĠIn Ġaddition , Ġfeatures Ġsuch Ġas Ġtypo , Ġhyper link , Ġand Ġemoji Ġthat Ġonly Ġappear Ġin Ġchat - based Ġdata Ġwill Ġneed Ġsome Ġdomain - specific Ġtechniques Ġto Ġprocess . </s> Inc identally , Ġthe Ġobjective Ġof Ġthe Ġchallenge Ġis Ġto Ġpredict Ġthe Ġemotion Ġfor Ġeach Ġutter ance . ĠJust , Ġaccording Ġto ĠEm otion X Ġ2019 Ġspecification , Ġthere Ġare Ġonly Ġfour Ġemotions Ġbe Ġselected Ġas Ġour Ġlabel Ġcandidates , Ġwhich Ġare ĠJoy , ĠSad ness , ĠAnger , Ġand ĠNeutral . ĠThese Ġemotions Ġwill Ġbe Ġconsidered Ġduring Ġperformance Ġevaluation . ĠThe Ġtechnical Ġdetail Ġwill Ġalso Ġbe Ġintroduced Ġand Ġdiscussed Ġin Ġfollowing ĠSection ĠSEC REF 13 Ġand ĠSection ĠSEC REF 26 . </s> Model ĠDescription </s> For Ġthis Ġchallenge , Ġwe Ġadapt ĠB ERT Ġwhich Ġis Ġproposed Ġby ĠB IB REF 5 Ġto Ġhelp Ġunderstand Ġthe Ġcontext Ġat Ġthe Ġsame Ġtime . ĠTechn ically , ĠB ERT , Ġdesigned Ġon Ġend - to - end Ġarchitecture , Ġis Ġa Ġdeep Ġpre - trained Ġtransformer Ġenc oder Ġthat Ġdynamically Ġprovides Ġlanguage Ġrepresentation Ġand ĠB ERT Ġalready Ġachieved Ġmultiple Ġstate - of - the - art Ġresults Ġon ĠGL UE Ġbenchmark ĠB IB REF 7 Ġand Ġmany Ġtasks . ĠA Ġquick Ġrecap Ġfor ĠB ERT 's Ġarchitecture Ġand Ġits Ġpre - training Ġtasks Ġwill Ġbe Ġillustrated Ġin Ġthe Ġfollowing Ġsubsections . </s> Model ĠDescription Ġ: :: ĠModel ĠArchitecture </s> BER T , Ġthe ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers , Ġconsists Ġof Ġseveral Ġtransformer Ġenc oder Ġlayers Ġthat Ġenable Ġthe Ġmodel Ġto Ġextract Ġvery Ġdeep Ġlanguage Ġfeatures Ġon Ġboth Ġtoken - level Ġand Ġsentence - level . ĠEach Ġtransformer Ġenc oder Ġcontains Ġmulti - head Ġself - att ention Ġlayers Ġthat Ġprovide Ġability Ġto Ġlearn Ġmultiple Ġattention Ġfeature Ġof Ġeach Ġword Ġfrom Ġtheir Ġbid irection al Ġcontext . ĠThe Ġtransformer Ġand Ġits Ġself - att ention Ġmechanism Ġare Ġproposed Ġby ĠB IB REF 8 . ĠThis Ġself - att ention Ġmechanism Ġcan Ġbe Ġinterpreted Ġas Ġa Ġkey - value Ġmapping Ġgiven Ġquery . ĠBy Ġgiven Ġthe Ġembed ding Ġvector Ġfor Ġtoken Ġinput , Ġthe Ġquery Ġ($ Q $ ), Ġkey Ġ($ K $ ) Ġand Ġvalue Ġ($ V $ ) Ġare Ġproduced Ġby Ġthe Ġprojection Ġfrom Ġeach Ġthree Ġparameter Ġmat rices Ġwhere Ġ$ W ^ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ {\ rm Ġmodel }} Ġ\ times Ġd _{ k } }, ĠW ^ K Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel } Ġ\ times Ġd _{ k }} $ Ġand Ġ$ W ^ V Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel } Ġ\ times Ġd _{ v }} $. ĠThe Ġself - att ention ĠB IB REF 8 Ġis Ġformally Ġrepresented Ġas : </s> The Ġ$ Ġd _ k Ġ= Ġd _ v Ġ= Ġd _{ \ rm Ġmodel } Ġ= Ġ1024 $ Ġin ĠB ERT Ġlarge Ġversion Ġand Ġ768 Ġin ĠB ERT Ġbase Ġversion . ĠOnce Ġmodel Ġcan Ġextract Ġattention Ġfeature , Ġwe Ġcan Ġextend Ġone Ġself - att ention Ġinto Ġmulti - head Ġself - att ention , Ġthis Ġextension Ġmakes Ġsub - space Ġfeatures Ġcan Ġbe Ġextracted Ġin Ġsame Ġtime Ġby Ġthis Ġmulti - head Ġconfiguration . ĠOverall , Ġthe Ġmulti - att ention Ġmechanism Ġis Ġadopt Ġfor Ġeach Ġtransformer Ġenc oder , Ġand Ġseveral Ġof Ġenc oder Ġlayer Ġwill Ġbe Ġstacked Ġtogether Ġto Ġform Ġa Ġdeep Ġtransformer Ġenc oder . </s> For Ġthe Ġmodel Ġinput , ĠB ERT Ġallow Ġus Ġtake Ġone Ġsentence Ġas Ġinput Ġsequence Ġor Ġtwo Ġsentences Ġtogether Ġas Ġone Ġinput Ġsequence , Ġand Ġthe Ġmaximum Ġlength Ġof Ġinput Ġsequence Ġis Ġ512 . ĠThe Ġway Ġthat ĠB ERT Ġwas Ġdesigned Ġis Ġfor Ġgiving Ġmodel Ġthe Ġsentence - level Ġand Ġtoken - level Ġunderstanding . ĠIn Ġtwo Ġsentences Ġcase , Ġa Ġspecial Ġtoken Ġ([ SE P ]) Ġwill Ġbe Ġinserted Ġbetween Ġtwo Ġsentences . ĠIn Ġaddition , Ġthe Ġfirst Ġinput Ġtoken Ġis Ġalso Ġa Ġspecial Ġtoken Ġ([ CL S ]), Ġand Ġits Ġcorresponding Ġo up ut Ġwill Ġbe Ġvector Ġplace Ġfor Ġclassification Ġduring Ġfine - tun ing . ĠThe Ġoutputs Ġof Ġthe Ġlast Ġenc oder Ġlayer Ġcorresponding Ġto Ġeach Ġinput Ġtoken Ġcan Ġbe Ġtreated Ġas Ġword Ġrepresentations Ġfor Ġeach Ġtoken , Ġand Ġthe Ġword Ġrepresentation Ġof Ġthe Ġfirst Ġtoken Ġ([ CL S ]) Ġwill Ġbe Ġconsider Ġas Ġclassification Ġ( output ) Ġrepresentation Ġfor Ġfurther Ġfine - tun ing Ġtasks . ĠIn ĠB ERT , Ġthis Ġvector Ġis Ġden oted Ġas Ġ$ C Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel }} Ġ$ , Ġand Ġa Ġclassification Ġlayer Ġis Ġden oted Ġas Ġ$ ĠW Ġ\ in Ġ\ math bb Ġ{ R } ^{ K Ġ\ times Ġd _{ \ rm Ġmodel }} $, Ġwhere Ġ$ K $ Ġis Ġnumber Ġof Ġclassification Ġlabels . ĠFinally , Ġthe Ġprediction Ġ$ P $ Ġof ĠB ERT Ġis Ġrepresented Ġas Ġ$ P Ġ= Ġ{\ rm Ġsoft max } ( CW ^ T ) $. </s> Model ĠDescription Ġ: :: ĠPre - training ĠT asks </s> In Ġpre - training , Ġint ead Ġof Ġusing Ġun id irection al Ġlanguage Ġmodels , ĠB ERT Ġdeveloped Ġtwo Ġpre - training Ġtasks : Ġ( 1 ) ĠMask ed ĠLM Ġ( cl o ze Ġtest ) Ġand Ġ( 2 ) ĠNext ĠSent ence ĠPrediction . ĠAt Ġthe Ġfirst Ġpre - training Ġtask , Ġbid irection al Ġlanguage Ġmodeling Ġcan Ġbe Ġdone Ġat Ġthis Ġclo ze - like Ġpre - training . ĠIn Ġdetail , Ġ15 % Ġtokens Ġof Ġinput Ġsequence Ġwill Ġbe Ġmasked Ġat Ġrandom Ġand Ġmodel Ġneed Ġto Ġpredict Ġthose Ġmasked Ġtokens . ĠThe Ġenc oder Ġwill Ġtry Ġto Ġlearn Ġcontextual Ġrepresentations Ġfrom Ġevery Ġgiven Ġtokens Ġdue Ġto Ġmask ing Ġtokens Ġat Ġrandom . ĠModel Ġwill Ġnot Ġknow Ġwhich Ġpart Ġof Ġthe Ġinput Ġis Ġgoing Ġto Ġbe Ġmasked , Ġso Ġthat Ġthe Ġinformation Ġof Ġeach Ġmasked Ġtokens Ġshould Ġbe Ġinferred Ġby Ġremaining Ġtokens . ĠAt ĠNext ĠSent ence ĠPrediction , Ġtwo Ġsentences Ġconc aten ated Ġtogether Ġwill Ġbe Ġconsidered Ġas Ġmodel Ġinput . ĠIn Ġorder Ġto Ġgive Ġmodel Ġa Ġgood Ġnature Ġlanguage Ġunderstanding , Ġknowing Ġrelationship Ġbetween Ġsentence Ġis Ġone Ġof Ġimportant Ġabilities . ĠWhen Ġgenerating Ġinput Ġsequences , Ġ50 % Ġof Ġtime Ġthe Ġsentence ĠB Ġis Ġactually Ġfollowed Ġby Ġsentence ĠA , Ġand Ġrest Ġ50 % Ġof Ġthe Ġtime Ġthe Ġsentence ĠB Ġwill Ġbe Ġpicked Ġrandomly Ġfrom Ġdataset , Ġand Ġmodel Ġneed Ġto Ġpredict Ġif Ġthe Ġsentence ĠB Ġis Ġnext Ġsentence Ġof Ġsentence ĠA . ĠThat Ġis , Ġthe Ġattention Ġinformation Ġwill Ġbe Ġshared Ġbetween Ġsentences . ĠSuch Ġsentence - level Ġunderstanding Ġmay Ġhave Ġdifficulties Ġto Ġbe Ġlearned Ġat Ġfirst Ġpre - training Ġtask Ġ( Mask ed ĠLM ), Ġtherefore , Ġthe Ġpre - training Ġtask Ġ( N SP ) Ġis Ġdeveloped Ġas Ġsecond Ġtraining Ġgoal Ġto Ġcapture Ġthe Ġcross Ġsentence Ġrelationship . </s> In Ġthis Ġcompetition , Ġlimited Ġby Ġthe Ġsize Ġof Ġdataset Ġand Ġthe Ġchallenge Ġin Ġcontextual Ġemotion Ġrecognition , Ġwe Ġconsider ĠB ERT Ġwith Ġboth Ġtwo Ġpre - training Ġtasks Ġcan Ġgive Ġa Ġgood Ġstarting Ġpoint Ġto Ġextract Ġemotion Ġchanging Ġduring Ġdialogue - like Ġconversation . ĠEspecially Ġthe Ġsecond Ġpre - training Ġtask , Ġit Ġmight Ġbe Ġmore Ġimportant Ġfor Ġdialogue - like Ġconversation Ġwhere Ġthe Ġemotion Ġmay Ġvarious Ġby Ġthe Ġcontext Ġof Ġcontinuous Ġutter ances . ĠThat Ġis , Ġgiven Ġa Ġset Ġof Ġcontinues Ġconversations , Ġthe Ġemotion Ġof Ġcurrent Ġutter ance Ġmight Ġbe Ġinfluenced Ġby Ġprevious Ġutter ance . ĠBy Ġthis Ġassumption Ġand Ġwith Ġsupporting Ġfrom Ġthe Ġexperiment Ġresults Ġof ĠB ERT , Ġwe Ġcan Ġtake Ġsentence ĠA Ġas Ġone - sent ence Ġcontext Ġand Ġconsider Ġsentence ĠB Ġas Ġthe Ġtarget Ġsentence Ġfor Ġemotion Ġprediction . ĠThe Ġdetail Ġwill Ġbe Ġdescribed Ġin ĠSection ĠSEC REF 4 . </s> Method ology </s> The Ġmain Ġgoal Ġof Ġthe Ġpresent Ġwork Ġis Ġto Ġpredict Ġthe Ġemotion Ġof Ġutter ance Ġwithin Ġthe Ġdialogue . ĠFollowing Ġare Ġfour Ġmajor Ġdifficulties Ġwe Ġconcern Ġabout : </s> The Ġemotion Ġof Ġthe Ġutter ances Ġdepends Ġnot Ġonly Ġon Ġthe Ġtext Ġbut Ġalso Ġon Ġthe Ġinteraction Ġhappened Ġearlier . </s> The Ġsource Ġof Ġthe Ġtwo Ġdatasets Ġare Ġdifferent . ĠFriends Ġis Ġspeech - based Ġdialog ues Ġand ĠEm otion Push Ġis Ġchat - based Ġdialog ues . ĠIt Ġmakes Ġdatasets Ġpossess Ġdifferent Ġcharacteristics . </s> There Ġare Ġonly Ġ$ 1 , 000 $ Ġdialog ues Ġin Ġboth Ġtraining Ġdatasets Ġwhich Ġare Ġnot Ġlarge Ġenough Ġfor Ġthe Ġstability Ġof Ġtraining Ġa Ġcomplex Ġneural - based Ġmodel . </s> The Ġprediction Ġtargets Ġ( em otion Ġlabels ) Ġare Ġhighly Ġun balanced . </s> The Ġproposed Ġapproach Ġis Ġsummarized Ġin ĠFigure ĠFIG REF 3 , Ġwhich Ġaims Ġto Ġovercome Ġthese Ġchallenges . ĠThe Ġframework Ġcould Ġbe Ġseparated Ġinto Ġthree Ġsteps Ġand Ġdescribed Ġas Ġfollow : </s> Method ology Ġ: :: ĠCa usal ĠUt ter ance ĠModel ing </s> Given Ġa Ġdialogue Ġ$ D ^{ ( i )} $ Ġwhich Ġincludes Ġsequence Ġof Ġutter ances Ġden oted Ġas Ġ$ D ^{ ( i )} =( u ^{ ( i )} _{ 1 }, Ġu ^{ ( i )} _{ 2 }, Ġ... , Ġu ^{ ( i )} _{ n }) $, Ġwhere Ġ$ i $ Ġis Ġthe Ġindex Ġin Ġdataset Ġand Ġ$ n $ Ġis Ġthe Ġnumber Ġof Ġutter ances Ġin Ġthe Ġgiven Ġdialogue . ĠIn Ġorder Ġto Ġconserve Ġthe Ġemotional Ġinformation Ġof Ġboth Ġutter ance Ġand Ġconversation , Ġwe Ġrearr ange Ġeach Ġtwo Ġconsecutive Ġutter ances Ġ$ u _{ t }, Ġu _{ t - 1 } $ Ġinto Ġa Ġsingle Ġsentence Ġrepresentation Ġ$ x _{ t } $ Ġas </s> The Ġcorresponding Ġsentence Ġrepresentation Ġcorpus Ġ$ X ^{ ( i )} $ Ġare Ġden oted Ġas Ġ$ X ^{ ( i )} =( x ^{ ( i )} _{ 1 }, Ġx ^{ ( i )} _{ 2 }, Ġ... , Ġx ^{ ( i )} _{ n }) $. ĠNote Ġthat Ġthe Ġfirst Ġutter ance Ġwithin Ġa Ġconversation Ġdoes Ġnot Ġhave Ġits Ġcausal Ġutter ance Ġ( pre vious Ġsentence ), Ġtherefore , Ġthe Ġcausal Ġutter ance Ġwill Ġbe Ġset Ġas Ġ[ None ]. ĠA Ġpractical Ġexample Ġof Ġsentence Ġrepresentation Ġis Ġshown Ġin ĠTable ĠT AB REF 11 . </s> Since Ġthe Ġcharacteristics Ġof Ġtwo Ġdatasets Ġare Ġnot Ġidentical , Ġwe Ġcustomize Ġdifferent Ġcausal Ġutter ance Ġmodeling Ġstrategies Ġto Ġrefine Ġthe Ġinformation Ġin Ġtext . </s> For ĠFriends , Ġthere Ġare Ġtwo Ġspecific Ġproperties . ĠThe Ġfirst Ġone Ġis Ġthat Ġmost Ġdialog ues Ġare Ġsurrounding Ġwith Ġthe Ġsix Ġmain Ġcharacters , Ġincluding ĠRachel , ĠMonica , ĠPh oe be , ĠJoey , ĠChandler , Ġand ĠRoss . ĠThe Ġutter ance Ġratio Ġof Ġgiven Ġby Ġthe Ġsix Ġroles Ġis Ġup Ġto Ġ$ 83 . 4 \ % $. ĠSecond , Ġthe Ġpersonal Ġcharacteristics Ġof Ġthe Ġsix Ġcharacters Ġare Ġvery Ġclear . ĠEach Ġleading Ġrole Ġhas Ġits Ġown Ġemotion Ġund ulated Ġrule . ĠTo Ġmake Ġuse Ġof Ġthese Ġfeatures , Ġwe Ġintroduce Ġthe Ġpersonality Ġtoken ization Ġwhich Ġhelp Ġlearning Ġthe Ġpersonality Ġof Ġthe Ġsix Ġcharacters . ĠPersonality Ġtoken ization Ġconc aten ate Ġthe Ġspeaker Ġand Ġsays Ġtokens Ġbefore Ġthe Ġinput Ġutter ance Ġif Ġthe Ġspeaker Ġis Ġone Ġof Ġthe Ġsix Ġcharacters . ĠThe Ġexample Ġis Ġshown Ġin ĠTable ĠT AB REF 12 . </s> For ĠEm otion Push , Ġthe Ġtext Ġare Ġinformal Ġchats Ġwhich Ġincluding Ġlike Ġslang , Ġacronym , Ġtypo , Ġhyper link , Ġand Ġemoji . ĠAnother Ġcharacteristic Ġis Ġthat Ġthe Ġspecific Ġname Ġentities Ġare Ġtoken ized Ġwith Ġrandom Ġindex . Ġ( e . g . ĠâĢ ľ organ ization _ 80 âĢ Ŀ , ĠâĢ ľ person _ 01 âĢ Ŀ , Ġand ĠâĢ ľ time _ 12 âĢ Ŀ ). ĠWe Ġconsider Ġsome Ġof Ġthese Ġinformal Ġtext Ġare Ġrelated Ġto Ġexpressing Ġemotion Ġsuch Ġas Ġrepeated Ġtyping , Ġpur posed Ġcapital ization , Ġand Ġemoji Ġ( e . g . ĠâĢ ľ : D âĢ Ŀ , ĠâĢ ľ :( âĢ Ŀ , Ġand ĠâĢ ľ < 3 âĢ Ŀ )). ĠTherefore , Ġwe Ġkeep Ġmost Ġinformal Ġexpressions Ġbut Ġonly Ġprocess Ġhyper links , Ġempty Ġutter ance , Ġand Ġname Ġentities Ġby Ġun ifying Ġthe Ġtokens . </s> Method ology Ġ: :: ĠModel ĠPre - training </s> Since Ġthe Ġsize Ġof Ġboth Ġdatasets Ġare Ġnot Ġlarge Ġenough Ġfor Ġcomplex Ġneural - based Ġmodel Ġtraining Ġas Ġwell Ġas ĠB ERT Ġmodel Ġis Ġonly Ġpre - train Ġon Ġformal Ġtext Ġdatasets , Ġthe Ġissues Ġof Ġover fitting Ġand Ġdomain Ġbias Ġare Ġimportant Ġconsiderations Ġfor Ġdesign Ġthe Ġpre - training Ġprocess . </s> To Ġavoid Ġour Ġmodel Ġover fitting Ġon Ġthe Ġtraining Ġdata Ġand Ġincrease Ġthe Ġunderstanding Ġof Ġinformal Ġtext , Ġwe Ġadapted ĠB ERT Ġand Ġderived Ġtwo Ġmodels , Ġnamely ĠFriends BER T Ġand ĠChat BER T , Ġwith Ġdifferent Ġpre - training Ġtasks Ġbefore Ġthe Ġformal Ġtraining Ġprocess Ġfor ĠFriends Ġand ĠEm otion Push Ġdataset , Ġrespectively . ĠThe Ġpre - training Ġstrategies Ġare Ġdescribed Ġbelow . </s> For Ġpre - training ĠFriends BER T , Ġwe Ġcollect Ġthe Ġcompleted Ġscripts Ġof Ġall Ġten Ġseasons Ġof ĠFriends ĠTV Ġshows Ġfrom Ġem ory nl p Ġwhich Ġincludes Ġ3 , 107 Ġscenes Ġwithin Ġ61 , 309 Ġutter ances . ĠAll Ġthe Ġutter ances Ġare Ġfollowed Ġthe Ġpre processing Ġmethods Ġmentions Ġabove Ġto Ġcompose Ġthe Ġcorpus Ġfor ĠMask ed Ġlanguage Ġmodel Ġpre - training Ġtask . ĠThe Ġconsequ ent Ġutter ances Ġin Ġthe Ġsame Ġscenes Ġare Ġconsidered Ġas Ġthe Ġconsequ ent Ġsentences Ġto Ġpre - train Ġthe ĠNext ĠSent ence ĠPrediction Ġtask . ĠIn Ġthe Ġpre - training Ġprocess , Ġthe Ġtraining Ġloss Ġis Ġthe Ġsum Ġof Ġthe Ġmean Ġlikelihood Ġof Ġtwo Ġpre - train Ġtasks . </s> For Ġpre - training ĠChat BER T , Ġwe Ġpre - train Ġour Ġmodel Ġon Ġthe ĠTwitter Ġdataset , Ġsince Ġthe Ġtext Ġand Ġwriting Ġstyle Ġon ĠTwitter Ġare Ġclose Ġto Ġthe Ġchat Ġtext Ġwhere Ġboth Ġmay Ġinvolved Ġwith Ġmany Ġinformal Ġwords Ġor Ġemot icons Ġas Ġwell . ĠThe ĠTwitter Ġemotion Ġdataset , Ġ8 Ġbasic Ġemotions Ġfrom Ġemotion Ġwheel ĠB IB REF 1 , Ġwas Ġcollected Ġby Ġtwitter Ġstreaming ĠAPI Ġwith Ġspecific Ġemotion - related Ġhasht ags , Ġsuch Ġas Ġ# anger , Ġ# joy , Ġ# cry , Ġ# s ad Ġand Ġetc . ĠThe Ġhasht ags Ġin Ġtweets Ġare Ġtreated Ġas Ġemotion Ġlabel Ġfor Ġmodel Ġfine - tun ing . ĠThe Ġtweets Ġwere Ġfine - gr ined Ġprocessing Ġfollowed Ġthe Ġrules Ġin ĠB IB REF 9 , ĠB IB REF 4 , Ġincluding Ġduplicate Ġtweets Ġremoving , Ġthe Ġemotion Ġhasht ags Ġmust Ġappearing Ġin Ġthe Ġlast Ġposition Ġof Ġa Ġtweet , Ġand Ġetc . ĠThe Ġstat is Ġof Ġtweets Ġwere Ġsummarized Ġin ĠTable ĠT AB REF 17 . ĠEach Ġtweet Ġand Ġcorresponding Ġemotion Ġlabel Ġcomp oses Ġan Ġemotion Ġclassification Ġdataset Ġfor Ġpre - training . </s> Method ology Ġ: :: ĠFine - tun ing </s> Since Ġour Ġemotion Ġrecognition Ġtask Ġis Ġtreated Ġas Ġa Ġsequence - level Ġclassification Ġtask , Ġthe Ġmodel Ġwould Ġbe Ġfine - tun ed Ġon Ġthe Ġprocessed Ġtraining Ġdata . ĠFollowing Ġthe ĠB ERT Ġconstruction , Ġwe Ġtake Ġthe Ġfirst Ġembed ding Ġvector Ġwhich Ġcorresponds Ġto Ġthe Ġspecial Ġtoken Ġ[ CL S ] Ġfrom Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe ĠTrans former Ġenc oder . ĠThis Ġvector Ġrepresents Ġthe Ġembed ding Ġvector Ġof Ġthe Ġcorresponding Ġconversation Ġutter ances Ġwhich Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ C } Ġ\ in Ġ\ math bb Ġ{ R } ^{ H } $, Ġwhere Ġ$ H $ Ġis Ġthe Ġembed ding Ġsize . ĠA Ġdense Ġneural Ġlayer Ġis Ġtreated Ġas Ġa Ġclassification Ġlayer Ġwhich Ġconsists Ġof Ġparameters Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K \ times ĠH } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K } $, Ġwhere Ġ$ K $ Ġis Ġthe Ġnumber Ġof Ġemotion Ġclass . ĠThe Ġemotion Ġprediction Ġprobabilities Ġ$\ math bf Ġ{ P } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K } $ Ġare Ġcomputed Ġby Ġa Ġsoft max Ġactivation Ġfunction Ġas </s> All Ġthe Ġparameters Ġin ĠB ERT Ġand Ġthe Ġclassification Ġlayer Ġwould Ġbe Ġfine - turned Ġtogether Ġto Ġminimize Ġthe ĠNegative ĠLog ĠLike lihood Ġ( N LL ) Ġloss Ġfunction , Ġas ĠEqu ation Ġ( DIS PLAY _ FORM 22 ), Ġbased Ġon Ġthe Ġground Ġtruth Ġemotion Ġlabel Ġ$ c $. </s> In Ġorder Ġto Ġtackle Ġthe Ġproblem Ġof Ġhighly Ġun balanced Ġemotion Ġlabels , Ġwe Ġapply Ġweighted Ġbalanced Ġwarming Ġon ĠN LL Ġloss Ġfunction , Ġas ĠEqu ation Ġ( DIS PLAY _ FORM 23 ), Ġin Ġthe Ġfirst Ġepoch Ġof Ġfine - tun ing Ġprocedure . </s> where Ġ$\ math bf Ġ{ w } $ Ġare Ġthe Ġweights Ġof Ġcorresponding Ġemotion Ġlabel Ġ$ c $ Ġwhich Ġare Ġcomputed Ġand Ġnormal ize Ġby Ġthe Ġfrequency Ġas </s> By Ġadding Ġthe Ġweighted Ġbalanced Ġwarming Ġon ĠN LL Ġloss , Ġthe Ġmodel Ġcould Ġlearn Ġto Ġpredict Ġthe Ġminor Ġemotions Ġ( e . g . Ġanger Ġand Ġsadness ) Ġearlier Ġand Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . ĠSince Ġthe Ġmajor Ġevaluation Ġmetrics Ġmicro ĠF 1 - score Ġis Ġeffect Ġby Ġthe Ġnumber Ġof Ġeach Ġlabel , Ġwe Ġonly Ġapply Ġthe Ġweighted Ġbalanced Ġwarming Ġin Ġfirst Ġepoch Ġto Ġoptimize Ġthe Ġperformance . </s> Exper iments </s> Since Ġthe ĠEm otion X Ġchallenge Ġonly Ġprovided Ġthe Ġgold Ġlabels Ġin Ġtraining Ġdata , Ġwe Ġpick Ġthe Ġbest Ġperformance Ġmodel Ġ( weights ) Ġto Ġpredict Ġthe Ġtesting Ġdata . ĠIn Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġexperiment Ġand Ġevaluation Ġresults . </s> Exper iments Ġ: :: ĠExperimental ĠSetup </s> The ĠEm otion X Ġchallenge Ġconsists Ġof Ġ$ 1 , 000 $ Ġdialog ues Ġfor Ġboth ĠFriends Ġand ĠEm otion Push . ĠIn Ġall Ġof Ġour Ġexperiments , Ġeach Ġdataset Ġis Ġseparated Ġinto Ġtop Ġ800 Ġdialog ues Ġfor Ġtraining Ġand Ġlast Ġ200 Ġdialog ues Ġfor Ġvalidation . ĠSince Ġthe ĠEm otion X Ġchallenge Ġconsiders Ġonly Ġthe Ġfour Ġemotions Ġ( anger , Ġjoy , Ġneutral , Ġand Ġsadness ) Ġin Ġthe Ġevaluation Ġstage , Ġwe Ġignore Ġall Ġthe Ġdata Ġpoint Ġcorresponding Ġto Ġother Ġemotions Ġdirectly . ĠThe Ġdetails Ġof Ġemotions Ġdistribution Ġare Ġshown Ġin ĠTable ĠT AB REF 18 . </s> The Ġhyper param eters Ġand Ġtraining Ġsetup Ġof Ġour Ġmodels Ġ( Friends BER T Ġand ĠChat BER T ) Ġare Ġshown Ġin ĠTable ĠT AB REF 25 . ĠSome Ġcommon Ġand Ġeasily Ġimplemented Ġmethods Ġare Ġselected Ġas Ġthe Ġbas elines Ġembed ding Ġmethods Ġand Ġclassification Ġmodels . ĠThe Ġbaseline Ġembed ding Ġmethods Ġare Ġincluding Ġbag - of - words Ġ( B OW ), Ġterm Ġfrequency âĢĵ in verse Ġdocument Ġfrequency Ġ( TF ID F ), Ġand Ġneural - based Ġword Ġembed ding . ĠThe Ġclassification Ġmodels Ġare Ġincluding ĠLog istic ĠReg ression Ġ( LR ), ĠRandom ĠForest Ġ( RF ), ĠText CNN ĠB IB REF 10 Ġwith Ġinitial Ġword Ġembed ding Ġas ĠGl o Ve ĠB IB REF 11 , Ġand Ġour Ġproposed Ġmodel . ĠAll Ġthe Ġexperiment Ġresults Ġare Ġbased Ġon Ġthe Ġbest Ġperformances Ġof Ġvalidation Ġresults . </s> Exper iments Ġ: :: ĠPerformance </s> The Ġexperiment Ġresults Ġof Ġvalidation Ġon ĠFriends Ġare Ġshown Ġin ĠTable ĠT AB REF 19 . ĠThe Ġproposed Ġmodel Ġand Ġbas elines Ġare Ġevaluated Ġbased Ġon Ġthe ĠPrecision Ġ( P .), ĠRecall Ġ( R .), Ġand ĠF 1 - me asure Ġ( F 1 ). </s> For Ġthe Ġtraditional Ġbas elines , Ġnamely ĠB OW Ġand ĠTF ID F , Ġwe Ġobserve Ġthat Ġthey Ġachieve Ġsurprising Ġhigh ĠF 1 Ġscores Ġaround Ġ$ 0 . 81 $, Ġhowever , Ġthe Ġscores Ġfor ĠAnger Ġand ĠSad ness Ġare Ġlower . ĠThis Ġexplains Ġthat Ġtraditional Ġapproaches Ġtend Ġto Ġpredict Ġthe Ġlabels Ġwith Ġlarge Ġsample Ġsize , Ġsuch Ġas ĠJoy Ġand ĠNeutral , Ġbut Ġfail Ġto Ġtake Ġof Ġscarce Ġsamples Ġeven Ġwhen Ġan Ġensemble Ġrandom Ġforest Ġclass ifier Ġis Ġadopted . ĠIn Ġorder Ġto Ġprevent Ġthe Ġun balanced Ġlearning , Ġwe Ġchoose Ġthe Ġweighted Ġloss Ġmechanism Ġfor Ġboth ĠText CNN Ġand Ġcausal Ġmodeling ĠText CNN Ġ( C - Text CNN ), Ġthese Ġmodels Ġsuffer Ġless Ġthan Ġthe Ġtraditional Ġbas elines Ġand Ġachieve Ġa Ġslightly Ġbalance Ġperformance , Ġwhere Ġthere Ġare Ġaround Ġ15 % Ġand Ġ7 % Ġimprovement Ġon ĠAnger Ġand ĠSad ness , Ġrespectively . ĠWe Ġfollowing Ġadopt Ġthe Ġcasual Ġutter ance Ġmodeling Ġto Ġoriginal ĠText CNN , Ġmapping Ġprevious Ġutter ance Ġas Ġwell Ġas Ġtarget Ġutter ance Ġinto Ġmodel . ĠThe Ġcausal Ġutter ance Ġmodeling Ġimprove Ġthe ĠC - Text CNN Ġover ĠText CNN Ġfor Ġ6 %, Ġ2 % Ġand Ġ1 % Ġon ĠAnger , ĠJoy Ġand Ġoverall ĠF 1 Ġscore . ĠMot ivated Ġfrom Ġthese Ġpreliminary Ġexperiments , Ġthe Ġproposed ĠFriends BER T Ġalso Ġadopt Ġthe Ġideas Ġof Ġboth Ġweighted Ġloss Ġand Ġcausal Ġutter ance Ġmodeling . ĠAs Ġcompared Ġto Ġthe Ġoriginal ĠB ERT , Ġsingle Ġsentence ĠB ERT Ġ( Friends BER T - base - s ), Ġthe Ġproposed ĠFriends BER T - base Ġimprove Ġ1 % Ġfor ĠJoy Ġand Ġoverall ĠF 1 , Ġand Ġ2 % Ġfor ĠSad ness . ĠFor Ġthe Ġfinal Ġvalidation Ġperformance , Ġour Ġproposed Ġapproach Ġachieves Ġthe Ġhighest Ġscores , Ġwhich Ġare Ġ$ 0 . 85 $ Ġand Ġ$ 0 . 86 $ Ġfor ĠFriends BER T - base Ġand ĠFriends BER T - large , Ġrespectively . </s> Overall , Ġthe Ġproposed ĠFriends BER T Ġsuccessfully Ġcaptures Ġthe Ġsentence - level Ġcontext - aw arded Ġinformation Ġand Ġoutper forms Ġall Ġthe Ġbas elines , Ġwhich Ġnot Ġonly Ġachieves Ġhigh Ġperformance Ġon Ġlarge Ġsample Ġlabels , Ġbut Ġalso Ġon Ġsmall Ġsample Ġlabels . ĠThe Ġsimilar Ġsettings Ġare Ġalso Ġadapted Ġto ĠEm otion Push Ġdataset Ġfor Ġthe Ġfinal Ġevaluation . </s> Exper iments Ġ: :: ĠEvaluation ĠResults </s> The Ġtesting Ġdataset Ġconsists Ġof Ġ240 Ġdialog ues Ġincluding Ġ$ 3 , 296 $ Ġand Ġ$ 3 , 536 $ Ġutter ances Ġin ĠFriends Ġand ĠEm otion Push Ġrespectively . ĠWe Ġre - train Ġour ĠFriends BER T Ġand ĠChat BER T Ġwith Ġtop Ġ920 Ġtraining Ġdialog ues Ġand Ġpredict Ġthe Ġevaluation Ġresults Ġusing Ġthe Ġmodel Ġperforming Ġthe Ġbest Ġvalidation Ġresults . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 29 Ġand ĠTable ĠT AB REF 30 . ĠThe Ġpresent Ġmethod Ġachieves Ġ$ 81 . 5 \ % $ Ġand Ġ$ 88 . 5 \ % $ Ġmicro ĠF 1 - score Ġon Ġthe Ġtesting Ġdataset Ġof ĠFriends Ġand ĠEm otion Push , Ġrespectively . </s> Conclusion Ġand ĠFuture Ġwork </s> In Ġthe Ġpresent Ġwork , Ġwe Ġpropose ĠFriends BER T Ġand ĠChat BER T Ġfor Ġthe Ġmulti - utter ance Ġemotion Ġrecognition Ġtask Ġon ĠEm otion L ines Ġdataset . ĠThe Ġproposed Ġmodels Ġare Ġadapted Ġfrom ĠB ERT ĠB IB REF 5 Ġwith Ġthree Ġmain Ġimprovement Ġduring Ġthe Ġmodel Ġtraining Ġprocedure , Ġwhich Ġare Ġthe Ġcausal Ġutter ance Ġmodeling Ġmechanism , Ġspecific Ġmodel Ġpre - training , Ġand Ġadapt Ġweighted Ġloss . ĠThe Ġcausal Ġutter ance Ġmodeling Ġtakes Ġthe Ġadvantages Ġof Ġthe Ġsentence - level Ġcontext Ġinformation Ġduring Ġmodel Ġinference . ĠThe Ġspecific Ġmodel Ġpre - training Ġhelps Ġto Ġagainst Ġthe Ġbias Ġin Ġdifferent Ġtext Ġdomain . ĠThe Ġweighted Ġloss Ġavoids Ġour Ġmodel Ġto Ġonly Ġpredict Ġon Ġlarge Ġsize Ġsample . ĠThe Ġeffectiveness Ġand Ġgeneral iz ability Ġof Ġthe Ġproposed Ġmethods Ġare Ġdemonstrated Ġfrom Ġthe Ġexperiments . </s> In Ġfuture Ġwork , Ġwe Ġconsider Ġto Ġinclude Ġthe Ġconditional Ġprob abil istic Ġconstraint Ġ$ P Ġ( {\ rm ĠEm o } _{ B } Ġ| Ġ\ hat {\ rm ĠEm o } _{ A }) $. ĠModel Ġshould Ġpredict Ġthe Ġemotion Ġbased Ġon Ġa Ġcertain Ġunderstanding Ġabout Ġcontext Ġemotions . ĠThis Ġmight Ġbe Ġmore Ġreasonable Ġfor Ġguiding Ġmodel Ġthan Ġjust Ġpredicting Ġemotion Ġof Ġ${ \ rm ĠSent ence } _ B $ Ġdirectly . ĠIn Ġaddition , Ġdue Ġto Ġthe Ġlimitation Ġof ĠB ERT Ġinput Ġformat , Ġambiguous Ġnumber Ġof Ġinput Ġsentences Ġis Ġnow Ġbecoming Ġan Ġimportant Ġdesign Ġrequirement Ġfor Ġour Ġfuture Ġwork . ĠAlso , Ġpersonality Ġembed ding Ġdevelopment Ġwill Ġbe Ġanother Ġfuture Ġwork Ġof Ġthe Ġemotion Ġrecognition . ĠThe Ġpersonality Ġembed ding Ġwill Ġbe Ġconsidered Ġas Ġsentence Ġembed ding Ġinjected Ġinto Ġword Ġembed ding , Ġand Ġit Ġseems Ġthis Ġadditional Ġinformation Ġcan Ġcontribute Ġsome Ġimprovement Ġpotentially .
Introduction Emotion detection has long been a topic of interest to scholars in natural language processing (NLP) domain. Researchers aim to recognize the emotion behind the text and distribute similar ones into the same group. Establishing an emotion classifier can not only understand each user's feeling but also be extended to various application, for example, the motivation behind a user's interests BIBREF0. Based on releasing of large text corpus on social media and the emotion categories proposed by BIBREF1, BIBREF2, numerous models have provided and achieved fabulous precision so far. For example, DeepMoji BIBREF3 which utilized transfer learning concept to enhance emotions and sarcasm understanding behind the target sentence. CARER BIBREF4 learned contextualized affect representations to make itself more sensitive to rare words and the scenario behind the texts. As methods become mature, text-based emotion detecting applications can be extended from a single utterance to a dialogue contributed by a series of utterances. Table TABREF2 illustrates the difference between single utterance and dialogue emotion recognition. The same utterances in Table TABREF2, even the same person said the same sentence, the emotion it convey may be various, which may depend on different background of the conversation, tone of speaking or personality. Therefore, for emotion detection, the information from preceding utterances in a conversation is relatively critical. In SocialNLP 2019 EmotionX, the challenge is to recognize emotions for all utterances in EmotionLines dataset, a dataset consists of dialogues. According to the needs for considering context at the same time, we develop two classification models, inspired by bidirectional encoder representations from transformers (BERT) BIBREF5, FriendsBERT and ChatBERT. In this paper, we introduce our approaches including causal utterance modeling, model pre-training, and fine-turning. Dataset EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues. The former comes from the scripts of the Friends TV sitcom. The other is made up of Facebook messenger chats. Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances. All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content. Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral. If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral”. For the datasets, there are properties worth additional mentioning. Although Friends and EmotionPush share the same data format, they are quite different in nature. Friends is a speech-based dataset which is annotated dialogues from the TV sitcom. It means most of the utterances are generated by the a few main characters. The personality of a character often affects the way of speaking, and therefore “who is the speaker" might provide extra clues for emotion prediction. In contrast, EmotionPush does not have this trait due to the anonymous mechanism. In addition, features such as typo, hyperlink, and emoji that only appear in chat-based data will need some domain-specific techniques to process. Incidentally, the objective of the challenge is to predict the emotion for each utterance. Just, according to EmotionX 2019 specification, there are only four emotions be selected as our label candidates, which are Joy, Sadness, Anger, and Neutral. These emotions will be considered during performance evaluation. The technical detail will also be introduced and discussed in following Section SECREF13 and Section SECREF26. Model Description For this challenge, we adapt BERT which is proposed by BIBREF5 to help understand the context at the same time. Technically, BERT, designed on end-to-end architecture, is a deep pre-trained transformer encoder that dynamically provides language representation and BERT already achieved multiple state-of-the-art results on GLUE benchmark BIBREF7 and many tasks. A quick recap for BERT's architecture and its pre-training tasks will be illustrated in the following subsections. Model Description ::: Model Architecture BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level. Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context. The transformer and its self-attention mechanism are proposed by BIBREF8. This self-attention mechanism can be interpreted as a key-value mapping given query. By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$. The self-attention BIBREF8 is formally represented as: The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version. Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration. Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder. For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512. The way that BERT was designed is for giving model the sentence-level and token-level understanding. In two sentences case, a special token ([SEP]) will be inserted between two sentences. In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning. The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks. In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels. Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$. Model Description ::: Pre-training Tasks In pre-training, intead of using unidirectional language models, BERT developed two pre-training tasks: (1) Masked LM (cloze test) and (2) Next Sentence Prediction. At the first pre-training task, bidirectional language modeling can be done at this cloze-like pre-training. In detail, 15% tokens of input sequence will be masked at random and model need to predict those masked tokens. The encoder will try to learn contextual representations from every given tokens due to masking tokens at random. Model will not know which part of the input is going to be masked, so that the information of each masked tokens should be inferred by remaining tokens. At Next Sentence Prediction, two sentences concatenated together will be considered as model input. In order to give model a good nature language understanding, knowing relationship between sentence is one of important abilities. When generating input sequences, 50% of time the sentence B is actually followed by sentence A, and rest 50% of the time the sentence B will be picked randomly from dataset, and model need to predict if the sentence B is next sentence of sentence A. That is, the attention information will be shared between sentences. Such sentence-level understanding may have difficulties to be learned at first pre-training task (Masked LM), therefore, the pre-training task (NSP) is developed as second training goal to capture the cross sentence relationship. In this competition, limited by the size of dataset and the challenge in contextual emotion recognition, we consider BERT with both two pre-training tasks can give a good starting point to extract emotion changing during dialogue-like conversation. Especially the second pre-training task, it might be more important for dialogue-like conversation where the emotion may various by the context of continuous utterances. That is, given a set of continues conversations, the emotion of current utterance might be influenced by previous utterance. By this assumption and with supporting from the experiment results of BERT, we can take sentence A as one-sentence context and consider sentence B as the target sentence for emotion prediction. The detail will be described in Section SECREF4. Methodology The main goal of the present work is to predict the emotion of utterance within the dialogue. Following are four major difficulties we concern about: The emotion of the utterances depends not only on the text but also on the interaction happened earlier. The source of the two datasets are different. Friends is speech-based dialogues and EmotionPush is chat-based dialogues. It makes datasets possess different characteristics. There are only $1,000$ dialogues in both training datasets which are not large enough for the stability of training a complex neural-based model. The prediction targets (emotion labels) are highly unbalanced. The proposed approach is summarized in Figure FIGREF3, which aims to overcome these challenges. The framework could be separated into three steps and described as follow: Methodology ::: Causal Utterance Modeling Given a dialogue $D^{(i)}$ which includes sequence of utterances denoted as $D^{(i)}=(u^{(i)}_{1}, u^{(i)}_{2}, ..., u^{(i)}_{n})$, where $i$ is the index in dataset and $n$ is the number of utterances in the given dialogue. In order to conserve the emotional information of both utterance and conversation, we rearrange each two consecutive utterances $u_{t}, u_{t-1}$ into a single sentence representation $x_{t}$ as The corresponding sentence representation corpus $X^{(i)}$ are denoted as $X^{(i)}=(x^{(i)}_{1}, x^{(i)}_{2}, ..., x^{(i)}_{n})$. Note that the first utterance within a conversation does not have its causal utterance (previous sentence), therefore, the causal utterance will be set as [None]. A practical example of sentence representation is shown in Table TABREF11. Since the characteristics of two datasets are not identical, we customize different causal utterance modeling strategies to refine the information in text. For Friends, there are two specific properties. The first one is that most dialogues are surrounding with the six main characters, including Rachel, Monica, Phoebe, Joey, Chandler, and Ross. The utterance ratio of given by the six roles is up to $83.4\%$. Second, the personal characteristics of the six characters are very clear. Each leading role has its own emotion undulated rule. To make use of these features, we introduce the personality tokenization which help learning the personality of the six characters. Personality tokenization concatenate the speaker and says tokens before the input utterance if the speaker is one of the six characters. The example is shown in Table TABREF12. For EmotionPush, the text are informal chats which including like slang, acronym, typo, hyperlink, and emoji. Another characteristic is that the specific name entities are tokenized with random index. (e.g. “organization_80”, “person_01”, and “time_12”). We consider some of these informal text are related to expressing emotion such as repeated typing, purposed capitalization, and emoji (e.g. “:D”, “:(”, and “<3”)). Therefore, we keep most informal expressions but only process hyperlinks, empty utterance, and name entities by unifying the tokens. Methodology ::: Model Pre-training Since the size of both datasets are not large enough for complex neural-based model training as well as BERT model is only pre-train on formal text datasets, the issues of overfitting and domain bias are important considerations for design the pre-training process. To avoid our model overfitting on the training data and increase the understanding of informal text, we adapted BERT and derived two models, namely FriendsBERT and ChatBERT, with different pre-training tasks before the formal training process for Friends and EmotionPush dataset, respectively. The pre-training strategies are described below. For pre-training FriendsBERT, we collect the completed scripts of all ten seasons of Friends TV shows from emorynlp which includes 3,107 scenes within 61,309 utterances. All the utterances are followed the preprocessing methods mentions above to compose the corpus for Masked language model pre-training task. The consequent utterances in the same scenes are considered as the consequent sentences to pre-train the Next Sentence Prediction task. In the pre-training process, the training loss is the sum of the mean likelihood of two pre-train tasks. For pre-training ChatBERT, we pre-train our model on the Twitter dataset, since the text and writing style on Twitter are close to the chat text where both may involved with many informal words or emoticons as well. The Twitter emotion dataset, 8 basic emotions from emotion wheel BIBREF1, was collected by twitter streaming API with specific emotion-related hashtags, such as #anger, #joy, #cry, #sad and etc. The hashtags in tweets are treated as emotion label for model fine-tuning. The tweets were fine-grined processing followed the rules in BIBREF9, BIBREF4, including duplicate tweets removing, the emotion hashtags must appearing in the last position of a tweet, and etc. The statis of tweets were summarized in Table TABREF17. Each tweet and corresponding emotion label composes an emotion classification dataset for pre-training. Methodology ::: Fine-tuning Since our emotion recognition task is treated as a sequence-level classification task, the model would be fine-tuned on the processed training data. Following the BERT construction, we take the first embedding vector which corresponds to the special token [CLS] from the final hidden state of the Transformer encoder. This vector represents the embedding vector of the corresponding conversation utterances which is denoted as $\mathbf {C} \in \mathbb {R}^{H}$, where $H$ is the embedding size. A dense neural layer is treated as a classification layer which consists of parameters $\mathbf {W} \in \mathbb {R}^{K\times H}$ and $\mathbf {b} \in \mathbb {R}^{K}$, where $K$ is the number of emotion class. The emotion prediction probabilities $\mathbf {P} \in \mathbb {R}^{K}$ are computed by a softmax activation function as All the parameters in BERT and the classification layer would be fine-turned together to minimize the Negative Log Likelihood (NLL) loss function, as Equation (DISPLAY_FORM22), based on the ground truth emotion label $c$. In order to tackle the problem of highly unbalanced emotion labels, we apply weighted balanced warming on NLL loss function, as Equation (DISPLAY_FORM23), in the first epoch of fine-tuning procedure. where $\mathbf {w}$ are the weights of corresponding emotion label $c$ which are computed and normalize by the frequency as By adding the weighted balanced warming on NLL loss, the model could learn to predict the minor emotions (e.g. anger and sadness) earlier and make the training process more stable. Since the major evaluation metrics micro F1-score is effect by the number of each label, we only apply the weighted balanced warming in first epoch to optimize the performance. Experiments Since the EmotionX challenge only provided the gold labels in training data, we pick the best performance model (weights) to predict the testing data. In this section, we present the experiment and evaluation results. Experiments ::: Experimental Setup The EmotionX challenge consists of $1,000$ dialogues for both Friends and EmotionPush. In all of our experiments, each dataset is separated into top 800 dialogues for training and last 200 dialogues for validation. Since the EmotionX challenge considers only the four emotions (anger, joy, neutral, and sadness) in the evaluation stage, we ignore all the data point corresponding to other emotions directly. The details of emotions distribution are shown in Table TABREF18. The hyperparameters and training setup of our models (FriendsBERT and ChatBERT) are shown in Table TABREF25. Some common and easily implemented methods are selected as the baselines embedding methods and classification models. The baseline embedding methods are including bag-of-words (BOW), term frequency–inverse document frequency (TFIDF), and neural-based word embedding. The classification models are including Logistic Regression (LR), Random Forest (RF), TextCNN BIBREF10 with initial word embedding as GloVe BIBREF11, and our proposed model. All the experiment results are based on the best performances of validation results. Experiments ::: Performance The experiment results of validation on Friends are shown in Table TABREF19. The proposed model and baselines are evaluated based on the Precision (P.), Recall (R.), and F1-measure (F1). For the traditional baselines, namely BOW and TFIDF, we observe that they achieve surprising high F1 scores around $0.81$, however, the scores for Anger and Sadness are lower. This explains that traditional approaches tend to predict the labels with large sample size, such as Joy and Neutral, but fail to take of scarce samples even when an ensemble random forest classifier is adopted. In order to prevent the unbalanced learning, we choose the weighted loss mechanism for both TextCNN and causal modeling TextCNN (C-TextCNN), these models suffer less than the traditional baselines and achieve a slightly balance performance, where there are around 15% and 7% improvement on Anger and Sadness, respectively. We following adopt the casual utterance modeling to original TextCNN, mapping previous utterance as well as target utterance into model. The causal utterance modeling improve the C-TextCNN over TextCNN for 6%, 2% and 1% on Anger, Joy and overall F1 score. Motivated from these preliminary experiments, the proposed FriendsBERT also adopt the ideas of both weighted loss and causal utterance modeling. As compared to the original BERT, single sentence BERT (FriendsBERT-base-s), the proposed FriendsBERT-base improve 1% for Joy and overall F1, and 2% for Sadness. For the final validation performance, our proposed approach achieves the highest scores, which are $0.85$ and $0.86$ for FriendsBERT-base and FriendsBERT-large, respectively. Overall, the proposed FriendsBERT successfully captures the sentence-level context-awarded information and outperforms all the baselines, which not only achieves high performance on large sample labels, but also on small sample labels. The similar settings are also adapted to EmotionPush dataset for the final evaluation. Experiments ::: Evaluation Results The testing dataset consists of 240 dialogues including $3,296$ and $3,536$ utterances in Friends and EmotionPush respectively. We re-train our FriendsBERT and ChatBERT with top 920 training dialogues and predict the evaluation results using the model performing the best validation results. The results are shown in Table TABREF29 and Table TABREF30. The present method achieves $81.5\%$ and $88.5\%$ micro F1-score on the testing dataset of Friends and EmotionPush, respectively. Conclusion and Future work In the present work, we propose FriendsBERT and ChatBERT for the multi-utterance emotion recognition task on EmotionLines dataset. The proposed models are adapted from BERT BIBREF5 with three main improvement during the model training procedure, which are the causal utterance modeling mechanism, specific model pre-training, and adapt weighted loss. The causal utterance modeling takes the advantages of the sentence-level context information during model inference. The specific model pre-training helps to against the bias in different text domain. The weighted loss avoids our model to only predict on large size sample. The effectiveness and generalizability of the proposed methods are demonstrated from the experiments. In future work, we consider to include the conditional probabilistic constraint $P ({\rm Emo}_{B} | \hat{\rm Emo}_{A})$. Model should predict the emotion based on a certain understanding about context emotions. This might be more reasonable for guiding model than just predicting emotion of ${\rm Sentence}_B$ directly. In addition, due to the limitation of BERT input format, ambiguous number of input sentences is now becoming an important design requirement for our future work. Also, personality embedding development will be another future work of the emotion recognition. The personality embedding will be considered as sentence embedding injected into word embedding, and it seems this additional information can contribute some improvement potentially.
what datasets were used?
78a4ec72d76f0a736a4a01369a42b092922203b6
what Ġdatasets Ġwere Ġused ?
Friends , ĠEm otion Push
<s> what Ġdatasets Ġwere Ġused ? </s> Introduction </s> Em otion Ġdetection Ġhas Ġlong Ġbeen Ġa Ġtopic Ġof Ġinterest Ġto Ġscholars Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġdomain . ĠResearchers Ġaim Ġto Ġrecognize Ġthe Ġemotion Ġbehind Ġthe Ġtext Ġand Ġdistribute Ġsimilar Ġones Ġinto Ġthe Ġsame Ġgroup . ĠEst ablish ing Ġan Ġemotion Ġclass ifier Ġcan Ġnot Ġonly Ġunderstand Ġeach Ġuser 's Ġfeeling Ġbut Ġalso Ġbe Ġextended Ġto Ġvarious Ġapplication , Ġfor Ġexample , Ġthe Ġmotivation Ġbehind Ġa Ġuser 's Ġinterests ĠB IB REF 0 . ĠBased Ġon Ġreleasing Ġof Ġlarge Ġtext Ġcorpus Ġon Ġsocial Ġmedia Ġand Ġthe Ġemotion Ġcategories Ġproposed Ġby ĠB IB REF 1 , ĠB IB REF 2 , Ġnumerous Ġmodels Ġhave Ġprovided Ġand Ġachieved Ġfabulous Ġprecision Ġso Ġfar . ĠFor Ġexample , ĠDeep Mo ji ĠB IB REF 3 Ġwhich Ġutilized Ġtransfer Ġlearning Ġconcept Ġto Ġenhance Ġemotions Ġand Ġsarc asm Ġunderstanding Ġbehind Ġthe Ġtarget Ġsentence . ĠCAR ER ĠB IB REF 4 Ġlearned Ġcontextual ized Ġaffect Ġrepresentations Ġto Ġmake Ġitself Ġmore Ġsensitive Ġto Ġrare Ġwords Ġand Ġthe Ġscenario Ġbehind Ġthe Ġtexts . </s> As Ġmethods Ġbecome Ġmature , Ġtext - based Ġemotion Ġdetecting Ġapplications Ġcan Ġbe Ġextended Ġfrom Ġa Ġsingle Ġutter ance Ġto Ġa Ġdialogue Ġcontributed Ġby Ġa Ġseries Ġof Ġutter ances . ĠTable ĠT AB REF 2 Ġillustrates Ġthe Ġdifference Ġbetween Ġsingle Ġutter ance Ġand Ġdialogue Ġemotion Ġrecognition . ĠThe Ġsame Ġutter ances Ġin ĠTable ĠT AB REF 2 , Ġeven Ġthe Ġsame Ġperson Ġsaid Ġthe Ġsame Ġsentence , Ġthe Ġemotion Ġit Ġconvey Ġmay Ġbe Ġvarious , Ġwhich Ġmay Ġdepend Ġon Ġdifferent Ġbackground Ġof Ġthe Ġconversation , Ġtone Ġof Ġspeaking Ġor Ġpersonality . ĠTherefore , Ġfor Ġemotion Ġdetection , Ġthe Ġinformation Ġfrom Ġpreceding Ġutter ances Ġin Ġa Ġconversation Ġis Ġrelatively Ġcritical . </s> In ĠSocial N LP Ġ2019 ĠEm otion X , Ġthe Ġchallenge Ġis Ġto Ġrecognize Ġemotions Ġfor Ġall Ġutter ances Ġin ĠEm otion L ines Ġdataset , Ġa Ġdataset Ġconsists Ġof Ġdialog ues . ĠAccording Ġto Ġthe Ġneeds Ġfor Ġconsidering Ġcontext Ġat Ġthe Ġsame Ġtime , Ġwe Ġdevelop Ġtwo Ġclassification Ġmodels , Ġinspired Ġby Ġbid irection al Ġenc oder Ġrepresentations Ġfrom Ġtransform ers Ġ( BER T ) ĠB IB REF 5 , ĠFriends BER T Ġand ĠChat BER T . ĠIn Ġthis Ġpaper , Ġwe Ġintroduce Ġour Ġapproaches Ġincluding Ġcausal Ġutter ance Ġmodeling , Ġmodel Ġpre - training , Ġand Ġfine - turn ing . </s> Dat as et </s> Em otion L ines ĠB IB REF 6 Ġis Ġa Ġdialogue Ġdataset Ġcomposed Ġof Ġtwo Ġsubs ets , ĠFriends Ġand ĠEm otion Push , Ġaccording Ġto Ġthe Ġsource Ġof Ġthe Ġdialog ues . ĠThe Ġformer Ġcomes Ġfrom Ġthe Ġscripts Ġof Ġthe ĠFriends ĠTV Ġsitcom . ĠThe Ġother Ġis Ġmade Ġup Ġof ĠFacebook Ġmessenger Ġchats . ĠEach Ġsubset Ġincludes Ġ$ 1 , 000 $ ĠEnglish Ġdialog ues , Ġand Ġeach Ġdialogue Ġcan Ġbe Ġfurther Ġdivided Ġinto Ġa Ġfew Ġconsecutive Ġutter ances . ĠAll Ġthe Ġutter ances Ġare Ġannot ated Ġby Ġfive Ġannot ators Ġon Ġa Ġcrowd - s ourcing Ġplatform Ġ( Amazon ĠMechanical ĠTurk ), Ġand Ġthe Ġlabeling Ġwork Ġis Ġonly Ġbased Ġon Ġthe Ġtextual Ġcontent . ĠAn not ator Ġvotes Ġfor Ġone Ġof Ġthe Ġseven Ġemotions , Ġnamely ĠEk man âĢ Ļ s Ġsix Ġbasic Ġemotions ĠB IB REF 1 , Ġplus Ġthe Ġneutral . ĠIf Ġnone Ġof Ġthe Ġemotion Ġgets Ġmore Ġthan Ġthree Ġvotes , Ġthe Ġutter ance Ġwill Ġbe Ġmarked Ġas ĠâĢ ľ non - neutral âĢ Ŀ . </s> For Ġthe Ġdatasets , Ġthere Ġare Ġproperties Ġworth Ġadditional Ġmentioning . ĠAlthough ĠFriends Ġand ĠEm otion Push Ġshare Ġthe Ġsame Ġdata Ġformat , Ġthey Ġare Ġquite Ġdifferent Ġin Ġnature . ĠFriends Ġis Ġa Ġspeech - based Ġdataset Ġwhich Ġis Ġannot ated Ġdialog ues Ġfrom Ġthe ĠTV Ġsitcom . ĠIt Ġmeans Ġmost Ġof Ġthe Ġutter ances Ġare Ġgenerated Ġby Ġthe Ġa Ġfew Ġmain Ġcharacters . ĠThe Ġpersonality Ġof Ġa Ġcharacter Ġoften Ġaffects Ġthe Ġway Ġof Ġspeaking , Ġand Ġtherefore ĠâĢ ľ who Ġis Ġthe Ġspeaker " Ġmight Ġprovide Ġextra Ġclues Ġfor Ġemotion Ġprediction . ĠIn Ġcontrast , ĠEm otion Push Ġdoes Ġnot Ġhave Ġthis Ġtrait Ġdue Ġto Ġthe Ġanonymous Ġmechanism . ĠIn Ġaddition , Ġfeatures Ġsuch Ġas Ġtypo , Ġhyper link , Ġand Ġemoji Ġthat Ġonly Ġappear Ġin Ġchat - based Ġdata Ġwill Ġneed Ġsome Ġdomain - specific Ġtechniques Ġto Ġprocess . </s> Inc identally , Ġthe Ġobjective Ġof Ġthe Ġchallenge Ġis Ġto Ġpredict Ġthe Ġemotion Ġfor Ġeach Ġutter ance . ĠJust , Ġaccording Ġto ĠEm otion X Ġ2019 Ġspecification , Ġthere Ġare Ġonly Ġfour Ġemotions Ġbe Ġselected Ġas Ġour Ġlabel Ġcandidates , Ġwhich Ġare ĠJoy , ĠSad ness , ĠAnger , Ġand ĠNeutral . ĠThese Ġemotions Ġwill Ġbe Ġconsidered Ġduring Ġperformance Ġevaluation . ĠThe Ġtechnical Ġdetail Ġwill Ġalso Ġbe Ġintroduced Ġand Ġdiscussed Ġin Ġfollowing ĠSection ĠSEC REF 13 Ġand ĠSection ĠSEC REF 26 . </s> Model ĠDescription </s> For Ġthis Ġchallenge , Ġwe Ġadapt ĠB ERT Ġwhich Ġis Ġproposed Ġby ĠB IB REF 5 Ġto Ġhelp Ġunderstand Ġthe Ġcontext Ġat Ġthe Ġsame Ġtime . ĠTechn ically , ĠB ERT , Ġdesigned Ġon Ġend - to - end Ġarchitecture , Ġis Ġa Ġdeep Ġpre - trained Ġtransformer Ġenc oder Ġthat Ġdynamically Ġprovides Ġlanguage Ġrepresentation Ġand ĠB ERT Ġalready Ġachieved Ġmultiple Ġstate - of - the - art Ġresults Ġon ĠGL UE Ġbenchmark ĠB IB REF 7 Ġand Ġmany Ġtasks . ĠA Ġquick Ġrecap Ġfor ĠB ERT 's Ġarchitecture Ġand Ġits Ġpre - training Ġtasks Ġwill Ġbe Ġillustrated Ġin Ġthe Ġfollowing Ġsubsections . </s> Model ĠDescription Ġ: :: ĠModel ĠArchitecture </s> BER T , Ġthe ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers , Ġconsists Ġof Ġseveral Ġtransformer Ġenc oder Ġlayers Ġthat Ġenable Ġthe Ġmodel Ġto Ġextract Ġvery Ġdeep Ġlanguage Ġfeatures Ġon Ġboth Ġtoken - level Ġand Ġsentence - level . ĠEach Ġtransformer Ġenc oder Ġcontains Ġmulti - head Ġself - att ention Ġlayers Ġthat Ġprovide Ġability Ġto Ġlearn Ġmultiple Ġattention Ġfeature Ġof Ġeach Ġword Ġfrom Ġtheir Ġbid irection al Ġcontext . ĠThe Ġtransformer Ġand Ġits Ġself - att ention Ġmechanism Ġare Ġproposed Ġby ĠB IB REF 8 . ĠThis Ġself - att ention Ġmechanism Ġcan Ġbe Ġinterpreted Ġas Ġa Ġkey - value Ġmapping Ġgiven Ġquery . ĠBy Ġgiven Ġthe Ġembed ding Ġvector Ġfor Ġtoken Ġinput , Ġthe Ġquery Ġ($ Q $ ), Ġkey Ġ($ K $ ) Ġand Ġvalue Ġ($ V $ ) Ġare Ġproduced Ġby Ġthe Ġprojection Ġfrom Ġeach Ġthree Ġparameter Ġmat rices Ġwhere Ġ$ W ^ Q Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ {\ rm Ġmodel }} Ġ\ times Ġd _{ k } }, ĠW ^ K Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel } Ġ\ times Ġd _{ k }} $ Ġand Ġ$ W ^ V Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel } Ġ\ times Ġd _{ v }} $. ĠThe Ġself - att ention ĠB IB REF 8 Ġis Ġformally Ġrepresented Ġas : </s> The Ġ$ Ġd _ k Ġ= Ġd _ v Ġ= Ġd _{ \ rm Ġmodel } Ġ= Ġ1024 $ Ġin ĠB ERT Ġlarge Ġversion Ġand Ġ768 Ġin ĠB ERT Ġbase Ġversion . ĠOnce Ġmodel Ġcan Ġextract Ġattention Ġfeature , Ġwe Ġcan Ġextend Ġone Ġself - att ention Ġinto Ġmulti - head Ġself - att ention , Ġthis Ġextension Ġmakes Ġsub - space Ġfeatures Ġcan Ġbe Ġextracted Ġin Ġsame Ġtime Ġby Ġthis Ġmulti - head Ġconfiguration . ĠOverall , Ġthe Ġmulti - att ention Ġmechanism Ġis Ġadopt Ġfor Ġeach Ġtransformer Ġenc oder , Ġand Ġseveral Ġof Ġenc oder Ġlayer Ġwill Ġbe Ġstacked Ġtogether Ġto Ġform Ġa Ġdeep Ġtransformer Ġenc oder . </s> For Ġthe Ġmodel Ġinput , ĠB ERT Ġallow Ġus Ġtake Ġone Ġsentence Ġas Ġinput Ġsequence Ġor Ġtwo Ġsentences Ġtogether Ġas Ġone Ġinput Ġsequence , Ġand Ġthe Ġmaximum Ġlength Ġof Ġinput Ġsequence Ġis Ġ512 . ĠThe Ġway Ġthat ĠB ERT Ġwas Ġdesigned Ġis Ġfor Ġgiving Ġmodel Ġthe Ġsentence - level Ġand Ġtoken - level Ġunderstanding . ĠIn Ġtwo Ġsentences Ġcase , Ġa Ġspecial Ġtoken Ġ([ SE P ]) Ġwill Ġbe Ġinserted Ġbetween Ġtwo Ġsentences . ĠIn Ġaddition , Ġthe Ġfirst Ġinput Ġtoken Ġis Ġalso Ġa Ġspecial Ġtoken Ġ([ CL S ]), Ġand Ġits Ġcorresponding Ġo up ut Ġwill Ġbe Ġvector Ġplace Ġfor Ġclassification Ġduring Ġfine - tun ing . ĠThe Ġoutputs Ġof Ġthe Ġlast Ġenc oder Ġlayer Ġcorresponding Ġto Ġeach Ġinput Ġtoken Ġcan Ġbe Ġtreated Ġas Ġword Ġrepresentations Ġfor Ġeach Ġtoken , Ġand Ġthe Ġword Ġrepresentation Ġof Ġthe Ġfirst Ġtoken Ġ([ CL S ]) Ġwill Ġbe Ġconsider Ġas Ġclassification Ġ( output ) Ġrepresentation Ġfor Ġfurther Ġfine - tun ing Ġtasks . ĠIn ĠB ERT , Ġthis Ġvector Ġis Ġden oted Ġas Ġ$ C Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _{ \ rm Ġmodel }} Ġ$ , Ġand Ġa Ġclassification Ġlayer Ġis Ġden oted Ġas Ġ$ ĠW Ġ\ in Ġ\ math bb Ġ{ R } ^{ K Ġ\ times Ġd _{ \ rm Ġmodel }} $, Ġwhere Ġ$ K $ Ġis Ġnumber Ġof Ġclassification Ġlabels . ĠFinally , Ġthe Ġprediction Ġ$ P $ Ġof ĠB ERT Ġis Ġrepresented Ġas Ġ$ P Ġ= Ġ{\ rm Ġsoft max } ( CW ^ T ) $. </s> Model ĠDescription Ġ: :: ĠPre - training ĠT asks </s> In Ġpre - training , Ġint ead Ġof Ġusing Ġun id irection al Ġlanguage Ġmodels , ĠB ERT Ġdeveloped Ġtwo Ġpre - training Ġtasks : Ġ( 1 ) ĠMask ed ĠLM Ġ( cl o ze Ġtest ) Ġand Ġ( 2 ) ĠNext ĠSent ence ĠPrediction . ĠAt Ġthe Ġfirst Ġpre - training Ġtask , Ġbid irection al Ġlanguage Ġmodeling Ġcan Ġbe Ġdone Ġat Ġthis Ġclo ze - like Ġpre - training . ĠIn Ġdetail , Ġ15 % Ġtokens Ġof Ġinput Ġsequence Ġwill Ġbe Ġmasked Ġat Ġrandom Ġand Ġmodel Ġneed Ġto Ġpredict Ġthose Ġmasked Ġtokens . ĠThe Ġenc oder Ġwill Ġtry Ġto Ġlearn Ġcontextual Ġrepresentations Ġfrom Ġevery Ġgiven Ġtokens Ġdue Ġto Ġmask ing Ġtokens Ġat Ġrandom . ĠModel Ġwill Ġnot Ġknow Ġwhich Ġpart Ġof Ġthe Ġinput Ġis Ġgoing Ġto Ġbe Ġmasked , Ġso Ġthat Ġthe Ġinformation Ġof Ġeach Ġmasked Ġtokens Ġshould Ġbe Ġinferred Ġby Ġremaining Ġtokens . ĠAt ĠNext ĠSent ence ĠPrediction , Ġtwo Ġsentences Ġconc aten ated Ġtogether Ġwill Ġbe Ġconsidered Ġas Ġmodel Ġinput . ĠIn Ġorder Ġto Ġgive Ġmodel Ġa Ġgood Ġnature Ġlanguage Ġunderstanding , Ġknowing Ġrelationship Ġbetween Ġsentence Ġis Ġone Ġof Ġimportant Ġabilities . ĠWhen Ġgenerating Ġinput Ġsequences , Ġ50 % Ġof Ġtime Ġthe Ġsentence ĠB Ġis Ġactually Ġfollowed Ġby Ġsentence ĠA , Ġand Ġrest Ġ50 % Ġof Ġthe Ġtime Ġthe Ġsentence ĠB Ġwill Ġbe Ġpicked Ġrandomly Ġfrom Ġdataset , Ġand Ġmodel Ġneed Ġto Ġpredict Ġif Ġthe Ġsentence ĠB Ġis Ġnext Ġsentence Ġof Ġsentence ĠA . ĠThat Ġis , Ġthe Ġattention Ġinformation Ġwill Ġbe Ġshared Ġbetween Ġsentences . ĠSuch Ġsentence - level Ġunderstanding Ġmay Ġhave Ġdifficulties Ġto Ġbe Ġlearned Ġat Ġfirst Ġpre - training Ġtask Ġ( Mask ed ĠLM ), Ġtherefore , Ġthe Ġpre - training Ġtask Ġ( N SP ) Ġis Ġdeveloped Ġas Ġsecond Ġtraining Ġgoal Ġto Ġcapture Ġthe Ġcross Ġsentence Ġrelationship . </s> In Ġthis Ġcompetition , Ġlimited Ġby Ġthe Ġsize Ġof Ġdataset Ġand Ġthe Ġchallenge Ġin Ġcontextual Ġemotion Ġrecognition , Ġwe Ġconsider ĠB ERT Ġwith Ġboth Ġtwo Ġpre - training Ġtasks Ġcan Ġgive Ġa Ġgood Ġstarting Ġpoint Ġto Ġextract Ġemotion Ġchanging Ġduring Ġdialogue - like Ġconversation . ĠEspecially Ġthe Ġsecond Ġpre - training Ġtask , Ġit Ġmight Ġbe Ġmore Ġimportant Ġfor Ġdialogue - like Ġconversation Ġwhere Ġthe Ġemotion Ġmay Ġvarious Ġby Ġthe Ġcontext Ġof Ġcontinuous Ġutter ances . ĠThat Ġis , Ġgiven Ġa Ġset Ġof Ġcontinues Ġconversations , Ġthe Ġemotion Ġof Ġcurrent Ġutter ance Ġmight Ġbe Ġinfluenced Ġby Ġprevious Ġutter ance . ĠBy Ġthis Ġassumption Ġand Ġwith Ġsupporting Ġfrom Ġthe Ġexperiment Ġresults Ġof ĠB ERT , Ġwe Ġcan Ġtake Ġsentence ĠA Ġas Ġone - sent ence Ġcontext Ġand Ġconsider Ġsentence ĠB Ġas Ġthe Ġtarget Ġsentence Ġfor Ġemotion Ġprediction . ĠThe Ġdetail Ġwill Ġbe Ġdescribed Ġin ĠSection ĠSEC REF 4 . </s> Method ology </s> The Ġmain Ġgoal Ġof Ġthe Ġpresent Ġwork Ġis Ġto Ġpredict Ġthe Ġemotion Ġof Ġutter ance Ġwithin Ġthe Ġdialogue . ĠFollowing Ġare Ġfour Ġmajor Ġdifficulties Ġwe Ġconcern Ġabout : </s> The Ġemotion Ġof Ġthe Ġutter ances Ġdepends Ġnot Ġonly Ġon Ġthe Ġtext Ġbut Ġalso Ġon Ġthe Ġinteraction Ġhappened Ġearlier . </s> The Ġsource Ġof Ġthe Ġtwo Ġdatasets Ġare Ġdifferent . ĠFriends Ġis Ġspeech - based Ġdialog ues Ġand ĠEm otion Push Ġis Ġchat - based Ġdialog ues . ĠIt Ġmakes Ġdatasets Ġpossess Ġdifferent Ġcharacteristics . </s> There Ġare Ġonly Ġ$ 1 , 000 $ Ġdialog ues Ġin Ġboth Ġtraining Ġdatasets Ġwhich Ġare Ġnot Ġlarge Ġenough Ġfor Ġthe Ġstability Ġof Ġtraining Ġa Ġcomplex Ġneural - based Ġmodel . </s> The Ġprediction Ġtargets Ġ( em otion Ġlabels ) Ġare Ġhighly Ġun balanced . </s> The Ġproposed Ġapproach Ġis Ġsummarized Ġin ĠFigure ĠFIG REF 3 , Ġwhich Ġaims Ġto Ġovercome Ġthese Ġchallenges . ĠThe Ġframework Ġcould Ġbe Ġseparated Ġinto Ġthree Ġsteps Ġand Ġdescribed Ġas Ġfollow : </s> Method ology Ġ: :: ĠCa usal ĠUt ter ance ĠModel ing </s> Given Ġa Ġdialogue Ġ$ D ^{ ( i )} $ Ġwhich Ġincludes Ġsequence Ġof Ġutter ances Ġden oted Ġas Ġ$ D ^{ ( i )} =( u ^{ ( i )} _{ 1 }, Ġu ^{ ( i )} _{ 2 }, Ġ... , Ġu ^{ ( i )} _{ n }) $, Ġwhere Ġ$ i $ Ġis Ġthe Ġindex Ġin Ġdataset Ġand Ġ$ n $ Ġis Ġthe Ġnumber Ġof Ġutter ances Ġin Ġthe Ġgiven Ġdialogue . ĠIn Ġorder Ġto Ġconserve Ġthe Ġemotional Ġinformation Ġof Ġboth Ġutter ance Ġand Ġconversation , Ġwe Ġrearr ange Ġeach Ġtwo Ġconsecutive Ġutter ances Ġ$ u _{ t }, Ġu _{ t - 1 } $ Ġinto Ġa Ġsingle Ġsentence Ġrepresentation Ġ$ x _{ t } $ Ġas </s> The Ġcorresponding Ġsentence Ġrepresentation Ġcorpus Ġ$ X ^{ ( i )} $ Ġare Ġden oted Ġas Ġ$ X ^{ ( i )} =( x ^{ ( i )} _{ 1 }, Ġx ^{ ( i )} _{ 2 }, Ġ... , Ġx ^{ ( i )} _{ n }) $. ĠNote Ġthat Ġthe Ġfirst Ġutter ance Ġwithin Ġa Ġconversation Ġdoes Ġnot Ġhave Ġits Ġcausal Ġutter ance Ġ( pre vious Ġsentence ), Ġtherefore , Ġthe Ġcausal Ġutter ance Ġwill Ġbe Ġset Ġas Ġ[ None ]. ĠA Ġpractical Ġexample Ġof Ġsentence Ġrepresentation Ġis Ġshown Ġin ĠTable ĠT AB REF 11 . </s> Since Ġthe Ġcharacteristics Ġof Ġtwo Ġdatasets Ġare Ġnot Ġidentical , Ġwe Ġcustomize Ġdifferent Ġcausal Ġutter ance Ġmodeling Ġstrategies Ġto Ġrefine Ġthe Ġinformation Ġin Ġtext . </s> For ĠFriends , Ġthere Ġare Ġtwo Ġspecific Ġproperties . ĠThe Ġfirst Ġone Ġis Ġthat Ġmost Ġdialog ues Ġare Ġsurrounding Ġwith Ġthe Ġsix Ġmain Ġcharacters , Ġincluding ĠRachel , ĠMonica , ĠPh oe be , ĠJoey , ĠChandler , Ġand ĠRoss . ĠThe Ġutter ance Ġratio Ġof Ġgiven Ġby Ġthe Ġsix Ġroles Ġis Ġup Ġto Ġ$ 83 . 4 \ % $. ĠSecond , Ġthe Ġpersonal Ġcharacteristics Ġof Ġthe Ġsix Ġcharacters Ġare Ġvery Ġclear . ĠEach Ġleading Ġrole Ġhas Ġits Ġown Ġemotion Ġund ulated Ġrule . ĠTo Ġmake Ġuse Ġof Ġthese Ġfeatures , Ġwe Ġintroduce Ġthe Ġpersonality Ġtoken ization Ġwhich Ġhelp Ġlearning Ġthe Ġpersonality Ġof Ġthe Ġsix Ġcharacters . ĠPersonality Ġtoken ization Ġconc aten ate Ġthe Ġspeaker Ġand Ġsays Ġtokens Ġbefore Ġthe Ġinput Ġutter ance Ġif Ġthe Ġspeaker Ġis Ġone Ġof Ġthe Ġsix Ġcharacters . ĠThe Ġexample Ġis Ġshown Ġin ĠTable ĠT AB REF 12 . </s> For ĠEm otion Push , Ġthe Ġtext Ġare Ġinformal Ġchats Ġwhich Ġincluding Ġlike Ġslang , Ġacronym , Ġtypo , Ġhyper link , Ġand Ġemoji . ĠAnother Ġcharacteristic Ġis Ġthat Ġthe Ġspecific Ġname Ġentities Ġare Ġtoken ized Ġwith Ġrandom Ġindex . Ġ( e . g . ĠâĢ ľ organ ization _ 80 âĢ Ŀ , ĠâĢ ľ person _ 01 âĢ Ŀ , Ġand ĠâĢ ľ time _ 12 âĢ Ŀ ). ĠWe Ġconsider Ġsome Ġof Ġthese Ġinformal Ġtext Ġare Ġrelated Ġto Ġexpressing Ġemotion Ġsuch Ġas Ġrepeated Ġtyping , Ġpur posed Ġcapital ization , Ġand Ġemoji Ġ( e . g . ĠâĢ ľ : D âĢ Ŀ , ĠâĢ ľ :( âĢ Ŀ , Ġand ĠâĢ ľ < 3 âĢ Ŀ )). ĠTherefore , Ġwe Ġkeep Ġmost Ġinformal Ġexpressions Ġbut Ġonly Ġprocess Ġhyper links , Ġempty Ġutter ance , Ġand Ġname Ġentities Ġby Ġun ifying Ġthe Ġtokens . </s> Method ology Ġ: :: ĠModel ĠPre - training </s> Since Ġthe Ġsize Ġof Ġboth Ġdatasets Ġare Ġnot Ġlarge Ġenough Ġfor Ġcomplex Ġneural - based Ġmodel Ġtraining Ġas Ġwell Ġas ĠB ERT Ġmodel Ġis Ġonly Ġpre - train Ġon Ġformal Ġtext Ġdatasets , Ġthe Ġissues Ġof Ġover fitting Ġand Ġdomain Ġbias Ġare Ġimportant Ġconsiderations Ġfor Ġdesign Ġthe Ġpre - training Ġprocess . </s> To Ġavoid Ġour Ġmodel Ġover fitting Ġon Ġthe Ġtraining Ġdata Ġand Ġincrease Ġthe Ġunderstanding Ġof Ġinformal Ġtext , Ġwe Ġadapted ĠB ERT Ġand Ġderived Ġtwo Ġmodels , Ġnamely ĠFriends BER T Ġand ĠChat BER T , Ġwith Ġdifferent Ġpre - training Ġtasks Ġbefore Ġthe Ġformal Ġtraining Ġprocess Ġfor ĠFriends Ġand ĠEm otion Push Ġdataset , Ġrespectively . ĠThe Ġpre - training Ġstrategies Ġare Ġdescribed Ġbelow . </s> For Ġpre - training ĠFriends BER T , Ġwe Ġcollect Ġthe Ġcompleted Ġscripts Ġof Ġall Ġten Ġseasons Ġof ĠFriends ĠTV Ġshows Ġfrom Ġem ory nl p Ġwhich Ġincludes Ġ3 , 107 Ġscenes Ġwithin Ġ61 , 309 Ġutter ances . ĠAll Ġthe Ġutter ances Ġare Ġfollowed Ġthe Ġpre processing Ġmethods Ġmentions Ġabove Ġto Ġcompose Ġthe Ġcorpus Ġfor ĠMask ed Ġlanguage Ġmodel Ġpre - training Ġtask . ĠThe Ġconsequ ent Ġutter ances Ġin Ġthe Ġsame Ġscenes Ġare Ġconsidered Ġas Ġthe Ġconsequ ent Ġsentences Ġto Ġpre - train Ġthe ĠNext ĠSent ence ĠPrediction Ġtask . ĠIn Ġthe Ġpre - training Ġprocess , Ġthe Ġtraining Ġloss Ġis Ġthe Ġsum Ġof Ġthe Ġmean Ġlikelihood Ġof Ġtwo Ġpre - train Ġtasks . </s> For Ġpre - training ĠChat BER T , Ġwe Ġpre - train Ġour Ġmodel Ġon Ġthe ĠTwitter Ġdataset , Ġsince Ġthe Ġtext Ġand Ġwriting Ġstyle Ġon ĠTwitter Ġare Ġclose Ġto Ġthe Ġchat Ġtext Ġwhere Ġboth Ġmay Ġinvolved Ġwith Ġmany Ġinformal Ġwords Ġor Ġemot icons Ġas Ġwell . ĠThe ĠTwitter Ġemotion Ġdataset , Ġ8 Ġbasic Ġemotions Ġfrom Ġemotion Ġwheel ĠB IB REF 1 , Ġwas Ġcollected Ġby Ġtwitter Ġstreaming ĠAPI Ġwith Ġspecific Ġemotion - related Ġhasht ags , Ġsuch Ġas Ġ# anger , Ġ# joy , Ġ# cry , Ġ# s ad Ġand Ġetc . ĠThe Ġhasht ags Ġin Ġtweets Ġare Ġtreated Ġas Ġemotion Ġlabel Ġfor Ġmodel Ġfine - tun ing . ĠThe Ġtweets Ġwere Ġfine - gr ined Ġprocessing Ġfollowed Ġthe Ġrules Ġin ĠB IB REF 9 , ĠB IB REF 4 , Ġincluding Ġduplicate Ġtweets Ġremoving , Ġthe Ġemotion Ġhasht ags Ġmust Ġappearing Ġin Ġthe Ġlast Ġposition Ġof Ġa Ġtweet , Ġand Ġetc . ĠThe Ġstat is Ġof Ġtweets Ġwere Ġsummarized Ġin ĠTable ĠT AB REF 17 . ĠEach Ġtweet Ġand Ġcorresponding Ġemotion Ġlabel Ġcomp oses Ġan Ġemotion Ġclassification Ġdataset Ġfor Ġpre - training . </s> Method ology Ġ: :: ĠFine - tun ing </s> Since Ġour Ġemotion Ġrecognition Ġtask Ġis Ġtreated Ġas Ġa Ġsequence - level Ġclassification Ġtask , Ġthe Ġmodel Ġwould Ġbe Ġfine - tun ed Ġon Ġthe Ġprocessed Ġtraining Ġdata . ĠFollowing Ġthe ĠB ERT Ġconstruction , Ġwe Ġtake Ġthe Ġfirst Ġembed ding Ġvector Ġwhich Ġcorresponds Ġto Ġthe Ġspecial Ġtoken Ġ[ CL S ] Ġfrom Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe ĠTrans former Ġenc oder . ĠThis Ġvector Ġrepresents Ġthe Ġembed ding Ġvector Ġof Ġthe Ġcorresponding Ġconversation Ġutter ances Ġwhich Ġis Ġden oted Ġas Ġ$\ math bf Ġ{ C } Ġ\ in Ġ\ math bb Ġ{ R } ^{ H } $, Ġwhere Ġ$ H $ Ġis Ġthe Ġembed ding Ġsize . ĠA Ġdense Ġneural Ġlayer Ġis Ġtreated Ġas Ġa Ġclassification Ġlayer Ġwhich Ġconsists Ġof Ġparameters Ġ$\ math bf Ġ{ W } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K \ times ĠH } $ Ġand Ġ$\ math bf Ġ{ b } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K } $, Ġwhere Ġ$ K $ Ġis Ġthe Ġnumber Ġof Ġemotion Ġclass . ĠThe Ġemotion Ġprediction Ġprobabilities Ġ$\ math bf Ġ{ P } Ġ\ in Ġ\ math bb Ġ{ R } ^{ K } $ Ġare Ġcomputed Ġby Ġa Ġsoft max Ġactivation Ġfunction Ġas </s> All Ġthe Ġparameters Ġin ĠB ERT Ġand Ġthe Ġclassification Ġlayer Ġwould Ġbe Ġfine - turned Ġtogether Ġto Ġminimize Ġthe ĠNegative ĠLog ĠLike lihood Ġ( N LL ) Ġloss Ġfunction , Ġas ĠEqu ation Ġ( DIS PLAY _ FORM 22 ), Ġbased Ġon Ġthe Ġground Ġtruth Ġemotion Ġlabel Ġ$ c $. </s> In Ġorder Ġto Ġtackle Ġthe Ġproblem Ġof Ġhighly Ġun balanced Ġemotion Ġlabels , Ġwe Ġapply Ġweighted Ġbalanced Ġwarming Ġon ĠN LL Ġloss Ġfunction , Ġas ĠEqu ation Ġ( DIS PLAY _ FORM 23 ), Ġin Ġthe Ġfirst Ġepoch Ġof Ġfine - tun ing Ġprocedure . </s> where Ġ$\ math bf Ġ{ w } $ Ġare Ġthe Ġweights Ġof Ġcorresponding Ġemotion Ġlabel Ġ$ c $ Ġwhich Ġare Ġcomputed Ġand Ġnormal ize Ġby Ġthe Ġfrequency Ġas </s> By Ġadding Ġthe Ġweighted Ġbalanced Ġwarming Ġon ĠN LL Ġloss , Ġthe Ġmodel Ġcould Ġlearn Ġto Ġpredict Ġthe Ġminor Ġemotions Ġ( e . g . Ġanger Ġand Ġsadness ) Ġearlier Ġand Ġmake Ġthe Ġtraining Ġprocess Ġmore Ġstable . ĠSince Ġthe Ġmajor Ġevaluation Ġmetrics Ġmicro ĠF 1 - score Ġis Ġeffect Ġby Ġthe Ġnumber Ġof Ġeach Ġlabel , Ġwe Ġonly Ġapply Ġthe Ġweighted Ġbalanced Ġwarming Ġin Ġfirst Ġepoch Ġto Ġoptimize Ġthe Ġperformance . </s> Exper iments </s> Since Ġthe ĠEm otion X Ġchallenge Ġonly Ġprovided Ġthe Ġgold Ġlabels Ġin Ġtraining Ġdata , Ġwe Ġpick Ġthe Ġbest Ġperformance Ġmodel Ġ( weights ) Ġto Ġpredict Ġthe Ġtesting Ġdata . ĠIn Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġexperiment Ġand Ġevaluation Ġresults . </s> Exper iments Ġ: :: ĠExperimental ĠSetup </s> The ĠEm otion X Ġchallenge Ġconsists Ġof Ġ$ 1 , 000 $ Ġdialog ues Ġfor Ġboth ĠFriends Ġand ĠEm otion Push . ĠIn Ġall Ġof Ġour Ġexperiments , Ġeach Ġdataset Ġis Ġseparated Ġinto Ġtop Ġ800 Ġdialog ues Ġfor Ġtraining Ġand Ġlast Ġ200 Ġdialog ues Ġfor Ġvalidation . ĠSince Ġthe ĠEm otion X Ġchallenge Ġconsiders Ġonly Ġthe Ġfour Ġemotions Ġ( anger , Ġjoy , Ġneutral , Ġand Ġsadness ) Ġin Ġthe Ġevaluation Ġstage , Ġwe Ġignore Ġall Ġthe Ġdata Ġpoint Ġcorresponding Ġto Ġother Ġemotions Ġdirectly . ĠThe Ġdetails Ġof Ġemotions Ġdistribution Ġare Ġshown Ġin ĠTable ĠT AB REF 18 . </s> The Ġhyper param eters Ġand Ġtraining Ġsetup Ġof Ġour Ġmodels Ġ( Friends BER T Ġand ĠChat BER T ) Ġare Ġshown Ġin ĠTable ĠT AB REF 25 . ĠSome Ġcommon Ġand Ġeasily Ġimplemented Ġmethods Ġare Ġselected Ġas Ġthe Ġbas elines Ġembed ding Ġmethods Ġand Ġclassification Ġmodels . ĠThe Ġbaseline Ġembed ding Ġmethods Ġare Ġincluding Ġbag - of - words Ġ( B OW ), Ġterm Ġfrequency âĢĵ in verse Ġdocument Ġfrequency Ġ( TF ID F ), Ġand Ġneural - based Ġword Ġembed ding . ĠThe Ġclassification Ġmodels Ġare Ġincluding ĠLog istic ĠReg ression Ġ( LR ), ĠRandom ĠForest Ġ( RF ), ĠText CNN ĠB IB REF 10 Ġwith Ġinitial Ġword Ġembed ding Ġas ĠGl o Ve ĠB IB REF 11 , Ġand Ġour Ġproposed Ġmodel . ĠAll Ġthe Ġexperiment Ġresults Ġare Ġbased Ġon Ġthe Ġbest Ġperformances Ġof Ġvalidation Ġresults . </s> Exper iments Ġ: :: ĠPerformance </s> The Ġexperiment Ġresults Ġof Ġvalidation Ġon ĠFriends Ġare Ġshown Ġin ĠTable ĠT AB REF 19 . ĠThe Ġproposed Ġmodel Ġand Ġbas elines Ġare Ġevaluated Ġbased Ġon Ġthe ĠPrecision Ġ( P .), ĠRecall Ġ( R .), Ġand ĠF 1 - me asure Ġ( F 1 ). </s> For Ġthe Ġtraditional Ġbas elines , Ġnamely ĠB OW Ġand ĠTF ID F , Ġwe Ġobserve Ġthat Ġthey Ġachieve Ġsurprising Ġhigh ĠF 1 Ġscores Ġaround Ġ$ 0 . 81 $, Ġhowever , Ġthe Ġscores Ġfor ĠAnger Ġand ĠSad ness Ġare Ġlower . ĠThis Ġexplains Ġthat Ġtraditional Ġapproaches Ġtend Ġto Ġpredict Ġthe Ġlabels Ġwith Ġlarge Ġsample Ġsize , Ġsuch Ġas ĠJoy Ġand ĠNeutral , Ġbut Ġfail Ġto Ġtake Ġof Ġscarce Ġsamples Ġeven Ġwhen Ġan Ġensemble Ġrandom Ġforest Ġclass ifier Ġis Ġadopted . ĠIn Ġorder Ġto Ġprevent Ġthe Ġun balanced Ġlearning , Ġwe Ġchoose Ġthe Ġweighted Ġloss Ġmechanism Ġfor Ġboth ĠText CNN Ġand Ġcausal Ġmodeling ĠText CNN Ġ( C - Text CNN ), Ġthese Ġmodels Ġsuffer Ġless Ġthan Ġthe Ġtraditional Ġbas elines Ġand Ġachieve Ġa Ġslightly Ġbalance Ġperformance , Ġwhere Ġthere Ġare Ġaround Ġ15 % Ġand Ġ7 % Ġimprovement Ġon ĠAnger Ġand ĠSad ness , Ġrespectively . ĠWe Ġfollowing Ġadopt Ġthe Ġcasual Ġutter ance Ġmodeling Ġto Ġoriginal ĠText CNN , Ġmapping Ġprevious Ġutter ance Ġas Ġwell Ġas Ġtarget Ġutter ance Ġinto Ġmodel . ĠThe Ġcausal Ġutter ance Ġmodeling Ġimprove Ġthe ĠC - Text CNN Ġover ĠText CNN Ġfor Ġ6 %, Ġ2 % Ġand Ġ1 % Ġon ĠAnger , ĠJoy Ġand Ġoverall ĠF 1 Ġscore . ĠMot ivated Ġfrom Ġthese Ġpreliminary Ġexperiments , Ġthe Ġproposed ĠFriends BER T Ġalso Ġadopt Ġthe Ġideas Ġof Ġboth Ġweighted Ġloss Ġand Ġcausal Ġutter ance Ġmodeling . ĠAs Ġcompared Ġto Ġthe Ġoriginal ĠB ERT , Ġsingle Ġsentence ĠB ERT Ġ( Friends BER T - base - s ), Ġthe Ġproposed ĠFriends BER T - base Ġimprove Ġ1 % Ġfor ĠJoy Ġand Ġoverall ĠF 1 , Ġand Ġ2 % Ġfor ĠSad ness . ĠFor Ġthe Ġfinal Ġvalidation Ġperformance , Ġour Ġproposed Ġapproach Ġachieves Ġthe Ġhighest Ġscores , Ġwhich Ġare Ġ$ 0 . 85 $ Ġand Ġ$ 0 . 86 $ Ġfor ĠFriends BER T - base Ġand ĠFriends BER T - large , Ġrespectively . </s> Overall , Ġthe Ġproposed ĠFriends BER T Ġsuccessfully Ġcaptures Ġthe Ġsentence - level Ġcontext - aw arded Ġinformation Ġand Ġoutper forms Ġall Ġthe Ġbas elines , Ġwhich Ġnot Ġonly Ġachieves Ġhigh Ġperformance Ġon Ġlarge Ġsample Ġlabels , Ġbut Ġalso Ġon Ġsmall Ġsample Ġlabels . ĠThe Ġsimilar Ġsettings Ġare Ġalso Ġadapted Ġto ĠEm otion Push Ġdataset Ġfor Ġthe Ġfinal Ġevaluation . </s> Exper iments Ġ: :: ĠEvaluation ĠResults </s> The Ġtesting Ġdataset Ġconsists Ġof Ġ240 Ġdialog ues Ġincluding Ġ$ 3 , 296 $ Ġand Ġ$ 3 , 536 $ Ġutter ances Ġin ĠFriends Ġand ĠEm otion Push Ġrespectively . ĠWe Ġre - train Ġour ĠFriends BER T Ġand ĠChat BER T Ġwith Ġtop Ġ920 Ġtraining Ġdialog ues Ġand Ġpredict Ġthe Ġevaluation Ġresults Ġusing Ġthe Ġmodel Ġperforming Ġthe Ġbest Ġvalidation Ġresults . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 29 Ġand ĠTable ĠT AB REF 30 . ĠThe Ġpresent Ġmethod Ġachieves Ġ$ 81 . 5 \ % $ Ġand Ġ$ 88 . 5 \ % $ Ġmicro ĠF 1 - score Ġon Ġthe Ġtesting Ġdataset Ġof ĠFriends Ġand ĠEm otion Push , Ġrespectively . </s> Conclusion Ġand ĠFuture Ġwork </s> In Ġthe Ġpresent Ġwork , Ġwe Ġpropose ĠFriends BER T Ġand ĠChat BER T Ġfor Ġthe Ġmulti - utter ance Ġemotion Ġrecognition Ġtask Ġon ĠEm otion L ines Ġdataset . ĠThe Ġproposed Ġmodels Ġare Ġadapted Ġfrom ĠB ERT ĠB IB REF 5 Ġwith Ġthree Ġmain Ġimprovement Ġduring Ġthe Ġmodel Ġtraining Ġprocedure , Ġwhich Ġare Ġthe Ġcausal Ġutter ance Ġmodeling Ġmechanism , Ġspecific Ġmodel Ġpre - training , Ġand Ġadapt Ġweighted Ġloss . ĠThe Ġcausal Ġutter ance Ġmodeling Ġtakes Ġthe Ġadvantages Ġof Ġthe Ġsentence - level Ġcontext Ġinformation Ġduring Ġmodel Ġinference . ĠThe Ġspecific Ġmodel Ġpre - training Ġhelps Ġto Ġagainst Ġthe Ġbias Ġin Ġdifferent Ġtext Ġdomain . ĠThe Ġweighted Ġloss Ġavoids Ġour Ġmodel Ġto Ġonly Ġpredict Ġon Ġlarge Ġsize Ġsample . ĠThe Ġeffectiveness Ġand Ġgeneral iz ability Ġof Ġthe Ġproposed Ġmethods Ġare Ġdemonstrated Ġfrom Ġthe Ġexperiments . </s> In Ġfuture Ġwork , Ġwe Ġconsider Ġto Ġinclude Ġthe Ġconditional Ġprob abil istic Ġconstraint Ġ$ P Ġ( {\ rm ĠEm o } _{ B } Ġ| Ġ\ hat {\ rm ĠEm o } _{ A }) $. ĠModel Ġshould Ġpredict Ġthe Ġemotion Ġbased Ġon Ġa Ġcertain Ġunderstanding Ġabout Ġcontext Ġemotions . ĠThis Ġmight Ġbe Ġmore Ġreasonable Ġfor Ġguiding Ġmodel Ġthan Ġjust Ġpredicting Ġemotion Ġof Ġ${ \ rm ĠSent ence } _ B $ Ġdirectly . ĠIn Ġaddition , Ġdue Ġto Ġthe Ġlimitation Ġof ĠB ERT Ġinput Ġformat , Ġambiguous Ġnumber Ġof Ġinput Ġsentences Ġis Ġnow Ġbecoming Ġan Ġimportant Ġdesign Ġrequirement Ġfor Ġour Ġfuture Ġwork . ĠAlso , Ġpersonality Ġembed ding Ġdevelopment Ġwill Ġbe Ġanother Ġfuture Ġwork Ġof Ġthe Ġemotion Ġrecognition . ĠThe Ġpersonality Ġembed ding Ġwill Ġbe Ġconsidered Ġas Ġsentence Ġembed ding Ġinjected Ġinto Ġword Ġembed ding , Ġand Ġit Ġseems Ġthis Ġadditional Ġinformation Ġcan Ġcontribute Ġsome Ġimprovement Ġpotentially .