Yes
BOOLEAN
Unanswerable
NONE
FLOAT SELECTED: Table 2: Results of final classification in Wang et al.
1907.09369
Introduction </s> There Ġhave Ġbeen Ġmany Ġadvances Ġin Ġmachine Ġlearning Ġmethods Ġwhich Ġhelp Ġmachines Ġunderstand Ġhuman Ġbehavior Ġbetter Ġthan Ġever . ĠOne Ġof Ġthe Ġmost Ġimportant Ġaspects Ġof Ġhuman Ġbehavior Ġis Ġemotion . ĠIf Ġmachines Ġcould Ġdetect Ġhuman Ġemotional Ġexpressions , Ġit Ġcould Ġbe Ġused Ġto Ġimprove Ġon Ġver ity Ġof Ġapplications Ġsuch Ġas Ġmarketing ĠB IB REF 0 Ġ, Ġhuman - computer Ġinteractions ĠB IB REF 1 Ġ, Ġpolitical Ġscience ĠB IB REF 2 Ġetc . </s> Em otion Ġin Ġhumans Ġis Ġcomplex Ġand Ġhard Ġto Ġdistinguish . ĠThere Ġhave Ġbeen Ġmany Ġemotional Ġmodels Ġin Ġpsychology Ġwhich Ġtried Ġto Ġclassify Ġand Ġpoint Ġout Ġbasic Ġhuman Ġemotions Ġsuch Ġas ĠEk man 's Ġ6 Ġbasic Ġemotions ĠB IB REF 3 Ġ, ĠPl utch ik 's Ġwheel Ġof Ġemotions ĠB IB REF 4 Ġ, Ġor ĠPar ro tt 's Ġthree - level Ġcategor ization Ġof Ġemotions ĠB IB REF 5 Ġ. ĠThese Ġvarieties Ġshow Ġthat Ġemotions Ġare Ġhard Ġto Ġdefine , Ġdistinguish , Ġand Ġcategor ize Ġeven Ġfor Ġhuman Ġexperts . </s> By Ġadding Ġthe Ġcomplexity Ġof Ġlanguage Ġand Ġthe Ġfact Ġthat Ġemotion Ġexpressions Ġare Ġvery Ġcomplex Ġand Ġcontext Ġdepend ant ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwe Ġcan Ġsee Ġwhy Ġdetecting Ġemotions Ġin Ġtextual Ġdata Ġis Ġa Ġchallenging Ġtask . ĠThis Ġdifficulty Ġcan Ġbe Ġseen Ġwhen Ġhuman Ġannot ators Ġtry Ġto Ġassign Ġemotional Ġlabels Ġto Ġthe Ġtext , Ġbut Ġusing Ġvarious Ġtechniques Ġthe Ġannotation Ġtask Ġcan Ġbe Ġaccomplished Ġwith Ġdesirable Ġagreement Ġamong Ġthe Ġannot ators ĠB IB REF 9 Ġ. </s> Related ĠWork </s> A Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġon Ġdetecting Ġemotion Ġin Ġspeech Ġor Ġvisual Ġdata ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠBut Ġdetecting Ġemotions Ġin Ġtextual Ġdata Ġis Ġa Ġrelatively Ġnew Ġarea Ġthat Ġdemands Ġmore Ġresearch . ĠThere Ġhave Ġbeen Ġmany Ġattempts Ġto Ġdetect Ġemotions Ġin Ġtext Ġusing Ġconventional Ġmachine Ġlearning Ġtechniques Ġand Ġhand crafted Ġfeatures Ġin Ġwhich Ġgiven Ġthe Ġdataset , Ġthe Ġauthors Ġtry Ġto Ġfind Ġthe Ġbest Ġfeature Ġset Ġthat Ġrepresents Ġthe Ġmost Ġand Ġthe Ġbest Ġinformation Ġabout Ġthe Ġtext , Ġthen Ġpassing Ġthe Ġconverted Ġtext Ġas Ġfeature Ġvectors Ġto Ġthe Ġclass ifier Ġfor Ġtraining ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. ĠDuring Ġthe Ġprocess Ġof Ġcreating Ġthe Ġfeature Ġset , Ġin Ġthese Ġmethods , Ġsome Ġof Ġthe Ġmost Ġimportant Ġinformation Ġin Ġthe Ġtext Ġsuch Ġas Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata , Ġand Ġthe Ġcontext Ġwill Ġbe Ġlost . </s> Considering Ġthe Ġcomplexity Ġof Ġthe Ġtask , Ġand Ġthe Ġfact Ġthat Ġthese Ġmodels Ġlose Ġa Ġlot Ġof Ġinformation Ġby Ġusing Ġsimpler Ġmodels Ġsuch Ġas Ġthe Ġbag Ġof Ġwords Ġmodel Ġ( B OW ) Ġor Ġlex icon Ġfeatures , Ġthese Ġattempts Ġlead Ġto Ġmethods Ġwhich Ġare Ġnot Ġreusable Ġand Ġgeneral izable . ĠFurther Ġimprovement Ġin Ġclassification Ġalgorithms , Ġand Ġtrying Ġout Ġnew Ġpaths Ġis Ġnecessary Ġin Ġorder Ġto Ġimprove Ġthe Ġperformance Ġof Ġemotion Ġdetection Ġmethods . ĠSome Ġsuggestions Ġthat Ġwere Ġless Ġpresent Ġin Ġthe Ġliterature , Ġare Ġto Ġdevelop Ġmethods Ġthat Ġgo Ġabove Ġlex ical Ġrepresentations Ġand Ġconsider Ġthe Ġflow Ġof Ġthe Ġlanguage . </s> Due Ġto Ġthis Ġsequential Ġnature , Ġrecurrent Ġand Ġconv olution al Ġneural Ġnetworks Ġhave Ġbeen Ġused Ġin Ġmany ĠN LP Ġtasks Ġand Ġwere Ġable Ġto Ġimprove Ġthe Ġperformance Ġin Ġa Ġvariety Ġof Ġclassification Ġtasks ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. ĠThere Ġhave Ġbeen Ġvery Ġfew Ġworks Ġin Ġusing Ġdeep Ġneural Ġnetwork Ġfor Ġemotion Ġdetection Ġin Ġtext ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ. ĠThese Ġmodels Ġcan Ġcapture Ġthe Ġcomplexity Ġan Ġcontext Ġof Ġthe Ġlanguage Ġbetter Ġnot Ġonly Ġby Ġkeeping Ġthe Ġsequential Ġinformation Ġbut Ġalso Ġby Ġcreating Ġhidden Ġrepresentation Ġfor Ġthe Ġtext Ġas Ġa Ġwhole Ġand Ġby Ġlearning Ġthe Ġimportant Ġfeatures Ġwithout Ġany Ġadditional Ġ( and Ġoften Ġincomplete ) Ġhuman - designed Ġfeatures . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġcreating Ġa Ġmodel Ġthat Ġcan Ġbetter Ġcapture Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġtext Ġ, Ġcan Ġsignificantly Ġimprove Ġthe Ġperformance Ġin Ġthe Ġhard Ġtask Ġof Ġemotion Ġdetection . ĠWe Ġshow Ġthis Ġby Ġusing Ġa Ġrecurrent Ġneural Ġnetwork - based Ġclass ifier Ġthat Ġcan Ġlearn Ġto Ġcreate Ġa Ġmore Ġinformative Ġlatent Ġrepresentation Ġof Ġthe Ġtarget Ġtext Ġas Ġa Ġwhole , Ġand Ġwe Ġshow Ġthat Ġthis Ġcan Ġimprove Ġthe Ġfinal Ġperformance Ġsignificantly . ĠBased Ġon Ġthat , Ġwe Ġsuggest Ġfocusing Ġon Ġmethod ologies Ġthat Ġincrease Ġthe Ġquality Ġof Ġthese Ġlatent Ġrepresentations Ġboth Ġcontext ually Ġand Ġemotionally , Ġcan Ġimprove Ġthe Ġperformance Ġof Ġthese Ġmodels . ĠBased Ġon Ġthis Ġassumption Ġwe Ġpropose Ġa Ġdeep Ġrecurrent Ġneural Ġnetwork Ġarchitecture Ġto Ġdetect Ġdiscrete Ġemotions Ġin Ġa Ġtweet Ġdataset . ĠThe Ġcode Ġcan Ġbe Ġaccessed Ġat ĠGitHub Ġ[ https :// github . com / arm int ab ari / Em otion - Det ection - R NN ]. </s> Bas eline ĠAppro aches </s> We Ġcompare Ġour Ġapproach Ġto Ġtwo Ġother , Ġthe Ġfirst Ġone Ġuses Ġalmost Ġthe Ġsame Ġtweet Ġdata Ġas Ġwe Ġuse Ġfor Ġtraining , Ġand Ġthe Ġsecond Ġone Ġis Ġthe ĠCrowd Fl ower Ġdataset Ġannot ated Ġfor Ġemotions . </s> In Ġthe Ġfirst Ġone ĠWang Ġet Ġal . ĠB IB REF 21 Ġdownloaded Ġover Ġ5 M Ġtweets Ġwhich Ġincluded Ġone Ġof Ġ131 Ġemotional Ġhasht ags Ġbased Ġon ĠPar ro tt 's Ġthree - level Ġcategor ization Ġof Ġemotions Ġin Ġseven Ġcategories : Ġjoy , Ġsadness , Ġanger , Ġlove , Ġfear , Ġthank fulness , Ġsurprise . ĠTo Ġassess Ġthe Ġquality Ġof Ġusing Ġhasht ags Ġas Ġlabels , Ġthe Ġsampled Ġ400 Ġtweets Ġrandomly Ġand Ġafter Ġcomparing Ġhuman Ġannotations Ġby Ġhashtag Ġlabels Ġthey Ġcame Ġup Ġwith Ġsimple Ġhe ur istics Ġto Ġincrease Ġthe Ġquality Ġof Ġlabeling Ġby Ġignoring Ġtweets Ġwith Ġquotations Ġand ĠURLs Ġand Ġonly Ġkeeping Ġtweets Ġwith Ġ5 Ġterms Ġor Ġmore Ġthat Ġhave Ġthe Ġemotional Ġhasht ags Ġat Ġthe Ġend Ġof Ġthe Ġtweets . ĠUsing Ġthese Ġrules Ġthey Ġextracted Ġaround Ġ2 . 5 M Ġtweets . ĠAfter Ġsampling Ġanother Ġ400 Ġrandom Ġtweets Ġand Ġcomparing Ġit Ġto Ġhuman Ġannotation Ġthe Ġsaw Ġthat Ġhasht ags Ġcan Ġclassify Ġthe Ġtweets Ġwith Ġ95 % Ġprecision . ĠThey Ġdid Ġsome Ġpre - processing Ġby Ġmaking Ġall Ġwords Ġlower - case , Ġreplaced Ġuser Ġmentions Ġwith Ġ@ user , Ġreplaced Ġletters / p unct uation Ġthat Ġis Ġrepeated Ġmore Ġthan Ġtwice Ġwith Ġthe Ġsame Ġtwo Ġletters / p unct uation Ġ( e . g ., Ġo oo oh ĠIN LINE FORM 0 Ġo oh , Ġ!! !!! ĠIN LINE FORM 1 Ġ!! ); Ġnormalized Ġsome Ġfrequently Ġused Ġinformal Ġexpressions Ġ( e . g ., Ġll ĠâĨĴ Ġwill , Ġd nt ĠIN LINE FORM 2 Ġdo Ġnot ); Ġand Ġstripped Ġhash Ġsymbols . ĠThey Ġused Ġa Ġsub - sample Ġof Ġtheir Ġdataset Ġto Ġfigure Ġout Ġthe Ġbest Ġapproaches Ġfor Ġclassification , Ġand Ġafter Ġtrying Ġtwo Ġdifferent Ġclass ifiers Ġ( mult in omial ĠNa ive ĠBay es Ġand ĠLI BL INE AR ) Ġand Ġ12 Ġdifferent Ġfeature Ġsets , Ġthey Ġgot Ġtheir Ġbest Ġresults Ġusing Ġlog istic Ġregression Ġbranch Ġfor ĠLI BL INE AR Ġclass ifier Ġand Ġa Ġfeature Ġset Ġconsist Ġof Ġn - gram ( n = 1 , 2 ), ĠLI WC Ġand ĠMP Q A Ġlex icons , ĠWord Net - A ffect Ġand ĠPOS Ġtags . </s> In Ġthe Ġsecond Ġone , Ġthe Ġreported Ġresults Ġare Ġfrom Ġa Ġpaper Ġby ĠB IB REF 33 Ġin Ġwhich Ġthey Ġused Ġmaximum Ġentropy Ġclass ifier Ġwith Ġbag Ġof Ġwords Ġmodel Ġto Ġclassify Ġvarious Ġemotional Ġdatasets . ĠHere Ġwe Ġonly Ġreport Ġpart Ġof Ġtheir Ġresult Ġfor ĠCrowd Fl ower Ġdataset Ġthat Ġcan Ġbe Ġmapped Ġto Ġone Ġof Ġour Ġseven Ġlabels . </s> Data Ġand Ġpreparation </s> There Ġare Ġnot Ġmany Ġfree Ġdatasets Ġavailable Ġfor Ġemotion Ġclassification . ĠMost Ġdatasets Ġare Ġsubject - specific Ġ( i . e . Ġnews Ġheadlines , Ġfairy Ġtails , Ġetc .) Ġand Ġnot Ġbig Ġenough Ġto Ġtrain Ġdeep Ġneural Ġnetworks . ĠHere Ġwe Ġuse Ġthe Ġtweet Ġdataset Ġcreated Ġby ĠWang Ġet Ġal . ĠAs Ġmentioned Ġin Ġthe Ġprevious Ġsection , Ġthey Ġhave Ġcollected Ġover Ġ2 Ġmillion Ġtweets Ġby Ġusing Ġhasht ags Ġfor Ġlabeling Ġtheir Ġdata . ĠThey Ġcreated Ġa Ġlist Ġof Ġwords Ġassociated Ġwith Ġ7 Ġemotions Ġ( six Ġemotions Ġfrom ĠB IB REF 34 Ġlove , Ġjoy , Ġsurprise , Ġanger , Ġsadness Ġfear Ġplus Ġthank fulness Ġ( See ĠTable ĠT AB REF 3 Ġ), Ġand Ġused Ġthe Ġlist Ġas Ġtheir Ġguide Ġto Ġlabel Ġthe Ġsampled Ġtweets Ġwith Ġacceptable Ġquality . </s> After Ġpre - processing , Ġthey Ġhave Ġused Ġ250 k Ġtweets Ġas Ġthe Ġtest Ġset , Ġaround Ġ250 k Ġas Ġdevelopment Ġtest Ġand Ġthe Ġrest Ġof Ġthe Ġdata Ġ( around Ġ2 M ) Ġas Ġtraining Ġdata . Ġtheir Ġbest Ġresults Ġusing ĠLI BL INE AR Ġclass ifier Ġand Ġa Ġfeature Ġset Ġcontaining Ġn - gram ( n = 1 , 2 ), ĠLI WC Ġand ĠMP Q A Ġlex icons , ĠWord Net - A ffect Ġand ĠPOS Ġtags Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 4 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġtheir Ġbest Ġresults Ġwere Ġfor Ġhigh Ġcount Ġemotions Ġlike Ġjoy Ġand Ġsadness Ġas Ġhigh Ġas Ġ72 . 1 Ġin ĠF - me asure Ġand Ġworst Ġresult Ġwas Ġfor Ġa Ġlow Ġcount Ġemotion Ġsurprise Ġwith ĠF - me asure Ġof Ġ13 . 9 . </s> As ĠTwitter Ġis Ġagainst Ġpol ishing Ġthis Ġmany Ġtweets , ĠWang Ġet Ġal . Ġprovided Ġthe Ġtweet Ġ ids Ġalong Ġwith Ġtheir Ġlabel . ĠFor Ġour Ġexperiment , Ġwe Ġretrieved Ġthe Ġtweets Ġin ĠWang Ġet Ġal .' s Ġdataset Ġby Ġtweet ĠIDs . ĠAs Ġthe Ġdataset Ġis Ġfrom Ġ7 Ġyears Ġago ĠWe Ġcould Ġonly Ġdownload Ġover Ġ1 . 3 Ġmillion Ġtweets Ġfrom Ġaround Ġ2 . 5 M Ġtweet ĠIDs Ġin Ġthe Ġdataset . ĠThe Ġdistribution Ġof Ġthe Ġdata Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 5 Ġ. </s> In Ġour Ġexperiment , Ġwe Ġused Ġsimpler Ġpre - processing Ġsteps Ġwhich Ġwill Ġbe Ġexplained Ġlater Ġon Ġin Ġthe Ġ" Exper iment " Ġsection . </s> Model </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġthe Ġdeep Ġneural Ġnetwork Ġarchitecture Ġthat Ġwe Ġused Ġto Ġclassify Ġemotions Ġin Ġthe Ġtweets Ġdataset . ĠEm otional Ġexpressions Ġare Ġmore Ġcomplex Ġand Ġcontext - dependent Ġeven Ġcompared Ġto Ġother Ġforms Ġof Ġexpressions Ġbased Ġmostly Ġon Ġthe Ġcomplexity Ġand Ġambiguity Ġof Ġhuman Ġemotions Ġand Ġemotional Ġexpressions Ġand Ġthe Ġhuge Ġimpact Ġof Ġcontext Ġon Ġthe Ġunderstanding Ġof Ġthe Ġexpressed Ġemotion . ĠThese Ġcomplexities Ġare Ġwhat Ġled Ġus Ġto Ġbelieve Ġlex icon - based Ġfeatures Ġlike Ġis Ġnormally Ġused Ġin Ġconventional Ġmachine Ġlearning Ġapproaches Ġare Ġunable Ġto Ġcapture Ġthe Ġintric acy Ġof Ġemotional Ġexpressions . </s> Our Ġarchitecture Ġwas Ġdesigned Ġto Ġshow Ġthat Ġusing Ġa Ġmodel Ġthat Ġcaptures Ġbetter Ġinformation Ġabout Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġthe Ġtext Ġcan Ġoutper form Ġlex icon - based Ġmethods Ġcommonly Ġused Ġin Ġthe Ġliterature . ĠAs Ġmentioned Ġin Ġthe ĠIntroduction , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġhave Ġbeen Ġshown Ġto Ġperform Ġwell Ġfor Ġthe Ġver ity Ġof Ġtasks Ġin ĠN LP , Ġespecially Ġclassification Ġtasks . ĠAnd Ġas Ġour Ġgoal Ġwas Ġto Ġcapture Ġmore Ġinformation Ġabout Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġthe Ġtext , Ġwe Ġdecided Ġto Ġuse Ġa Ġmodel Ġbased Ġon Ġbid irection al ĠR NN , Ġspecifically Ġa Ġbid irection al ĠGR U Ġnetwork Ġto Ġanalyze Ġthe Ġtweets . </s> For Ġbuilding Ġthe Ġemotion Ġclass ifier , Ġwe Ġhave Ġdecided Ġto Ġuse Ġ7 Ġbinary Ġclass ifiers - one Ġfor Ġeach Ġemotion - Ġeach Ġof Ġwhich Ġuses Ġthe Ġsame Ġarchitecture Ġfor Ġdetecting Ġa Ġspecific Ġemotion . ĠYou Ġcan Ġsee Ġthe Ġplot Ġdiagram Ġof Ġthe Ġmodel Ġin ĠFigure ĠFIG REF 6 Ġ. ĠThe Ġfirst Ġlayer Ġconsists Ġof Ġan Ġembed ding Ġlookup Ġlayer Ġthat Ġwill Ġnot Ġchange Ġduring Ġtraining Ġand Ġwill Ġbe Ġused Ġto Ġconvert Ġeach Ġterm Ġto Ġits Ġcorresponding Ġembed ding Ġvector . ĠIn Ġour Ġexperiments , Ġwe Ġtried Ġvarious Ġword Ġembed ding Ġmodels Ġbut Ġsaw Ġlittle Ġdifference Ġin Ġtheir Ġperformance . ĠHere Ġwe Ġreport Ġthe Ġresults Ġfor Ġtwo Ġwhich Ġhad Ġthe Ġbest Ġperformance Ġamong Ġall , ĠConcept Net ĠNumber batch ĠB IB REF 35 Ġand Ġfast Text ĠB IB REF 36 Ġboth Ġhad Ġ300 Ġdimensions . </s> As Ġnone Ġof Ġour Ġtweets Ġhad Ġmore Ġthan Ġ35 Ġterms , Ġwe Ġset Ġthe Ġsize Ġof Ġthe Ġembed ding Ġlayer Ġto Ġ35 Ġand Ġadded Ġpadding Ġto Ġshorter Ġtweets . ĠThe Ġoutput Ġof Ġthis Ġlayer Ġgoes Ġto Ġa Ġbid irection al ĠGR U Ġlayer Ġselected Ġto Ġcapture Ġthe Ġentirety Ġof Ġeach Ġtweet Ġbefore Ġpassing Ġits Ġoutput Ġforward . ĠThe Ġgoal Ġis Ġto Ġcreate Ġan Ġintermediate Ġrepresentation Ġfor Ġthe Ġtweets Ġthat Ġcapture Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata . ĠFor Ġthe Ġnext Ġstep , Ġwe Ġuse Ġa Ġconc aten ation Ġof Ġglobal Ġmax - pool ing Ġand Ġaverage - pool ing Ġlayers Ġ( with Ġa Ġwindow Ġsize Ġof Ġtwo ). ĠThen Ġa Ġmax - pool ing Ġwas Ġused Ġto Ġextract Ġthe Ġmost Ġimportant Ġfeatures Ġform Ġthe ĠGR U Ġoutput Ġand Ġan Ġaverage - pool ing Ġlayer Ġwas Ġused Ġto Ġconsiders Ġall Ġfeatures Ġto Ġcreate Ġa Ġrepresentation Ġfor Ġthe Ġtext Ġas Ġa Ġwhole . ĠThese Ġpartial Ġrepresentations Ġare Ġthen Ġwere Ġconc aten ated Ġto Ġcreate Ġout Ġfinal Ġhidden Ġrepresentation . ĠFor Ġclassification , Ġthe Ġoutput Ġof Ġthe Ġconc aten ation Ġis Ġpassed Ġto Ġa Ġdense Ġclassification Ġlayer Ġwith Ġ70 Ġnodes Ġalong Ġwith Ġa Ġdrop out Ġlayer Ġwith Ġa Ġrate Ġof Ġ50 % Ġto Ġprevent Ġover - fitting . ĠThe Ġfinal Ġlayer Ġis Ġa Ġs igm oid Ġlayer Ġthat Ġgenerates Ġthe Ġfinal Ġoutput Ġof Ġthe Ġclass ifier Ġreturning Ġthe Ġclass Ġprobability . </s> Exper iment </s> Min imal Ġpre - processing Ġwas Ġdone Ġby Ġconverting Ġtext Ġto Ġlower Ġcase , Ġremoving Ġthe Ġhasht ags Ġat Ġthe Ġend Ġof Ġtweets Ġand Ġseparating Ġeach Ġpunct uation Ġfrom Ġthe Ġconnected Ġtoken Ġ( e . g ., Ġawesome !! ĠIN LINE FORM 0 Ġawesome Ġ!! ) Ġand Ġreplacing Ġcomma Ġand Ġnew - line Ġcharacters Ġwith Ġwhite Ġspace . ĠThe Ġtext , Ġthen , Ġwas Ġtoken ized Ġusing ĠT ensor Flow - K er as Ġtoken izer . ĠTop ĠN Ġterms Ġwere Ġselected Ġand Ġadded Ġto Ġour Ġdictionary Ġwhere ĠN = 100 k Ġfor Ġhigher Ġcount Ġemotions Ġjoy , Ġsadness , Ġanger , Ġlove Ġand ĠN = 50 k Ġfor Ġthank fulness Ġand Ġfear Ġand ĠN = 25 k Ġfor Ġsurprise . ĠSeven Ġbinary Ġclass ifiers Ġwere Ġtrained Ġfor Ġthe Ġseven Ġemotions Ġwith Ġa Ġbatch Ġsize Ġof Ġ250 Ġand Ġfor Ġ20 Ġepoch s Ġwith Ġbinary Ġcross - ent ropy Ġas Ġthe Ġobjective Ġfunction Ġand ĠAdam Ġoptim izer . ĠThe Ġarchitecture Ġof Ġthe Ġmodel Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 6 Ġ. ĠFor Ġtraining Ġeach Ġclass ifier , Ġa Ġbalanced Ġdataset Ġwas Ġcreated Ġwith Ġselecting Ġall Ġtweets Ġfrom Ġthe Ġtarget Ġset Ġas Ġclass Ġ1 Ġand Ġa Ġrandom Ġsample Ġof Ġthe Ġsame Ġsize Ġfrom Ġother Ġclasses Ġas Ġclass Ġ0 . ĠFor Ġeach Ġclass ifier , Ġ80 % Ġof Ġthe Ġdata Ġwas Ġrandomly Ġselected Ġas Ġthe Ġtraining Ġset , Ġand Ġ10 % Ġfor Ġthe Ġvalidation Ġset , Ġand Ġ10 % Ġas Ġthe Ġtest Ġset . ĠAs Ġmentioned Ġbefore Ġwe Ġused Ġthe Ġtwo Ġembed ding Ġmodels , ĠConcept Net ĠNumber batch Ġand Ġfast Text Ġas Ġthe Ġtwo Ġmore Ġmodern Ġpre - trained Ġword Ġvector Ġspaces Ġto Ġsee Ġhow Ġchanging Ġthe Ġembed ding Ġlayer Ġcan Ġaffect Ġthe Ġperformance . ĠThe Ġresult Ġof Ġcomparison Ġamong Ġdifferent Ġembed d ings Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 10 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe Ġbest Ġperformance Ġwas Ġdivided Ġbetween Ġthe Ġtwo Ġembed ding Ġmodels Ġwith Ġminor Ġperformance Ġvariations . </s> The Ġcomparison Ġof Ġour Ġresult Ġwith ĠWang Ġet Ġal . Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 9 Ġ. Ġas Ġshown , Ġthe Ġresults Ġfrom Ġour Ġmodel Ġshows Ġsignificant Ġimprovement Ġfrom Ġ10 % Ġincrease Ġin ĠF - me asure Ġfor Ġa Ġhigh Ġcount Ġemotion Ġjoy Ġup Ġto Ġ61 . 7 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġa Ġlow Ġcount Ġemotion Ġsurprise . Ġon Ġaverage Ġwe Ġshowed Ġ26 . 8 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġall Ġcategories Ġand Ġmore Ġinterestingly Ġour Ġresult Ġshows Ġvery Ġlittle Ġvariance Ġbetween Ġdifferent Ġemotions Ġcompare Ġto Ġresults Ġreported Ġby ĠWang Ġet Ġal . </s> Model ĠPerform ances Ġon ĠNew ĠDat as et </s> To Ġasses Ġthe Ġperformance Ġof Ġthese Ġmodels Ġon Ġa Ġtotally Ġunseen Ġdata , Ġwe Ġtried Ġto Ġclassify Ġthe ĠCrowd Fl ower Ġemotional Ġtweets Ġdataset . ĠThe ĠCrowd Fl ower Ġdataset Ġconsists Ġof Ġ40 k Ġtweets Ġannot ated Ġvia Ġcrowd - s ourcing Ġeach Ġwith Ġa Ġsingle Ġemotional Ġlabel . ĠThis Ġdataset Ġis Ġconsidered Ġa Ġhard Ġdataset Ġto Ġclassify Ġwith Ġa Ġlot Ġof Ġnoise . ĠThe Ġdistribution Ġof Ġthe Ġdataset Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 18 Ġ. ĠThe Ġlabeling Ġon Ġthis Ġdataset Ġis Ġnon - standard , Ġso Ġwe Ġused Ġthe Ġfollowing Ġmapping Ġfor Ġlabels : </s> s ad ness ĠIN LINE FORM 0 Ġsadness </s> w orry ĠIN LINE FORM 0 Ġfear </s> h appiness ĠIN LINE FORM 0 Ġjoy </s> love ĠIN LINE FORM 0 Ġlove </s> sur prise ĠIN LINE FORM 0 Ġsurprise </s> anger ĠIN LINE FORM 0 Ġanger </s> We Ġthen Ġclassified Ġemotions Ġusing Ġthe Ġpre - trained Ġmodels Ġand Ġemotionally Ġfitted Ġfast Text Ġembed ding . ĠThe Ġresult Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 19 Ġ. ĠThe Ġbaseline Ġresults Ġare Ġfrom ĠB IB REF 33 Ġdone Ġusing ĠB OW Ġmodel Ġand Ġmaximum Ġentropy Ġclass ifier . ĠWe Ġsaw Ġa Ġhuge Ġimprovement Ġfrom Ġ26 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġthe Ġemotion Ġjoy Ġ( h appiness ) Ġup Ġto Ġ57 Ġpoint Ġincrease Ġfor Ġsurprise Ġwith Ġtotal Ġaverage Ġincrease Ġof Ġ38 . 6 Ġpoints . ĠB ost an Ġand ĠKling er Ġdid Ġnot Ġreport Ġclassification Ġresults Ġfor Ġthe Ġemotion Ġlove Ġso Ġwe Ġdid Ġnot Ġinclude Ġit Ġin Ġthe Ġaverage . ĠThese Ġresults Ġshow Ġthat Ġour Ġtrained Ġmodels Ġperform Ġexceptionally Ġon Ġa Ġtotally Ġnew Ġdataset Ġwith Ġa Ġdifferent Ġmethod Ġof Ġannotation . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġhave Ġshown Ġthat Ġusing Ġthe Ġdesigned ĠR NN Ġbased Ġnetwork Ġwe Ġcould Ġincrease Ġthe Ġperformance Ġof Ġclassification Ġdramatically . ĠWe Ġshowed Ġthat Ġkeeping Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata Ġcan Ġbe Ġhugely Ġbeneficial Ġwhen Ġworking Ġwith Ġtextual Ġdata Ġespecially Ġfaced Ġwith Ġthe Ġhard Ġtask Ġof Ġdetecting Ġmore Ġcomplex Ġphenomena Ġlike Ġemotions . ĠWe Ġaccomplished Ġthat Ġby Ġusing Ġa Ġrecurrent Ġnetwork Ġin Ġthe Ġprocess Ġof Ġgenerating Ġour Ġhidden Ġrepresentation . ĠWe Ġhave Ġalso Ġused Ġa Ġmax - pool ing Ġlayer Ġto Ġcapture Ġthe Ġmost Ġrelevant Ġfeatures Ġand Ġan Ġaverage Ġpool ing Ġlayer Ġto Ġcapture Ġthe Ġtext Ġas Ġa Ġwhole Ġproving Ġthat Ġwe Ġcan Ġachieve Ġbetter Ġperformance Ġby Ġfocusing Ġon Ġcreating Ġa Ġmore Ġinformative Ġhidden Ġrepresentation . ĠIn Ġfuture Ġwe Ġcan Ġfocus Ġon Ġimproving Ġthese Ġrepresentations Ġfor Ġexample Ġby Ġusing Ġattention Ġnetworks ĠB IB REF 37 Ġ, ĠB IB REF 38 Ġto Ġcapture Ġa Ġmore Ġcontextual Ġrepresentation Ġor Ġusing Ġlanguage Ġmodel Ġbased Ġmethods Ġlike ĠB ERT ĠB IB REF 39 Ġthat Ġhas Ġbeen Ġshown Ġvery Ġsuccessful Ġin Ġvarious ĠN LP Ġtasks .
Introduction There have been many advances in machine learning methods which help machines understand human behavior better than ever. One of the most important aspects of human behavior is emotion. If machines could detect human emotional expressions, it could be used to improve on verity of applications such as marketing BIBREF0 , human-computer interactions BIBREF1 , political science BIBREF2 etc. Emotion in humans is complex and hard to distinguish. There have been many emotional models in psychology which tried to classify and point out basic human emotions such as Ekman's 6 basic emotions BIBREF3 , Plutchik's wheel of emotions BIBREF4 , or Parrott's three-level categorization of emotions BIBREF5 . These varieties show that emotions are hard to define, distinguish, and categorize even for human experts. By adding the complexity of language and the fact that emotion expressions are very complex and context dependant BIBREF6 , BIBREF7 , BIBREF8 , we can see why detecting emotions in textual data is a challenging task. This difficulty can be seen when human annotators try to assign emotional labels to the text, but using various techniques the annotation task can be accomplished with desirable agreement among the annotators BIBREF9 . Related Work A lot of work has been done on detecting emotion in speech or visual data BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . But detecting emotions in textual data is a relatively new area that demands more research. There have been many attempts to detect emotions in text using conventional machine learning techniques and handcrafted features in which given the dataset, the authors try to find the best feature set that represents the most and the best information about the text, then passing the converted text as feature vectors to the classifier for training BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 . During the process of creating the feature set, in these methods, some of the most important information in the text such as the sequential nature of the data, and the context will be lost. Considering the complexity of the task, and the fact that these models lose a lot of information by using simpler models such as the bag of words model (BOW) or lexicon features, these attempts lead to methods which are not reusable and generalizable. Further improvement in classification algorithms, and trying out new paths is necessary in order to improve the performance of emotion detection methods. Some suggestions that were less present in the literature, are to develop methods that go above lexical representations and consider the flow of the language. Due to this sequential nature, recurrent and convolutional neural networks have been used in many NLP tasks and were able to improve the performance in a variety of classification tasks BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 . There have been very few works in using deep neural network for emotion detection in text BIBREF31 , BIBREF32 . These models can capture the complexity an context of the language better not only by keeping the sequential information but also by creating hidden representation for the text as a whole and by learning the important features without any additional (and often incomplete) human-designed features. In this work, we argue that creating a model that can better capture the context and sequential nature of text , can significantly improve the performance in the hard task of emotion detection. We show this by using a recurrent neural network-based classifier that can learn to create a more informative latent representation of the target text as a whole, and we show that this can improve the final performance significantly. Based on that, we suggest focusing on methodologies that increase the quality of these latent representations both contextually and emotionally, can improve the performance of these models. Based on this assumption we propose a deep recurrent neural network architecture to detect discrete emotions in a tweet dataset. The code can be accessed at GitHub [https://github.com/armintabari/Emotion-Detection-RNN]. Baseline Approaches We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions. In the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise. To assess the quality of using hashtags as labels, the sampled 400 tweets randomly and after comparing human annotations by hashtag labels they came up with simple heuristics to increase the quality of labeling by ignoring tweets with quotations and URLs and only keeping tweets with 5 terms or more that have the emotional hashtags at the end of the tweets. Using these rules they extracted around 2.5M tweets. After sampling another 400 random tweets and comparing it to human annotation the saw that hashtags can classify the tweets with 95% precision. They did some pre-processing by making all words lower-case, replaced user mentions with @user, replaced letters/punctuation that is repeated more than twice with the same two letters/punctuation (e.g., ooooh INLINEFORM0 ooh, !!!!! INLINEFORM1 !!); normalized some frequently used informal expressions (e.g., ll → will, dnt INLINEFORM2 do not); and stripped hash symbols. They used a sub-sample of their dataset to figure out the best approaches for classification, and after trying two different classifiers (multinomial Naive Bayes and LIBLINEAR) and 12 different feature sets, they got their best results using logistic regression branch for LIBLINEAR classifier and a feature set consist of n-gram(n=1,2), LIWC and MPQA lexicons, WordNet-Affect and POS tags. In the second one, the reported results are from a paper by BIBREF33 in which they used maximum entropy classifier with bag of words model to classify various emotional datasets. Here we only report part of their result for CrowdFlower dataset that can be mapped to one of our seven labels. Data and preparation There are not many free datasets available for emotion classification. Most datasets are subject-specific (i.e. news headlines, fairy tails, etc.) and not big enough to train deep neural networks. Here we use the tweet dataset created by Wang et al. As mentioned in the previous section, they have collected over 2 million tweets by using hashtags for labeling their data. They created a list of words associated with 7 emotions (six emotions from BIBREF34 love, joy, surprise, anger, sadness fear plus thankfulness (See Table TABREF3 ), and used the list as their guide to label the sampled tweets with acceptable quality. After pre-processing, they have used 250k tweets as the test set, around 250k as development test and the rest of the data (around 2M) as training data. their best results using LIBLINEAR classifier and a feature set containing n-gram(n=1,2), LIWC and MPQA lexicons, WordNet-Affect and POS tags can be seen in Table TABREF4 . It can be seen that their best results were for high count emotions like joy and sadness as high as 72.1 in F-measure and worst result was for a low count emotion surprise with F-measure of 13.9. As Twitter is against polishing this many tweets, Wang et al. provided the tweet ids along with their label. For our experiment, we retrieved the tweets in Wang et al.'s dataset by tweet IDs. As the dataset is from 7 years ago We could only download over 1.3 million tweets from around 2.5M tweet IDs in the dataset. The distribution of the data can be seen in Table TABREF5 . In our experiment, we used simpler pre-processing steps which will be explained later on in the "Experiment" section. Model In this section, we introduce the deep neural network architecture that we used to classify emotions in the tweets dataset. Emotional expressions are more complex and context-dependent even compared to other forms of expressions based mostly on the complexity and ambiguity of human emotions and emotional expressions and the huge impact of context on the understanding of the expressed emotion. These complexities are what led us to believe lexicon-based features like is normally used in conventional machine learning approaches are unable to capture the intricacy of emotional expressions. Our architecture was designed to show that using a model that captures better information about the context and sequential nature of the text can outperform lexicon-based methods commonly used in the literature. As mentioned in the Introduction, Recurrent Neural Networks (RNNs) have been shown to perform well for the verity of tasks in NLP, especially classification tasks. And as our goal was to capture more information about the context and sequential nature of the text, we decided to use a model based on bidirectional RNN, specifically a bidirectional GRU network to analyze the tweets. For building the emotion classifier, we have decided to use 7 binary classifiers-one for each emotion- each of which uses the same architecture for detecting a specific emotion. You can see the plot diagram of the model in Figure FIGREF6 . The first layer consists of an embedding lookup layer that will not change during training and will be used to convert each term to its corresponding embedding vector. In our experiments, we tried various word embedding models but saw little difference in their performance. Here we report the results for two which had the best performance among all, ConceptNet Numberbatch BIBREF35 and fastText BIBREF36 both had 300 dimensions. As none of our tweets had more than 35 terms, we set the size of the embedding layer to 35 and added padding to shorter tweets. The output of this layer goes to a bidirectional GRU layer selected to capture the entirety of each tweet before passing its output forward. The goal is to create an intermediate representation for the tweets that capture the sequential nature of the data. For the next step, we use a concatenation of global max-pooling and average-pooling layers (with a window size of two). Then a max-pooling was used to extract the most important features form the GRU output and an average-pooling layer was used to considers all features to create a representation for the text as a whole. These partial representations are then were concatenated to create out final hidden representation. For classification, the output of the concatenation is passed to a dense classification layer with 70 nodes along with a dropout layer with a rate of 50% to prevent over-fitting. The final layer is a sigmoid layer that generates the final output of the classifier returning the class probability. Experiment Minimal pre-processing was done by converting text to lower case, removing the hashtags at the end of tweets and separating each punctuation from the connected token (e.g., awesome!! INLINEFORM0 awesome !!) and replacing comma and new-line characters with white space. The text, then, was tokenized using TensorFlow-Keras tokenizer. Top N terms were selected and added to our dictionary where N=100k for higher count emotions joy, sadness, anger, love and N=50k for thankfulness and fear and N=25k for surprise. Seven binary classifiers were trained for the seven emotions with a batch size of 250 and for 20 epochs with binary cross-entropy as the objective function and Adam optimizer. The architecture of the model can be seen in Figure FIGREF6 . For training each classifier, a balanced dataset was created with selecting all tweets from the target set as class 1 and a random sample of the same size from other classes as class 0. For each classifier, 80% of the data was randomly selected as the training set, and 10% for the validation set, and 10% as the test set. As mentioned before we used the two embedding models, ConceptNet Numberbatch and fastText as the two more modern pre-trained word vector spaces to see how changing the embedding layer can affect the performance. The result of comparison among different embeddings can be seen in Table TABREF10 . It can be seen that the best performance was divided between the two embedding models with minor performance variations. The comparison of our result with Wang et al. can be seen in Table TABREF9 . as shown, the results from our model shows significant improvement from 10% increase in F-measure for a high count emotion joy up to 61.7 point increase in F-measure for a low count emotion surprise. on average we showed 26.8 point increase in F-measure for all categories and more interestingly our result shows very little variance between different emotions compare to results reported by Wang et al. Model Performances on New Dataset To asses the performance of these models on a totally unseen data, we tried to classify the CrowdFlower emotional tweets dataset. The CrowdFlower dataset consists of 40k tweets annotated via crowd-sourcing each with a single emotional label. This dataset is considered a hard dataset to classify with a lot of noise. The distribution of the dataset can be seen in Table TABREF18 . The labeling on this dataset is non-standard, so we used the following mapping for labels: sadness INLINEFORM0 sadness worry INLINEFORM0 fear happiness INLINEFORM0 joy love INLINEFORM0 love surprise INLINEFORM0 surprise anger INLINEFORM0 anger We then classified emotions using the pre-trained models and emotionally fitted fastText embedding. The result can be seen in Table TABREF19 . The baseline results are from BIBREF33 done using BOW model and maximum entropy classifier. We saw a huge improvement from 26 point increase in F-measure for the emotion joy (happiness) up to 57 point increase for surprise with total average increase of 38.6 points. Bostan and Klinger did not report classification results for the emotion love so we did not include it in the average. These results show that our trained models perform exceptionally on a totally new dataset with a different method of annotation. Conclusion and Future Work In this paper, we have shown that using the designed RNN based network we could increase the performance of classification dramatically. We showed that keeping the sequential nature of the data can be hugely beneficial when working with textual data especially faced with the hard task of detecting more complex phenomena like emotions. We accomplished that by using a recurrent network in the process of generating our hidden representation. We have also used a max-pooling layer to capture the most relevant features and an average pooling layer to capture the text as a whole proving that we can achieve better performance by focusing on creating a more informative hidden representation. In future we can focus on improving these representations for example by using attention networks BIBREF37 , BIBREF38 to capture a more contextual representation or using language model based methods like BERT BIBREF39 that has been shown very successful in various NLP tasks.
Do they report results only on English data?
fd2c6c26fd0ab3c10aae4f2550c5391576a77491
Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish Ġdata ?
Yes
<s> Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish Ġdata ? </s> Introduction </s> There Ġhave Ġbeen Ġmany Ġadvances Ġin Ġmachine Ġlearning Ġmethods Ġwhich Ġhelp Ġmachines Ġunderstand Ġhuman Ġbehavior Ġbetter Ġthan Ġever . ĠOne Ġof Ġthe Ġmost Ġimportant Ġaspects Ġof Ġhuman Ġbehavior Ġis Ġemotion . ĠIf Ġmachines Ġcould Ġdetect Ġhuman Ġemotional Ġexpressions , Ġit Ġcould Ġbe Ġused Ġto Ġimprove Ġon Ġver ity Ġof Ġapplications Ġsuch Ġas Ġmarketing ĠB IB REF 0 Ġ, Ġhuman - computer Ġinteractions ĠB IB REF 1 Ġ, Ġpolitical Ġscience ĠB IB REF 2 Ġetc . </s> Em otion Ġin Ġhumans Ġis Ġcomplex Ġand Ġhard Ġto Ġdistinguish . ĠThere Ġhave Ġbeen Ġmany Ġemotional Ġmodels Ġin Ġpsychology Ġwhich Ġtried Ġto Ġclassify Ġand Ġpoint Ġout Ġbasic Ġhuman Ġemotions Ġsuch Ġas ĠEk man 's Ġ6 Ġbasic Ġemotions ĠB IB REF 3 Ġ, ĠPl utch ik 's Ġwheel Ġof Ġemotions ĠB IB REF 4 Ġ, Ġor ĠPar ro tt 's Ġthree - level Ġcategor ization Ġof Ġemotions ĠB IB REF 5 Ġ. ĠThese Ġvarieties Ġshow Ġthat Ġemotions Ġare Ġhard Ġto Ġdefine , Ġdistinguish , Ġand Ġcategor ize Ġeven Ġfor Ġhuman Ġexperts . </s> By Ġadding Ġthe Ġcomplexity Ġof Ġlanguage Ġand Ġthe Ġfact Ġthat Ġemotion Ġexpressions Ġare Ġvery Ġcomplex Ġand Ġcontext Ġdepend ant ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġwe Ġcan Ġsee Ġwhy Ġdetecting Ġemotions Ġin Ġtextual Ġdata Ġis Ġa Ġchallenging Ġtask . ĠThis Ġdifficulty Ġcan Ġbe Ġseen Ġwhen Ġhuman Ġannot ators Ġtry Ġto Ġassign Ġemotional Ġlabels Ġto Ġthe Ġtext , Ġbut Ġusing Ġvarious Ġtechniques Ġthe Ġannotation Ġtask Ġcan Ġbe Ġaccomplished Ġwith Ġdesirable Ġagreement Ġamong Ġthe Ġannot ators ĠB IB REF 9 Ġ. </s> Related ĠWork </s> A Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġon Ġdetecting Ġemotion Ġin Ġspeech Ġor Ġvisual Ġdata ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠBut Ġdetecting Ġemotions Ġin Ġtextual Ġdata Ġis Ġa Ġrelatively Ġnew Ġarea Ġthat Ġdemands Ġmore Ġresearch . ĠThere Ġhave Ġbeen Ġmany Ġattempts Ġto Ġdetect Ġemotions Ġin Ġtext Ġusing Ġconventional Ġmachine Ġlearning Ġtechniques Ġand Ġhand crafted Ġfeatures Ġin Ġwhich Ġgiven Ġthe Ġdataset , Ġthe Ġauthors Ġtry Ġto Ġfind Ġthe Ġbest Ġfeature Ġset Ġthat Ġrepresents Ġthe Ġmost Ġand Ġthe Ġbest Ġinformation Ġabout Ġthe Ġtext , Ġthen Ġpassing Ġthe Ġconverted Ġtext Ġas Ġfeature Ġvectors Ġto Ġthe Ġclass ifier Ġfor Ġtraining ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. ĠDuring Ġthe Ġprocess Ġof Ġcreating Ġthe Ġfeature Ġset , Ġin Ġthese Ġmethods , Ġsome Ġof Ġthe Ġmost Ġimportant Ġinformation Ġin Ġthe Ġtext Ġsuch Ġas Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata , Ġand Ġthe Ġcontext Ġwill Ġbe Ġlost . </s> Considering Ġthe Ġcomplexity Ġof Ġthe Ġtask , Ġand Ġthe Ġfact Ġthat Ġthese Ġmodels Ġlose Ġa Ġlot Ġof Ġinformation Ġby Ġusing Ġsimpler Ġmodels Ġsuch Ġas Ġthe Ġbag Ġof Ġwords Ġmodel Ġ( B OW ) Ġor Ġlex icon Ġfeatures , Ġthese Ġattempts Ġlead Ġto Ġmethods Ġwhich Ġare Ġnot Ġreusable Ġand Ġgeneral izable . ĠFurther Ġimprovement Ġin Ġclassification Ġalgorithms , Ġand Ġtrying Ġout Ġnew Ġpaths Ġis Ġnecessary Ġin Ġorder Ġto Ġimprove Ġthe Ġperformance Ġof Ġemotion Ġdetection Ġmethods . ĠSome Ġsuggestions Ġthat Ġwere Ġless Ġpresent Ġin Ġthe Ġliterature , Ġare Ġto Ġdevelop Ġmethods Ġthat Ġgo Ġabove Ġlex ical Ġrepresentations Ġand Ġconsider Ġthe Ġflow Ġof Ġthe Ġlanguage . </s> Due Ġto Ġthis Ġsequential Ġnature , Ġrecurrent Ġand Ġconv olution al Ġneural Ġnetworks Ġhave Ġbeen Ġused Ġin Ġmany ĠN LP Ġtasks Ġand Ġwere Ġable Ġto Ġimprove Ġthe Ġperformance Ġin Ġa Ġvariety Ġof Ġclassification Ġtasks ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. ĠThere Ġhave Ġbeen Ġvery Ġfew Ġworks Ġin Ġusing Ġdeep Ġneural Ġnetwork Ġfor Ġemotion Ġdetection Ġin Ġtext ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ. ĠThese Ġmodels Ġcan Ġcapture Ġthe Ġcomplexity Ġan Ġcontext Ġof Ġthe Ġlanguage Ġbetter Ġnot Ġonly Ġby Ġkeeping Ġthe Ġsequential Ġinformation Ġbut Ġalso Ġby Ġcreating Ġhidden Ġrepresentation Ġfor Ġthe Ġtext Ġas Ġa Ġwhole Ġand Ġby Ġlearning Ġthe Ġimportant Ġfeatures Ġwithout Ġany Ġadditional Ġ( and Ġoften Ġincomplete ) Ġhuman - designed Ġfeatures . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġcreating Ġa Ġmodel Ġthat Ġcan Ġbetter Ġcapture Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġtext Ġ, Ġcan Ġsignificantly Ġimprove Ġthe Ġperformance Ġin Ġthe Ġhard Ġtask Ġof Ġemotion Ġdetection . ĠWe Ġshow Ġthis Ġby Ġusing Ġa Ġrecurrent Ġneural Ġnetwork - based Ġclass ifier Ġthat Ġcan Ġlearn Ġto Ġcreate Ġa Ġmore Ġinformative Ġlatent Ġrepresentation Ġof Ġthe Ġtarget Ġtext Ġas Ġa Ġwhole , Ġand Ġwe Ġshow Ġthat Ġthis Ġcan Ġimprove Ġthe Ġfinal Ġperformance Ġsignificantly . ĠBased Ġon Ġthat , Ġwe Ġsuggest Ġfocusing Ġon Ġmethod ologies Ġthat Ġincrease Ġthe Ġquality Ġof Ġthese Ġlatent Ġrepresentations Ġboth Ġcontext ually Ġand Ġemotionally , Ġcan Ġimprove Ġthe Ġperformance Ġof Ġthese Ġmodels . ĠBased Ġon Ġthis Ġassumption Ġwe Ġpropose Ġa Ġdeep Ġrecurrent Ġneural Ġnetwork Ġarchitecture Ġto Ġdetect Ġdiscrete Ġemotions Ġin Ġa Ġtweet Ġdataset . ĠThe Ġcode Ġcan Ġbe Ġaccessed Ġat ĠGitHub Ġ[ https :// github . com / arm int ab ari / Em otion - Det ection - R NN ]. </s> Bas eline ĠAppro aches </s> We Ġcompare Ġour Ġapproach Ġto Ġtwo Ġother , Ġthe Ġfirst Ġone Ġuses Ġalmost Ġthe Ġsame Ġtweet Ġdata Ġas Ġwe Ġuse Ġfor Ġtraining , Ġand Ġthe Ġsecond Ġone Ġis Ġthe ĠCrowd Fl ower Ġdataset Ġannot ated Ġfor Ġemotions . </s> In Ġthe Ġfirst Ġone ĠWang Ġet Ġal . ĠB IB REF 21 Ġdownloaded Ġover Ġ5 M Ġtweets Ġwhich Ġincluded Ġone Ġof Ġ131 Ġemotional Ġhasht ags Ġbased Ġon ĠPar ro tt 's Ġthree - level Ġcategor ization Ġof Ġemotions Ġin Ġseven Ġcategories : Ġjoy , Ġsadness , Ġanger , Ġlove , Ġfear , Ġthank fulness , Ġsurprise . ĠTo Ġassess Ġthe Ġquality Ġof Ġusing Ġhasht ags Ġas Ġlabels , Ġthe Ġsampled Ġ400 Ġtweets Ġrandomly Ġand Ġafter Ġcomparing Ġhuman Ġannotations Ġby Ġhashtag Ġlabels Ġthey Ġcame Ġup Ġwith Ġsimple Ġhe ur istics Ġto Ġincrease Ġthe Ġquality Ġof Ġlabeling Ġby Ġignoring Ġtweets Ġwith Ġquotations Ġand ĠURLs Ġand Ġonly Ġkeeping Ġtweets Ġwith Ġ5 Ġterms Ġor Ġmore Ġthat Ġhave Ġthe Ġemotional Ġhasht ags Ġat Ġthe Ġend Ġof Ġthe Ġtweets . ĠUsing Ġthese Ġrules Ġthey Ġextracted Ġaround Ġ2 . 5 M Ġtweets . ĠAfter Ġsampling Ġanother Ġ400 Ġrandom Ġtweets Ġand Ġcomparing Ġit Ġto Ġhuman Ġannotation Ġthe Ġsaw Ġthat Ġhasht ags Ġcan Ġclassify Ġthe Ġtweets Ġwith Ġ95 % Ġprecision . ĠThey Ġdid Ġsome Ġpre - processing Ġby Ġmaking Ġall Ġwords Ġlower - case , Ġreplaced Ġuser Ġmentions Ġwith Ġ@ user , Ġreplaced Ġletters / p unct uation Ġthat Ġis Ġrepeated Ġmore Ġthan Ġtwice Ġwith Ġthe Ġsame Ġtwo Ġletters / p unct uation Ġ( e . g ., Ġo oo oh ĠIN LINE FORM 0 Ġo oh , Ġ!! !!! ĠIN LINE FORM 1 Ġ!! ); Ġnormalized Ġsome Ġfrequently Ġused Ġinformal Ġexpressions Ġ( e . g ., Ġll ĠâĨĴ Ġwill , Ġd nt ĠIN LINE FORM 2 Ġdo Ġnot ); Ġand Ġstripped Ġhash Ġsymbols . ĠThey Ġused Ġa Ġsub - sample Ġof Ġtheir Ġdataset Ġto Ġfigure Ġout Ġthe Ġbest Ġapproaches Ġfor Ġclassification , Ġand Ġafter Ġtrying Ġtwo Ġdifferent Ġclass ifiers Ġ( mult in omial ĠNa ive ĠBay es Ġand ĠLI BL INE AR ) Ġand Ġ12 Ġdifferent Ġfeature Ġsets , Ġthey Ġgot Ġtheir Ġbest Ġresults Ġusing Ġlog istic Ġregression Ġbranch Ġfor ĠLI BL INE AR Ġclass ifier Ġand Ġa Ġfeature Ġset Ġconsist Ġof Ġn - gram ( n = 1 , 2 ), ĠLI WC Ġand ĠMP Q A Ġlex icons , ĠWord Net - A ffect Ġand ĠPOS Ġtags . </s> In Ġthe Ġsecond Ġone , Ġthe Ġreported Ġresults Ġare Ġfrom Ġa Ġpaper Ġby ĠB IB REF 33 Ġin Ġwhich Ġthey Ġused Ġmaximum Ġentropy Ġclass ifier Ġwith Ġbag Ġof Ġwords Ġmodel Ġto Ġclassify Ġvarious Ġemotional Ġdatasets . ĠHere Ġwe Ġonly Ġreport Ġpart Ġof Ġtheir Ġresult Ġfor ĠCrowd Fl ower Ġdataset Ġthat Ġcan Ġbe Ġmapped Ġto Ġone Ġof Ġour Ġseven Ġlabels . </s> Data Ġand Ġpreparation </s> There Ġare Ġnot Ġmany Ġfree Ġdatasets Ġavailable Ġfor Ġemotion Ġclassification . ĠMost Ġdatasets Ġare Ġsubject - specific Ġ( i . e . Ġnews Ġheadlines , Ġfairy Ġtails , Ġetc .) Ġand Ġnot Ġbig Ġenough Ġto Ġtrain Ġdeep Ġneural Ġnetworks . ĠHere Ġwe Ġuse Ġthe Ġtweet Ġdataset Ġcreated Ġby ĠWang Ġet Ġal . ĠAs Ġmentioned Ġin Ġthe Ġprevious Ġsection , Ġthey Ġhave Ġcollected Ġover Ġ2 Ġmillion Ġtweets Ġby Ġusing Ġhasht ags Ġfor Ġlabeling Ġtheir Ġdata . ĠThey Ġcreated Ġa Ġlist Ġof Ġwords Ġassociated Ġwith Ġ7 Ġemotions Ġ( six Ġemotions Ġfrom ĠB IB REF 34 Ġlove , Ġjoy , Ġsurprise , Ġanger , Ġsadness Ġfear Ġplus Ġthank fulness Ġ( See ĠTable ĠT AB REF 3 Ġ), Ġand Ġused Ġthe Ġlist Ġas Ġtheir Ġguide Ġto Ġlabel Ġthe Ġsampled Ġtweets Ġwith Ġacceptable Ġquality . </s> After Ġpre - processing , Ġthey Ġhave Ġused Ġ250 k Ġtweets Ġas Ġthe Ġtest Ġset , Ġaround Ġ250 k Ġas Ġdevelopment Ġtest Ġand Ġthe Ġrest Ġof Ġthe Ġdata Ġ( around Ġ2 M ) Ġas Ġtraining Ġdata . Ġtheir Ġbest Ġresults Ġusing ĠLI BL INE AR Ġclass ifier Ġand Ġa Ġfeature Ġset Ġcontaining Ġn - gram ( n = 1 , 2 ), ĠLI WC Ġand ĠMP Q A Ġlex icons , ĠWord Net - A ffect Ġand ĠPOS Ġtags Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 4 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġtheir Ġbest Ġresults Ġwere Ġfor Ġhigh Ġcount Ġemotions Ġlike Ġjoy Ġand Ġsadness Ġas Ġhigh Ġas Ġ72 . 1 Ġin ĠF - me asure Ġand Ġworst Ġresult Ġwas Ġfor Ġa Ġlow Ġcount Ġemotion Ġsurprise Ġwith ĠF - me asure Ġof Ġ13 . 9 . </s> As ĠTwitter Ġis Ġagainst Ġpol ishing Ġthis Ġmany Ġtweets , ĠWang Ġet Ġal . Ġprovided Ġthe Ġtweet Ġ ids Ġalong Ġwith Ġtheir Ġlabel . ĠFor Ġour Ġexperiment , Ġwe Ġretrieved Ġthe Ġtweets Ġin ĠWang Ġet Ġal .' s Ġdataset Ġby Ġtweet ĠIDs . ĠAs Ġthe Ġdataset Ġis Ġfrom Ġ7 Ġyears Ġago ĠWe Ġcould Ġonly Ġdownload Ġover Ġ1 . 3 Ġmillion Ġtweets Ġfrom Ġaround Ġ2 . 5 M Ġtweet ĠIDs Ġin Ġthe Ġdataset . ĠThe Ġdistribution Ġof Ġthe Ġdata Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 5 Ġ. </s> In Ġour Ġexperiment , Ġwe Ġused Ġsimpler Ġpre - processing Ġsteps Ġwhich Ġwill Ġbe Ġexplained Ġlater Ġon Ġin Ġthe Ġ" Exper iment " Ġsection . </s> Model </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġthe Ġdeep Ġneural Ġnetwork Ġarchitecture Ġthat Ġwe Ġused Ġto Ġclassify Ġemotions Ġin Ġthe Ġtweets Ġdataset . ĠEm otional Ġexpressions Ġare Ġmore Ġcomplex Ġand Ġcontext - dependent Ġeven Ġcompared Ġto Ġother Ġforms Ġof Ġexpressions Ġbased Ġmostly Ġon Ġthe Ġcomplexity Ġand Ġambiguity Ġof Ġhuman Ġemotions Ġand Ġemotional Ġexpressions Ġand Ġthe Ġhuge Ġimpact Ġof Ġcontext Ġon Ġthe Ġunderstanding Ġof Ġthe Ġexpressed Ġemotion . ĠThese Ġcomplexities Ġare Ġwhat Ġled Ġus Ġto Ġbelieve Ġlex icon - based Ġfeatures Ġlike Ġis Ġnormally Ġused Ġin Ġconventional Ġmachine Ġlearning Ġapproaches Ġare Ġunable Ġto Ġcapture Ġthe Ġintric acy Ġof Ġemotional Ġexpressions . </s> Our Ġarchitecture Ġwas Ġdesigned Ġto Ġshow Ġthat Ġusing Ġa Ġmodel Ġthat Ġcaptures Ġbetter Ġinformation Ġabout Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġthe Ġtext Ġcan Ġoutper form Ġlex icon - based Ġmethods Ġcommonly Ġused Ġin Ġthe Ġliterature . ĠAs Ġmentioned Ġin Ġthe ĠIntroduction , ĠRec urrent ĠNeural ĠNetworks Ġ( R NN s ) Ġhave Ġbeen Ġshown Ġto Ġperform Ġwell Ġfor Ġthe Ġver ity Ġof Ġtasks Ġin ĠN LP , Ġespecially Ġclassification Ġtasks . ĠAnd Ġas Ġour Ġgoal Ġwas Ġto Ġcapture Ġmore Ġinformation Ġabout Ġthe Ġcontext Ġand Ġsequential Ġnature Ġof Ġthe Ġtext , Ġwe Ġdecided Ġto Ġuse Ġa Ġmodel Ġbased Ġon Ġbid irection al ĠR NN , Ġspecifically Ġa Ġbid irection al ĠGR U Ġnetwork Ġto Ġanalyze Ġthe Ġtweets . </s> For Ġbuilding Ġthe Ġemotion Ġclass ifier , Ġwe Ġhave Ġdecided Ġto Ġuse Ġ7 Ġbinary Ġclass ifiers - one Ġfor Ġeach Ġemotion - Ġeach Ġof Ġwhich Ġuses Ġthe Ġsame Ġarchitecture Ġfor Ġdetecting Ġa Ġspecific Ġemotion . ĠYou Ġcan Ġsee Ġthe Ġplot Ġdiagram Ġof Ġthe Ġmodel Ġin ĠFigure ĠFIG REF 6 Ġ. ĠThe Ġfirst Ġlayer Ġconsists Ġof Ġan Ġembed ding Ġlookup Ġlayer Ġthat Ġwill Ġnot Ġchange Ġduring Ġtraining Ġand Ġwill Ġbe Ġused Ġto Ġconvert Ġeach Ġterm Ġto Ġits Ġcorresponding Ġembed ding Ġvector . ĠIn Ġour Ġexperiments , Ġwe Ġtried Ġvarious Ġword Ġembed ding Ġmodels Ġbut Ġsaw Ġlittle Ġdifference Ġin Ġtheir Ġperformance . ĠHere Ġwe Ġreport Ġthe Ġresults Ġfor Ġtwo Ġwhich Ġhad Ġthe Ġbest Ġperformance Ġamong Ġall , ĠConcept Net ĠNumber batch ĠB IB REF 35 Ġand Ġfast Text ĠB IB REF 36 Ġboth Ġhad Ġ300 Ġdimensions . </s> As Ġnone Ġof Ġour Ġtweets Ġhad Ġmore Ġthan Ġ35 Ġterms , Ġwe Ġset Ġthe Ġsize Ġof Ġthe Ġembed ding Ġlayer Ġto Ġ35 Ġand Ġadded Ġpadding Ġto Ġshorter Ġtweets . ĠThe Ġoutput Ġof Ġthis Ġlayer Ġgoes Ġto Ġa Ġbid irection al ĠGR U Ġlayer Ġselected Ġto Ġcapture Ġthe Ġentirety Ġof Ġeach Ġtweet Ġbefore Ġpassing Ġits Ġoutput Ġforward . ĠThe Ġgoal Ġis Ġto Ġcreate Ġan Ġintermediate Ġrepresentation Ġfor Ġthe Ġtweets Ġthat Ġcapture Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata . ĠFor Ġthe Ġnext Ġstep , Ġwe Ġuse Ġa Ġconc aten ation Ġof Ġglobal Ġmax - pool ing Ġand Ġaverage - pool ing Ġlayers Ġ( with Ġa Ġwindow Ġsize Ġof Ġtwo ). ĠThen Ġa Ġmax - pool ing Ġwas Ġused Ġto Ġextract Ġthe Ġmost Ġimportant Ġfeatures Ġform Ġthe ĠGR U Ġoutput Ġand Ġan Ġaverage - pool ing Ġlayer Ġwas Ġused Ġto Ġconsiders Ġall Ġfeatures Ġto Ġcreate Ġa Ġrepresentation Ġfor Ġthe Ġtext Ġas Ġa Ġwhole . ĠThese Ġpartial Ġrepresentations Ġare Ġthen Ġwere Ġconc aten ated Ġto Ġcreate Ġout Ġfinal Ġhidden Ġrepresentation . ĠFor Ġclassification , Ġthe Ġoutput Ġof Ġthe Ġconc aten ation Ġis Ġpassed Ġto Ġa Ġdense Ġclassification Ġlayer Ġwith Ġ70 Ġnodes Ġalong Ġwith Ġa Ġdrop out Ġlayer Ġwith Ġa Ġrate Ġof Ġ50 % Ġto Ġprevent Ġover - fitting . ĠThe Ġfinal Ġlayer Ġis Ġa Ġs igm oid Ġlayer Ġthat Ġgenerates Ġthe Ġfinal Ġoutput Ġof Ġthe Ġclass ifier Ġreturning Ġthe Ġclass Ġprobability . </s> Exper iment </s> Min imal Ġpre - processing Ġwas Ġdone Ġby Ġconverting Ġtext Ġto Ġlower Ġcase , Ġremoving Ġthe Ġhasht ags Ġat Ġthe Ġend Ġof Ġtweets Ġand Ġseparating Ġeach Ġpunct uation Ġfrom Ġthe Ġconnected Ġtoken Ġ( e . g ., Ġawesome !! ĠIN LINE FORM 0 Ġawesome Ġ!! ) Ġand Ġreplacing Ġcomma Ġand Ġnew - line Ġcharacters Ġwith Ġwhite Ġspace . ĠThe Ġtext , Ġthen , Ġwas Ġtoken ized Ġusing ĠT ensor Flow - K er as Ġtoken izer . ĠTop ĠN Ġterms Ġwere Ġselected Ġand Ġadded Ġto Ġour Ġdictionary Ġwhere ĠN = 100 k Ġfor Ġhigher Ġcount Ġemotions Ġjoy , Ġsadness , Ġanger , Ġlove Ġand ĠN = 50 k Ġfor Ġthank fulness Ġand Ġfear Ġand ĠN = 25 k Ġfor Ġsurprise . ĠSeven Ġbinary Ġclass ifiers Ġwere Ġtrained Ġfor Ġthe Ġseven Ġemotions Ġwith Ġa Ġbatch Ġsize Ġof Ġ250 Ġand Ġfor Ġ20 Ġepoch s Ġwith Ġbinary Ġcross - ent ropy Ġas Ġthe Ġobjective Ġfunction Ġand ĠAdam Ġoptim izer . ĠThe Ġarchitecture Ġof Ġthe Ġmodel Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 6 Ġ. ĠFor Ġtraining Ġeach Ġclass ifier , Ġa Ġbalanced Ġdataset Ġwas Ġcreated Ġwith Ġselecting Ġall Ġtweets Ġfrom Ġthe Ġtarget Ġset Ġas Ġclass Ġ1 Ġand Ġa Ġrandom Ġsample Ġof Ġthe Ġsame Ġsize Ġfrom Ġother Ġclasses Ġas Ġclass Ġ0 . ĠFor Ġeach Ġclass ifier , Ġ80 % Ġof Ġthe Ġdata Ġwas Ġrandomly Ġselected Ġas Ġthe Ġtraining Ġset , Ġand Ġ10 % Ġfor Ġthe Ġvalidation Ġset , Ġand Ġ10 % Ġas Ġthe Ġtest Ġset . ĠAs Ġmentioned Ġbefore Ġwe Ġused Ġthe Ġtwo Ġembed ding Ġmodels , ĠConcept Net ĠNumber batch Ġand Ġfast Text Ġas Ġthe Ġtwo Ġmore Ġmodern Ġpre - trained Ġword Ġvector Ġspaces Ġto Ġsee Ġhow Ġchanging Ġthe Ġembed ding Ġlayer Ġcan Ġaffect Ġthe Ġperformance . ĠThe Ġresult Ġof Ġcomparison Ġamong Ġdifferent Ġembed d ings Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 10 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe Ġbest Ġperformance Ġwas Ġdivided Ġbetween Ġthe Ġtwo Ġembed ding Ġmodels Ġwith Ġminor Ġperformance Ġvariations . </s> The Ġcomparison Ġof Ġour Ġresult Ġwith ĠWang Ġet Ġal . Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 9 Ġ. Ġas Ġshown , Ġthe Ġresults Ġfrom Ġour Ġmodel Ġshows Ġsignificant Ġimprovement Ġfrom Ġ10 % Ġincrease Ġin ĠF - me asure Ġfor Ġa Ġhigh Ġcount Ġemotion Ġjoy Ġup Ġto Ġ61 . 7 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġa Ġlow Ġcount Ġemotion Ġsurprise . Ġon Ġaverage Ġwe Ġshowed Ġ26 . 8 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġall Ġcategories Ġand Ġmore Ġinterestingly Ġour Ġresult Ġshows Ġvery Ġlittle Ġvariance Ġbetween Ġdifferent Ġemotions Ġcompare Ġto Ġresults Ġreported Ġby ĠWang Ġet Ġal . </s> Model ĠPerform ances Ġon ĠNew ĠDat as et </s> To Ġasses Ġthe Ġperformance Ġof Ġthese Ġmodels Ġon Ġa Ġtotally Ġunseen Ġdata , Ġwe Ġtried Ġto Ġclassify Ġthe ĠCrowd Fl ower Ġemotional Ġtweets Ġdataset . ĠThe ĠCrowd Fl ower Ġdataset Ġconsists Ġof Ġ40 k Ġtweets Ġannot ated Ġvia Ġcrowd - s ourcing Ġeach Ġwith Ġa Ġsingle Ġemotional Ġlabel . ĠThis Ġdataset Ġis Ġconsidered Ġa Ġhard Ġdataset Ġto Ġclassify Ġwith Ġa Ġlot Ġof Ġnoise . ĠThe Ġdistribution Ġof Ġthe Ġdataset Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 18 Ġ. ĠThe Ġlabeling Ġon Ġthis Ġdataset Ġis Ġnon - standard , Ġso Ġwe Ġused Ġthe Ġfollowing Ġmapping Ġfor Ġlabels : </s> s ad ness ĠIN LINE FORM 0 Ġsadness </s> w orry ĠIN LINE FORM 0 Ġfear </s> h appiness ĠIN LINE FORM 0 Ġjoy </s> love ĠIN LINE FORM 0 Ġlove </s> sur prise ĠIN LINE FORM 0 Ġsurprise </s> anger ĠIN LINE FORM 0 Ġanger </s> We Ġthen Ġclassified Ġemotions Ġusing Ġthe Ġpre - trained Ġmodels Ġand Ġemotionally Ġfitted Ġfast Text Ġembed ding . ĠThe Ġresult Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 19 Ġ. ĠThe Ġbaseline Ġresults Ġare Ġfrom ĠB IB REF 33 Ġdone Ġusing ĠB OW Ġmodel Ġand Ġmaximum Ġentropy Ġclass ifier . ĠWe Ġsaw Ġa Ġhuge Ġimprovement Ġfrom Ġ26 Ġpoint Ġincrease Ġin ĠF - me asure Ġfor Ġthe Ġemotion Ġjoy Ġ( h appiness ) Ġup Ġto Ġ57 Ġpoint Ġincrease Ġfor Ġsurprise Ġwith Ġtotal Ġaverage Ġincrease Ġof Ġ38 . 6 Ġpoints . ĠB ost an Ġand ĠKling er Ġdid Ġnot Ġreport Ġclassification Ġresults Ġfor Ġthe Ġemotion Ġlove Ġso Ġwe Ġdid Ġnot Ġinclude Ġit Ġin Ġthe Ġaverage . ĠThese Ġresults Ġshow Ġthat Ġour Ġtrained Ġmodels Ġperform Ġexceptionally Ġon Ġa Ġtotally Ġnew Ġdataset Ġwith Ġa Ġdifferent Ġmethod Ġof Ġannotation . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġhave Ġshown Ġthat Ġusing Ġthe Ġdesigned ĠR NN Ġbased Ġnetwork Ġwe Ġcould Ġincrease Ġthe Ġperformance Ġof Ġclassification Ġdramatically . ĠWe Ġshowed Ġthat Ġkeeping Ġthe Ġsequential Ġnature Ġof Ġthe Ġdata Ġcan Ġbe Ġhugely Ġbeneficial Ġwhen Ġworking Ġwith Ġtextual Ġdata Ġespecially Ġfaced Ġwith Ġthe Ġhard Ġtask Ġof Ġdetecting Ġmore Ġcomplex Ġphenomena Ġlike Ġemotions . ĠWe Ġaccomplished Ġthat Ġby Ġusing Ġa Ġrecurrent Ġnetwork Ġin Ġthe Ġprocess Ġof Ġgenerating Ġour Ġhidden Ġrepresentation . ĠWe Ġhave Ġalso Ġused Ġa Ġmax - pool ing Ġlayer Ġto Ġcapture Ġthe Ġmost Ġrelevant Ġfeatures Ġand Ġan Ġaverage Ġpool ing Ġlayer Ġto Ġcapture Ġthe Ġtext Ġas Ġa Ġwhole Ġproving Ġthat Ġwe Ġcan Ġachieve Ġbetter Ġperformance Ġby Ġfocusing Ġon Ġcreating Ġa Ġmore Ġinformative Ġhidden Ġrepresentation . ĠIn Ġfuture Ġwe Ġcan Ġfocus Ġon Ġimproving Ġthese Ġrepresentations Ġfor Ġexample Ġby Ġusing Ġattention Ġnetworks ĠB IB REF 37 Ġ, ĠB IB REF 38 Ġto Ġcapture Ġa Ġmore Ġcontextual Ġrepresentation Ġor Ġusing Ġlanguage Ġmodel Ġbased Ġmethods Ġlike ĠB ERT ĠB IB REF 39 Ġthat Ġhas Ġbeen Ġshown Ġvery Ġsuccessful Ġin Ġvarious ĠN LP Ġtasks .