WASSA-2017 Shared Task on Emotion Intensity
EXTRACTIVE
We would like to thank the organizers of the WASSA-2017 Shared Task on Emotion Intensity, for providing the data, the guidelines and timely support.
1708.06185
Introduction </s> Twitter , Ġa Ġmicro - blog ging Ġand Ġsocial Ġnetworking Ġsite Ġhas Ġemerged Ġas Ġa Ġplatform Ġwhere Ġpeople Ġexpress Ġthemselves Ġand Ġreact Ġto Ġevents Ġin Ġreal - time . ĠIt Ġis Ġestimated Ġthat Ġnearly Ġ500 Ġmillion Ġtweets Ġare Ġsent Ġper Ġday Ġ. ĠTwitter Ġdata Ġis Ġparticularly Ġinteresting Ġbecause Ġof Ġits Ġpeculiar Ġnature Ġwhere Ġpeople Ġconvey Ġmessages Ġin Ġshort Ġsentences Ġusing Ġhasht ags , Ġemot icons , Ġem oj is Ġetc . ĠIn Ġaddition , Ġeach Ġtweet Ġhas Ġmeta Ġdata Ġlike Ġlocation Ġand Ġlanguage Ġused Ġby Ġthe Ġsender . ĠIt 's Ġchallenging Ġto Ġanalyze Ġthis Ġdata Ġbecause Ġthe Ġtweets Ġmight Ġnot Ġbe Ġgram matically Ġcorrect Ġand Ġthe Ġusers Ġtend Ġto Ġuse Ġinformal Ġand Ġslang Ġwords Ġall Ġthe Ġtime . ĠHence , Ġthis Ġposes Ġan Ġinteresting Ġproblem Ġfor ĠN LP Ġresearchers . ĠAny Ġadvances Ġin Ġusing Ġthis Ġabundant Ġand Ġdiverse Ġdata Ġcan Ġhelp Ġunderstand Ġand Ġanalyze Ġinformation Ġabout Ġa Ġperson , Ġan Ġevent , Ġa Ġproduct , Ġan Ġorganization Ġor Ġa Ġcountry Ġas Ġa Ġwhole . ĠMany Ġnotable Ġuse Ġcases Ġof Ġthe Ġtwitter Ġcan Ġbe Ġfound Ġhere . </s> Along Ġthe Ġsimilar Ġlines , ĠThe ĠTask Ġ1 Ġof ĠWAS SA - 2017 ĠB IB REF 0 Ġposes Ġa Ġproblem Ġof Ġfinding Ġemotion Ġintensity Ġof Ġfour Ġemotions Ġnamely Ġanger , Ġfear , Ġjoy , Ġsadness Ġfrom Ġtweets . ĠIn Ġthis Ġpaper , Ġwe Ġdescribe Ġour Ġapproach Ġand Ġexperiments Ġto Ġsolve Ġthis Ġproblem . ĠThe Ġrest Ġof Ġthe Ġpaper Ġis Ġlaid Ġout Ġas Ġfollows : ĠSection Ġ2 Ġdescribes Ġthe Ġsystem Ġarchitecture , ĠSection Ġ3 Ġreports Ġresults Ġand Ġinference Ġfrom Ġdifferent Ġexperiments , Ġwhile ĠSection Ġ4 Ġpoints Ġto Ġways Ġthat Ġthe Ġproblem Ġcan Ġbe Ġfurther Ġexplored . </s> Pre processing </s> The Ġpre processing Ġstep Ġmod ifies Ġthe Ġraw Ġtweets Ġbefore Ġthey Ġare Ġpassed Ġto Ġfeature Ġextraction . ĠTwe ets Ġare Ġprocessed Ġusing Ġtweet oken ize Ġtool . ĠTwitter Ġspecific Ġfeatures Ġare Ġreplaced Ġas Ġfollows : Ġusername Ġhandles Ġto ĠUS ER NAME , Ġphone Ġnumbers Ġto ĠPH ON EN UM BER , Ġnumbers Ġto ĠNUM BER , ĠURLs Ġto ĠURL Ġand Ġtimes Ġto ĠTIME . ĠA Ġcontinuous Ġsequence Ġof Ġem oj is Ġis Ġbroken Ġinto Ġindividual Ġtokens . ĠFinally , Ġall Ġtokens Ġare Ġconverted Ġto Ġlower case . </s> Feature ĠExt raction </s> Many Ġtasks Ġrelated Ġto Ġsentiment Ġor Ġemotion Ġanalysis Ġdepend Ġupon Ġaffect , Ġopinion , Ġsentiment , Ġsense Ġand Ġemotion Ġlex icons . ĠThese Ġlex icons Ġassociate Ġwords Ġto Ġcorresponding Ġsentiment Ġor Ġemotion Ġmetrics . ĠOn Ġthe Ġother Ġhand , Ġthe Ġsemantic Ġmeaning Ġof Ġwords , Ġsentences , Ġand Ġdocuments Ġare Ġpreserved Ġand Ġcompact ly Ġrepresented Ġusing Ġlow Ġdimensional Ġvectors ĠB IB REF 1 Ġinstead Ġof Ġone Ġhot Ġencoding Ġvectors Ġwhich Ġare Ġsparse Ġand Ġhigh Ġdimensional . ĠFinally , Ġthere Ġare Ġtraditional ĠN LP Ġfeatures Ġlike Ġword ĠN - gram s , Ġcharacter ĠN - gram s , ĠPart - Of - Spe ech ĠN - gram s Ġand Ġword Ġclusters Ġwhich Ġare Ġknown Ġto Ġperform Ġwell Ġon Ġvarious Ġtasks . </s> Based Ġon Ġthese Ġobservations , Ġthe Ġfeature Ġextraction Ġstep Ġis Ġimplemented Ġas Ġa Ġunion Ġof Ġdifferent Ġindependent Ġfeature Ġextract ors Ġ( fe atur izers ) Ġin Ġa Ġlight - weight Ġand Ġeasy Ġto Ġuse ĠPython Ġprogram ĠEm o Int Ġ. ĠIt Ġcomprises Ġof Ġall Ġfeatures Ġavailable Ġin Ġthe Ġbaseline Ġmodel ĠB IB REF 2 Ġalong Ġwith Ġadditional Ġfeature Ġextract ors Ġand Ġbi - gram Ġsupport . ĠFour teen Ġsuch Ġfeature Ġextract ors Ġhave Ġbeen Ġimplemented Ġwhich Ġcan Ġbe Ġclub bed Ġinto Ġ3 Ġmajor Ġcategories : </s> [ no item se p ] </s> Lex icon ĠFeatures </s> Word ĠV ect ors </s> Sy ntax ĠFeatures </s> Lex icon ĠFeatures : ĠAF IN N ĠB IB REF 3 Ġword Ġlist Ġare Ġmanually Ġrated Ġfor Ġval ence Ġwith Ġan Ġinteger Ġbetween Ġ- 5 Ġ( Neg ative ĠSent iment ) Ġand Ġ+ 5 Ġ( Pos itive ĠSent iment ). ĠBing ĠLiu ĠB IB REF 4 Ġopinion Ġlex icon Ġextract Ġopinion Ġon Ġcustomer Ġreviews . Ġ+/- Effect Word Net ĠB IB REF 5 Ġby ĠMP Q A Ġgroup Ġare Ġsense Ġlevel Ġlex icons . ĠThe ĠN RC ĠAffect ĠInt ensity ĠB IB REF 6 Ġlex icons Ġprovide Ġreal Ġvalued Ġaffect Ġintensity . ĠN RC ĠWord - Em otion ĠAssociation ĠLex icon ĠB IB REF 7 Ġcontains Ġ8 Ġsense Ġlevel Ġassociations Ġ( anger , Ġfear , Ġanticipation , Ġtrust , Ġsurprise , Ġsadness , Ġjoy , Ġand Ġdisgust ) Ġand Ġ2 Ġsentiment Ġlevel Ġassociations Ġ( negative Ġand Ġpositive ). ĠExp anded ĠN RC ĠWord - Em otion ĠAssociation ĠLex icon ĠB IB REF 8 Ġexpands Ġthe ĠN RC Ġword - em otion Ġassociation Ġlex icon Ġfor Ġtwitter Ġspecific Ġlanguage . ĠN RC ĠHash tag ĠEm otion ĠLex icon ĠB IB REF 9 Ġcontains Ġemotion Ġword Ġassociations Ġcomputed Ġon Ġemotion Ġlabeled Ġtwitter Ġcorpus Ġvia ĠHash tags . ĠN RC ĠHash tag ĠSent iment ĠLex icon Ġand ĠSent iment 140 ĠLex icon ĠB IB REF 10 Ġcontains Ġsentiment Ġword Ġassociations Ġcomputed Ġon Ġtwitter Ġcorpus Ġvia ĠHash tags Ġand ĠEm otic ons . ĠSent i Word Net ĠB IB REF 11 Ġassigns Ġto Ġeach Ġsyn set Ġof ĠWord Net Ġthree Ġsentiment Ġscores : Ġpos itivity , Ġnegativity , Ġobject ivity . ĠNeg ation Ġlex icons Ġcollections Ġare Ġused Ġto Ġcount Ġthe Ġtotal Ġoccurrence Ġof Ġnegative Ġwords . ĠIn Ġaddition Ġto Ġthese , ĠSent i Strength ĠB IB REF 12 Ġapplication Ġwhich Ġestimates Ġthe Ġstrength Ġof Ġpositive Ġand Ġnegative Ġsentiment Ġfrom Ġtweets Ġis Ġalso Ġadded . </s> Word ĠV ect ors : ĠWe Ġfocus Ġprimarily Ġon Ġthe Ġword Ġvector Ġrepresentations Ġ( word Ġembed d ings ) Ġcreated Ġspecifically Ġusing Ġthe Ġtwitter Ġdataset . ĠGl o Ve ĠB IB REF 13 Ġis Ġan Ġun super vised Ġlearning Ġalgorithm Ġfor Ġobtaining Ġvector Ġrepresentations Ġfor Ġwords . Ġ200 - dimensional ĠGl o Ve Ġembed d ings Ġtrained Ġon Ġ2 ĠBillion Ġtweets Ġare Ġintegrated . ĠEdinburgh Ġembed d ings ĠB IB REF 14 Ġare Ġobtained Ġby Ġtraining Ġskip - gram Ġmodel Ġon ĠEdinburgh Ġcorpus ĠB IB REF 15 Ġ. ĠSince Ġtweets Ġare Ġabundant Ġwith Ġem oj is , ĠEm oji Ġembed d ings ĠB IB REF 16 Ġwhich Ġare Ġlearned Ġfrom Ġthe Ġemoji Ġdescriptions Ġhave Ġbeen Ġused . ĠEmb edd ings Ġfor Ġeach Ġtweet Ġare Ġobtained Ġby Ġsum ming Ġup Ġindividual Ġword Ġvectors Ġand Ġthen Ġdividing Ġby Ġthe Ġnumber Ġof Ġtokens Ġin Ġthe Ġtweet . </s> Sy nt actic ĠFeatures : ĠSynt ax Ġspecific Ġfeatures Ġsuch Ġas ĠWord ĠN - gram s , ĠPart - Of - Spe ech ĠN - gram s ĠB IB REF 17 Ġ, ĠBrown ĠCluster ĠN - gram s ĠB IB REF 18 Ġobtained Ġusing ĠTweet N LP Ġproject Ġhave Ġbeen Ġintegrated Ġinto Ġthe Ġsystem . </s> The Ġfinal Ġfeature Ġvector Ġis Ġthe Ġconc aten ation Ġof Ġall Ġthe Ġindividual Ġfeatures . ĠFor Ġexample , Ġwe Ġconc aten ate Ġaverage Ġword Ġvectors , Ġsum Ġof ĠN RC ĠAffect ĠInt ens ities , Ġnumber Ġof Ġpositive Ġand Ġnegative ĠBing ĠLiu Ġlex icons , Ġnumber Ġof Ġneg ation Ġwords Ġand Ġso Ġon Ġto Ġget Ġfinal Ġfeature Ġvector . ĠThe Ġscaling Ġof Ġfinal Ġfeatures Ġis Ġnot Ġrequired Ġwhen Ġused Ġwith Ġgradient Ġboosted Ġtrees . ĠHowever , Ġscaling Ġsteps Ġlike Ġstandard Ġscaling Ġ( zero Ġmean Ġand Ġunit Ġnormal ) Ġmay Ġbe Ġbeneficial Ġfor Ġneural Ġnetworks Ġas Ġthe Ġoptim izers Ġwork Ġwell Ġwhen Ġthe Ġdata Ġis Ġcentered Ġaround Ġorigin . </s> A Ġtotal Ġof Ġfourteen Ġdifferent Ġfeature Ġextract ors Ġhave Ġbeen Ġimplemented , Ġall Ġof Ġwhich Ġcan Ġbe Ġenabled Ġor Ġdisabled Ġindividually Ġto Ġextract Ġfeatures Ġfrom Ġa Ġgiven Ġtweet . </s> Reg ression </s> The Ġdev Ġdata Ġset ĠB IB REF 19 Ġin Ġthe Ġcompetition Ġwas Ġsmall Ġhence , Ġthe Ġtrain Ġand Ġdev Ġsets Ġwere Ġmerged Ġto Ġperform Ġ10 - fold Ġcross Ġvalidation . ĠOn Ġeach Ġfold , Ġa Ġmodel Ġwas Ġtrained Ġand Ġthe Ġpredictions Ġwere Ġcollected Ġon Ġthe Ġremaining Ġdataset . ĠThe Ġpredictions Ġare Ġaveraged Ġacross Ġall Ġthe Ġfolds Ġto Ġgeneral ize Ġthe Ġsolution Ġand Ġprevent Ġover - fitting . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġ, Ġdifferent Ġcombinations Ġof Ġfeature Ġextract ors Ġwere Ġused . ĠAfter Ġperforming Ġfeature Ġextraction , Ġthe Ġdata Ġwas Ġthen Ġpassed Ġto Ġvarious Ġregress ors ĠSupport ĠVector ĠReg ression , ĠAda Boost , ĠRandom Forest Reg ressor , Ġand , ĠB agging Reg ressor Ġof Ġsk learn ĠB IB REF 20 Ġ. ĠFinally , Ġthe Ġchosen Ġtop Ġperforming Ġmodels Ġhad Ġthe Ġleast Ġerror Ġon Ġevaluation Ġmetrics Ġnamely ĠPearson 's ĠCor relation ĠCo efficient Ġand ĠSpear man 's Ġrank - order Ġcorrelation . </s> Parameter ĠOptim ization </s> In Ġorder Ġto Ġfind Ġthe Ġoptimal Ġparameter Ġvalues Ġfor Ġthe ĠEm o Int Ġsystem , Ġan Ġextensive Ġgrid Ġsearch Ġwas Ġperformed Ġthrough Ġthe Ġsc ik it - Learn Ġframework Ġover Ġall Ġsubs ets Ġof Ġthe Ġtraining Ġset Ġ( sh uff led ), Ġusing Ġstrat ified Ġ10 - fold Ġcross Ġvalidation Ġand Ġoptimizing Ġthe ĠPearson 's ĠCor relation Ġscore . ĠBest Ġcross - valid ation Ġresults Ġwere Ġobtained Ġusing ĠAda Boost Ġmeta Ġreg ressor Ġwith Ġbase Ġreg ressor Ġas ĠX GB o ost ĠB IB REF 21 Ġwith Ġ1000 Ġestim ators Ġand Ġ0 . 1 Ġlearning Ġrate . ĠExper iments Ġand Ġanalysis Ġof Ġresults Ġare Ġpresented Ġin Ġthe Ġnext Ġsection . </s> Exper imental ĠResults </s> As Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġvarious Ġsyntax Ġfeatures Ġwere Ġused Ġnamely , ĠPart - of - Spe ech Ġtags , Ġbrown Ġclusters Ġof ĠTweet N LP Ġproject . ĠHowever , Ġthese Ġdidn 't Ġperform Ġwell Ġin Ġcross Ġvalidation . ĠHence , Ġthey Ġwere Ġdropped Ġfrom Ġthe Ġfinal Ġsystem . ĠWhile Ġperforming Ġgrid - search Ġas Ġmentioned Ġin ĠSection ĠSEC REF 14 Ġ, Ġkeeping Ġall Ġthe Ġlex icon Ġbased Ġfeatures Ġsame , Ġchoice Ġof Ġcombination Ġof Ġemoji Ġvector Ġand Ġword Ġvectors Ġare Ġvaried Ġto Ġminimize Ġcross Ġvalidation Ġmetric . ĠTable ĠT AB REF 16 Ġdescribes Ġthe Ġresults Ġfor Ġexperiments Ġconducted Ġwith Ġdifferent Ġcombinations Ġof Ġword Ġvectors . ĠEm oji Ġembed d ings ĠB IB REF 16 Ġgive Ġbetter Ġresults Ġthan Ġusing Ġplain ĠGl o Ve Ġand ĠEdinburgh Ġembed d ings . ĠEdinburgh Ġembed d ings Ġoutper form ĠGl o Ve Ġembed d ings Ġin ĠJoy Ġand ĠSad ness Ġcategory Ġbut Ġlag Ġbehind Ġin ĠAnger Ġand ĠFear Ġcategory . ĠThe Ġofficial Ġsubmission Ġcomprised Ġof Ġthe Ġtop - performing Ġmodel Ġfor Ġeach Ġemotion Ġcategory . ĠThis Ġsystem Ġranked Ġ3 Ġfor Ġthe Ġentire Ġtest Ġdataset Ġand Ġ2 Ġfor Ġthe Ġsubset Ġof Ġthe Ġtest Ġdata Ġformed Ġby Ġtaking Ġevery Ġinstance Ġwith Ġa Ġgold Ġemotion Ġintensity Ġscore Ġgreater Ġthan Ġor Ġequal Ġto Ġ0 . 5 . ĠPost Ġcompetition , Ġexperiments Ġwere Ġperformed Ġon Ġen semb ling Ġdiverse Ġmodels Ġfor Ġimproving Ġthe Ġaccuracy . ĠAn Ġensemble Ġobtained Ġby Ġaveraging Ġthe Ġresults Ġof Ġthe Ġtop Ġ2 Ġperforming Ġmodels Ġoutper forms Ġall Ġthe Ġindividual Ġmodels . </s> Feature ĠImport ance </s> The Ġrelative Ġfeature Ġimportance Ġcan Ġbe Ġassessed Ġby Ġthe Ġrelative Ġdepth Ġof Ġthe Ġfeature Ġused Ġas Ġa Ġdecision Ġnode Ġin Ġthe Ġtree . ĠFeatures Ġused Ġat Ġthe Ġtop Ġof Ġthe Ġtree Ġcontribute Ġto Ġthe Ġfinal Ġprediction Ġdecision Ġof Ġa Ġlarger Ġfraction Ġof Ġthe Ġinput Ġsamples . ĠThe Ġexpected Ġfraction Ġof Ġthe Ġsamples Ġthey Ġcontribute Ġto Ġcan Ġthus Ġbe Ġused Ġas Ġan Ġestimate Ġof Ġthe Ġrelative Ġimportance Ġof Ġthe Ġfeatures . ĠBy Ġaveraging Ġthe Ġmeasure Ġover Ġseveral Ġrandomized Ġtrees , Ġthe Ġvariance Ġof Ġthe Ġestimate Ġcan Ġbe Ġreduced Ġand Ġused Ġas Ġa Ġmeasure Ġof Ġrelative Ġfeature Ġimportance . ĠIn ĠFigure ĠFIG REF 18 Ġfeature Ġimportance Ġgraphs Ġare Ġplotted Ġfor Ġeach Ġemotion Ġto Ġinfer Ġwhich Ġfeatures Ġare Ġplaying Ġthe Ġmajor Ġrole Ġin Ġidentifying Ġemotional Ġintensity Ġin Ġtweets . Ġ+/- Effect Word Net ĠB IB REF 5 Ġ, ĠN RC ĠHash tag ĠSent iment ĠLex icon , ĠSent iment 140 ĠLex icon ĠB IB REF 10 Ġand ĠN RC ĠHash tag ĠEm otion ĠLex icon ĠB IB REF 9 Ġare Ġplaying Ġthe Ġmost Ġimportant Ġrole . </s> System ĠLim itations </s> It Ġis Ġimportant Ġto Ġunderstand Ġhow Ġthe Ġmodel Ġperforms Ġin Ġdifferent Ġscenarios . ĠTable ĠT AB REF 20 Ġanaly zes Ġwhen Ġthe Ġsystem Ġperforms Ġthe Ġbest Ġand Ġworst Ġfor Ġeach Ġemotion . ĠSince Ġthe Ġfeatures Ġused Ġare Ġmostly Ġlex icon Ġbased , Ġthe Ġsystem Ġhas Ġdifficulties Ġin Ġcapturing Ġthe Ġoverall Ġsentiment Ġand Ġit Ġleads Ġto Ġampl ifying Ġor Ġvanishing Ġintensity Ġsignals . ĠFor Ġinstance , Ġin Ġexample Ġ4 Ġof Ġfear Ġlouder Ġand Ġshaking Ġlex icons Ġimply Ġfear Ġbut Ġoverall Ġsentence Ġdoesn 't Ġimply Ġfear . ĠA Ġsimilar Ġpattern Ġcan Ġbe Ġfound Ġin Ġthe Ġ4 Ġexample Ġof ĠAnger Ġand Ġ3 Ġexample Ġof ĠJoy . ĠThe Ġsystem Ġhas Ġdifficulties Ġin Ġunderstanding Ġof Ġsarcastic Ġtweets , Ġfor Ġinstance , Ġin Ġthe Ġ3 Ġtweet Ġof ĠAnger Ġthe Ġuser Ġexpressed Ġanger Ġbut Ġused Ġlol Ġwhich Ġis Ġused Ġin Ġa Ġpositive Ġsense Ġmost Ġof Ġthe Ġtimes Ġand Ġhence Ġthe Ġsystem Ġdid Ġa Ġbad Ġjob Ġat Ġpredicting Ġintensity . ĠThe Ġsystem Ġalso Ġfails Ġin Ġpredicting Ġsentences Ġhaving Ġdeeper Ġemotion Ġand Ġsentiment Ġwhich Ġhumans Ġcan Ġunderstand Ġwith Ġa Ġlittle Ġcontext . ĠFor Ġexample , Ġin Ġsample Ġ4 Ġof Ġsadness , Ġthe Ġtweet Ġrefers Ġto Ġpost Ġtravel Ġblues Ġwhich Ġhumans Ġcan Ġunderstand . ĠBut Ġwith Ġlittle Ġcontext , Ġit Ġis Ġdifficult Ġfor Ġthe Ġsystem Ġto Ġaccurately Ġestimate Ġthe Ġintensity . ĠThe Ġperformance Ġis Ġpoor Ġwith Ġvery Ġshort Ġsentences Ġas Ġthere Ġare Ġfewer Ġindicators Ġto Ġprovide Ġa Ġreasonable Ġestimate . </s> Future ĠWork Ġ& ĠConclusion </s> The Ġpaper Ġstudies Ġthe Ġeffectiveness Ġof Ġvarious Ġaffect Ġlex icons Ġword Ġembed d ings Ġto Ġestimate Ġemotional Ġintensity Ġin Ġtweets . ĠA Ġlight - weight Ġeasy Ġto Ġuse Ġaffect Ġcomputing Ġframework Ġ( E mo Int ) Ġto Ġfacilitate Ġease Ġof Ġexperimenting Ġwith Ġvarious Ġlex icon Ġfeatures Ġfor Ġtext Ġtasks Ġis Ġopen - s ourced . ĠIt Ġprovides Ġplug Ġand Ġplay Ġaccess Ġto Ġvarious Ġfeature Ġextract ors Ġand Ġhandy Ġscripts Ġfor Ġcreating Ġen semb les . </s> Few Ġproblems Ġexplained Ġin Ġthe Ġanalysis Ġsection Ġcan Ġbe Ġresolved Ġwith Ġthe Ġhelp Ġof Ġsentence Ġembed d ings Ġwhich Ġtake Ġthe Ġcontext Ġinformation Ġinto Ġconsideration . ĠThe Ġfeatures Ġused Ġin Ġthe Ġsystem Ġare Ġgeneric Ġenough Ġto Ġuse Ġthem Ġin Ġother Ġaffect ive Ġcomputing Ġtasks Ġon Ġsocial Ġmedia Ġtext , Ġnot Ġjust Ġtweet Ġdata . ĠAnother Ġinteresting Ġfeature Ġof Ġlex icon - based Ġsystems Ġis Ġtheir Ġgood Ġrun - time Ġperformance Ġduring Ġprediction , Ġfuture Ġwork Ġto Ġbenchmark Ġthe Ġperformance Ġof Ġthe Ġsystem Ġcan Ġprove Ġvital Ġfor Ġdeploying Ġin Ġa Ġreal - world Ġsetting . </s> Acknowled gement </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġorganizers Ġof Ġthe ĠWAS SA - 2017 ĠShared ĠTask Ġon ĠEm otion ĠInt ensity , Ġfor Ġproviding Ġthe Ġdata , Ġthe Ġguidelines Ġand Ġtimely Ġsupport .
Introduction Twitter, a micro-blogging and social networking site has emerged as a platform where people express themselves and react to events in real-time. It is estimated that nearly 500 million tweets are sent per day . Twitter data is particularly interesting because of its peculiar nature where people convey messages in short sentences using hashtags, emoticons, emojis etc. In addition, each tweet has meta data like location and language used by the sender. It's challenging to analyze this data because the tweets might not be grammatically correct and the users tend to use informal and slang words all the time. Hence, this poses an interesting problem for NLP researchers. Any advances in using this abundant and diverse data can help understand and analyze information about a person, an event, a product, an organization or a country as a whole. Many notable use cases of the twitter can be found here. Along the similar lines, The Task 1 of WASSA-2017 BIBREF0 poses a problem of finding emotion intensity of four emotions namely anger, fear, joy, sadness from tweets. In this paper, we describe our approach and experiments to solve this problem. The rest of the paper is laid out as follows: Section 2 describes the system architecture, Section 3 reports results and inference from different experiments, while Section 4 points to ways that the problem can be further explored. Preprocessing The preprocessing step modifies the raw tweets before they are passed to feature extraction. Tweets are processed using tweetokenize tool. Twitter specific features are replaced as follows: username handles to USERNAME, phone numbers to PHONENUMBER, numbers to NUMBER, URLs to URL and times to TIME. A continuous sequence of emojis is broken into individual tokens. Finally, all tokens are converted to lowercase. Feature Extraction Many tasks related to sentiment or emotion analysis depend upon affect, opinion, sentiment, sense and emotion lexicons. These lexicons associate words to corresponding sentiment or emotion metrics. On the other hand, the semantic meaning of words, sentences, and documents are preserved and compactly represented using low dimensional vectors BIBREF1 instead of one hot encoding vectors which are sparse and high dimensional. Finally, there are traditional NLP features like word N-grams, character N-grams, Part-Of-Speech N-grams and word clusters which are known to perform well on various tasks. Based on these observations, the feature extraction step is implemented as a union of different independent feature extractors (featurizers) in a light-weight and easy to use Python program EmoInt . It comprises of all features available in the baseline model BIBREF2 along with additional feature extractors and bi-gram support. Fourteen such feature extractors have been implemented which can be clubbed into 3 major categories: [noitemsep] Lexicon Features Word Vectors Syntax Features Lexicon Features: AFINN BIBREF3 word list are manually rated for valence with an integer between -5 (Negative Sentiment) and +5 (Positive Sentiment). Bing Liu BIBREF4 opinion lexicon extract opinion on customer reviews. +/-EffectWordNet BIBREF5 by MPQA group are sense level lexicons. The NRC Affect Intensity BIBREF6 lexicons provide real valued affect intensity. NRC Word-Emotion Association Lexicon BIBREF7 contains 8 sense level associations (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and 2 sentiment level associations (negative and positive). Expanded NRC Word-Emotion Association Lexicon BIBREF8 expands the NRC word-emotion association lexicon for twitter specific language. NRC Hashtag Emotion Lexicon BIBREF9 contains emotion word associations computed on emotion labeled twitter corpus via Hashtags. NRC Hashtag Sentiment Lexicon and Sentiment140 Lexicon BIBREF10 contains sentiment word associations computed on twitter corpus via Hashtags and Emoticons. SentiWordNet BIBREF11 assigns to each synset of WordNet three sentiment scores: positivity, negativity, objectivity. Negation lexicons collections are used to count the total occurrence of negative words. In addition to these, SentiStrength BIBREF12 application which estimates the strength of positive and negative sentiment from tweets is also added. Word Vectors: We focus primarily on the word vector representations (word embeddings) created specifically using the twitter dataset. GloVe BIBREF13 is an unsupervised learning algorithm for obtaining vector representations for words. 200-dimensional GloVe embeddings trained on 2 Billion tweets are integrated. Edinburgh embeddings BIBREF14 are obtained by training skip-gram model on Edinburgh corpus BIBREF15 . Since tweets are abundant with emojis, Emoji embeddings BIBREF16 which are learned from the emoji descriptions have been used. Embeddings for each tweet are obtained by summing up individual word vectors and then dividing by the number of tokens in the tweet. Syntactic Features: Syntax specific features such as Word N-grams, Part-Of-Speech N-grams BIBREF17 , Brown Cluster N-grams BIBREF18 obtained using TweetNLP project have been integrated into the system. The final feature vector is the concatenation of all the individual features. For example, we concatenate average word vectors, sum of NRC Affect Intensities, number of positive and negative Bing Liu lexicons, number of negation words and so on to get final feature vector. The scaling of final features is not required when used with gradient boosted trees. However, scaling steps like standard scaling (zero mean and unit normal) may be beneficial for neural networks as the optimizers work well when the data is centered around origin. A total of fourteen different feature extractors have been implemented, all of which can be enabled or disabled individually to extract features from a given tweet. Regression The dev data set BIBREF19 in the competition was small hence, the train and dev sets were merged to perform 10-fold cross validation. On each fold, a model was trained and the predictions were collected on the remaining dataset. The predictions are averaged across all the folds to generalize the solution and prevent over-fitting. As described in Section SECREF6 , different combinations of feature extractors were used. After performing feature extraction, the data was then passed to various regressors Support Vector Regression, AdaBoost, RandomForestRegressor, and, BaggingRegressor of sklearn BIBREF20 . Finally, the chosen top performing models had the least error on evaluation metrics namely Pearson's Correlation Coefficient and Spearman's rank-order correlation. Parameter Optimization In order to find the optimal parameter values for the EmoInt system, an extensive grid search was performed through the scikit-Learn framework over all subsets of the training set (shuffled), using stratified 10-fold cross validation and optimizing the Pearson's Correlation score. Best cross-validation results were obtained using AdaBoost meta regressor with base regressor as XGBoost BIBREF21 with 1000 estimators and 0.1 learning rate. Experiments and analysis of results are presented in the next section. Experimental Results As described in Section SECREF6 various syntax features were used namely, Part-of-Speech tags, brown clusters of TweetNLP project. However, these didn't perform well in cross validation. Hence, they were dropped from the final system. While performing grid-search as mentioned in Section SECREF14 , keeping all the lexicon based features same, choice of combination of emoji vector and word vectors are varied to minimize cross validation metric. Table TABREF16 describes the results for experiments conducted with different combinations of word vectors. Emoji embeddings BIBREF16 give better results than using plain GloVe and Edinburgh embeddings. Edinburgh embeddings outperform GloVe embeddings in Joy and Sadness category but lag behind in Anger and Fear category. The official submission comprised of the top-performing model for each emotion category. This system ranked 3 for the entire test dataset and 2 for the subset of the test data formed by taking every instance with a gold emotion intensity score greater than or equal to 0.5. Post competition, experiments were performed on ensembling diverse models for improving the accuracy. An ensemble obtained by averaging the results of the top 2 performing models outperforms all the individual models. Feature Importance The relative feature importance can be assessed by the relative depth of the feature used as a decision node in the tree. Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features. By averaging the measure over several randomized trees, the variance of the estimate can be reduced and used as a measure of relative feature importance. In Figure FIGREF18 feature importance graphs are plotted for each emotion to infer which features are playing the major role in identifying emotional intensity in tweets. +/-EffectWordNet BIBREF5 , NRC Hashtag Sentiment Lexicon, Sentiment140 Lexicon BIBREF10 and NRC Hashtag Emotion Lexicon BIBREF9 are playing the most important role. System Limitations It is important to understand how the model performs in different scenarios. Table TABREF20 analyzes when the system performs the best and worst for each emotion. Since the features used are mostly lexicon based, the system has difficulties in capturing the overall sentiment and it leads to amplifying or vanishing intensity signals. For instance, in example 4 of fear louder and shaking lexicons imply fear but overall sentence doesn't imply fear. A similar pattern can be found in the 4 example of Anger and 3 example of Joy. The system has difficulties in understanding of sarcastic tweets, for instance, in the 3 tweet of Anger the user expressed anger but used lol which is used in a positive sense most of the times and hence the system did a bad job at predicting intensity. The system also fails in predicting sentences having deeper emotion and sentiment which humans can understand with a little context. For example, in sample 4 of sadness, the tweet refers to post travel blues which humans can understand. But with little context, it is difficult for the system to accurately estimate the intensity. The performance is poor with very short sentences as there are fewer indicators to provide a reasonable estimate. Future Work & Conclusion The paper studies the effectiveness of various affect lexicons word embeddings to estimate emotional intensity in tweets. A light-weight easy to use affect computing framework (EmoInt) to facilitate ease of experimenting with various lexicon features for text tasks is open-sourced. It provides plug and play access to various feature extractors and handy scripts for creating ensembles. Few problems explained in the analysis section can be resolved with the help of sentence embeddings which take the context information into consideration. The features used in the system are generic enough to use them in other affective computing tasks on social media text, not just tweet data. Another interesting feature of lexicon-based systems is their good run-time performance during prediction, future work to benchmark the performance of the system can prove vital for deploying in a real-world setting. Acknowledgement We would like to thank the organizers of the WASSA-2017 Shared Task on Emotion Intensity, for providing the data, the guidelines and timely support.
what dataset was used?
e48e750743aef36529fbea4328b8253dbe928b4d
what Ġdataset Ġwas Ġused ?
W AS SA - 2017 ĠShared ĠTask Ġon ĠEm otion ĠInt ensity
<s> what Ġdataset Ġwas Ġused ? </s> Introduction </s> Twitter , Ġa Ġmicro - blog ging Ġand Ġsocial Ġnetworking Ġsite Ġhas Ġemerged Ġas Ġa Ġplatform Ġwhere Ġpeople Ġexpress Ġthemselves Ġand Ġreact Ġto Ġevents Ġin Ġreal - time . ĠIt Ġis Ġestimated Ġthat Ġnearly Ġ500 Ġmillion Ġtweets Ġare Ġsent Ġper Ġday Ġ. ĠTwitter Ġdata Ġis Ġparticularly Ġinteresting Ġbecause Ġof Ġits Ġpeculiar Ġnature Ġwhere Ġpeople Ġconvey Ġmessages Ġin Ġshort Ġsentences Ġusing Ġhasht ags , Ġemot icons , Ġem oj is Ġetc . ĠIn Ġaddition , Ġeach Ġtweet Ġhas Ġmeta Ġdata Ġlike Ġlocation Ġand Ġlanguage Ġused Ġby Ġthe Ġsender . ĠIt 's Ġchallenging Ġto Ġanalyze Ġthis Ġdata Ġbecause Ġthe Ġtweets Ġmight Ġnot Ġbe Ġgram matically Ġcorrect Ġand Ġthe Ġusers Ġtend Ġto Ġuse Ġinformal Ġand Ġslang Ġwords Ġall Ġthe Ġtime . ĠHence , Ġthis Ġposes Ġan Ġinteresting Ġproblem Ġfor ĠN LP Ġresearchers . ĠAny Ġadvances Ġin Ġusing Ġthis Ġabundant Ġand Ġdiverse Ġdata Ġcan Ġhelp Ġunderstand Ġand Ġanalyze Ġinformation Ġabout Ġa Ġperson , Ġan Ġevent , Ġa Ġproduct , Ġan Ġorganization Ġor Ġa Ġcountry Ġas Ġa Ġwhole . ĠMany Ġnotable Ġuse Ġcases Ġof Ġthe Ġtwitter Ġcan Ġbe Ġfound Ġhere . </s> Along Ġthe Ġsimilar Ġlines , ĠThe ĠTask Ġ1 Ġof ĠWAS SA - 2017 ĠB IB REF 0 Ġposes Ġa Ġproblem Ġof Ġfinding Ġemotion Ġintensity Ġof Ġfour Ġemotions Ġnamely Ġanger , Ġfear , Ġjoy , Ġsadness Ġfrom Ġtweets . ĠIn Ġthis Ġpaper , Ġwe Ġdescribe Ġour Ġapproach Ġand Ġexperiments Ġto Ġsolve Ġthis Ġproblem . ĠThe Ġrest Ġof Ġthe Ġpaper Ġis Ġlaid Ġout Ġas Ġfollows : ĠSection Ġ2 Ġdescribes Ġthe Ġsystem Ġarchitecture , ĠSection Ġ3 Ġreports Ġresults Ġand Ġinference Ġfrom Ġdifferent Ġexperiments , Ġwhile ĠSection Ġ4 Ġpoints Ġto Ġways Ġthat Ġthe Ġproblem Ġcan Ġbe Ġfurther Ġexplored . </s> Pre processing </s> The Ġpre processing Ġstep Ġmod ifies Ġthe Ġraw Ġtweets Ġbefore Ġthey Ġare Ġpassed Ġto Ġfeature Ġextraction . ĠTwe ets Ġare Ġprocessed Ġusing Ġtweet oken ize Ġtool . ĠTwitter Ġspecific Ġfeatures Ġare Ġreplaced Ġas Ġfollows : Ġusername Ġhandles Ġto ĠUS ER NAME , Ġphone Ġnumbers Ġto ĠPH ON EN UM BER , Ġnumbers Ġto ĠNUM BER , ĠURLs Ġto ĠURL Ġand Ġtimes Ġto ĠTIME . ĠA Ġcontinuous Ġsequence Ġof Ġem oj is Ġis Ġbroken Ġinto Ġindividual Ġtokens . ĠFinally , Ġall Ġtokens Ġare Ġconverted Ġto Ġlower case . </s> Feature ĠExt raction </s> Many Ġtasks Ġrelated Ġto Ġsentiment Ġor Ġemotion Ġanalysis Ġdepend Ġupon Ġaffect , Ġopinion , Ġsentiment , Ġsense Ġand Ġemotion Ġlex icons . ĠThese Ġlex icons Ġassociate Ġwords Ġto Ġcorresponding Ġsentiment Ġor Ġemotion Ġmetrics . ĠOn Ġthe Ġother Ġhand , Ġthe Ġsemantic Ġmeaning Ġof Ġwords , Ġsentences , Ġand Ġdocuments Ġare Ġpreserved Ġand Ġcompact ly Ġrepresented Ġusing Ġlow Ġdimensional Ġvectors ĠB IB REF 1 Ġinstead Ġof Ġone Ġhot Ġencoding Ġvectors Ġwhich Ġare Ġsparse Ġand Ġhigh Ġdimensional . ĠFinally , Ġthere Ġare Ġtraditional ĠN LP Ġfeatures Ġlike Ġword ĠN - gram s , Ġcharacter ĠN - gram s , ĠPart - Of - Spe ech ĠN - gram s Ġand Ġword Ġclusters Ġwhich Ġare Ġknown Ġto Ġperform Ġwell Ġon Ġvarious Ġtasks . </s> Based Ġon Ġthese Ġobservations , Ġthe Ġfeature Ġextraction Ġstep Ġis Ġimplemented Ġas Ġa Ġunion Ġof Ġdifferent Ġindependent Ġfeature Ġextract ors Ġ( fe atur izers ) Ġin Ġa Ġlight - weight Ġand Ġeasy Ġto Ġuse ĠPython Ġprogram ĠEm o Int Ġ. ĠIt Ġcomprises Ġof Ġall Ġfeatures Ġavailable Ġin Ġthe Ġbaseline Ġmodel ĠB IB REF 2 Ġalong Ġwith Ġadditional Ġfeature Ġextract ors Ġand Ġbi - gram Ġsupport . ĠFour teen Ġsuch Ġfeature Ġextract ors Ġhave Ġbeen Ġimplemented Ġwhich Ġcan Ġbe Ġclub bed Ġinto Ġ3 Ġmajor Ġcategories : </s> [ no item se p ] </s> Lex icon ĠFeatures </s> Word ĠV ect ors </s> Sy ntax ĠFeatures </s> Lex icon ĠFeatures : ĠAF IN N ĠB IB REF 3 Ġword Ġlist Ġare Ġmanually Ġrated Ġfor Ġval ence Ġwith Ġan Ġinteger Ġbetween Ġ- 5 Ġ( Neg ative ĠSent iment ) Ġand Ġ+ 5 Ġ( Pos itive ĠSent iment ). ĠBing ĠLiu ĠB IB REF 4 Ġopinion Ġlex icon Ġextract Ġopinion Ġon Ġcustomer Ġreviews . Ġ+/- Effect Word Net ĠB IB REF 5 Ġby ĠMP Q A Ġgroup Ġare Ġsense Ġlevel Ġlex icons . ĠThe ĠN RC ĠAffect ĠInt ensity ĠB IB REF 6 Ġlex icons Ġprovide Ġreal Ġvalued Ġaffect Ġintensity . ĠN RC ĠWord - Em otion ĠAssociation ĠLex icon ĠB IB REF 7 Ġcontains Ġ8 Ġsense Ġlevel Ġassociations Ġ( anger , Ġfear , Ġanticipation , Ġtrust , Ġsurprise , Ġsadness , Ġjoy , Ġand Ġdisgust ) Ġand Ġ2 Ġsentiment Ġlevel Ġassociations Ġ( negative Ġand Ġpositive ). ĠExp anded ĠN RC ĠWord - Em otion ĠAssociation ĠLex icon ĠB IB REF 8 Ġexpands Ġthe ĠN RC Ġword - em otion Ġassociation Ġlex icon Ġfor Ġtwitter Ġspecific Ġlanguage . ĠN RC ĠHash tag ĠEm otion ĠLex icon ĠB IB REF 9 Ġcontains Ġemotion Ġword Ġassociations Ġcomputed Ġon Ġemotion Ġlabeled Ġtwitter Ġcorpus Ġvia ĠHash tags . ĠN RC ĠHash tag ĠSent iment ĠLex icon Ġand ĠSent iment 140 ĠLex icon ĠB IB REF 10 Ġcontains Ġsentiment Ġword Ġassociations Ġcomputed Ġon Ġtwitter Ġcorpus Ġvia ĠHash tags Ġand ĠEm otic ons . ĠSent i Word Net ĠB IB REF 11 Ġassigns Ġto Ġeach Ġsyn set Ġof ĠWord Net Ġthree Ġsentiment Ġscores : Ġpos itivity , Ġnegativity , Ġobject ivity . ĠNeg ation Ġlex icons Ġcollections Ġare Ġused Ġto Ġcount Ġthe Ġtotal Ġoccurrence Ġof Ġnegative Ġwords . ĠIn Ġaddition Ġto Ġthese , ĠSent i Strength ĠB IB REF 12 Ġapplication Ġwhich Ġestimates Ġthe Ġstrength Ġof Ġpositive Ġand Ġnegative Ġsentiment Ġfrom Ġtweets Ġis Ġalso Ġadded . </s> Word ĠV ect ors : ĠWe Ġfocus Ġprimarily Ġon Ġthe Ġword Ġvector Ġrepresentations Ġ( word Ġembed d ings ) Ġcreated Ġspecifically Ġusing Ġthe Ġtwitter Ġdataset . ĠGl o Ve ĠB IB REF 13 Ġis Ġan Ġun super vised Ġlearning Ġalgorithm Ġfor Ġobtaining Ġvector Ġrepresentations Ġfor Ġwords . Ġ200 - dimensional ĠGl o Ve Ġembed d ings Ġtrained Ġon Ġ2 ĠBillion Ġtweets Ġare Ġintegrated . ĠEdinburgh Ġembed d ings ĠB IB REF 14 Ġare Ġobtained Ġby Ġtraining Ġskip - gram Ġmodel Ġon ĠEdinburgh Ġcorpus ĠB IB REF 15 Ġ. ĠSince Ġtweets Ġare Ġabundant Ġwith Ġem oj is , ĠEm oji Ġembed d ings ĠB IB REF 16 Ġwhich Ġare Ġlearned Ġfrom Ġthe Ġemoji Ġdescriptions Ġhave Ġbeen Ġused . ĠEmb edd ings Ġfor Ġeach Ġtweet Ġare Ġobtained Ġby Ġsum ming Ġup Ġindividual Ġword Ġvectors Ġand Ġthen Ġdividing Ġby Ġthe Ġnumber Ġof Ġtokens Ġin Ġthe Ġtweet . </s> Sy nt actic ĠFeatures : ĠSynt ax Ġspecific Ġfeatures Ġsuch Ġas ĠWord ĠN - gram s , ĠPart - Of - Spe ech ĠN - gram s ĠB IB REF 17 Ġ, ĠBrown ĠCluster ĠN - gram s ĠB IB REF 18 Ġobtained Ġusing ĠTweet N LP Ġproject Ġhave Ġbeen Ġintegrated Ġinto Ġthe Ġsystem . </s> The Ġfinal Ġfeature Ġvector Ġis Ġthe Ġconc aten ation Ġof Ġall Ġthe Ġindividual Ġfeatures . ĠFor Ġexample , Ġwe Ġconc aten ate Ġaverage Ġword Ġvectors , Ġsum Ġof ĠN RC ĠAffect ĠInt ens ities , Ġnumber Ġof Ġpositive Ġand Ġnegative ĠBing ĠLiu Ġlex icons , Ġnumber Ġof Ġneg ation Ġwords Ġand Ġso Ġon Ġto Ġget Ġfinal Ġfeature Ġvector . ĠThe Ġscaling Ġof Ġfinal Ġfeatures Ġis Ġnot Ġrequired Ġwhen Ġused Ġwith Ġgradient Ġboosted Ġtrees . ĠHowever , Ġscaling Ġsteps Ġlike Ġstandard Ġscaling Ġ( zero Ġmean Ġand Ġunit Ġnormal ) Ġmay Ġbe Ġbeneficial Ġfor Ġneural Ġnetworks Ġas Ġthe Ġoptim izers Ġwork Ġwell Ġwhen Ġthe Ġdata Ġis Ġcentered Ġaround Ġorigin . </s> A Ġtotal Ġof Ġfourteen Ġdifferent Ġfeature Ġextract ors Ġhave Ġbeen Ġimplemented , Ġall Ġof Ġwhich Ġcan Ġbe Ġenabled Ġor Ġdisabled Ġindividually Ġto Ġextract Ġfeatures Ġfrom Ġa Ġgiven Ġtweet . </s> Reg ression </s> The Ġdev Ġdata Ġset ĠB IB REF 19 Ġin Ġthe Ġcompetition Ġwas Ġsmall Ġhence , Ġthe Ġtrain Ġand Ġdev Ġsets Ġwere Ġmerged Ġto Ġperform Ġ10 - fold Ġcross Ġvalidation . ĠOn Ġeach Ġfold , Ġa Ġmodel Ġwas Ġtrained Ġand Ġthe Ġpredictions Ġwere Ġcollected Ġon Ġthe Ġremaining Ġdataset . ĠThe Ġpredictions Ġare Ġaveraged Ġacross Ġall Ġthe Ġfolds Ġto Ġgeneral ize Ġthe Ġsolution Ġand Ġprevent Ġover - fitting . ĠAs Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġ, Ġdifferent Ġcombinations Ġof Ġfeature Ġextract ors Ġwere Ġused . ĠAfter Ġperforming Ġfeature Ġextraction , Ġthe Ġdata Ġwas Ġthen Ġpassed Ġto Ġvarious Ġregress ors ĠSupport ĠVector ĠReg ression , ĠAda Boost , ĠRandom Forest Reg ressor , Ġand , ĠB agging Reg ressor Ġof Ġsk learn ĠB IB REF 20 Ġ. ĠFinally , Ġthe Ġchosen Ġtop Ġperforming Ġmodels Ġhad Ġthe Ġleast Ġerror Ġon Ġevaluation Ġmetrics Ġnamely ĠPearson 's ĠCor relation ĠCo efficient Ġand ĠSpear man 's Ġrank - order Ġcorrelation . </s> Parameter ĠOptim ization </s> In Ġorder Ġto Ġfind Ġthe Ġoptimal Ġparameter Ġvalues Ġfor Ġthe ĠEm o Int Ġsystem , Ġan Ġextensive Ġgrid Ġsearch Ġwas Ġperformed Ġthrough Ġthe Ġsc ik it - Learn Ġframework Ġover Ġall Ġsubs ets Ġof Ġthe Ġtraining Ġset Ġ( sh uff led ), Ġusing Ġstrat ified Ġ10 - fold Ġcross Ġvalidation Ġand Ġoptimizing Ġthe ĠPearson 's ĠCor relation Ġscore . ĠBest Ġcross - valid ation Ġresults Ġwere Ġobtained Ġusing ĠAda Boost Ġmeta Ġreg ressor Ġwith Ġbase Ġreg ressor Ġas ĠX GB o ost ĠB IB REF 21 Ġwith Ġ1000 Ġestim ators Ġand Ġ0 . 1 Ġlearning Ġrate . ĠExper iments Ġand Ġanalysis Ġof Ġresults Ġare Ġpresented Ġin Ġthe Ġnext Ġsection . </s> Exper imental ĠResults </s> As Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġvarious Ġsyntax Ġfeatures Ġwere Ġused Ġnamely , ĠPart - of - Spe ech Ġtags , Ġbrown Ġclusters Ġof ĠTweet N LP Ġproject . ĠHowever , Ġthese Ġdidn 't Ġperform Ġwell Ġin Ġcross Ġvalidation . ĠHence , Ġthey Ġwere Ġdropped Ġfrom Ġthe Ġfinal Ġsystem . ĠWhile Ġperforming Ġgrid - search Ġas Ġmentioned Ġin ĠSection ĠSEC REF 14 Ġ, Ġkeeping Ġall Ġthe Ġlex icon Ġbased Ġfeatures Ġsame , Ġchoice Ġof Ġcombination Ġof Ġemoji Ġvector Ġand Ġword Ġvectors Ġare Ġvaried Ġto Ġminimize Ġcross Ġvalidation Ġmetric . ĠTable ĠT AB REF 16 Ġdescribes Ġthe Ġresults Ġfor Ġexperiments Ġconducted Ġwith Ġdifferent Ġcombinations Ġof Ġword Ġvectors . ĠEm oji Ġembed d ings ĠB IB REF 16 Ġgive Ġbetter Ġresults Ġthan Ġusing Ġplain ĠGl o Ve Ġand ĠEdinburgh Ġembed d ings . ĠEdinburgh Ġembed d ings Ġoutper form ĠGl o Ve Ġembed d ings Ġin ĠJoy Ġand ĠSad ness Ġcategory Ġbut Ġlag Ġbehind Ġin ĠAnger Ġand ĠFear Ġcategory . ĠThe Ġofficial Ġsubmission Ġcomprised Ġof Ġthe Ġtop - performing Ġmodel Ġfor Ġeach Ġemotion Ġcategory . ĠThis Ġsystem Ġranked Ġ3 Ġfor Ġthe Ġentire Ġtest Ġdataset Ġand Ġ2 Ġfor Ġthe Ġsubset Ġof Ġthe Ġtest Ġdata Ġformed Ġby Ġtaking Ġevery Ġinstance Ġwith Ġa Ġgold Ġemotion Ġintensity Ġscore Ġgreater Ġthan Ġor Ġequal Ġto Ġ0 . 5 . ĠPost Ġcompetition , Ġexperiments Ġwere Ġperformed Ġon Ġen semb ling Ġdiverse Ġmodels Ġfor Ġimproving Ġthe Ġaccuracy . ĠAn Ġensemble Ġobtained Ġby Ġaveraging Ġthe Ġresults Ġof Ġthe Ġtop Ġ2 Ġperforming Ġmodels Ġoutper forms Ġall Ġthe Ġindividual Ġmodels . </s> Feature ĠImport ance </s> The Ġrelative Ġfeature Ġimportance Ġcan Ġbe Ġassessed Ġby Ġthe Ġrelative Ġdepth Ġof Ġthe Ġfeature Ġused Ġas Ġa Ġdecision Ġnode Ġin Ġthe Ġtree . ĠFeatures Ġused Ġat Ġthe Ġtop Ġof Ġthe Ġtree Ġcontribute Ġto Ġthe Ġfinal Ġprediction Ġdecision Ġof Ġa Ġlarger Ġfraction Ġof Ġthe Ġinput Ġsamples . ĠThe Ġexpected Ġfraction Ġof Ġthe Ġsamples Ġthey Ġcontribute Ġto Ġcan Ġthus Ġbe Ġused Ġas Ġan Ġestimate Ġof Ġthe Ġrelative Ġimportance Ġof Ġthe Ġfeatures . ĠBy Ġaveraging Ġthe Ġmeasure Ġover Ġseveral Ġrandomized Ġtrees , Ġthe Ġvariance Ġof Ġthe Ġestimate Ġcan Ġbe Ġreduced Ġand Ġused Ġas Ġa Ġmeasure Ġof Ġrelative Ġfeature Ġimportance . ĠIn ĠFigure ĠFIG REF 18 Ġfeature Ġimportance Ġgraphs Ġare Ġplotted Ġfor Ġeach Ġemotion Ġto Ġinfer Ġwhich Ġfeatures Ġare Ġplaying Ġthe Ġmajor Ġrole Ġin Ġidentifying Ġemotional Ġintensity Ġin Ġtweets . Ġ+/- Effect Word Net ĠB IB REF 5 Ġ, ĠN RC ĠHash tag ĠSent iment ĠLex icon , ĠSent iment 140 ĠLex icon ĠB IB REF 10 Ġand ĠN RC ĠHash tag ĠEm otion ĠLex icon ĠB IB REF 9 Ġare Ġplaying Ġthe Ġmost Ġimportant Ġrole . </s> System ĠLim itations </s> It Ġis Ġimportant Ġto Ġunderstand Ġhow Ġthe Ġmodel Ġperforms Ġin Ġdifferent Ġscenarios . ĠTable ĠT AB REF 20 Ġanaly zes Ġwhen Ġthe Ġsystem Ġperforms Ġthe Ġbest Ġand Ġworst Ġfor Ġeach Ġemotion . ĠSince Ġthe Ġfeatures Ġused Ġare Ġmostly Ġlex icon Ġbased , Ġthe Ġsystem Ġhas Ġdifficulties Ġin Ġcapturing Ġthe Ġoverall Ġsentiment Ġand Ġit Ġleads Ġto Ġampl ifying Ġor Ġvanishing Ġintensity Ġsignals . ĠFor Ġinstance , Ġin Ġexample Ġ4 Ġof Ġfear Ġlouder Ġand Ġshaking Ġlex icons Ġimply Ġfear Ġbut Ġoverall Ġsentence Ġdoesn 't Ġimply Ġfear . ĠA Ġsimilar Ġpattern Ġcan Ġbe Ġfound Ġin Ġthe Ġ4 Ġexample Ġof ĠAnger Ġand Ġ3 Ġexample Ġof ĠJoy . ĠThe Ġsystem Ġhas Ġdifficulties Ġin Ġunderstanding Ġof Ġsarcastic Ġtweets , Ġfor Ġinstance , Ġin Ġthe Ġ3 Ġtweet Ġof ĠAnger Ġthe Ġuser Ġexpressed Ġanger Ġbut Ġused Ġlol Ġwhich Ġis Ġused Ġin Ġa Ġpositive Ġsense Ġmost Ġof Ġthe Ġtimes Ġand Ġhence Ġthe Ġsystem Ġdid Ġa Ġbad Ġjob Ġat Ġpredicting Ġintensity . ĠThe Ġsystem Ġalso Ġfails Ġin Ġpredicting Ġsentences Ġhaving Ġdeeper Ġemotion Ġand Ġsentiment Ġwhich Ġhumans Ġcan Ġunderstand Ġwith Ġa Ġlittle Ġcontext . ĠFor Ġexample , Ġin Ġsample Ġ4 Ġof Ġsadness , Ġthe Ġtweet Ġrefers Ġto Ġpost Ġtravel Ġblues Ġwhich Ġhumans Ġcan Ġunderstand . ĠBut Ġwith Ġlittle Ġcontext , Ġit Ġis Ġdifficult Ġfor Ġthe Ġsystem Ġto Ġaccurately Ġestimate Ġthe Ġintensity . ĠThe Ġperformance Ġis Ġpoor Ġwith Ġvery Ġshort Ġsentences Ġas Ġthere Ġare Ġfewer Ġindicators Ġto Ġprovide Ġa Ġreasonable Ġestimate . </s> Future ĠWork Ġ& ĠConclusion </s> The Ġpaper Ġstudies Ġthe Ġeffectiveness Ġof Ġvarious Ġaffect Ġlex icons Ġword Ġembed d ings Ġto Ġestimate Ġemotional Ġintensity Ġin Ġtweets . ĠA Ġlight - weight Ġeasy Ġto Ġuse Ġaffect Ġcomputing Ġframework Ġ( E mo Int ) Ġto Ġfacilitate Ġease Ġof Ġexperimenting Ġwith Ġvarious Ġlex icon Ġfeatures Ġfor Ġtext Ġtasks Ġis Ġopen - s ourced . ĠIt Ġprovides Ġplug Ġand Ġplay Ġaccess Ġto Ġvarious Ġfeature Ġextract ors Ġand Ġhandy Ġscripts Ġfor Ġcreating Ġen semb les . </s> Few Ġproblems Ġexplained Ġin Ġthe Ġanalysis Ġsection Ġcan Ġbe Ġresolved Ġwith Ġthe Ġhelp Ġof Ġsentence Ġembed d ings Ġwhich Ġtake Ġthe Ġcontext Ġinformation Ġinto Ġconsideration . ĠThe Ġfeatures Ġused Ġin Ġthe Ġsystem Ġare Ġgeneric Ġenough Ġto Ġuse Ġthem Ġin Ġother Ġaffect ive Ġcomputing Ġtasks Ġon Ġsocial Ġmedia Ġtext , Ġnot Ġjust Ġtweet Ġdata . ĠAnother Ġinteresting Ġfeature Ġof Ġlex icon - based Ġsystems Ġis Ġtheir Ġgood Ġrun - time Ġperformance Ġduring Ġprediction , Ġfuture Ġwork Ġto Ġbenchmark Ġthe Ġperformance Ġof Ġthe Ġsystem Ġcan Ġprove Ġvital Ġfor Ġdeploying Ġin Ġa Ġreal - world Ġsetting . </s> Acknowled gement </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġorganizers Ġof Ġthe ĠWAS SA - 2017 ĠShared ĠTask Ġon ĠEm otion ĠInt ensity , Ġfor Ġproviding Ġthe Ġdata , Ġthe Ġguidelines Ġand Ġtimely Ġsupport .