Affective Text, Fairy Tales, ISEAR
EXTRACTIVE
 Affective Text dataset, Fairy Tales dataset, ISEAR dataset
EXTRACTIVE
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. In order to compare our performance to state-of-the-art results, we have used them as well. In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7 . A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation.
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. In order to compare our performance to state-of-the-art results, we have used them as well. In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7 . A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation.
1611.02988
Introduction </s> This Ġwork Ġis Ġlic enced Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLic ence . ĠLic ence Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> In Ġthe Ġspirit Ġof Ġthe Ġbre vity Ġof Ġsocial Ġmedia 's Ġmessages Ġand Ġreactions , Ġpeople Ġhave Ġgot Ġused Ġto Ġexpress Ġfeelings Ġminim ally Ġand Ġsymbol ically , Ġas Ġwith Ġhasht ags Ġon ĠTwitter Ġand ĠInstagram . ĠOn ĠFacebook , Ġpeople Ġtend Ġto Ġbe Ġmore Ġword y , Ġbut Ġposts Ġnormally Ġreceive Ġmore Ġsimple ĠâĢ ľ l ikes âĢ Ŀ Ġthan Ġlonger Ġcomments . ĠSince ĠFebruary Ġ2016 , ĠFacebook Ġusers Ġcan Ġexpress Ġspecific Ġemotions Ġin Ġresponse Ġto Ġa Ġpost Ġthanks Ġto Ġthe Ġnewly Ġintroduced Ġreaction Ġfeature Ġ( see ĠSection ĠSEC REF 2 Ġ), Ġso Ġthat Ġnow Ġa Ġpost Ġcan Ġbe Ġword lessly Ġmarked Ġwith Ġan Ġexpression Ġof Ġsay ĠâĢ ľ joy " Ġor ĠâĢ ľ sur prise " Ġrather Ġthan Ġa Ġgeneric ĠâĢ ľ like âĢ Ŀ . </s> It Ġhas Ġbeen Ġobserved Ġthat Ġthis Ġnew Ġfeature Ġhelps ĠFacebook Ġto Ġknow Ġmuch Ġmore Ġabout Ġtheir Ġusers Ġand Ġexploit Ġthis Ġinformation Ġfor Ġtargeted Ġadvertising ĠB IB REF 0 Ġ, Ġbut Ġinterest Ġin Ġpeople 's Ġopinions Ġand Ġhow Ġthey Ġfeel Ġisn 't Ġlimited Ġto Ġcommercial Ġreasons , Ġas Ġit Ġinvests Ġsocial Ġmonitoring , Ġtoo , Ġincluding Ġhealth Ġcare Ġand Ġeducation ĠB IB REF 1 Ġ. ĠHowever , Ġemotions Ġand Ġopinions Ġare Ġnot Ġalways Ġexpressed Ġthis Ġexplicitly , Ġso Ġthat Ġthere Ġis Ġhigh Ġinterest Ġin Ġdeveloping Ġsystems Ġtowards Ġtheir Ġautomatic Ġdetection . ĠCreating Ġmanually Ġannot ated Ġdatasets Ġlarge Ġenough Ġto Ġtrain Ġsupervised Ġmodels Ġis Ġnot Ġonly Ġcostly , Ġbut Ġalso âĢĶ especially Ġin Ġthe Ġcase Ġof Ġopinions Ġand Ġemotions âĢĶ diff icult , Ġdue Ġto Ġthe Ġintrinsic Ġsubject ivity Ġof Ġthe Ġtask ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠTherefore , Ġresearch Ġhas Ġfocused Ġon Ġun super vised Ġmethods Ġenriched Ġwith Ġinformation Ġderived Ġfrom Ġlex ica , Ġwhich Ġare Ġmanually Ġcreated ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠSince Ġgo 2009 twitter Ġhave Ġshown Ġthat Ġhappy Ġand Ġsad Ġemot icons Ġcan Ġbe Ġsuccessfully Ġused Ġas Ġsignals Ġfor Ġsentiment Ġlabels , Ġdistant Ġsupervision , Ġi . e . Ġusing Ġsome Ġreasonably Ġsafe Ġsignals Ġas Ġproxies Ġfor Ġautomatically Ġlab elling Ġtraining Ġdata ĠB IB REF 5 Ġ, Ġhas Ġbeen Ġused Ġalso Ġfor Ġemotion Ġrecognition , Ġfor Ġexample Ġexploiting Ġboth Ġemot icons Ġand ĠTwitter Ġhasht ags ĠB IB REF 6 Ġ, Ġbut Ġmainly Ġtowards Ġcreating Ġemotion Ġlex ica . Ġm oh ammad 2015 using Ġuse Ġhasht ags , Ġexperimenting Ġalso Ġwith Ġhighly Ġfine - gr ained Ġemotion Ġsets Ġ( up Ġto Ġalmost Ġ600 Ġemotion Ġlabels ), Ġto Ġcreate Ġthe Ġlarge ĠHash tag ĠEm otion ĠLex icon . ĠEm otic ons Ġare Ġused Ġas Ġproxies Ġalso Ġby Ġhalls m arm ult i , Ġwho Ġuse Ġdistributed Ġvector Ġrepresentations Ġto Ġfind Ġwhich Ġwords Ġare Ġinterchangeable Ġwith Ġemot icons Ġbut Ġalso Ġwhich Ġemot icons Ġare Ġused Ġin Ġa Ġsimilar Ġcontext . </s> We Ġtake Ġadvantage Ġof Ġdistant Ġsupervision Ġby Ġusing ĠFacebook Ġreactions Ġas Ġproxies Ġfor Ġemotion Ġlabels , Ġwhich Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġhasn 't Ġbeen Ġdone Ġyet , Ġand Ġwe Ġtrain Ġa Ġset Ġof ĠSupport ĠVector ĠMachine Ġmodels Ġfor Ġemotion Ġrecognition . ĠOur Ġmodels , Ġdifferently Ġfrom Ġexisting Ġones , Ġexploit Ġinformation Ġwhich Ġis Ġacquired Ġentirely Ġautomatically , Ġand Ġachieve Ġcompetitive Ġor Ġeven Ġstate - of - the - art Ġresults Ġfor Ġsome Ġof Ġthe Ġemotion Ġlabels Ġon Ġexisting , Ġstandard Ġevaluation Ġdatasets . ĠFor Ġexplanatory Ġpurposes , Ġrelated Ġwork Ġis Ġdiscussed Ġfurther Ġand Ġmore Ġin Ġdetail Ġwhen Ġwe Ġdescribe Ġthe Ġbenchmarks Ġfor Ġevaluation Ġ( Section ĠSEC REF 3 Ġ) Ġand Ġwhen Ġwe Ġcompare Ġour Ġmodels Ġto Ġexisting Ġones Ġ( Section ĠSEC REF 5 Ġ). ĠWe Ġalso Ġexplore Ġand Ġdiscuss Ġhow Ġchoosing Ġdifferent Ġsets Ġof ĠFacebook Ġpages Ġas Ġtraining Ġdata Ġprovides Ġan Ġintrinsic Ġdomain - adapt ation Ġmethod . </s> Facebook Ġreactions Ġas Ġlabels </s> For Ġyears , Ġon ĠFacebook Ġpeople Ġcould Ġleave Ġcomments Ġto Ġposts , Ġand Ġalso ĠâĢ ľ like âĢ Ŀ Ġthem , Ġby Ġusing Ġa Ġthumbs - up Ġfeature Ġto Ġexplicitly Ġexpress Ġa Ġgeneric , Ġrather Ġunders pec ified , Ġapproval . ĠA ĠâĢ ľ like âĢ Ŀ Ġcould Ġthus Ġmean ĠâĢ ľ I Ġlike Ġwhat Ġyou Ġsaid ", Ġbut Ġalso ĠâĢ ľ I Ġlike Ġthat Ġyou Ġbring Ġup Ġsuch Ġtopic Ġ( though ĠI Ġfind Ġthe Ġcontent Ġof Ġthe Ġarticle Ġyou Ġlinked Ġannoying ) ". </s> In ĠFebruary Ġ2016 , Ġafter Ġa Ġshort Ġtrial , ĠFacebook Ġmade Ġa Ġmore Ġexplicit Ġreaction Ġfeature Ġavailable Ġworld - wide . ĠRather Ġthan Ġallowing Ġfor Ġthe Ġunders pec ified ĠâĢ ľ like âĢ Ŀ Ġas Ġthe Ġonly Ġword less Ġresponse Ġto Ġa Ġpost , Ġa Ġset Ġof Ġsix Ġmore Ġspecific Ġreactions Ġwas Ġintroduced , Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ: ĠLike , ĠLove , ĠH aha , ĠWow , ĠSad Ġand ĠAngry . ĠWe Ġuse Ġsuch Ġreactions Ġas Ġproxies Ġfor Ġemotion Ġlabels Ġassociated Ġto Ġposts . </s> We Ġcollected ĠFacebook Ġposts Ġand Ġtheir Ġcorresponding Ġreactions Ġfrom Ġpublic Ġpages Ġusing Ġthe ĠFacebook ĠAPI , Ġwhich Ġwe Ġaccessed Ġvia Ġthe ĠFacebook - sd k Ġpython Ġlibrary . ĠWe Ġchose Ġdifferent Ġpages Ġ( and Ġtherefore Ġdomains Ġand Ġstances ), Ġaiming Ġat Ġa Ġbalanced Ġand Ġvaried Ġdataset , Ġbut Ġwe Ġdid Ġso Ġmainly Ġbased Ġon Ġintuition Ġ( see ĠSection ĠSEC REF 4 Ġ) Ġand Ġwith Ġan Ġeye Ġto Ġthe Ġnature Ġof Ġthe Ġdatasets Ġavailable Ġfor Ġevaluation Ġ( see ĠSection ĠSEC REF 5 Ġ). ĠThe Ġchoice Ġof Ġwhich Ġpages Ġto Ġselect Ġposts Ġfrom Ġis Ġfar Ġfrom Ġtrivial , Ġand Ġwe Ġbelieve Ġthis Ġis Ġactually Ġan Ġinteresting Ġaspect Ġof Ġour Ġapproach , Ġas Ġby Ġusing Ġdifferent ĠFacebook Ġpages Ġone Ġcan Ġintrinsically Ġtackle Ġthe Ġdomain - adapt ation Ġproblem Ġ( See ĠSection ĠSEC REF 6 Ġfor Ġfurther Ġdiscussion Ġon Ġthis ). ĠThe Ġfinal Ġcollection Ġof ĠFacebook Ġpages Ġfor Ġthe Ġexperiments Ġdescribed Ġin Ġthis Ġpaper Ġis Ġas Ġfollows : ĠFox News , ĠCNN , ĠESPN , ĠNew ĠYork ĠTimes , ĠTime Ġmagazine , ĠHuffington ĠPost ĠWeird ĠNews , ĠThe ĠGuardian , ĠCartoon ĠNetwork , ĠCooking ĠLight , ĠHome ĠCooking ĠAdventure , ĠJustin ĠBieber , ĠNickel ode on , ĠSponge b ob , ĠDisney . </s> Note Ġthat Ġthankful Ġwas Ġonly Ġavailable Ġduring Ġspecific Ġtime Ġspans Ġrelated Ġto Ġcertain Ġevents , Ġas ĠMother 's ĠDay Ġin ĠMay Ġ2016 . </s> For Ġeach Ġpage , Ġwe Ġdownloaded Ġthe Ġlatest Ġ1000 Ġposts , Ġor Ġthe Ġmaximum Ġavailable Ġif Ġthere Ġare Ġfewer , Ġfrom ĠFebruary Ġ2016 , Ġretrieving Ġthe Ġcounts Ġof Ġreactions Ġfor Ġeach Ġpost . ĠThe Ġoutput Ġis Ġa ĠJSON Ġfile Ġcontaining Ġa Ġlist Ġof Ġdiction aries Ġwith Ġa Ġtimestamp , Ġthe Ġpost Ġand Ġa Ġreaction Ġvector Ġwith Ġfrequency Ġvalues , Ġwhich Ġindicate Ġhow Ġmany Ġusers Ġused Ġthat Ġreaction Ġin Ġresponse Ġto Ġthe Ġpost Ġ( Figure ĠFIG REF 3 Ġ). ĠThe Ġresulting Ġemotion Ġvectors Ġmust Ġthen Ġbe Ġturned Ġinto Ġan Ġemotion Ġlabel . </s> In Ġthe Ġcontext Ġof Ġthis Ġexperiment , Ġwe Ġmade Ġthe Ġsimple Ġdecision Ġof Ġassoci ating Ġto Ġeach Ġpost Ġthe Ġemotion Ġwith Ġthe Ġhighest Ġcount , Ġignoring Ġlike Ġas Ġit Ġis Ġthe Ġdefault Ġand Ġmost Ġgeneric Ġreaction Ġpeople Ġtend Ġto Ġuse . ĠTherefore , Ġfor Ġexample , Ġto Ġthe Ġfirst Ġpost Ġin ĠFigure ĠFIG REF 3 Ġ, Ġwe Ġwould Ġassociate Ġthe Ġlabel Ġsad , Ġas Ġit Ġhas Ġthe Ġhighest Ġscore Ġ( 284 ) Ġamong Ġthe Ġmeaningful Ġemotions Ġwe Ġconsider , Ġthough Ġit Ġalso Ġhas Ġnon - zero Ġscores Ġfor Ġother Ġemotions . ĠAt Ġthis Ġstage , Ġwe Ġdidn 't Ġperform Ġany Ġother Ġentropy - based Ġselection Ġof Ġposts , Ġto Ġbe Ġinvestigated Ġin Ġfuture Ġwork . </s> Em otion Ġdatasets </s> Three Ġdatasets Ġannot ated Ġwith Ġemotions Ġare Ġcommonly Ġused Ġfor Ġthe Ġdevelopment Ġand Ġevaluation Ġof Ġemotion Ġdetection Ġsystems , Ġnamely Ġthe ĠAffect ive ĠText Ġdataset , Ġthe ĠFairy ĠTales Ġdataset , Ġand Ġthe ĠIS EAR Ġdataset . ĠIn Ġorder Ġto Ġcompare Ġour Ġperformance Ġto Ġstate - of - the - art Ġresults , Ġwe Ġhave Ġused Ġthem Ġas Ġwell . ĠIn Ġthis ĠSection , Ġin Ġaddition Ġto Ġa Ġdescription Ġof Ġeach Ġdataset , Ġwe Ġprovide Ġan Ġoverview Ġof Ġthe Ġemotions Ġused , Ġtheir Ġdistribution , Ġand Ġhow Ġwe Ġmapped Ġthem Ġto Ġthose Ġwe Ġobtained Ġfrom ĠFacebook Ġposts Ġin ĠSection ĠSEC REF 7 Ġ. ĠA Ġsummary Ġis Ġprovided Ġin ĠTable ĠT AB REF 8 Ġ, Ġwhich Ġalso Ġshows , Ġin Ġthe Ġbottom Ġrow , Ġwhat Ġrole Ġeach Ġdataset Ġhas Ġin Ġour Ġexperiments : Ġapart Ġfrom Ġthe Ġdevelopment Ġportion Ġof Ġthe ĠAffect ive ĠText , Ġwhich Ġwe Ġused Ġto Ġdevelop Ġour Ġmodels Ġ( Section ĠSEC REF 4 Ġ), Ġall Ġthree Ġhave Ġbeen Ġused Ġas Ġbenchmarks Ġfor Ġour Ġevaluation . </s> A ffect ive ĠText Ġdataset </s> Task Ġ14 Ġat ĠSem E val Ġ2007 ĠB IB REF 7 Ġwas Ġconcerned Ġwith Ġthe Ġclassification Ġof Ġemotions Ġand Ġval ence Ġin Ġnews Ġheadlines . ĠThe Ġheadlines Ġwhere Ġcollected Ġfrom Ġseveral Ġnews Ġwebsites Ġincluding ĠGoogle Ġnews , ĠThe ĠNew ĠYork ĠTimes , ĠBBC ĠNews Ġand ĠCNN . ĠThe Ġused Ġemotion Ġlabels Ġwere ĠAnger , ĠDis g ust , ĠFear , ĠJoy , ĠSad ness , ĠSurprise , Ġin Ġline Ġwith Ġthe Ġsix Ġbasic Ġemotions Ġof ĠEk man 's Ġstandard Ġmodel ĠB IB REF 8 Ġ. ĠVal ence Ġwas Ġto Ġbe Ġdetermined Ġas Ġpositive Ġor Ġnegative . ĠClassification Ġof Ġemotion Ġand Ġval ence Ġwere Ġtreated Ġas Ġseparate Ġtasks . ĠEm otion Ġlabels Ġwere Ġnot Ġconsidered Ġas Ġmut ut ally Ġexclusive , Ġand Ġeach Ġemotion Ġwas Ġassigned Ġa Ġscore Ġfrom Ġ0 Ġto Ġ100 . ĠTraining / develop ing Ġdata Ġamounted Ġto Ġ250 Ġannot ated Ġheadlines Ġ( A ffect ive Ġdevelopment ), Ġwhile Ġsystems Ġwere Ġevaluated Ġon Ġanother Ġ1000 Ġ( A ffect ive Ġtest ). ĠEvaluation Ġwas Ġdone Ġusing Ġtwo Ġdifferent Ġmethods : Ġa Ġfine - gr ained Ġevaluation Ġusing ĠPearson 's Ġr Ġto Ġmeasure Ġthe Ġcorrelation Ġbetween Ġthe Ġsystem Ġscores Ġand Ġthe Ġgold Ġstandard ; Ġand Ġa Ġcoarse - gr ained Ġmethod Ġwhere Ġeach Ġemotion Ġscore Ġwas Ġconverted Ġto Ġa Ġbinary Ġlabel , Ġand Ġprecision , Ġrecall , Ġand Ġf - score Ġwere Ġcomputed Ġto Ġassess Ġperformance . ĠAs Ġit Ġis Ġdone Ġin Ġmost Ġworks Ġthat Ġuse Ġthis Ġdataset ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 9 Ġ, Ġwe Ġalso Ġtreat Ġthis Ġas Ġa Ġclassification Ġproblem Ġ( co arse - gr ained ). ĠThis Ġdataset Ġhas Ġbeen Ġextensively Ġused Ġfor Ġthe Ġevaluation Ġof Ġvarious Ġun super vised Ġmethods ĠB IB REF 2 Ġ, Ġbut Ġalso Ġfor Ġtesting Ġdifferent Ġsupervised Ġlearning Ġtechniques Ġand Ġfeature Ġport ability ĠB IB REF 10 Ġ. </s> F airy ĠTales Ġdataset </s> This Ġis Ġa Ġdataset Ġcollected Ġby Ġal m 2008 aff ect , Ġwhere Ġabout Ġ1 , 000 Ġsentences Ġfrom Ġfairy Ġtales Ġ( by ĠB . ĠPotter , ĠH . C . ĠAndersen Ġand ĠGrimm ) Ġwere Ġannot ated Ġwith Ġthe Ġsame Ġsix Ġemotions Ġof Ġthe ĠAffect ive ĠText Ġdataset , Ġthough Ġwith Ġdifferent Ġnames : ĠAngry , ĠDis g usted , ĠFear ful , ĠHappy , ĠSad , Ġand ĠSur pr ised . ĠIn Ġmost Ġworks Ġthat Ġuse Ġthis Ġdataset ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 9 Ġ, Ġonly Ġsentences Ġwhere Ġall Ġannot ators Ġagreed Ġare Ġused , Ġand Ġthe Ġlabels Ġangry Ġand Ġdisgusted Ġare Ġmerged . ĠWe Ġadopt Ġthe Ġsame Ġchoices . </s> IS EAR </s> The ĠIS EAR Ġ( International ĠSurvey Ġon ĠEm otion ĠAnt ec ed ents Ġand ĠRe actions ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ) Ġis Ġa Ġdataset Ġcreated Ġin Ġthe Ġcontext Ġof Ġa Ġpsychology Ġproject Ġof Ġthe Ġ1990 s , Ġby Ġcollecting Ġquestion naires Ġanswered Ġby Ġpeople Ġwith Ġdifferent Ġcultural Ġbackgrounds . ĠThe Ġmain Ġaim Ġof Ġthis Ġproject Ġwas Ġto Ġgather Ġinsights Ġin Ġcross - cultural Ġaspects Ġof Ġemotional Ġreactions . ĠStudent Ġrespondents , Ġboth Ġpsychologists Ġand Ġnon - psych ologists , Ġwere Ġasked Ġto Ġreport Ġsituations Ġin Ġwhich Ġthey Ġhad Ġexperienced Ġall Ġof Ġseven Ġmajor Ġemotions Ġ( joy , Ġfear , Ġanger , Ġsadness , Ġdisgust , Ġshame Ġand Ġguilt ). ĠIn Ġeach Ġcase , Ġthe Ġquestions Ġcovered Ġthe Ġway Ġthey Ġhad Ġappra ised Ġa Ġgiven Ġsituation Ġand Ġhow Ġthey Ġreacted . ĠThe Ġfinal Ġdataset Ġcontains Ġreports Ġby Ġapproximately Ġ3000 Ġrespondents Ġfrom Ġall Ġover Ġthe Ġworld , Ġfor Ġa Ġtotal Ġof Ġ7 665 Ġsentences Ġlabelled Ġwith Ġan Ġemotion , Ġmaking Ġthis Ġthe Ġlargest Ġdataset Ġout Ġof Ġthe Ġthree Ġwe Ġuse . </s> Overview Ġof Ġdatasets Ġand Ġemotions </s> We Ġsummar ise Ġdatasets Ġand Ġemotion Ġdistribution Ġfrom Ġtwo Ġviewpoints . ĠFirst , Ġbecause Ġthere Ġare Ġdifferent Ġsets Ġof Ġemotions Ġlabels Ġin Ġthe Ġdatasets Ġand ĠFacebook Ġdata , Ġwe Ġneed Ġto Ġprovide Ġa Ġmapping Ġand Ġderive Ġa Ġsubset Ġof Ġemotions Ġthat Ġwe Ġare Ġgoing Ġto Ġuse Ġfor Ġthe Ġexperiments . ĠThis Ġis Ġshown Ġin ĠTable ĠT AB REF 8 Ġ, Ġwhere Ġin Ġthe ĠâĢ ľ M apped âĢ Ŀ Ġcolumn Ġwe Ġreport Ġthe Ġfinal Ġemotions Ġwe Ġuse Ġin Ġthis Ġpaper : Ġanger , Ġjoy , Ġsadness , Ġsurprise . ĠAll Ġlabels Ġin Ġeach Ġdataset Ġare Ġmapped Ġto Ġthese Ġfinal Ġemotions , Ġwhich Ġare Ġtherefore Ġthe Ġlabels Ġwe Ġuse Ġfor Ġtraining Ġand Ġtesting Ġour Ġmodels . </s> Second , Ġthe Ġdistribution Ġof Ġthe Ġemotions Ġfor Ġeach Ġdataset Ġis Ġdifferent , Ġas Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 9 Ġ. </s> In ĠFigure ĠFIG REF 9 Ġwe Ġalso Ġprovide Ġthe Ġdistribution Ġof Ġthe Ġemotions Ġanger , Ġjoy , Ġsadness , Ġsurprise Ġper ĠFacebook Ġpage , Ġin Ġterms Ġof Ġnumber Ġof Ġposts Ġ( rec all Ġthat Ġwe Ġassign Ġto Ġa Ġpost Ġthe Ġlabel Ġcorresponding Ġto Ġthe Ġmajority Ġemotion Ġassociated Ġto Ġit , Ġsee ĠSection ĠSEC REF 2 Ġ). ĠWe Ġcan Ġobserve Ġthat Ġfor Ġexample Ġpages Ġabout Ġnews Ġtend Ġto Ġhave Ġmore Ġsadness Ġand Ġanger Ġposts , Ġwhile Ġpages Ġabout Ġcooking Ġand Ġtv - shows Ġhave Ġa Ġhigh Ġpercentage Ġof Ġjoy Ġposts . ĠWe Ġwill Ġuse Ġthis Ġinformation Ġto Ġfind Ġthe Ġbest Ġset Ġof Ġpages Ġfor Ġa Ġgiven Ġtarget Ġdomain Ġ( see ĠSection ĠSEC REF 5 Ġ). </s> Model </s> There Ġare Ġtwo Ġmain Ġdecisions Ġto Ġbe Ġtaken Ġin Ġdeveloping Ġour Ġmodel : Ġ( i ) Ġwhich ĠFacebook Ġpages Ġto Ġselect Ġas Ġtraining Ġdata , Ġand Ġ( ii ) Ġwhich Ġfeatures Ġto Ġuse Ġto Ġtrain Ġthe Ġmodel , Ġwhich Ġwe Ġdiscuss Ġbelow . ĠSpecifically , Ġwe Ġfirst Ġset Ġon Ġa Ġsubset Ġof Ġpages Ġand Ġthen Ġexperiment Ġwith Ġfeatures . ĠFurther Ġexploration Ġof Ġthe Ġinteraction Ġbetween Ġchoice Ġof Ġpages Ġand Ġchoice Ġof Ġfeatures Ġis Ġleft Ġto Ġfuture Ġwork , Ġand Ġpartly Ġdiscussed Ġin ĠSection ĠSEC REF 6 Ġ. ĠFor Ġdevelopment , Ġwe Ġuse Ġa Ġsmall Ġportion Ġof Ġthe ĠAffect ive Ġdata Ġset Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ, Ġthat Ġis Ġthe Ġportion Ġthat Ġhad Ġbeen Ġreleased Ġas Ġdevelopment Ġset Ġfor ĠSem E val 's Ġ2007 ĠTask Ġ14 ĠB IB REF 7 Ġ, Ġwhich Ġcontains Ġ250 Ġannot ated Ġsentences Ġ( A ffect ive Ġdevelopment , ĠSection ĠSEC REF 4 Ġ). ĠAll Ġresults Ġreported Ġin Ġthis Ġsection Ġare Ġon Ġthis Ġdataset . ĠThe Ġtest Ġset Ġof ĠTask Ġ14 Ġas Ġwell Ġas Ġthe Ġother Ġtwo Ġdatasets Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġwill Ġbe Ġused Ġto Ġevaluate Ġthe Ġfinal Ġmodels Ġ( Section ĠSEC REF 4 Ġ). </s> Select ing ĠFacebook Ġpages </s> Although Ġpage Ġselection Ġis Ġa Ġcrucial Ġingredient Ġof Ġthis Ġapproach , Ġwhich Ġwe Ġbelieve Ġcalls Ġfor Ġfurther Ġand Ġdeeper , Ġdedicated Ġinvestigation , Ġfor Ġthe Ġexperiments Ġdescribed Ġhere Ġwe Ġtook Ġa Ġrather Ġsimple Ġapproach . ĠFirst , Ġwe Ġselected Ġthe Ġpages Ġthat Ġwould Ġprovide Ġtraining Ġdata Ġbased Ġon Ġintuition Ġand Ġavailability , Ġthen Ġchose Ġdifferent Ġcombinations Ġaccording Ġto Ġresults Ġof Ġa Ġbasic Ġmodel Ġrun Ġon Ġdevelopment Ġdata , Ġand Ġeventually Ġtested Ġfeature Ġcombinations , Ġstill Ġon Ġthe Ġdevelopment Ġset . </s> For Ġthe Ġsake Ġof Ġsimplicity Ġand Ġtransparency , Ġwe Ġfirst Ġtrained Ġan ĠS VM Ġwith Ġa Ġsimple Ġbag - of - words Ġmodel Ġand Ġdefault Ġparameters Ġas Ġper Ġthe ĠSci kit - learn Ġimplementation ĠB IB REF 13 Ġon Ġdifferent Ġcombinations Ġof Ġpages . ĠBased Ġon Ġresults Ġof Ġthe Ġattempted Ġcombinations Ġas Ġwell Ġas Ġon Ġthe Ġdistribution Ġof Ġemotions Ġin Ġthe Ġdevelopment Ġdataset Ġ( Figure ĠFIG REF 9 Ġ), Ġwe Ġselected Ġa Ġbest Ġmodel Ġ( B - M ), Ġnamely Ġthe Ġcombined Ġset Ġof ĠTime , ĠThe ĠGuardian Ġand ĠDisney , Ġwhich Ġyields Ġthe Ġhighest Ġresults Ġon Ġdevelopment Ġdata . ĠTime Ġand ĠThe ĠGuardian Ġperform Ġwell Ġon Ġmost Ġemotions Ġbut ĠDisney Ġhelps Ġto Ġboost Ġthe Ġperformance Ġfor Ġthe ĠJoy Ġclass . </s> Features </s> In Ġselecting Ġappropriate Ġfeatures , Ġwe Ġmainly Ġrelied Ġon Ġprevious Ġwork Ġand Ġintuition . ĠWe Ġexperimented Ġwith Ġdifferent Ġcombinations , Ġand Ġall Ġtests Ġwere Ġstill Ġdone Ġon ĠAffect ive Ġdevelopment , Ġusing Ġthe Ġpages Ġfor Ġthe Ġbest Ġmodel Ġ( B - M ) Ġdescribed Ġabove Ġas Ġtraining Ġdata . ĠResults Ġare Ġin ĠTable ĠT AB REF 20 Ġ. ĠFuture Ġwork Ġwill Ġfurther Ġexplore Ġthe Ġsimultaneous Ġselection Ġof Ġfeatures Ġand Ġpage Ġcombinations . </s> We Ġuse Ġa Ġset Ġof Ġbasic Ġtext - based Ġfeatures Ġto Ġcapture Ġthe Ġemotion Ġclass . ĠThese Ġinclude Ġa Ġtf - id f Ġbag - of - words Ġfeature , Ġword Ġ( 2 - 3 ) Ġand Ġcharacter Ġ( 2 - 5 ) Ġn gram s , Ġand Ġfeatures Ġrelated Ġto Ġthe Ġpresence Ġof Ġneg ation Ġwords , Ġand Ġto Ġthe Ġusage Ġof Ġpunct uation . </s> This Ġfeature Ġis Ġused Ġin Ġall Ġun super vised Ġmodels Ġas Ġa Ġsource Ġof Ġinformation , Ġand Ġwe Ġmainly Ġinclude Ġit Ġto Ġassess Ġits Ġcontribution , Ġbut Ġeventually Ġdo Ġnot Ġuse Ġit Ġin Ġour Ġfinal Ġmodel . </s> We Ġused Ġthe ĠN RC 10 ĠLex icon Ġbecause Ġit Ġperformed Ġbest Ġin Ġthe Ġexperiments Ġby ĠB IB REF 10 Ġ, Ġwhich Ġis Ġbuilt Ġaround Ġthe Ġemotions Ġanger , Ġanticipation , Ġdisgust , Ġfear , Ġjoy , Ġsadness , Ġand Ġsurprise , Ġand Ġthe Ġval ence Ġvalues Ġpositive Ġand Ġnegative . ĠFor Ġeach Ġword Ġin Ġthe Ġlex icon , Ġa Ġboolean Ġvalue Ġindicating Ġpresence Ġor Ġabsence Ġis Ġassociated Ġto Ġeach Ġemotion . ĠFor Ġa Ġwhole Ġsentence , Ġa Ġglobal Ġscore Ġper Ġemotion Ġcan Ġbe Ġobtained Ġby Ġsum ming Ġthe Ġvectors Ġfor Ġall Ġcontent Ġwords Ġof Ġthat Ġsentence Ġincluded Ġin Ġthe Ġlex icon , Ġand Ġused Ġas Ġfeature . </s> As Ġadditional Ġfeature , Ġwe Ġalso Ġincluded ĠWord ĠEmb edd ings , Ġnamely Ġdistributed Ġrepresentations Ġof Ġwords Ġin Ġa Ġvector Ġspace , Ġwhich Ġhave Ġbeen Ġexceptionally Ġsuccessful Ġin Ġboosting Ġperformance Ġin Ġa Ġplethora Ġof ĠN LP Ġtasks . ĠWe Ġuse Ġthree Ġdifferent Ġembed d ings : </s> Google Ġembed d ings : Ġpre - trained Ġembed d ings Ġtrained Ġon ĠGoogle ĠNews Ġand Ġobtained Ġwith Ġthe Ġskip - gram Ġarchitecture Ġdescribed Ġin ĠB IB REF 14 Ġ. ĠThis Ġmodel Ġcontains Ġ300 - dimensional Ġvectors Ġfor Ġ3 Ġmillion Ġwords Ġand Ġphrases . </s> Facebook Ġembed d ings : Ġembed d ings Ġthat Ġwe Ġtrained Ġon Ġour Ġscrap ed ĠFacebook Ġpages Ġfor Ġa Ġtotal Ġof Ġ20 , 000 Ġsentences . ĠUsing Ġthe Ġg ens im Ġlibrary ĠB IB REF 15 Ġ, Ġwe Ġtrained Ġthe Ġembed d ings Ġwith Ġthe Ġfollowing Ġparameters : Ġwindow Ġsize Ġof Ġ5 , Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġdimension ality Ġof Ġ100 . ĠWe Ġfiltered Ġout Ġwords Ġwith Ġfrequency Ġlower Ġthan Ġ2 Ġoccurrences . </s> Ret ro fitted Ġembed d ings : ĠRetro fitting ĠB IB REF 16 Ġhas Ġbeen Ġshown Ġas Ġa Ġsimple Ġbut Ġefficient Ġway Ġof Ġinforming Ġtrained Ġembed d ings Ġwith Ġadditional Ġinformation Ġderived Ġfrom Ġsome Ġlex ical Ġresource , Ġrather Ġthan Ġincluding Ġit Ġdirectly Ġat Ġthe Ġtraining Ġstage , Ġas Ġit 's Ġdone Ġfor Ġexample Ġto Ġcreate Ġsense - aware ĠB IB REF 17 Ġor Ġsentiment - aware ĠB IB REF 18 Ġembed d ings . ĠIn Ġthis Ġwork , Ġwe Ġretro fit Ġgeneral Ġembed d ings Ġto Ġinclude Ġinformation Ġabout Ġemotions , Ġso Ġthat Ġemotion - similar Ġwords Ġcan Ġget Ġcloser Ġin Ġspace . ĠBoth Ġthe ĠGoogle Ġas Ġwell Ġas Ġour ĠFacebook Ġembed d ings Ġwere Ġretro fitted Ġwith Ġlex ical Ġinformation Ġobtained Ġfrom Ġthe ĠN RC 10 ĠLex icon Ġmentioned Ġabove , Ġwhich Ġprovides Ġemotion - similar ity Ġfor Ġeach Ġtoken . ĠNote Ġthat Ġdifferently Ġfrom Ġthe Ġprevious Ġtwo Ġtypes Ġof Ġembed d ings , Ġthe Ġretro fitted Ġones Ġdo Ġrely Ġon Ġhand crafted Ġinformation Ġin Ġthe Ġform Ġof Ġa Ġlex ical Ġresource . </s> Results Ġon Ġdevelopment Ġset </s> We Ġreport Ġprecision , Ġrecall , Ġand Ġf - score Ġon Ġthe Ġdevelopment Ġset . ĠThe Ġaverage Ġf - score Ġis Ġreported Ġas Ġmicro - average , Ġto Ġbetter Ġaccount Ġfor Ġthe Ġskewed Ġdistribution Ġof Ġthe Ġclasses Ġas Ġwell Ġas Ġin Ġaccordance Ġto Ġwhat Ġis Ġusually Ġreported Ġfor Ġthis Ġtask ĠB IB REF 19 Ġ. </s> From ĠTable ĠT AB REF 20 Ġwe Ġdraw Ġthree Ġmain Ġobservations . ĠFirst , Ġa Ġsimple Ġtf - id f Ġbag - of - word Ġmode Ġworks Ġalready Ġvery Ġwell , Ġto Ġthe Ġpoint Ġthat Ġthe Ġother Ġtextual Ġand Ġlex icon - based Ġfeatures Ġdon 't Ġseem Ġto Ġcontribute Ġto Ġthe Ġoverall Ġf - score Ġ( 0 . 368 ), Ġalthough Ġthere Ġis Ġa Ġrather Ġsubstantial Ġvariation Ġof Ġscores Ġper Ġclass . ĠSecond , ĠGoogle Ġembed d ings Ġperform Ġa Ġlot Ġbetter Ġthan ĠFacebook Ġembed d ings , Ġand Ġthis Ġis Ġlikely Ġdue Ġto Ġthe Ġsize Ġof Ġthe Ġcorpus Ġused Ġfor Ġtraining . ĠRetro fitting Ġdoesn 't Ġseem Ġto Ġhelp Ġat Ġall Ġfor Ġthe ĠGoogle Ġembed d ings , Ġbut Ġit Ġdoes Ġboost Ġthe ĠFacebook Ġembed d ings , Ġleading Ġto Ġthink Ġthat Ġwith Ġlittle Ġdata , Ġmore Ġaccurate Ġtask - related Ġinformation Ġis Ġhelping , Ġbut Ġcorpus Ġsize Ġmatters Ġmost . ĠThird , Ġin Ġcombination Ġwith Ġembed d ings , Ġall Ġfeatures Ġwork Ġbetter Ġthan Ġjust Ġusing Ġtf - id f , Ġbut Ġremoving Ġthe ĠLex icon Ġfeature , Ġwhich Ġis Ġthe Ġonly Ġone Ġbased Ġon Ġhand - crafted Ġresources , Ġyields Ġeven Ġbetter Ġresults . ĠThen Ġour Ġbest Ġmodel Ġ( B - M ) Ġon Ġdevelopment Ġdata Ġrelies Ġentirely Ġon Ġautomatically Ġobtained Ġinformation , Ġboth Ġin Ġterms Ġof Ġtraining Ġdata Ġas Ġwell Ġas Ġfeatures . </s> Results </s> In ĠTable ĠT AB REF 26 Ġwe Ġreport Ġthe Ġresults Ġof Ġour Ġmodel Ġon Ġthe Ġthree Ġdatasets Ġstandard ly Ġused Ġfor Ġthe Ġevaluation Ġof Ġemotion Ġclassification , Ġwhich Ġwe Ġhave Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ. </s> Our ĠB - M Ġmodel Ġrelies Ġon Ġsubs ets Ġof ĠFacebook Ġpages Ġfor Ġtraining , Ġwhich Ġwere Ġchosen Ġaccording Ġto Ġtheir Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġas Ġwell Ġas Ġon Ġthe Ġobservation Ġof Ġemotions Ġdistribution Ġon Ġdifferent Ġpages Ġand Ġin Ġthe Ġdifferent Ġdatasets , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ. ĠThe Ġfeature Ġset Ġwe Ġuse Ġis Ġour Ġbest Ġon Ġthe Ġdevelopment Ġset , Ġnamely Ġall Ġthe Ġfeatures Ġplus ĠGoogle - based Ġembed d ings , Ġbut Ġexcluding Ġthe Ġlex icon . ĠThis Ġmakes Ġour Ġapproach Ġcompletely Ġindependent Ġof Ġany Ġmanual Ġannotation Ġor Ġhand crafted Ġresource . ĠOur Ġmodel 's Ġperformance Ġis Ġcompared Ġto Ġthe Ġfollowing Ġsystems , Ġfor Ġwhich Ġresults Ġare Ġreported Ġin Ġthe Ġreferred Ġliterature . ĠPlease Ġnote Ġthat Ġno Ġother Ġexisting Ġmodel Ġwas Ġre - im ple mented , Ġand Ġresults Ġare Ġthose Ġreported Ġin Ġthe Ġrespective Ġpapers . </s> Discussion , Ġconclusions Ġand Ġfuture Ġwork </s> We Ġhave Ġexplored Ġthe Ġpotential Ġof Ġusing ĠFacebook Ġreactions Ġin Ġa Ġdistant Ġsupervised Ġsetting Ġto Ġperform Ġemotion Ġclassification . ĠThe Ġevaluation Ġon Ġstandard Ġbenchmarks Ġshows Ġthat Ġmodels Ġtrained Ġas Ġsuch , Ġespecially Ġwhen Ġenhanced Ġwith Ġcontinuous Ġvector Ġrepresentations , Ġcan Ġachieve Ġcompetitive Ġresults Ġwithout Ġrelying Ġon Ġany Ġhand crafted Ġresource . ĠAn Ġinteresting Ġaspect Ġof Ġour Ġapproach Ġis Ġthe Ġview Ġto Ġdomain Ġadaptation Ġvia Ġthe Ġselection Ġof ĠFacebook Ġpages Ġto Ġbe Ġused Ġas Ġtraining Ġdata . </s> We Ġbelieve Ġthat Ġthis Ġapproach Ġhas Ġa Ġlot Ġof Ġpotential , Ġand Ġwe Ġsee Ġthe Ġfollowing Ġdirections Ġfor Ġimprovement . ĠFeature - wise , Ġwe Ġwant Ġto Ġtrain Ġemotion - aware Ġembed d ings , Ġin Ġthe Ġvein Ġof Ġwork Ġby Ġtang : 14 , Ġand Ġi ac ob acci 2015 sense mb ed . ĠRetro fitting ĠFB - embed d ings Ġtrained Ġon Ġa Ġlarger Ġcorpus Ġmight Ġalso Ġbe Ġsuccessful , Ġbut Ġwould Ġrely Ġon Ġan Ġexternal Ġlex icon . </s> The Ġlargest Ġroom Ġfor Ġyielding Ġnot Ġonly Ġbetter Ġresults Ġbut Ġalso Ġinteresting Ġinsights Ġon Ġextensions Ġof Ġthis Ġapproach Ġlies Ġin Ġthe Ġchoice Ġof Ġtraining Ġinstances , Ġboth Ġin Ġterms Ġof ĠFacebook Ġpages Ġto Ġget Ġposts Ġfrom , Ġas Ġwell Ġas Ġin Ġwhich Ġposts Ġto Ġselect Ġfrom Ġthe Ġgiven Ġpages . ĠFor Ġthe Ġlatter , Ġone Ġcould Ġfor Ġexample Ġonly Ġselect Ġposts Ġthat Ġhave Ġa Ġcertain Ġlength , Ġignore Ġposts Ġthat Ġare Ġonly Ġquotes Ġor Ġcapt ions Ġto Ġimages , Ġor Ġexpand Ġposts Ġby Ġincluding Ġcontent Ġfrom Ġlinked Ġhtml Ġpages , Ġwhich Ġmight Ġprovide Ġlarger Ġand Ġbetter Ġcontexts ĠB IB REF 23 Ġ. ĠAdditionally , Ġand Ġmost Ġimportantly , Ġone Ġcould Ġuse Ġan Ġentropy - based Ġmeasure Ġto Ġselect Ġonly Ġposts Ġthat Ġhave Ġa Ġstrong Ġemotion Ġrather Ġthan Ġjust Ġconsidering Ġthe Ġmajority Ġemotion Ġas Ġtraining Ġlabel . ĠFor Ġthe Ġformer , Ġnamely Ġthe Ġchoice Ġof ĠFacebook Ġpages , Ġwhich Ġwe Ġbelieve Ġdeserves Ġthe Ġmost Ġinvestigation , Ġone Ġcould Ġexplore Ġseveral Ġavenues , Ġespecially Ġin Ġrelation Ġto Ġstance - based Ġissues ĠB IB REF 24 Ġ. ĠIn Ġour Ġdataset , Ġfor Ġexample , Ġa Ġpost Ġabout ĠChile Ġbeating ĠColombia Ġin Ġa Ġfootball Ġmatch Ġduring Ġthe ĠCop a ĠAmerica Ġhad Ġvery Ġcontradictory Ġreactions , Ġdepending Ġon Ġwhich Ġside Ġreaders Ġwould Ġcheer Ġfor . ĠSimilarly , Ġthe Ġvery Ġsame Ġpolitical Ġevent , Ġfor Ġexample , Ġwould Ġget Ġvery Ġdifferent Ġreactions Ġfrom Ġreaders Ġif Ġit Ġwas Ġposted Ġon ĠFox ĠNews Ġor ĠThe ĠLate ĠNight ĠShow , Ġas Ġthe Ġtarget Ġaudience Ġis Ġlikely Ġto Ġfeel Ġvery Ġdifferently Ġabout Ġthe Ġsame Ġissue . ĠThis Ġalso Ġbrings Ġup Ġtheoretical Ġissues Ġrelated Ġmore Ġgenerally Ġto Ġthe Ġdefinition Ġof Ġthe Ġemotion Ġdetection Ġtask , Ġas Ġit 's Ġstrongly Ġdependent Ġon Ġpersonal Ġtraits Ġof Ġthe Ġaudience . ĠAlso , Ġin Ġthis Ġwork , Ġpages Ġinitially Ġselected Ġon Ġavailability Ġand Ġintuition Ġwere Ġfurther Ġgrouped Ġinto Ġsets Ġto Ġmake Ġtraining Ġdata Ġaccording Ġto Ġperformance Ġon Ġdevelopment Ġdata , Ġand Ġlabel Ġdistribution . ĠAnother Ġcriterion Ġto Ġbe Ġexploited Ġwould Ġbe Ġvocabulary Ġoverlap Ġbetween Ġthe Ġpages Ġand Ġthe Ġdatasets . </s> Lastly , Ġwe Ġcould Ġdevelop Ġsingle Ġmodels Ġfor Ġeach Ġemotion , Ġtreating Ġthe Ġproblem Ġas Ġa Ġmulti - label Ġtask . ĠThis Ġwould Ġeven Ġbetter Ġreflect Ġthe Ġambiguity Ġand Ġsubject ivity Ġintrinsic Ġto Ġassigning Ġemotions Ġto Ġtext , Ġwhere Ġcontent Ġcould Ġbe Ġat Ġsame Ġtime Ġjoyful Ġor Ġsad , Ġdepending Ġon Ġthe Ġreader . </s> Acknowled gements </s> In Ġaddition Ġto Ġthe Ġanonymous Ġreviewers , Ġwe Ġwant Ġto Ġthank ĠLucia ĠPass aro Ġand ĠBarbara ĠPl ank Ġfor Ġinsightful Ġdiscussions , Ġand Ġfor Ġproviding Ġcomments Ġon Ġdraft Ġversions Ġof Ġthis Ġpaper .
Introduction This work is licenced under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram. On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments. Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like”. It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1 . However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection. Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3 . Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4 . Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i.e. using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica. mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon. Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context. We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition. Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets. For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 ). We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method. Facebook reactions as labels For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval. A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)". In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide. Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry. We use such reactions as proxies for emotion labels associated to posts. We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library. We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 ). The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this). The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney. Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016. For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post. The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 ). The resulting emotion vectors must then be turned into an emotion label. In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use. Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions. At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work. Emotion datasets Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. In order to compare our performance to state-of-the-art results, we have used them as well. In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7 . A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation. Affective Text dataset Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines. The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN. The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8 . Valence was to be determined as positive or negative. Classification of emotion and valence were treated as separate tasks. Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100. Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test). Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance. As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained). This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10 . Fairy Tales dataset This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B. Potter, H.C. Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised. In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged. We adopt the same choices. ISEAR The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds. The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions. Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt). In each case, the questions covered the way they had appraised a given situation and how they reacted. The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use. Overview of datasets and emotions We summarise datasets and emotion distribution from two viewpoints. First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments. This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise. All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models. Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9 . In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 ). We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts. We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ). Model There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below. Specifically, we first set on a subset of pages and then experiment with features. Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6 . For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 ). All results reported in this section are on this dataset. The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ). Selecting Facebook pages Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach. First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set. For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages. Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data. Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class. Features In selecting appropriate features, we mainly relied on previous work and intuition. We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data. Results are in Table TABREF20 . Future work will further explore the simultaneous selection of features and page combinations. We use a set of basic text-based features to capture the emotion class. These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation. This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model. We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative. For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion. For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature. As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks. We use three different embeddings: Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14 . This model contains 300-dimensional vectors for 3 million words and phrases. Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences. Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0.01 and dimensionality of 100. We filtered out words with frequency lower than 2 occurrences. Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings. In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space. Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token. Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource. Results on development set We report precision, recall, and f-score on the development set. The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19 . From Table TABREF20 we draw three main observations. First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0.368), although there is a rather substantial variation of scores per class. Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training. Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most. Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results. Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features. Results In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3 . Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4 . The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon. This makes our approach completely independent of any manual annotation or handcrafted resource. Our model's performance is compared to the following systems, for which results are reported in the referred literature. Please note that no other existing model was re-implemented, and results are those reported in the respective papers. Discussion, conclusions and future work We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification. The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource. An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data. We believe that this approach has a lot of potential, and we see the following directions for improvement. Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed. Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon. The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages. For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23 . Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label. For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24 . In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for. Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue. This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience. Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution. Another criterion to be exploited would be vocabulary overlap between the pages and the datasets. Lastly, we could develop single models for each emotion, treating the problem as a multi-label task. This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader. Acknowledgements In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper.
Which existing benchmarks did they compare to?
f875337f2ecd686cd7789e111174d0f14972638d
Which Ġexisting Ġbenchmarks Ġdid Ġthey Ġcompare Ġto ?
A ffect ive ĠText , ĠFairy ĠTales , ĠIS EAR
<s> Which Ġexisting Ġbenchmarks Ġdid Ġthey Ġcompare Ġto ? </s> Introduction </s> This Ġwork Ġis Ġlic enced Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLic ence . ĠLic ence Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> In Ġthe Ġspirit Ġof Ġthe Ġbre vity Ġof Ġsocial Ġmedia 's Ġmessages Ġand Ġreactions , Ġpeople Ġhave Ġgot Ġused Ġto Ġexpress Ġfeelings Ġminim ally Ġand Ġsymbol ically , Ġas Ġwith Ġhasht ags Ġon ĠTwitter Ġand ĠInstagram . ĠOn ĠFacebook , Ġpeople Ġtend Ġto Ġbe Ġmore Ġword y , Ġbut Ġposts Ġnormally Ġreceive Ġmore Ġsimple ĠâĢ ľ l ikes âĢ Ŀ Ġthan Ġlonger Ġcomments . ĠSince ĠFebruary Ġ2016 , ĠFacebook Ġusers Ġcan Ġexpress Ġspecific Ġemotions Ġin Ġresponse Ġto Ġa Ġpost Ġthanks Ġto Ġthe Ġnewly Ġintroduced Ġreaction Ġfeature Ġ( see ĠSection ĠSEC REF 2 Ġ), Ġso Ġthat Ġnow Ġa Ġpost Ġcan Ġbe Ġword lessly Ġmarked Ġwith Ġan Ġexpression Ġof Ġsay ĠâĢ ľ joy " Ġor ĠâĢ ľ sur prise " Ġrather Ġthan Ġa Ġgeneric ĠâĢ ľ like âĢ Ŀ . </s> It Ġhas Ġbeen Ġobserved Ġthat Ġthis Ġnew Ġfeature Ġhelps ĠFacebook Ġto Ġknow Ġmuch Ġmore Ġabout Ġtheir Ġusers Ġand Ġexploit Ġthis Ġinformation Ġfor Ġtargeted Ġadvertising ĠB IB REF 0 Ġ, Ġbut Ġinterest Ġin Ġpeople 's Ġopinions Ġand Ġhow Ġthey Ġfeel Ġisn 't Ġlimited Ġto Ġcommercial Ġreasons , Ġas Ġit Ġinvests Ġsocial Ġmonitoring , Ġtoo , Ġincluding Ġhealth Ġcare Ġand Ġeducation ĠB IB REF 1 Ġ. ĠHowever , Ġemotions Ġand Ġopinions Ġare Ġnot Ġalways Ġexpressed Ġthis Ġexplicitly , Ġso Ġthat Ġthere Ġis Ġhigh Ġinterest Ġin Ġdeveloping Ġsystems Ġtowards Ġtheir Ġautomatic Ġdetection . ĠCreating Ġmanually Ġannot ated Ġdatasets Ġlarge Ġenough Ġto Ġtrain Ġsupervised Ġmodels Ġis Ġnot Ġonly Ġcostly , Ġbut Ġalso âĢĶ especially Ġin Ġthe Ġcase Ġof Ġopinions Ġand Ġemotions âĢĶ diff icult , Ġdue Ġto Ġthe Ġintrinsic Ġsubject ivity Ġof Ġthe Ġtask ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠTherefore , Ġresearch Ġhas Ġfocused Ġon Ġun super vised Ġmethods Ġenriched Ġwith Ġinformation Ġderived Ġfrom Ġlex ica , Ġwhich Ġare Ġmanually Ġcreated ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠSince Ġgo 2009 twitter Ġhave Ġshown Ġthat Ġhappy Ġand Ġsad Ġemot icons Ġcan Ġbe Ġsuccessfully Ġused Ġas Ġsignals Ġfor Ġsentiment Ġlabels , Ġdistant Ġsupervision , Ġi . e . Ġusing Ġsome Ġreasonably Ġsafe Ġsignals Ġas Ġproxies Ġfor Ġautomatically Ġlab elling Ġtraining Ġdata ĠB IB REF 5 Ġ, Ġhas Ġbeen Ġused Ġalso Ġfor Ġemotion Ġrecognition , Ġfor Ġexample Ġexploiting Ġboth Ġemot icons Ġand ĠTwitter Ġhasht ags ĠB IB REF 6 Ġ, Ġbut Ġmainly Ġtowards Ġcreating Ġemotion Ġlex ica . Ġm oh ammad 2015 using Ġuse Ġhasht ags , Ġexperimenting Ġalso Ġwith Ġhighly Ġfine - gr ained Ġemotion Ġsets Ġ( up Ġto Ġalmost Ġ600 Ġemotion Ġlabels ), Ġto Ġcreate Ġthe Ġlarge ĠHash tag ĠEm otion ĠLex icon . ĠEm otic ons Ġare Ġused Ġas Ġproxies Ġalso Ġby Ġhalls m arm ult i , Ġwho Ġuse Ġdistributed Ġvector Ġrepresentations Ġto Ġfind Ġwhich Ġwords Ġare Ġinterchangeable Ġwith Ġemot icons Ġbut Ġalso Ġwhich Ġemot icons Ġare Ġused Ġin Ġa Ġsimilar Ġcontext . </s> We Ġtake Ġadvantage Ġof Ġdistant Ġsupervision Ġby Ġusing ĠFacebook Ġreactions Ġas Ġproxies Ġfor Ġemotion Ġlabels , Ġwhich Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġhasn 't Ġbeen Ġdone Ġyet , Ġand Ġwe Ġtrain Ġa Ġset Ġof ĠSupport ĠVector ĠMachine Ġmodels Ġfor Ġemotion Ġrecognition . ĠOur Ġmodels , Ġdifferently Ġfrom Ġexisting Ġones , Ġexploit Ġinformation Ġwhich Ġis Ġacquired Ġentirely Ġautomatically , Ġand Ġachieve Ġcompetitive Ġor Ġeven Ġstate - of - the - art Ġresults Ġfor Ġsome Ġof Ġthe Ġemotion Ġlabels Ġon Ġexisting , Ġstandard Ġevaluation Ġdatasets . ĠFor Ġexplanatory Ġpurposes , Ġrelated Ġwork Ġis Ġdiscussed Ġfurther Ġand Ġmore Ġin Ġdetail Ġwhen Ġwe Ġdescribe Ġthe Ġbenchmarks Ġfor Ġevaluation Ġ( Section ĠSEC REF 3 Ġ) Ġand Ġwhen Ġwe Ġcompare Ġour Ġmodels Ġto Ġexisting Ġones Ġ( Section ĠSEC REF 5 Ġ). ĠWe Ġalso Ġexplore Ġand Ġdiscuss Ġhow Ġchoosing Ġdifferent Ġsets Ġof ĠFacebook Ġpages Ġas Ġtraining Ġdata Ġprovides Ġan Ġintrinsic Ġdomain - adapt ation Ġmethod . </s> Facebook Ġreactions Ġas Ġlabels </s> For Ġyears , Ġon ĠFacebook Ġpeople Ġcould Ġleave Ġcomments Ġto Ġposts , Ġand Ġalso ĠâĢ ľ like âĢ Ŀ Ġthem , Ġby Ġusing Ġa Ġthumbs - up Ġfeature Ġto Ġexplicitly Ġexpress Ġa Ġgeneric , Ġrather Ġunders pec ified , Ġapproval . ĠA ĠâĢ ľ like âĢ Ŀ Ġcould Ġthus Ġmean ĠâĢ ľ I Ġlike Ġwhat Ġyou Ġsaid ", Ġbut Ġalso ĠâĢ ľ I Ġlike Ġthat Ġyou Ġbring Ġup Ġsuch Ġtopic Ġ( though ĠI Ġfind Ġthe Ġcontent Ġof Ġthe Ġarticle Ġyou Ġlinked Ġannoying ) ". </s> In ĠFebruary Ġ2016 , Ġafter Ġa Ġshort Ġtrial , ĠFacebook Ġmade Ġa Ġmore Ġexplicit Ġreaction Ġfeature Ġavailable Ġworld - wide . ĠRather Ġthan Ġallowing Ġfor Ġthe Ġunders pec ified ĠâĢ ľ like âĢ Ŀ Ġas Ġthe Ġonly Ġword less Ġresponse Ġto Ġa Ġpost , Ġa Ġset Ġof Ġsix Ġmore Ġspecific Ġreactions Ġwas Ġintroduced , Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ: ĠLike , ĠLove , ĠH aha , ĠWow , ĠSad Ġand ĠAngry . ĠWe Ġuse Ġsuch Ġreactions Ġas Ġproxies Ġfor Ġemotion Ġlabels Ġassociated Ġto Ġposts . </s> We Ġcollected ĠFacebook Ġposts Ġand Ġtheir Ġcorresponding Ġreactions Ġfrom Ġpublic Ġpages Ġusing Ġthe ĠFacebook ĠAPI , Ġwhich Ġwe Ġaccessed Ġvia Ġthe ĠFacebook - sd k Ġpython Ġlibrary . ĠWe Ġchose Ġdifferent Ġpages Ġ( and Ġtherefore Ġdomains Ġand Ġstances ), Ġaiming Ġat Ġa Ġbalanced Ġand Ġvaried Ġdataset , Ġbut Ġwe Ġdid Ġso Ġmainly Ġbased Ġon Ġintuition Ġ( see ĠSection ĠSEC REF 4 Ġ) Ġand Ġwith Ġan Ġeye Ġto Ġthe Ġnature Ġof Ġthe Ġdatasets Ġavailable Ġfor Ġevaluation Ġ( see ĠSection ĠSEC REF 5 Ġ). ĠThe Ġchoice Ġof Ġwhich Ġpages Ġto Ġselect Ġposts Ġfrom Ġis Ġfar Ġfrom Ġtrivial , Ġand Ġwe Ġbelieve Ġthis Ġis Ġactually Ġan Ġinteresting Ġaspect Ġof Ġour Ġapproach , Ġas Ġby Ġusing Ġdifferent ĠFacebook Ġpages Ġone Ġcan Ġintrinsically Ġtackle Ġthe Ġdomain - adapt ation Ġproblem Ġ( See ĠSection ĠSEC REF 6 Ġfor Ġfurther Ġdiscussion Ġon Ġthis ). ĠThe Ġfinal Ġcollection Ġof ĠFacebook Ġpages Ġfor Ġthe Ġexperiments Ġdescribed Ġin Ġthis Ġpaper Ġis Ġas Ġfollows : ĠFox News , ĠCNN , ĠESPN , ĠNew ĠYork ĠTimes , ĠTime Ġmagazine , ĠHuffington ĠPost ĠWeird ĠNews , ĠThe ĠGuardian , ĠCartoon ĠNetwork , ĠCooking ĠLight , ĠHome ĠCooking ĠAdventure , ĠJustin ĠBieber , ĠNickel ode on , ĠSponge b ob , ĠDisney . </s> Note Ġthat Ġthankful Ġwas Ġonly Ġavailable Ġduring Ġspecific Ġtime Ġspans Ġrelated Ġto Ġcertain Ġevents , Ġas ĠMother 's ĠDay Ġin ĠMay Ġ2016 . </s> For Ġeach Ġpage , Ġwe Ġdownloaded Ġthe Ġlatest Ġ1000 Ġposts , Ġor Ġthe Ġmaximum Ġavailable Ġif Ġthere Ġare Ġfewer , Ġfrom ĠFebruary Ġ2016 , Ġretrieving Ġthe Ġcounts Ġof Ġreactions Ġfor Ġeach Ġpost . ĠThe Ġoutput Ġis Ġa ĠJSON Ġfile Ġcontaining Ġa Ġlist Ġof Ġdiction aries Ġwith Ġa Ġtimestamp , Ġthe Ġpost Ġand Ġa Ġreaction Ġvector Ġwith Ġfrequency Ġvalues , Ġwhich Ġindicate Ġhow Ġmany Ġusers Ġused Ġthat Ġreaction Ġin Ġresponse Ġto Ġthe Ġpost Ġ( Figure ĠFIG REF 3 Ġ). ĠThe Ġresulting Ġemotion Ġvectors Ġmust Ġthen Ġbe Ġturned Ġinto Ġan Ġemotion Ġlabel . </s> In Ġthe Ġcontext Ġof Ġthis Ġexperiment , Ġwe Ġmade Ġthe Ġsimple Ġdecision Ġof Ġassoci ating Ġto Ġeach Ġpost Ġthe Ġemotion Ġwith Ġthe Ġhighest Ġcount , Ġignoring Ġlike Ġas Ġit Ġis Ġthe Ġdefault Ġand Ġmost Ġgeneric Ġreaction Ġpeople Ġtend Ġto Ġuse . ĠTherefore , Ġfor Ġexample , Ġto Ġthe Ġfirst Ġpost Ġin ĠFigure ĠFIG REF 3 Ġ, Ġwe Ġwould Ġassociate Ġthe Ġlabel Ġsad , Ġas Ġit Ġhas Ġthe Ġhighest Ġscore Ġ( 284 ) Ġamong Ġthe Ġmeaningful Ġemotions Ġwe Ġconsider , Ġthough Ġit Ġalso Ġhas Ġnon - zero Ġscores Ġfor Ġother Ġemotions . ĠAt Ġthis Ġstage , Ġwe Ġdidn 't Ġperform Ġany Ġother Ġentropy - based Ġselection Ġof Ġposts , Ġto Ġbe Ġinvestigated Ġin Ġfuture Ġwork . </s> Em otion Ġdatasets </s> Three Ġdatasets Ġannot ated Ġwith Ġemotions Ġare Ġcommonly Ġused Ġfor Ġthe Ġdevelopment Ġand Ġevaluation Ġof Ġemotion Ġdetection Ġsystems , Ġnamely Ġthe ĠAffect ive ĠText Ġdataset , Ġthe ĠFairy ĠTales Ġdataset , Ġand Ġthe ĠIS EAR Ġdataset . ĠIn Ġorder Ġto Ġcompare Ġour Ġperformance Ġto Ġstate - of - the - art Ġresults , Ġwe Ġhave Ġused Ġthem Ġas Ġwell . ĠIn Ġthis ĠSection , Ġin Ġaddition Ġto Ġa Ġdescription Ġof Ġeach Ġdataset , Ġwe Ġprovide Ġan Ġoverview Ġof Ġthe Ġemotions Ġused , Ġtheir Ġdistribution , Ġand Ġhow Ġwe Ġmapped Ġthem Ġto Ġthose Ġwe Ġobtained Ġfrom ĠFacebook Ġposts Ġin ĠSection ĠSEC REF 7 Ġ. ĠA Ġsummary Ġis Ġprovided Ġin ĠTable ĠT AB REF 8 Ġ, Ġwhich Ġalso Ġshows , Ġin Ġthe Ġbottom Ġrow , Ġwhat Ġrole Ġeach Ġdataset Ġhas Ġin Ġour Ġexperiments : Ġapart Ġfrom Ġthe Ġdevelopment Ġportion Ġof Ġthe ĠAffect ive ĠText , Ġwhich Ġwe Ġused Ġto Ġdevelop Ġour Ġmodels Ġ( Section ĠSEC REF 4 Ġ), Ġall Ġthree Ġhave Ġbeen Ġused Ġas Ġbenchmarks Ġfor Ġour Ġevaluation . </s> A ffect ive ĠText Ġdataset </s> Task Ġ14 Ġat ĠSem E val Ġ2007 ĠB IB REF 7 Ġwas Ġconcerned Ġwith Ġthe Ġclassification Ġof Ġemotions Ġand Ġval ence Ġin Ġnews Ġheadlines . ĠThe Ġheadlines Ġwhere Ġcollected Ġfrom Ġseveral Ġnews Ġwebsites Ġincluding ĠGoogle Ġnews , ĠThe ĠNew ĠYork ĠTimes , ĠBBC ĠNews Ġand ĠCNN . ĠThe Ġused Ġemotion Ġlabels Ġwere ĠAnger , ĠDis g ust , ĠFear , ĠJoy , ĠSad ness , ĠSurprise , Ġin Ġline Ġwith Ġthe Ġsix Ġbasic Ġemotions Ġof ĠEk man 's Ġstandard Ġmodel ĠB IB REF 8 Ġ. ĠVal ence Ġwas Ġto Ġbe Ġdetermined Ġas Ġpositive Ġor Ġnegative . ĠClassification Ġof Ġemotion Ġand Ġval ence Ġwere Ġtreated Ġas Ġseparate Ġtasks . ĠEm otion Ġlabels Ġwere Ġnot Ġconsidered Ġas Ġmut ut ally Ġexclusive , Ġand Ġeach Ġemotion Ġwas Ġassigned Ġa Ġscore Ġfrom Ġ0 Ġto Ġ100 . ĠTraining / develop ing Ġdata Ġamounted Ġto Ġ250 Ġannot ated Ġheadlines Ġ( A ffect ive Ġdevelopment ), Ġwhile Ġsystems Ġwere Ġevaluated Ġon Ġanother Ġ1000 Ġ( A ffect ive Ġtest ). ĠEvaluation Ġwas Ġdone Ġusing Ġtwo Ġdifferent Ġmethods : Ġa Ġfine - gr ained Ġevaluation Ġusing ĠPearson 's Ġr Ġto Ġmeasure Ġthe Ġcorrelation Ġbetween Ġthe Ġsystem Ġscores Ġand Ġthe Ġgold Ġstandard ; Ġand Ġa Ġcoarse - gr ained Ġmethod Ġwhere Ġeach Ġemotion Ġscore Ġwas Ġconverted Ġto Ġa Ġbinary Ġlabel , Ġand Ġprecision , Ġrecall , Ġand Ġf - score Ġwere Ġcomputed Ġto Ġassess Ġperformance . ĠAs Ġit Ġis Ġdone Ġin Ġmost Ġworks Ġthat Ġuse Ġthis Ġdataset ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 9 Ġ, Ġwe Ġalso Ġtreat Ġthis Ġas Ġa Ġclassification Ġproblem Ġ( co arse - gr ained ). ĠThis Ġdataset Ġhas Ġbeen Ġextensively Ġused Ġfor Ġthe Ġevaluation Ġof Ġvarious Ġun super vised Ġmethods ĠB IB REF 2 Ġ, Ġbut Ġalso Ġfor Ġtesting Ġdifferent Ġsupervised Ġlearning Ġtechniques Ġand Ġfeature Ġport ability ĠB IB REF 10 Ġ. </s> F airy ĠTales Ġdataset </s> This Ġis Ġa Ġdataset Ġcollected Ġby Ġal m 2008 aff ect , Ġwhere Ġabout Ġ1 , 000 Ġsentences Ġfrom Ġfairy Ġtales Ġ( by ĠB . ĠPotter , ĠH . C . ĠAndersen Ġand ĠGrimm ) Ġwere Ġannot ated Ġwith Ġthe Ġsame Ġsix Ġemotions Ġof Ġthe ĠAffect ive ĠText Ġdataset , Ġthough Ġwith Ġdifferent Ġnames : ĠAngry , ĠDis g usted , ĠFear ful , ĠHappy , ĠSad , Ġand ĠSur pr ised . ĠIn Ġmost Ġworks Ġthat Ġuse Ġthis Ġdataset ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 9 Ġ, Ġonly Ġsentences Ġwhere Ġall Ġannot ators Ġagreed Ġare Ġused , Ġand Ġthe Ġlabels Ġangry Ġand Ġdisgusted Ġare Ġmerged . ĠWe Ġadopt Ġthe Ġsame Ġchoices . </s> IS EAR </s> The ĠIS EAR Ġ( International ĠSurvey Ġon ĠEm otion ĠAnt ec ed ents Ġand ĠRe actions ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ) Ġis Ġa Ġdataset Ġcreated Ġin Ġthe Ġcontext Ġof Ġa Ġpsychology Ġproject Ġof Ġthe Ġ1990 s , Ġby Ġcollecting Ġquestion naires Ġanswered Ġby Ġpeople Ġwith Ġdifferent Ġcultural Ġbackgrounds . ĠThe Ġmain Ġaim Ġof Ġthis Ġproject Ġwas Ġto Ġgather Ġinsights Ġin Ġcross - cultural Ġaspects Ġof Ġemotional Ġreactions . ĠStudent Ġrespondents , Ġboth Ġpsychologists Ġand Ġnon - psych ologists , Ġwere Ġasked Ġto Ġreport Ġsituations Ġin Ġwhich Ġthey Ġhad Ġexperienced Ġall Ġof Ġseven Ġmajor Ġemotions Ġ( joy , Ġfear , Ġanger , Ġsadness , Ġdisgust , Ġshame Ġand Ġguilt ). ĠIn Ġeach Ġcase , Ġthe Ġquestions Ġcovered Ġthe Ġway Ġthey Ġhad Ġappra ised Ġa Ġgiven Ġsituation Ġand Ġhow Ġthey Ġreacted . ĠThe Ġfinal Ġdataset Ġcontains Ġreports Ġby Ġapproximately Ġ3000 Ġrespondents Ġfrom Ġall Ġover Ġthe Ġworld , Ġfor Ġa Ġtotal Ġof Ġ7 665 Ġsentences Ġlabelled Ġwith Ġan Ġemotion , Ġmaking Ġthis Ġthe Ġlargest Ġdataset Ġout Ġof Ġthe Ġthree Ġwe Ġuse . </s> Overview Ġof Ġdatasets Ġand Ġemotions </s> We Ġsummar ise Ġdatasets Ġand Ġemotion Ġdistribution Ġfrom Ġtwo Ġviewpoints . ĠFirst , Ġbecause Ġthere Ġare Ġdifferent Ġsets Ġof Ġemotions Ġlabels Ġin Ġthe Ġdatasets Ġand ĠFacebook Ġdata , Ġwe Ġneed Ġto Ġprovide Ġa Ġmapping Ġand Ġderive Ġa Ġsubset Ġof Ġemotions Ġthat Ġwe Ġare Ġgoing Ġto Ġuse Ġfor Ġthe Ġexperiments . ĠThis Ġis Ġshown Ġin ĠTable ĠT AB REF 8 Ġ, Ġwhere Ġin Ġthe ĠâĢ ľ M apped âĢ Ŀ Ġcolumn Ġwe Ġreport Ġthe Ġfinal Ġemotions Ġwe Ġuse Ġin Ġthis Ġpaper : Ġanger , Ġjoy , Ġsadness , Ġsurprise . ĠAll Ġlabels Ġin Ġeach Ġdataset Ġare Ġmapped Ġto Ġthese Ġfinal Ġemotions , Ġwhich Ġare Ġtherefore Ġthe Ġlabels Ġwe Ġuse Ġfor Ġtraining Ġand Ġtesting Ġour Ġmodels . </s> Second , Ġthe Ġdistribution Ġof Ġthe Ġemotions Ġfor Ġeach Ġdataset Ġis Ġdifferent , Ġas Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 9 Ġ. </s> In ĠFigure ĠFIG REF 9 Ġwe Ġalso Ġprovide Ġthe Ġdistribution Ġof Ġthe Ġemotions Ġanger , Ġjoy , Ġsadness , Ġsurprise Ġper ĠFacebook Ġpage , Ġin Ġterms Ġof Ġnumber Ġof Ġposts Ġ( rec all Ġthat Ġwe Ġassign Ġto Ġa Ġpost Ġthe Ġlabel Ġcorresponding Ġto Ġthe Ġmajority Ġemotion Ġassociated Ġto Ġit , Ġsee ĠSection ĠSEC REF 2 Ġ). ĠWe Ġcan Ġobserve Ġthat Ġfor Ġexample Ġpages Ġabout Ġnews Ġtend Ġto Ġhave Ġmore Ġsadness Ġand Ġanger Ġposts , Ġwhile Ġpages Ġabout Ġcooking Ġand Ġtv - shows Ġhave Ġa Ġhigh Ġpercentage Ġof Ġjoy Ġposts . ĠWe Ġwill Ġuse Ġthis Ġinformation Ġto Ġfind Ġthe Ġbest Ġset Ġof Ġpages Ġfor Ġa Ġgiven Ġtarget Ġdomain Ġ( see ĠSection ĠSEC REF 5 Ġ). </s> Model </s> There Ġare Ġtwo Ġmain Ġdecisions Ġto Ġbe Ġtaken Ġin Ġdeveloping Ġour Ġmodel : Ġ( i ) Ġwhich ĠFacebook Ġpages Ġto Ġselect Ġas Ġtraining Ġdata , Ġand Ġ( ii ) Ġwhich Ġfeatures Ġto Ġuse Ġto Ġtrain Ġthe Ġmodel , Ġwhich Ġwe Ġdiscuss Ġbelow . ĠSpecifically , Ġwe Ġfirst Ġset Ġon Ġa Ġsubset Ġof Ġpages Ġand Ġthen Ġexperiment Ġwith Ġfeatures . ĠFurther Ġexploration Ġof Ġthe Ġinteraction Ġbetween Ġchoice Ġof Ġpages Ġand Ġchoice Ġof Ġfeatures Ġis Ġleft Ġto Ġfuture Ġwork , Ġand Ġpartly Ġdiscussed Ġin ĠSection ĠSEC REF 6 Ġ. ĠFor Ġdevelopment , Ġwe Ġuse Ġa Ġsmall Ġportion Ġof Ġthe ĠAffect ive Ġdata Ġset Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ, Ġthat Ġis Ġthe Ġportion Ġthat Ġhad Ġbeen Ġreleased Ġas Ġdevelopment Ġset Ġfor ĠSem E val 's Ġ2007 ĠTask Ġ14 ĠB IB REF 7 Ġ, Ġwhich Ġcontains Ġ250 Ġannot ated Ġsentences Ġ( A ffect ive Ġdevelopment , ĠSection ĠSEC REF 4 Ġ). ĠAll Ġresults Ġreported Ġin Ġthis Ġsection Ġare Ġon Ġthis Ġdataset . ĠThe Ġtest Ġset Ġof ĠTask Ġ14 Ġas Ġwell Ġas Ġthe Ġother Ġtwo Ġdatasets Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġwill Ġbe Ġused Ġto Ġevaluate Ġthe Ġfinal Ġmodels Ġ( Section ĠSEC REF 4 Ġ). </s> Select ing ĠFacebook Ġpages </s> Although Ġpage Ġselection Ġis Ġa Ġcrucial Ġingredient Ġof Ġthis Ġapproach , Ġwhich Ġwe Ġbelieve Ġcalls Ġfor Ġfurther Ġand Ġdeeper , Ġdedicated Ġinvestigation , Ġfor Ġthe Ġexperiments Ġdescribed Ġhere Ġwe Ġtook Ġa Ġrather Ġsimple Ġapproach . ĠFirst , Ġwe Ġselected Ġthe Ġpages Ġthat Ġwould Ġprovide Ġtraining Ġdata Ġbased Ġon Ġintuition Ġand Ġavailability , Ġthen Ġchose Ġdifferent Ġcombinations Ġaccording Ġto Ġresults Ġof Ġa Ġbasic Ġmodel Ġrun Ġon Ġdevelopment Ġdata , Ġand Ġeventually Ġtested Ġfeature Ġcombinations , Ġstill Ġon Ġthe Ġdevelopment Ġset . </s> For Ġthe Ġsake Ġof Ġsimplicity Ġand Ġtransparency , Ġwe Ġfirst Ġtrained Ġan ĠS VM Ġwith Ġa Ġsimple Ġbag - of - words Ġmodel Ġand Ġdefault Ġparameters Ġas Ġper Ġthe ĠSci kit - learn Ġimplementation ĠB IB REF 13 Ġon Ġdifferent Ġcombinations Ġof Ġpages . ĠBased Ġon Ġresults Ġof Ġthe Ġattempted Ġcombinations Ġas Ġwell Ġas Ġon Ġthe Ġdistribution Ġof Ġemotions Ġin Ġthe Ġdevelopment Ġdataset Ġ( Figure ĠFIG REF 9 Ġ), Ġwe Ġselected Ġa Ġbest Ġmodel Ġ( B - M ), Ġnamely Ġthe Ġcombined Ġset Ġof ĠTime , ĠThe ĠGuardian Ġand ĠDisney , Ġwhich Ġyields Ġthe Ġhighest Ġresults Ġon Ġdevelopment Ġdata . ĠTime Ġand ĠThe ĠGuardian Ġperform Ġwell Ġon Ġmost Ġemotions Ġbut ĠDisney Ġhelps Ġto Ġboost Ġthe Ġperformance Ġfor Ġthe ĠJoy Ġclass . </s> Features </s> In Ġselecting Ġappropriate Ġfeatures , Ġwe Ġmainly Ġrelied Ġon Ġprevious Ġwork Ġand Ġintuition . ĠWe Ġexperimented Ġwith Ġdifferent Ġcombinations , Ġand Ġall Ġtests Ġwere Ġstill Ġdone Ġon ĠAffect ive Ġdevelopment , Ġusing Ġthe Ġpages Ġfor Ġthe Ġbest Ġmodel Ġ( B - M ) Ġdescribed Ġabove Ġas Ġtraining Ġdata . ĠResults Ġare Ġin ĠTable ĠT AB REF 20 Ġ. ĠFuture Ġwork Ġwill Ġfurther Ġexplore Ġthe Ġsimultaneous Ġselection Ġof Ġfeatures Ġand Ġpage Ġcombinations . </s> We Ġuse Ġa Ġset Ġof Ġbasic Ġtext - based Ġfeatures Ġto Ġcapture Ġthe Ġemotion Ġclass . ĠThese Ġinclude Ġa Ġtf - id f Ġbag - of - words Ġfeature , Ġword Ġ( 2 - 3 ) Ġand Ġcharacter Ġ( 2 - 5 ) Ġn gram s , Ġand Ġfeatures Ġrelated Ġto Ġthe Ġpresence Ġof Ġneg ation Ġwords , Ġand Ġto Ġthe Ġusage Ġof Ġpunct uation . </s> This Ġfeature Ġis Ġused Ġin Ġall Ġun super vised Ġmodels Ġas Ġa Ġsource Ġof Ġinformation , Ġand Ġwe Ġmainly Ġinclude Ġit Ġto Ġassess Ġits Ġcontribution , Ġbut Ġeventually Ġdo Ġnot Ġuse Ġit Ġin Ġour Ġfinal Ġmodel . </s> We Ġused Ġthe ĠN RC 10 ĠLex icon Ġbecause Ġit Ġperformed Ġbest Ġin Ġthe Ġexperiments Ġby ĠB IB REF 10 Ġ, Ġwhich Ġis Ġbuilt Ġaround Ġthe Ġemotions Ġanger , Ġanticipation , Ġdisgust , Ġfear , Ġjoy , Ġsadness , Ġand Ġsurprise , Ġand Ġthe Ġval ence Ġvalues Ġpositive Ġand Ġnegative . ĠFor Ġeach Ġword Ġin Ġthe Ġlex icon , Ġa Ġboolean Ġvalue Ġindicating Ġpresence Ġor Ġabsence Ġis Ġassociated Ġto Ġeach Ġemotion . ĠFor Ġa Ġwhole Ġsentence , Ġa Ġglobal Ġscore Ġper Ġemotion Ġcan Ġbe Ġobtained Ġby Ġsum ming Ġthe Ġvectors Ġfor Ġall Ġcontent Ġwords Ġof Ġthat Ġsentence Ġincluded Ġin Ġthe Ġlex icon , Ġand Ġused Ġas Ġfeature . </s> As Ġadditional Ġfeature , Ġwe Ġalso Ġincluded ĠWord ĠEmb edd ings , Ġnamely Ġdistributed Ġrepresentations Ġof Ġwords Ġin Ġa Ġvector Ġspace , Ġwhich Ġhave Ġbeen Ġexceptionally Ġsuccessful Ġin Ġboosting Ġperformance Ġin Ġa Ġplethora Ġof ĠN LP Ġtasks . ĠWe Ġuse Ġthree Ġdifferent Ġembed d ings : </s> Google Ġembed d ings : Ġpre - trained Ġembed d ings Ġtrained Ġon ĠGoogle ĠNews Ġand Ġobtained Ġwith Ġthe Ġskip - gram Ġarchitecture Ġdescribed Ġin ĠB IB REF 14 Ġ. ĠThis Ġmodel Ġcontains Ġ300 - dimensional Ġvectors Ġfor Ġ3 Ġmillion Ġwords Ġand Ġphrases . </s> Facebook Ġembed d ings : Ġembed d ings Ġthat Ġwe Ġtrained Ġon Ġour Ġscrap ed ĠFacebook Ġpages Ġfor Ġa Ġtotal Ġof Ġ20 , 000 Ġsentences . ĠUsing Ġthe Ġg ens im Ġlibrary ĠB IB REF 15 Ġ, Ġwe Ġtrained Ġthe Ġembed d ings Ġwith Ġthe Ġfollowing Ġparameters : Ġwindow Ġsize Ġof Ġ5 , Ġlearning Ġrate Ġof Ġ0 . 01 Ġand Ġdimension ality Ġof Ġ100 . ĠWe Ġfiltered Ġout Ġwords Ġwith Ġfrequency Ġlower Ġthan Ġ2 Ġoccurrences . </s> Ret ro fitted Ġembed d ings : ĠRetro fitting ĠB IB REF 16 Ġhas Ġbeen Ġshown Ġas Ġa Ġsimple Ġbut Ġefficient Ġway Ġof Ġinforming Ġtrained Ġembed d ings Ġwith Ġadditional Ġinformation Ġderived Ġfrom Ġsome Ġlex ical Ġresource , Ġrather Ġthan Ġincluding Ġit Ġdirectly Ġat Ġthe Ġtraining Ġstage , Ġas Ġit 's Ġdone Ġfor Ġexample Ġto Ġcreate Ġsense - aware ĠB IB REF 17 Ġor Ġsentiment - aware ĠB IB REF 18 Ġembed d ings . ĠIn Ġthis Ġwork , Ġwe Ġretro fit Ġgeneral Ġembed d ings Ġto Ġinclude Ġinformation Ġabout Ġemotions , Ġso Ġthat Ġemotion - similar Ġwords Ġcan Ġget Ġcloser Ġin Ġspace . ĠBoth Ġthe ĠGoogle Ġas Ġwell Ġas Ġour ĠFacebook Ġembed d ings Ġwere Ġretro fitted Ġwith Ġlex ical Ġinformation Ġobtained Ġfrom Ġthe ĠN RC 10 ĠLex icon Ġmentioned Ġabove , Ġwhich Ġprovides Ġemotion - similar ity Ġfor Ġeach Ġtoken . ĠNote Ġthat Ġdifferently Ġfrom Ġthe Ġprevious Ġtwo Ġtypes Ġof Ġembed d ings , Ġthe Ġretro fitted Ġones Ġdo Ġrely Ġon Ġhand crafted Ġinformation Ġin Ġthe Ġform Ġof Ġa Ġlex ical Ġresource . </s> Results Ġon Ġdevelopment Ġset </s> We Ġreport Ġprecision , Ġrecall , Ġand Ġf - score Ġon Ġthe Ġdevelopment Ġset . ĠThe Ġaverage Ġf - score Ġis Ġreported Ġas Ġmicro - average , Ġto Ġbetter Ġaccount Ġfor Ġthe Ġskewed Ġdistribution Ġof Ġthe Ġclasses Ġas Ġwell Ġas Ġin Ġaccordance Ġto Ġwhat Ġis Ġusually Ġreported Ġfor Ġthis Ġtask ĠB IB REF 19 Ġ. </s> From ĠTable ĠT AB REF 20 Ġwe Ġdraw Ġthree Ġmain Ġobservations . ĠFirst , Ġa Ġsimple Ġtf - id f Ġbag - of - word Ġmode Ġworks Ġalready Ġvery Ġwell , Ġto Ġthe Ġpoint Ġthat Ġthe Ġother Ġtextual Ġand Ġlex icon - based Ġfeatures Ġdon 't Ġseem Ġto Ġcontribute Ġto Ġthe Ġoverall Ġf - score Ġ( 0 . 368 ), Ġalthough Ġthere Ġis Ġa Ġrather Ġsubstantial Ġvariation Ġof Ġscores Ġper Ġclass . ĠSecond , ĠGoogle Ġembed d ings Ġperform Ġa Ġlot Ġbetter Ġthan ĠFacebook Ġembed d ings , Ġand Ġthis Ġis Ġlikely Ġdue Ġto Ġthe Ġsize Ġof Ġthe Ġcorpus Ġused Ġfor Ġtraining . ĠRetro fitting Ġdoesn 't Ġseem Ġto Ġhelp Ġat Ġall Ġfor Ġthe ĠGoogle Ġembed d ings , Ġbut Ġit Ġdoes Ġboost Ġthe ĠFacebook Ġembed d ings , Ġleading Ġto Ġthink Ġthat Ġwith Ġlittle Ġdata , Ġmore Ġaccurate Ġtask - related Ġinformation Ġis Ġhelping , Ġbut Ġcorpus Ġsize Ġmatters Ġmost . ĠThird , Ġin Ġcombination Ġwith Ġembed d ings , Ġall Ġfeatures Ġwork Ġbetter Ġthan Ġjust Ġusing Ġtf - id f , Ġbut Ġremoving Ġthe ĠLex icon Ġfeature , Ġwhich Ġis Ġthe Ġonly Ġone Ġbased Ġon Ġhand - crafted Ġresources , Ġyields Ġeven Ġbetter Ġresults . ĠThen Ġour Ġbest Ġmodel Ġ( B - M ) Ġon Ġdevelopment Ġdata Ġrelies Ġentirely Ġon Ġautomatically Ġobtained Ġinformation , Ġboth Ġin Ġterms Ġof Ġtraining Ġdata Ġas Ġwell Ġas Ġfeatures . </s> Results </s> In ĠTable ĠT AB REF 26 Ġwe Ġreport Ġthe Ġresults Ġof Ġour Ġmodel Ġon Ġthe Ġthree Ġdatasets Ġstandard ly Ġused Ġfor Ġthe Ġevaluation Ġof Ġemotion Ġclassification , Ġwhich Ġwe Ġhave Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ. </s> Our ĠB - M Ġmodel Ġrelies Ġon Ġsubs ets Ġof ĠFacebook Ġpages Ġfor Ġtraining , Ġwhich Ġwere Ġchosen Ġaccording Ġto Ġtheir Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġas Ġwell Ġas Ġon Ġthe Ġobservation Ġof Ġemotions Ġdistribution Ġon Ġdifferent Ġpages Ġand Ġin Ġthe Ġdifferent Ġdatasets , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 4 Ġ. ĠThe Ġfeature Ġset Ġwe Ġuse Ġis Ġour Ġbest Ġon Ġthe Ġdevelopment Ġset , Ġnamely Ġall Ġthe Ġfeatures Ġplus ĠGoogle - based Ġembed d ings , Ġbut Ġexcluding Ġthe Ġlex icon . ĠThis Ġmakes Ġour Ġapproach Ġcompletely Ġindependent Ġof Ġany Ġmanual Ġannotation Ġor Ġhand crafted Ġresource . ĠOur Ġmodel 's Ġperformance Ġis Ġcompared Ġto Ġthe Ġfollowing Ġsystems , Ġfor Ġwhich Ġresults Ġare Ġreported Ġin Ġthe Ġreferred Ġliterature . ĠPlease Ġnote Ġthat Ġno Ġother Ġexisting Ġmodel Ġwas Ġre - im ple mented , Ġand Ġresults Ġare Ġthose Ġreported Ġin Ġthe Ġrespective Ġpapers . </s> Discussion , Ġconclusions Ġand Ġfuture Ġwork </s> We Ġhave Ġexplored Ġthe Ġpotential Ġof Ġusing ĠFacebook Ġreactions Ġin Ġa Ġdistant Ġsupervised Ġsetting Ġto Ġperform Ġemotion Ġclassification . ĠThe Ġevaluation Ġon Ġstandard Ġbenchmarks Ġshows Ġthat Ġmodels Ġtrained Ġas Ġsuch , Ġespecially Ġwhen Ġenhanced Ġwith Ġcontinuous Ġvector Ġrepresentations , Ġcan Ġachieve Ġcompetitive Ġresults Ġwithout Ġrelying Ġon Ġany Ġhand crafted Ġresource . ĠAn Ġinteresting Ġaspect Ġof Ġour Ġapproach Ġis Ġthe Ġview Ġto Ġdomain Ġadaptation Ġvia Ġthe Ġselection Ġof ĠFacebook Ġpages Ġto Ġbe Ġused Ġas Ġtraining Ġdata . </s> We Ġbelieve Ġthat Ġthis Ġapproach Ġhas Ġa Ġlot Ġof Ġpotential , Ġand Ġwe Ġsee Ġthe Ġfollowing Ġdirections Ġfor Ġimprovement . ĠFeature - wise , Ġwe Ġwant Ġto Ġtrain Ġemotion - aware Ġembed d ings , Ġin Ġthe Ġvein Ġof Ġwork Ġby Ġtang : 14 , Ġand Ġi ac ob acci 2015 sense mb ed . ĠRetro fitting ĠFB - embed d ings Ġtrained Ġon Ġa Ġlarger Ġcorpus Ġmight Ġalso Ġbe Ġsuccessful , Ġbut Ġwould Ġrely Ġon Ġan Ġexternal Ġlex icon . </s> The Ġlargest Ġroom Ġfor Ġyielding Ġnot Ġonly Ġbetter Ġresults Ġbut Ġalso Ġinteresting Ġinsights Ġon Ġextensions Ġof Ġthis Ġapproach Ġlies Ġin Ġthe Ġchoice Ġof Ġtraining Ġinstances , Ġboth Ġin Ġterms Ġof ĠFacebook Ġpages Ġto Ġget Ġposts Ġfrom , Ġas Ġwell Ġas Ġin Ġwhich Ġposts Ġto Ġselect Ġfrom Ġthe Ġgiven Ġpages . ĠFor Ġthe Ġlatter , Ġone Ġcould Ġfor Ġexample Ġonly Ġselect Ġposts Ġthat Ġhave Ġa Ġcertain Ġlength , Ġignore Ġposts Ġthat Ġare Ġonly Ġquotes Ġor Ġcapt ions Ġto Ġimages , Ġor Ġexpand Ġposts Ġby Ġincluding Ġcontent Ġfrom Ġlinked Ġhtml Ġpages , Ġwhich Ġmight Ġprovide Ġlarger Ġand Ġbetter Ġcontexts ĠB IB REF 23 Ġ. ĠAdditionally , Ġand Ġmost Ġimportantly , Ġone Ġcould Ġuse Ġan Ġentropy - based Ġmeasure Ġto Ġselect Ġonly Ġposts Ġthat Ġhave Ġa Ġstrong Ġemotion Ġrather Ġthan Ġjust Ġconsidering Ġthe Ġmajority Ġemotion Ġas Ġtraining Ġlabel . ĠFor Ġthe Ġformer , Ġnamely Ġthe Ġchoice Ġof ĠFacebook Ġpages , Ġwhich Ġwe Ġbelieve Ġdeserves Ġthe Ġmost Ġinvestigation , Ġone Ġcould Ġexplore Ġseveral Ġavenues , Ġespecially Ġin Ġrelation Ġto Ġstance - based Ġissues ĠB IB REF 24 Ġ. ĠIn Ġour Ġdataset , Ġfor Ġexample , Ġa Ġpost Ġabout ĠChile Ġbeating ĠColombia Ġin Ġa Ġfootball Ġmatch Ġduring Ġthe ĠCop a ĠAmerica Ġhad Ġvery Ġcontradictory Ġreactions , Ġdepending Ġon Ġwhich Ġside Ġreaders Ġwould Ġcheer Ġfor . ĠSimilarly , Ġthe Ġvery Ġsame Ġpolitical Ġevent , Ġfor Ġexample , Ġwould Ġget Ġvery Ġdifferent Ġreactions Ġfrom Ġreaders Ġif Ġit Ġwas Ġposted Ġon ĠFox ĠNews Ġor ĠThe ĠLate ĠNight ĠShow , Ġas Ġthe Ġtarget Ġaudience Ġis Ġlikely Ġto Ġfeel Ġvery Ġdifferently Ġabout Ġthe Ġsame Ġissue . ĠThis Ġalso Ġbrings Ġup Ġtheoretical Ġissues Ġrelated Ġmore Ġgenerally Ġto Ġthe Ġdefinition Ġof Ġthe Ġemotion Ġdetection Ġtask , Ġas Ġit 's Ġstrongly Ġdependent Ġon Ġpersonal Ġtraits Ġof Ġthe Ġaudience . ĠAlso , Ġin Ġthis Ġwork , Ġpages Ġinitially Ġselected Ġon Ġavailability Ġand Ġintuition Ġwere Ġfurther Ġgrouped Ġinto Ġsets Ġto Ġmake Ġtraining Ġdata Ġaccording Ġto Ġperformance Ġon Ġdevelopment Ġdata , Ġand Ġlabel Ġdistribution . ĠAnother Ġcriterion Ġto Ġbe Ġexploited Ġwould Ġbe Ġvocabulary Ġoverlap Ġbetween Ġthe Ġpages Ġand Ġthe Ġdatasets . </s> Lastly , Ġwe Ġcould Ġdevelop Ġsingle Ġmodels Ġfor Ġeach Ġemotion , Ġtreating Ġthe Ġproblem Ġas Ġa Ġmulti - label Ġtask . ĠThis Ġwould Ġeven Ġbetter Ġreflect Ġthe Ġambiguity Ġand Ġsubject ivity Ġintrinsic Ġto Ġassigning Ġemotions Ġto Ġtext , Ġwhere Ġcontent Ġcould Ġbe Ġat Ġsame Ġtime Ġjoyful Ġor Ġsad , Ġdepending Ġon Ġthe Ġreader . </s> Acknowled gements </s> In Ġaddition Ġto Ġthe Ġanonymous Ġreviewers , Ġwe Ġwant Ġto Ġthank ĠLucia ĠPass aro Ġand ĠBarbara ĠPl ank Ġfor Ġinsightful Ġdiscussions , Ġand Ġfor Ġproviding Ġcomments Ġon Ġdraft Ġversions Ġof Ġthis Ġpaper .