Answer with content missing: (Subscript 1: "We did not participate in subtask 5 (E-c)") Authors participated in EI-Reg, EI-Oc, V-Reg and V-Oc subtasks.
ABSTRACTIVE
Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets. This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used. Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made. We conclude with some possible ideas for future work.
1805.10824
Introduction </s> Understanding Ġthe Ġemotions Ġexpressed Ġin Ġa Ġtext Ġor Ġmessage Ġis Ġof Ġhigh Ġrelevance Ġnowadays . ĠCompanies Ġare Ġinterested Ġin Ġthis Ġto Ġget Ġan Ġunderstanding Ġof Ġthe Ġsentiment Ġof Ġtheir Ġcurrent Ġcustomers Ġregarding Ġtheir Ġproducts Ġand Ġthe Ġsentiment Ġof Ġtheir Ġpotential Ġcustomers Ġto Ġattract Ġnew Ġones . ĠMoreover , Ġchanges Ġin Ġa Ġproduct Ġor Ġa Ġcompany Ġmay Ġalso Ġaffect Ġthe Ġsentiment Ġof Ġa Ġcustomer . ĠHowever , Ġthe Ġintensity Ġof Ġan Ġemotion Ġis Ġcrucial Ġin Ġdetermining Ġthe Ġurgency Ġand Ġimportance Ġof Ġthat Ġsentiment . ĠIf Ġsomeone Ġis Ġonly Ġslightly Ġhappy Ġabout Ġa Ġproduct , Ġis Ġa Ġcustomer Ġwilling Ġto Ġbuy Ġit Ġagain ? ĠConversely , Ġif Ġsomeone Ġis Ġvery Ġangry Ġabout Ġcustomer Ġservice , Ġhis Ġor Ġher Ġcomplaint Ġmight Ġbe Ġgiven Ġpriority Ġover Ġsomewhat Ġmild er Ġcomplaints . </s> BI B REF 0 Ġpresent Ġfour Ġtasks Ġin Ġwhich Ġsystems Ġhave Ġto Ġautomatically Ġdetermine Ġthe Ġintensity Ġof Ġemotions Ġ( E I ) Ġor Ġthe Ġintensity Ġof Ġthe Ġsentiment Ġ( Val ence ) Ġof Ġtweets Ġin Ġthe Ġlanguages ĠEnglish , ĠArabic , Ġand ĠSpanish . ĠThe Ġgoal Ġis Ġto Ġeither Ġpredict Ġa Ġcontinuous Ġregression Ġ( reg ) Ġvalue Ġor Ġto Ġdo Ġord inal Ġclassification Ġ( oc ) Ġbased Ġon Ġa Ġnumber Ġof Ġpred efined Ġcategories . ĠThe ĠE I Ġtasks Ġhave Ġseparate Ġtraining Ġsets Ġfor Ġfour Ġdifferent Ġemotions : Ġanger , Ġfear , Ġjoy Ġand Ġsadness . ĠDue Ġto Ġthe Ġlarge Ġnumber Ġof Ġsubt asks Ġand Ġthe Ġfact Ġthat Ġthis Ġlanguage Ġdoes Ġnot Ġhave Ġmany Ġresources Ġreadily Ġavailable , Ġwe Ġonly Ġfocus Ġon Ġthe ĠSpanish Ġsubt asks . ĠOur Ġwork Ġmakes Ġthe Ġfollowing Ġcontributions : </s> Our Ġsubmissions Ġranked Ġsecond Ġ( E I - Reg ), Ġsecond Ġ( E I - O c ), Ġfourth Ġ( V - Reg ) Ġand Ġfifth Ġ( V - O c ), Ġdemonstrating Ġthat Ġthe Ġproposed Ġmethod Ġis Ġaccurate Ġin Ġautomatically Ġdetermining Ġthe Ġintensity Ġof Ġemotions Ġand Ġsentiment Ġof ĠSpanish Ġtweets . ĠThis Ġpaper Ġwill Ġfirst Ġfocus Ġon Ġthe Ġdatasets , Ġthe Ġdata Ġgeneration Ġprocedure , Ġand Ġthe Ġtechniques Ġand Ġtools Ġused . ĠThen Ġwe Ġpresent Ġthe Ġresults Ġin Ġdetail , Ġafter Ġwhich Ġwe Ġperform Ġa Ġsmall Ġerror Ġanalysis Ġon Ġthe Ġlargest Ġmistakes Ġour Ġmodel Ġmade . ĠWe Ġconclude Ġwith Ġsome Ġpossible Ġideas Ġfor Ġfuture Ġwork . </s> Data </s> For Ġeach Ġtask , Ġthe Ġtraining Ġdata Ġthat Ġwas Ġmade Ġavailable Ġby Ġthe Ġorganizers Ġis Ġused , Ġwhich Ġis Ġa Ġselection Ġof Ġtweets Ġwith Ġfor Ġeach Ġtweet Ġa Ġlabel Ġdescribing Ġthe Ġintensity Ġof Ġthe Ġemotion Ġor Ġsentiment ĠB IB REF 1 Ġ. ĠLinks Ġand Ġus ern ames Ġwere Ġreplaced Ġby Ġthe Ġgeneral Ġtokens ĠURL Ġand Ġ@ username , Ġafter Ġwhich Ġthe Ġtweets Ġwere Ġtoken ized Ġby Ġusing ĠTweet Token izer . ĠAll Ġtext Ġwas Ġlower c ased . ĠIn Ġa Ġpost - processing Ġstep , Ġit Ġwas Ġensured Ġthat Ġeach Ġemoji Ġis Ġtoken ized Ġas Ġa Ġsingle Ġtoken . </s> Word ĠEmb edd ings </s> To Ġbe Ġable Ġto Ġtrain Ġword Ġembed d ings , ĠSpanish Ġtweets Ġwere Ġscrap ed Ġbetween ĠNovember Ġ8 , Ġ2017 Ġand ĠJanuary Ġ12 , Ġ2018 . ĠWe Ġchose Ġto Ġcreate Ġour Ġown Ġembed d ings Ġinstead Ġof Ġusing Ġpre - trained Ġembed d ings , Ġbecause Ġthis Ġway Ġthe Ġembed d ings Ġwould Ġresemble Ġthe Ġprovided Ġdata Ġset : Ġboth Ġare Ġbased Ġon ĠTwitter Ġdata . ĠAdded Ġto Ġthis Ġset Ġwas Ġthe ĠAffect Ġin ĠTwe ets ĠDist ant ĠSuper vision ĠCorpus Ġ( DIS C ) Ġmade Ġavailable Ġby Ġthe Ġorganizers ĠB IB REF 0 Ġand Ġa Ġset Ġof Ġ4 . 1 Ġmillion Ġtweets Ġfrom Ġ2015 , Ġobtained Ġfrom ĠB IB REF 2 Ġ. ĠAfter Ġremoving Ġduplicate Ġtweets Ġand Ġtweets Ġwith Ġfewer Ġthan Ġten Ġtokens , Ġthis Ġresulted Ġin Ġa Ġset Ġof Ġ58 . 7 Ġmillion Ġtweets , Ġcontaining Ġ1 . 1 Ġbillion Ġtokens . ĠThe Ġtweets Ġwere Ġpre process ed Ġusing Ġthe Ġmethod Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġ. ĠThe Ġword Ġembed d ings Ġwere Ġcreated Ġusing Ġword 2 vec Ġin Ġthe Ġg ens im Ġlibrary ĠB IB REF 3 Ġ, Ġusing ĠCB OW , Ġa Ġwindow Ġsize Ġof Ġ40 Ġand Ġa Ġminimum Ġcount Ġof Ġ5 . ĠThe Ġfeature Ġvectors Ġfor Ġeach Ġtweet Ġwere Ġthen Ġcreated Ġby Ġusing Ġthe ĠAffect ive Twe ets ĠWE KA Ġpackage ĠB IB REF 4 Ġ. </s> Trans l ating ĠLex icons </s> Most Ġlex ical Ġresources Ġfor Ġsentiment Ġanalysis Ġare Ġin ĠEnglish . ĠTo Ġstill Ġbe Ġable Ġto Ġbenefit Ġfrom Ġthese Ġsources , Ġthe Ġlex icons Ġin Ġthe ĠAffect ive Twe ets Ġpackage Ġwere Ġtranslated Ġto ĠSpanish , Ġusing Ġthe Ġmachine Ġtranslation Ġplatform ĠA pert ium ĠB IB REF 5 Ġ. </s> All Ġlex icons Ġfrom Ġthe ĠAffect ive Twe ets Ġpackage Ġwere Ġtranslated , Ġexcept Ġfor ĠSent i Strength . ĠInstead Ġof Ġtranslating Ġthis Ġlex icon , Ġthe ĠEnglish Ġversion Ġwas Ġreplaced Ġby Ġthe ĠSpanish Ġvariant Ġmade Ġavailable Ġby ĠB IB REF 6 Ġ. </s> For Ġeach Ġsubt ask , Ġthe Ġoptimal Ġcombination Ġof Ġlex icons Ġwas Ġdetermined . ĠThis Ġwas Ġdone Ġby Ġfirst Ġcalculating Ġthe Ġbenefits Ġof Ġadding Ġeach Ġlex icon Ġindividually , Ġafter Ġwhich Ġonly Ġbeneficial Ġlex icons Ġwere Ġadded Ġuntil Ġthe Ġscore Ġdid Ġnot Ġincrease Ġanymore Ġ( e . g . Ġafter Ġadding Ġthe Ġbest Ġfour Ġlex icons Ġthe Ġfifth Ġone Ġdid Ġnot Ġhelp Ġanymore , Ġso Ġonly Ġfour Ġwere Ġadded ). ĠThe Ġtests Ġwere Ġperformed Ġusing Ġa Ġdefault ĠS VM Ġmodel , Ġwith Ġthe Ġset Ġof Ġword Ġembed d ings Ġdescribed Ġin Ġthe Ġprevious Ġsection . ĠEach Ġsubt ask Ġthus Ġuses Ġa Ġdifferent Ġset Ġof Ġlex icons Ġ( see ĠTable ĠT AB REF 1 Ġfor Ġan Ġoverview Ġof Ġthe Ġlex icons Ġused Ġin Ġour Ġfinal Ġensemble ). ĠFor Ġeach Ġsubt ask , Ġthis Ġresulted Ġin Ġa Ġ( mod est ) Ġincrease Ġon Ġthe Ġdevelopment Ġset , Ġbetween Ġ0 . 01 Ġand Ġ0 . 05 . </s> Trans l ating ĠData </s> The Ġtraining Ġset Ġprovided Ġby ĠB IB REF 0 Ġis Ġnot Ġvery Ġlarge , Ġso Ġit Ġwas Ġinteresting Ġto Ġfind Ġa Ġway Ġto Ġaugment Ġthe Ġtraining Ġset . ĠA Ġpossible Ġmethod Ġis Ġto Ġsimply Ġtranslate Ġthe Ġdatasets Ġinto Ġother Ġlanguages , Ġleaving Ġthe Ġlabels Ġintact . ĠSince Ġthe Ġpresent Ġstudy Ġfocuses Ġon ĠSpanish Ġtweets , Ġall Ġtweets Ġfrom Ġthe ĠEnglish Ġdatasets Ġwere Ġtranslated Ġinto ĠSpanish . ĠThis Ġnew Ġset Ġof ĠâĢ ľ Spanish âĢ Ŀ Ġdata Ġwas Ġthen Ġadded Ġto Ġour Ġoriginal Ġtraining Ġset . ĠAgain , Ġthe Ġmachine Ġtranslation Ġplatform ĠA pert ium ĠB IB REF 5 Ġwas Ġused Ġfor Ġthe Ġtranslation Ġof Ġthe Ġdatasets . </s> Al gorith ms ĠUsed </s> Three Ġtypes Ġof Ġmodels Ġwere Ġused Ġin Ġour Ġsystem , Ġa Ġfeed - forward Ġneural Ġnetwork , Ġan ĠL ST M Ġnetwork Ġand Ġan ĠS VM Ġreg ressor . ĠThe Ġneural Ġnets Ġwere Ġinspired Ġby Ġthe Ġwork Ġof ĠPr ay as ĠB IB REF 7 Ġin Ġthe Ġprevious Ġshared Ġtask . ĠDifferent Ġregression Ġalgorithms Ġ( e . g . ĠAda Boost , ĠX GB o ost ) Ġwere Ġalso Ġtried Ġdue Ġto Ġthe Ġsuccess Ġof ĠSe er Net ĠB IB REF 8 Ġ, Ġbut Ġour Ġstudy Ġwas Ġnot Ġable Ġto Ġreproduce Ġtheir Ġresults Ġfor ĠSpanish . </s> For Ġboth Ġthe ĠL ST M Ġnetwork Ġand Ġthe Ġfeed - forward Ġnetwork , Ġa Ġparameter Ġsearch Ġwas Ġdone Ġfor Ġthe Ġnumber Ġof Ġlayers , Ġthe Ġnumber Ġof Ġnodes Ġand Ġdrop out Ġused . ĠThis Ġwas Ġdone Ġfor Ġeach Ġsubt ask , Ġi . e . Ġdifferent Ġtasks Ġcan Ġhave Ġa Ġdifferent Ġnumber Ġof Ġlayers . ĠAll Ġmodels Ġwere Ġimplemented Ġusing ĠKer as ĠB IB REF 9 Ġ. ĠAfter Ġthe Ġbest Ġparameter Ġsettings Ġwere Ġfound , Ġthe Ġresults Ġof Ġ10 Ġsystem Ġruns Ġto Ġproduce Ġour Ġpredictions Ġwere Ġaveraged Ġ( note Ġthat Ġthis Ġis Ġdifferent Ġfrom Ġaveraging Ġour Ġdifferent Ġtype Ġof Ġmodels Ġin ĠSection ĠSEC REF 16 Ġ). ĠFor Ġthe ĠS VM Ġ( im ple mented Ġin Ġsc ik it - learn ĠB IB REF 10 Ġ), Ġthe ĠRB F Ġkernel Ġwas Ġused Ġand Ġa Ġparameter Ġsearch Ġwas Ġconducted Ġfor Ġe ps ilon . ĠDet ailed Ġparameter Ġsettings Ġfor Ġeach Ġsubt ask Ġare Ġshown Ġin ĠTable ĠT AB REF 12 Ġ. ĠEach Ġparameter Ġsearch Ġwas Ġperformed Ġusing Ġ10 - fold Ġcross Ġvalidation , Ġas Ġto Ġnot Ġover fit Ġon Ġthe Ġdevelopment Ġset . </s> Sem i - super vised ĠLearning </s> One Ġof Ġthe Ġaims Ġof Ġthis Ġstudy Ġwas Ġto Ġsee Ġif Ġusing Ġsemi - super vised Ġlearning Ġis Ġbeneficial Ġfor Ġemotion Ġintensity Ġtasks . ĠFor Ġthis Ġpurpose , Ġthe ĠDIS C ĠB IB REF 0 Ġcorpus Ġwas Ġused . ĠThis Ġcorpus Ġwas Ġcreated Ġby Ġquer ying Ġcertain Ġemotion - related Ġwords , Ġwhich Ġmakes Ġit Ġvery Ġsuitable Ġas Ġa Ġsemi - super vised Ġcorpus . ĠHowever , Ġthe Ġspecific Ġemotion Ġthe Ġtweet Ġbelonged Ġto Ġwas Ġnot Ġmade Ġpublic . ĠTherefore , Ġa Ġmethod Ġwas Ġapplied Ġto Ġautomatically Ġassign Ġthe Ġtweets Ġto Ġan Ġemotion Ġby Ġcomparing Ġour Ġscrap ed Ġtweets Ġto Ġthis Ġnew Ġdata Ġset . </s> First , Ġin Ġan Ġattempt Ġto Ġobtain Ġthe Ġquery - terms , Ġwe Ġselected Ġthe Ġ100 Ġwords Ġwhich Ġoccurred Ġmost Ġfrequently Ġin Ġthe ĠDIS C Ġcorpus , Ġin Ġcomparison Ġwith Ġtheir Ġfrequencies Ġin Ġour Ġown Ġscrap ed Ġtweets Ġcorpus . ĠWords Ġthat Ġwere Ġclearly Ġnot Ġindicators Ġof Ġemotion Ġwere Ġremoved . ĠThe Ġrest Ġwas Ġannot ated Ġper Ġemotion Ġor Ġremoved Ġif Ġit Ġwas Ġunclear Ġto Ġwhich Ġemotion Ġthe Ġword Ġbelonged . ĠThis Ġallowed Ġus Ġto Ġcreate Ġsilver Ġdatasets Ġper Ġemotion , Ġassigning Ġtweets Ġto Ġan Ġemotion Ġif Ġan Ġannot ated Ġemotion - word Ġoccurred Ġin Ġthe Ġtweet . </s> Our Ġsemi - super vised Ġapproach Ġis Ġquite Ġstraightforward : Ġfirst Ġa Ġmodel Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġand Ġthen Ġthis Ġmodel Ġis Ġused Ġto Ġpredict Ġthe Ġlabels Ġof Ġthe Ġsilver Ġdata . ĠThis Ġsilver Ġdata Ġis Ġthen Ġsimply Ġadded Ġto Ġour Ġtraining Ġset , Ġafter Ġwhich Ġthe Ġmodel Ġis Ġret rained . ĠHowever , Ġan Ġextra Ġstep Ġis Ġapplied Ġto Ġensure Ġthat Ġthe Ġsilver Ġdata Ġis Ġof Ġreasonable Ġquality . ĠInstead Ġof Ġtraining Ġa Ġsingle Ġmodel Ġinitially , Ġten Ġdifferent Ġmodels Ġwere Ġtrained Ġwhich Ġpredict Ġthe Ġlabels Ġof Ġthe Ġsilver Ġinstances . ĠIf Ġthe Ġhighest Ġand Ġlowest Ġprediction Ġdo Ġnot Ġdiffer Ġmore Ġthan Ġa Ġcertain Ġthreshold Ġthe Ġsilver Ġinstance Ġis Ġmaintained , Ġotherwise Ġit Ġis Ġdiscarded . </s> This Ġresults Ġin Ġtwo Ġparameters Ġthat Ġcould Ġbe Ġoptimized : Ġthe Ġthreshold Ġand Ġthe Ġnumber Ġof Ġsilver Ġinstances Ġthat Ġwould Ġbe Ġadded . ĠThis Ġmethod Ġcan Ġbe Ġapplied Ġto Ġboth Ġthe ĠL ST M Ġand Ġfeed - forward Ġnetworks Ġthat Ġwere Ġused . ĠAn Ġoverview Ġof Ġthe Ġcharacteristics Ġof Ġour Ġdata Ġset Ġwith Ġthe Ġfinal Ġparameter Ġsettings Ġis Ġshown Ġin ĠTable ĠT AB REF 14 Ġ. ĠUsually , Ġonly Ġa Ġsmall Ġsubset Ġof Ġdata Ġwas Ġadded Ġto Ġour Ġtraining Ġset , Ġmeaning Ġthat Ġmost Ġof Ġthe Ġsilver Ġdata Ġis Ġnot Ġused Ġin Ġthe Ġexperiments . ĠNote Ġthat Ġsince Ġonly Ġthe Ġemotions Ġwere Ġannot ated , Ġthis Ġmethod Ġis Ġonly Ġapplicable Ġto Ġthe ĠE I Ġtasks . </s> En semb ling </s> To Ġboost Ġperformance , Ġthe ĠS VM , ĠL ST M , Ġand Ġfeed - forward Ġmodels Ġwere Ġcombined Ġinto Ġan Ġensemble . ĠFor Ġboth Ġthe ĠL ST M Ġand Ġfeed - forward Ġapproach , Ġthree Ġdifferent Ġmodels Ġwere Ġtrained . ĠThe Ġfirst Ġmodel Ġwas Ġtrained Ġon Ġthe Ġtraining Ġdata Ġ( regular ), Ġthe Ġsecond Ġmodel Ġwas Ġtrained Ġon Ġboth Ġthe Ġtraining Ġand Ġtranslated Ġtraining Ġdata Ġ( trans lated ) Ġand Ġthe Ġthird Ġone Ġwas Ġtrained Ġon Ġboth Ġthe Ġtraining Ġdata Ġand Ġthe Ġsemi - super vised Ġdata Ġ( silver ). ĠDue Ġto Ġthe Ġnature Ġof Ġthe ĠS VM Ġalgorithm , Ġsemi - super vised Ġlearning Ġdoes Ġnot Ġhelp , Ġso Ġonly Ġthe Ġregular Ġand Ġtranslated Ġmodel Ġwere Ġtrained Ġin Ġthis Ġcase . ĠThis Ġresults Ġin Ġ8 Ġdifferent Ġmodels Ġper Ġsubt ask . ĠNote Ġthat Ġfor Ġthe Ġval ence Ġtasks Ġno Ġsilver Ġtraining Ġdata Ġwas Ġobtained , Ġmeaning Ġthat Ġfor Ġthose Ġtasks Ġthe Ġsemi - super vised Ġmodels Ġcould Ġnot Ġbe Ġused . </s> Per Ġtask , Ġthe ĠL ST M Ġand Ġfeed - forward Ġmodel 's Ġpredictions Ġwere Ġaveraged Ġover Ġ10 Ġprediction Ġruns . ĠSub sequently , Ġthe Ġpredictions Ġof Ġall Ġindividual Ġmodels Ġwere Ġcombined Ġinto Ġan Ġaverage . ĠFinally , Ġmodels Ġwere Ġremoved Ġfrom Ġthe Ġensemble Ġin Ġa Ġstep wise Ġmanner Ġif Ġthe Ġremoval Ġincreased Ġthe Ġaverage Ġscore . ĠThis Ġwas Ġdone Ġbased Ġon Ġtheir Ġoriginal Ġscores , Ġi . e . Ġstarting Ġout Ġby Ġtrying Ġto Ġremove Ġthe Ġworst Ġindividual Ġmodel Ġand Ġworking Ġour Ġway Ġup Ġto Ġthe Ġbest Ġmodel . ĠWe Ġonly Ġconsider Ġit Ġan Ġincrease Ġin Ġscore Ġif Ġthe Ġdifference Ġis Ġlarger Ġthan Ġ0 . 002 Ġ( i . e . Ġthe Ġdifference Ġbetween Ġ0 . 7 16 Ġand Ġ0 . 718 ). ĠIf Ġat Ġsome Ġpoint Ġthe Ġscore Ġdoes Ġnot Ġincrease Ġand Ġwe Ġare Ġtherefore Ġunable Ġto Ġremove Ġa Ġmodel , Ġthe Ġprocess Ġis Ġstopped Ġand Ġour Ġbest Ġensemble Ġof Ġmodels Ġhas Ġbeen Ġfound . ĠThis Ġprocess Ġuses Ġthe Ġscores Ġon Ġthe Ġdevelopment Ġset Ġof Ġdifferent Ġcombinations Ġof Ġmodels . ĠNote Ġthat Ġthis Ġmeans Ġthat Ġthe Ġen semb les Ġfor Ġdifferent Ġsubt asks Ġcan Ġcontain Ġdifferent Ġsets Ġof Ġmodels . ĠThe Ġfinal Ġmodel Ġselections Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 17 Ġ. </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 18 Ġshows Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġof Ġall Ġindividuals Ġmodels , Ġdistinguishing Ġthe Ġthree Ġtypes Ġof Ġtraining : Ġregular Ġ( r ), Ġtranslated Ġ( t ) Ġand Ġsemi - super vised Ġ( s ). ĠIn ĠTables ĠT AB REF 17 Ġand ĠT AB REF 18 Ġ, Ġthe Ġletter Ġbehind Ġeach Ġmodel Ġ( e . g . ĠS VM - r , ĠL ST M - r ) Ġcorresponds Ġto Ġthe Ġtype Ġof Ġtraining Ġused . ĠComp aring Ġthe Ġregular Ġand Ġtranslated Ġcolumns Ġfor Ġthe Ġthree Ġalgorithms , Ġit Ġshows Ġthat Ġin Ġ22 Ġout Ġof Ġ30 Ġcases , Ġusing Ġtranslated Ġinstances Ġas Ġextra Ġtraining Ġdata Ġresulted Ġin Ġan Ġimprovement . ĠFor Ġthe Ġsemi - super vised Ġlearning Ġapproach , Ġan Ġimprovement Ġis Ġfound Ġin Ġ15 Ġout Ġof Ġ16 Ġcases . ĠMoreover , Ġour Ġbest Ġindividual Ġmodel Ġfor Ġeach Ġsubt ask Ġ( bold ed Ġscores Ġin ĠTable ĠT AB REF 18 Ġ) Ġis Ġalways Ġeither Ġa Ġtranslated Ġor Ġsemi - super vised Ġmodel . ĠTable ĠT AB REF 18 Ġalso Ġshows Ġthat , Ġin Ġgeneral , Ġour Ġfeed - forward Ġnetwork Ġobtained Ġthe Ġbest Ġresults , Ġhaving Ġthe Ġhighest ĠF - score Ġfor Ġ8 Ġout Ġof Ġ10 Ġsubt asks . </s> However , ĠTable ĠT AB REF 19 Ġshows Ġthat Ġthese Ġscores Ġcan Ġstill Ġbe Ġimproved Ġby Ġaveraging Ġor Ġen semb ling Ġthe Ġindividual Ġmodels . ĠOn Ġthe Ġdev Ġset , Ġaveraging Ġour Ġ8 Ġindividual Ġmodels Ġresults Ġin Ġa Ġbetter Ġscore Ġfor Ġ8 Ġout Ġof Ġ10 Ġsubt asks , Ġwhile Ġcreating Ġan Ġensemble Ġbeats Ġall Ġof Ġthe Ġindividual Ġmodels Ġas Ġwell Ġas Ġthe Ġaverage Ġfor Ġeach Ġsubt ask . ĠOn Ġthe Ġtest Ġset , Ġhowever , Ġonly Ġa Ġsmall Ġincrease Ġin Ġscore Ġ( if Ġany ) Ġis Ġfound Ġfor Ġstep wise Ġen semb ling , Ġcompared Ġto Ġaveraging . ĠEven Ġthough Ġthe Ġresults Ġdo Ġnot Ġget Ġworse , Ġwe Ġcannot Ġconclude Ġthat Ġstep wise Ġen semb ling Ġis Ġa Ġbetter Ġmethod Ġthan Ġsimply Ġaveraging . </s> Our Ġofficial Ġscores Ġ( column ĠEns ĠTest Ġin ĠTable ĠT AB REF 19 Ġ) Ġhave Ġplaced Ġus Ġsecond Ġ( E I - Reg , ĠE I - O c ), Ġfourth Ġ( V - Reg ) Ġand Ġfifth Ġ( V - O c ) Ġon Ġthe ĠSem E val ĠA IT - 2018 Ġleader board . ĠHowever , Ġit Ġis Ġevident Ġthat Ġthe Ġresults Ġobtained Ġon Ġthe Ġtest Ġset Ġare Ġnot Ġalways Ġin Ġline Ġwith Ġthose Ġachieved Ġon Ġthe Ġdevelopment Ġset . ĠEspecially Ġon Ġthe Ġanger Ġsubt ask Ġfor Ġboth ĠE I - Reg Ġand ĠE I - O c , Ġthe Ġscores Ġare Ġconsiderably Ġlower Ġon Ġthe Ġtest Ġset Ġin Ġcomparison Ġwith Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset . ĠTherefore , Ġa Ġsmall Ġerror Ġanalysis Ġwas Ġperformed Ġon Ġthe Ġinstances Ġwhere Ġour Ġfinal Ġmodel Ġmade Ġthe Ġlargest Ġerrors . </s> Error ĠAnalysis </s> Due Ġto Ġsome Ġlarge Ġdifferences Ġbetween Ġour Ġresults Ġon Ġthe Ġdev Ġand Ġtest Ġset Ġof Ġthis Ġtask , Ġwe Ġperformed Ġa Ġsmall Ġerror Ġanalysis Ġin Ġorder Ġto Ġsee Ġwhat Ġcaused Ġthese Ġdifferences . ĠFor ĠE I - Reg - anger , Ġthe Ġgold Ġlabels Ġwere Ġcompared Ġto Ġour Ġown Ġpredictions , Ġand Ġwe Ġmanually Ġchecked Ġ50 Ġinstances Ġfor Ġwhich Ġour Ġsystem Ġmade Ġthe Ġlargest Ġerrors . </s> Some Ġexamples Ġthat Ġwere Ġindicative Ġof Ġthe Ġshortcomings Ġof Ġour Ġsystem Ġare Ġshown Ġin ĠTable ĠT AB REF 20 Ġ. ĠFirst Ġof Ġall , Ġour Ġsystem Ġdid Ġnot Ġtake Ġinto Ġaccount Ġcapital ization . ĠThe Ġimplications Ġof Ġthis Ġare Ġshown Ġin Ġthe Ġfirst Ġsentence , Ġwhere Ġcapital ization Ġintens ifies Ġthe Ġemotion Ġused Ġin Ġthe Ġsentence . ĠIn Ġthe Ġsecond Ġsentence , Ġthe Ġname ĠImper ator ĠFur ios a Ġis Ġnot Ġunderstood . ĠSince Ġour Ġtexts Ġwere Ġlower c ased , Ġour Ġsystem Ġwas Ġunable Ġto Ġcapture Ġthe Ġnamed Ġentity Ġand Ġthought Ġthe Ġsentence Ġwas Ġabout Ġan Ġangry Ġemperor Ġinstead . ĠIn Ġthe Ġthird Ġsentence , Ġour Ġsystem Ġfails Ġto Ġcapture Ġthat Ġwhen Ġyou Ġare Ġso Ġangry Ġthat Ġit Ġmakes Ġyou Ġlaugh , Ġit Ġresults Ġin Ġa Ġreduced Ġintensity Ġof Ġthe Ġan gr iness . ĠFinally , Ġin Ġthe Ġfourth Ġsentence , Ġit Ġis Ġthe Ġfigur ative Ġlanguage Ġme Ġinf la Ġla Ġv ena Ġ( it Ġinfl ates Ġmy Ġvein ) Ġthat Ġthe Ġsystem Ġis Ġnot Ġable Ġto Ġunderstand . </s> The Ġfirst Ġtwo Ġerror - c ategories Ġmight Ġbe Ġsolved Ġby Ġincluding Ġsmart Ġfeatures Ġregarding Ġcapital ization Ġand Ġnamed Ġentity Ġrecognition . ĠHowever , Ġthe Ġlast Ġtwo Ġcategories Ġare Ġproblems Ġof Ġnatural Ġlanguage Ġunderstanding Ġand Ġwill Ġbe Ġvery Ġdifficult Ġto Ġfix . </s> Conclusion </s> To Ġconclude , Ġthe Ġpresent Ġstudy Ġdescribed Ġour Ġsubmission Ġfor Ġthe ĠSem eval Ġ2018 ĠShared ĠTask Ġon ĠAffect Ġin ĠTwe ets . ĠWe Ġparticipated Ġin Ġfour ĠSpanish Ġsubt asks Ġand Ġour Ġsubmissions Ġranked Ġsecond , Ġsecond , Ġfourth Ġand Ġfifth Ġplace . ĠOur Ġstudy Ġaimed Ġto Ġinvestigate Ġwhether Ġthe Ġautomatic Ġgeneration Ġof Ġadditional Ġtraining Ġdata Ġthrough Ġtranslation Ġand Ġsemi - super vised Ġlearning , Ġas Ġwell Ġas Ġthe Ġcreation Ġof Ġstep wise Ġen semb les , Ġincrease Ġthe Ġperformance Ġof Ġour ĠSpanish - language Ġmodels . ĠStrong Ġsupport Ġwas Ġfound Ġfor Ġthe Ġtranslation Ġand Ġsemi - super vised Ġlearning Ġapproaches ; Ġour Ġbest Ġmodels Ġfor Ġall Ġsubt asks Ġuse Ġeither Ġone Ġof Ġthese Ġapproaches . ĠThese Ġresults Ġsuggest Ġthat Ġboth Ġof Ġthese Ġadditional Ġdata Ġresources Ġare Ġbeneficial Ġwhen Ġdetermining Ġemotion Ġintensity Ġ( for ĠSpanish ). ĠHowever , Ġthe Ġcreation Ġof Ġa Ġstep wise Ġensemble Ġfrom Ġthe Ġbest Ġmodels Ġdid Ġnot Ġresult Ġin Ġbetter Ġperformance Ġcompared Ġto Ġsimply Ġaveraging Ġthe Ġmodels . ĠIn Ġaddition , Ġsome Ġsigns Ġof Ġover fitting Ġon Ġthe Ġdev Ġset Ġwere Ġfound . ĠIn Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġapply Ġthe Ġmethods Ġ( translation Ġand Ġsemi - super vised Ġlearning ) Ġused Ġon ĠSpanish Ġon Ġother Ġlow - resource Ġlanguages Ġand Ġpotentially Ġalso Ġon Ġother Ġtasks .
Introduction Understanding the emotions expressed in a text or message is of high relevance nowadays. Companies are interested in this to get an understanding of the sentiment of their current customers regarding their products and the sentiment of their potential customers to attract new ones. Moreover, changes in a product or a company may also affect the sentiment of a customer. However, the intensity of an emotion is crucial in determining the urgency and importance of that sentiment. If someone is only slightly happy about a product, is a customer willing to buy it again? Conversely, if someone is very angry about customer service, his or her complaint might be given priority over somewhat milder complaints. BIBREF0 present four tasks in which systems have to automatically determine the intensity of emotions (EI) or the intensity of the sentiment (Valence) of tweets in the languages English, Arabic, and Spanish. The goal is to either predict a continuous regression (reg) value or to do ordinal classification (oc) based on a number of predefined categories. The EI tasks have separate training sets for four different emotions: anger, fear, joy and sadness. Due to the large number of subtasks and the fact that this language does not have many resources readily available, we only focus on the Spanish subtasks. Our work makes the following contributions: Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets. This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used. Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made. We conclude with some possible ideas for future work. Data For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1 . Links and usernames were replaced by the general tokens URL and @username, after which the tweets were tokenized by using TweetTokenizer. All text was lowercased. In a post-processing step, it was ensured that each emoji is tokenized as a single token. Word Embeddings To be able to train word embeddings, Spanish tweets were scraped between November 8, 2017 and January 12, 2018. We chose to create our own embeddings instead of using pre-trained embeddings, because this way the embeddings would resemble the provided data set: both are based on Twitter data. Added to this set was the Affect in Tweets Distant Supervision Corpus (DISC) made available by the organizers BIBREF0 and a set of 4.1 million tweets from 2015, obtained from BIBREF2 . After removing duplicate tweets and tweets with fewer than ten tokens, this resulted in a set of 58.7 million tweets, containing 1.1 billion tokens. The tweets were preprocessed using the method described in Section SECREF6 . The word embeddings were created using word2vec in the gensim library BIBREF3 , using CBOW, a window size of 40 and a minimum count of 5. The feature vectors for each tweet were then created by using the AffectiveTweets WEKA package BIBREF4 . Translating Lexicons Most lexical resources for sentiment analysis are in English. To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5 . All lexicons from the AffectiveTweets package were translated, except for SentiStrength. Instead of translating this lexicon, the English version was replaced by the Spanish variant made available by BIBREF6 . For each subtask, the optimal combination of lexicons was determined. This was done by first calculating the benefits of adding each lexicon individually, after which only beneficial lexicons were added until the score did not increase anymore (e.g. after adding the best four lexicons the fifth one did not help anymore, so only four were added). The tests were performed using a default SVM model, with the set of word embeddings described in the previous section. Each subtask thus uses a different set of lexicons (see Table TABREF1 for an overview of the lexicons used in our final ensemble). For each subtask, this resulted in a (modest) increase on the development set, between 0.01 and 0.05. Translating Data The training set provided by BIBREF0 is not very large, so it was interesting to find a way to augment the training set. A possible method is to simply translate the datasets into other languages, leaving the labels intact. Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish. This new set of “Spanish” data was then added to our original training set. Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets. Algorithms Used Three types of models were used in our system, a feed-forward neural network, an LSTM network and an SVM regressor. The neural nets were inspired by the work of Prayas BIBREF7 in the previous shared task. Different regression algorithms (e.g. AdaBoost, XGBoost) were also tried due to the success of SeerNet BIBREF8 , but our study was not able to reproduce their results for Spanish. For both the LSTM network and the feed-forward network, a parameter search was done for the number of layers, the number of nodes and dropout used. This was done for each subtask, i.e. different tasks can have a different number of layers. All models were implemented using Keras BIBREF9 . After the best parameter settings were found, the results of 10 system runs to produce our predictions were averaged (note that this is different from averaging our different type of models in Section SECREF16 ). For the SVM (implemented in scikit-learn BIBREF10 ), the RBF kernel was used and a parameter search was conducted for epsilon. Detailed parameter settings for each subtask are shown in Table TABREF12 . Each parameter search was performed using 10-fold cross validation, as to not overfit on the development set. Semi-supervised Learning One of the aims of this study was to see if using semi-supervised learning is beneficial for emotion intensity tasks. For this purpose, the DISC BIBREF0 corpus was used. This corpus was created by querying certain emotion-related words, which makes it very suitable as a semi-supervised corpus. However, the specific emotion the tweet belonged to was not made public. Therefore, a method was applied to automatically assign the tweets to an emotion by comparing our scraped tweets to this new data set. First, in an attempt to obtain the query-terms, we selected the 100 words which occurred most frequently in the DISC corpus, in comparison with their frequencies in our own scraped tweets corpus. Words that were clearly not indicators of emotion were removed. The rest was annotated per emotion or removed if it was unclear to which emotion the word belonged. This allowed us to create silver datasets per emotion, assigning tweets to an emotion if an annotated emotion-word occurred in the tweet. Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data. This silver data is then simply added to our training set, after which the model is retrained. However, an extra step is applied to ensure that the silver data is of reasonable quality. Instead of training a single model initially, ten different models were trained which predict the labels of the silver instances. If the highest and lowest prediction do not differ more than a certain threshold the silver instance is maintained, otherwise it is discarded. This results in two parameters that could be optimized: the threshold and the number of silver instances that would be added. This method can be applied to both the LSTM and feed-forward networks that were used. An overview of the characteristics of our data set with the final parameter settings is shown in Table TABREF14 . Usually, only a small subset of data was added to our training set, meaning that most of the silver data is not used in the experiments. Note that since only the emotions were annotated, this method is only applicable to the EI tasks. Ensembling To boost performance, the SVM, LSTM, and feed-forward models were combined into an ensemble. For both the LSTM and feed-forward approach, three different models were trained. The first model was trained on the training data (regular), the second model was trained on both the training and translated training data (translated) and the third one was trained on both the training data and the semi-supervised data (silver). Due to the nature of the SVM algorithm, semi-supervised learning does not help, so only the regular and translated model were trained in this case. This results in 8 different models per subtask. Note that for the valence tasks no silver training data was obtained, meaning that for those tasks the semi-supervised models could not be used. Per task, the LSTM and feed-forward model's predictions were averaged over 10 prediction runs. Subsequently, the predictions of all individual models were combined into an average. Finally, models were removed from the ensemble in a stepwise manner if the removal increased the average score. This was done based on their original scores, i.e. starting out by trying to remove the worst individual model and working our way up to the best model. We only consider it an increase in score if the difference is larger than 0.002 (i.e. the difference between 0.716 and 0.718). If at some point the score does not increase and we are therefore unable to remove a model, the process is stopped and our best ensemble of models has been found. This process uses the scores on the development set of different combinations of models. Note that this means that the ensembles for different subtasks can contain different sets of models. The final model selections can be found in Table TABREF17 . Results and Discussion Table TABREF18 shows the results on the development set of all individuals models, distinguishing the three types of training: regular (r), translated (t) and semi-supervised (s). In Tables TABREF17 and TABREF18 , the letter behind each model (e.g. SVM-r, LSTM-r) corresponds to the type of training used. Comparing the regular and translated columns for the three algorithms, it shows that in 22 out of 30 cases, using translated instances as extra training data resulted in an improvement. For the semi-supervised learning approach, an improvement is found in 15 out of 16 cases. Moreover, our best individual model for each subtask (bolded scores in Table TABREF18 ) is always either a translated or semi-supervised model. Table TABREF18 also shows that, in general, our feed-forward network obtained the best results, having the highest F-score for 8 out of 10 subtasks. However, Table TABREF19 shows that these scores can still be improved by averaging or ensembling the individual models. On the dev set, averaging our 8 individual models results in a better score for 8 out of 10 subtasks, while creating an ensemble beats all of the individual models as well as the average for each subtask. On the test set, however, only a small increase in score (if any) is found for stepwise ensembling, compared to averaging. Even though the results do not get worse, we cannot conclude that stepwise ensembling is a better method than simply averaging. Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard. However, it is evident that the results obtained on the test set are not always in line with those achieved on the development set. Especially on the anger subtask for both EI-Reg and EI-Oc, the scores are considerably lower on the test set in comparison with the results on the development set. Therefore, a small error analysis was performed on the instances where our final model made the largest errors. Error Analysis Due to some large differences between our results on the dev and test set of this task, we performed a small error analysis in order to see what caused these differences. For EI-Reg-anger, the gold labels were compared to our own predictions, and we manually checked 50 instances for which our system made the largest errors. Some examples that were indicative of the shortcomings of our system are shown in Table TABREF20 . First of all, our system did not take into account capitalization. The implications of this are shown in the first sentence, where capitalization intensifies the emotion used in the sentence. In the second sentence, the name Imperator Furiosa is not understood. Since our texts were lowercased, our system was unable to capture the named entity and thought the sentence was about an angry emperor instead. In the third sentence, our system fails to capture that when you are so angry that it makes you laugh, it results in a reduced intensity of the angriness. Finally, in the fourth sentence, it is the figurative language me infla la vena (it inflates my vein) that the system is not able to understand. The first two error-categories might be solved by including smart features regarding capitalization and named entity recognition. However, the last two categories are problems of natural language understanding and will be very difficult to fix. Conclusion To conclude, the present study described our submission for the Semeval 2018 Shared Task on Affect in Tweets. We participated in four Spanish subtasks and our submissions ranked second, second, fourth and fifth place. Our study aimed to investigate whether the automatic generation of additional training data through translation and semi-supervised learning, as well as the creation of stepwise ensembles, increase the performance of our Spanish-language models. Strong support was found for the translation and semi-supervised learning approaches; our best models for all subtasks use either one of these approaches. These results suggest that both of these additional data resources are beneficial when determining emotion intensity (for Spanish). However, the creation of a stepwise ensemble from the best models did not result in better performance compared to simply averaging the models. In addition, some signs of overfitting on the dev set were found. In future work, we would like to apply the methods (translation and semi-supervised learning) used on Spanish on other low-resource languages and potentially also on other tasks.
What subtasks did they participate in?
e1ab11885f72b4658263a60751d956ba661c1d61
What Ġsubt asks Ġdid Ġthey Ġparticipate Ġin ?
Answer Ġwith Ġcontent Ġmissing : Ġ( Sub script Ġ1 : Ġ" We Ġdid Ġnot Ġparticipate Ġin Ġsubt ask Ġ5 Ġ( E - c ) ") ĠAuthors Ġparticipated Ġin ĠE I - Reg , ĠE I - O c , ĠV - Reg Ġand ĠV - O c Ġsubt asks .
<s> What Ġsubt asks Ġdid Ġthey Ġparticipate Ġin ? </s> Introduction </s> Understanding Ġthe Ġemotions Ġexpressed Ġin Ġa Ġtext Ġor Ġmessage Ġis Ġof Ġhigh Ġrelevance Ġnowadays . ĠCompanies Ġare Ġinterested Ġin Ġthis Ġto Ġget Ġan Ġunderstanding Ġof Ġthe Ġsentiment Ġof Ġtheir Ġcurrent Ġcustomers Ġregarding Ġtheir Ġproducts Ġand Ġthe Ġsentiment Ġof Ġtheir Ġpotential Ġcustomers Ġto Ġattract Ġnew Ġones . ĠMoreover , Ġchanges Ġin Ġa Ġproduct Ġor Ġa Ġcompany Ġmay Ġalso Ġaffect Ġthe Ġsentiment Ġof Ġa Ġcustomer . ĠHowever , Ġthe Ġintensity Ġof Ġan Ġemotion Ġis Ġcrucial Ġin Ġdetermining Ġthe Ġurgency Ġand Ġimportance Ġof Ġthat Ġsentiment . ĠIf Ġsomeone Ġis Ġonly Ġslightly Ġhappy Ġabout Ġa Ġproduct , Ġis Ġa Ġcustomer Ġwilling Ġto Ġbuy Ġit Ġagain ? ĠConversely , Ġif Ġsomeone Ġis Ġvery Ġangry Ġabout Ġcustomer Ġservice , Ġhis Ġor Ġher Ġcomplaint Ġmight Ġbe Ġgiven Ġpriority Ġover Ġsomewhat Ġmild er Ġcomplaints . </s> BI B REF 0 Ġpresent Ġfour Ġtasks Ġin Ġwhich Ġsystems Ġhave Ġto Ġautomatically Ġdetermine Ġthe Ġintensity Ġof Ġemotions Ġ( E I ) Ġor Ġthe Ġintensity Ġof Ġthe Ġsentiment Ġ( Val ence ) Ġof Ġtweets Ġin Ġthe Ġlanguages ĠEnglish , ĠArabic , Ġand ĠSpanish . ĠThe Ġgoal Ġis Ġto Ġeither Ġpredict Ġa Ġcontinuous Ġregression Ġ( reg ) Ġvalue Ġor Ġto Ġdo Ġord inal Ġclassification Ġ( oc ) Ġbased Ġon Ġa Ġnumber Ġof Ġpred efined Ġcategories . ĠThe ĠE I Ġtasks Ġhave Ġseparate Ġtraining Ġsets Ġfor Ġfour Ġdifferent Ġemotions : Ġanger , Ġfear , Ġjoy Ġand Ġsadness . ĠDue Ġto Ġthe Ġlarge Ġnumber Ġof Ġsubt asks Ġand Ġthe Ġfact Ġthat Ġthis Ġlanguage Ġdoes Ġnot Ġhave Ġmany Ġresources Ġreadily Ġavailable , Ġwe Ġonly Ġfocus Ġon Ġthe ĠSpanish Ġsubt asks . ĠOur Ġwork Ġmakes Ġthe Ġfollowing Ġcontributions : </s> Our Ġsubmissions Ġranked Ġsecond Ġ( E I - Reg ), Ġsecond Ġ( E I - O c ), Ġfourth Ġ( V - Reg ) Ġand Ġfifth Ġ( V - O c ), Ġdemonstrating Ġthat Ġthe Ġproposed Ġmethod Ġis Ġaccurate Ġin Ġautomatically Ġdetermining Ġthe Ġintensity Ġof Ġemotions Ġand Ġsentiment Ġof ĠSpanish Ġtweets . ĠThis Ġpaper Ġwill Ġfirst Ġfocus Ġon Ġthe Ġdatasets , Ġthe Ġdata Ġgeneration Ġprocedure , Ġand Ġthe Ġtechniques Ġand Ġtools Ġused . ĠThen Ġwe Ġpresent Ġthe Ġresults Ġin Ġdetail , Ġafter Ġwhich Ġwe Ġperform Ġa Ġsmall Ġerror Ġanalysis Ġon Ġthe Ġlargest Ġmistakes Ġour Ġmodel Ġmade . ĠWe Ġconclude Ġwith Ġsome Ġpossible Ġideas Ġfor Ġfuture Ġwork . </s> Data </s> For Ġeach Ġtask , Ġthe Ġtraining Ġdata Ġthat Ġwas Ġmade Ġavailable Ġby Ġthe Ġorganizers Ġis Ġused , Ġwhich Ġis Ġa Ġselection Ġof Ġtweets Ġwith Ġfor Ġeach Ġtweet Ġa Ġlabel Ġdescribing Ġthe Ġintensity Ġof Ġthe Ġemotion Ġor Ġsentiment ĠB IB REF 1 Ġ. ĠLinks Ġand Ġus ern ames Ġwere Ġreplaced Ġby Ġthe Ġgeneral Ġtokens ĠURL Ġand Ġ@ username , Ġafter Ġwhich Ġthe Ġtweets Ġwere Ġtoken ized Ġby Ġusing ĠTweet Token izer . ĠAll Ġtext Ġwas Ġlower c ased . ĠIn Ġa Ġpost - processing Ġstep , Ġit Ġwas Ġensured Ġthat Ġeach Ġemoji Ġis Ġtoken ized Ġas Ġa Ġsingle Ġtoken . </s> Word ĠEmb edd ings </s> To Ġbe Ġable Ġto Ġtrain Ġword Ġembed d ings , ĠSpanish Ġtweets Ġwere Ġscrap ed Ġbetween ĠNovember Ġ8 , Ġ2017 Ġand ĠJanuary Ġ12 , Ġ2018 . ĠWe Ġchose Ġto Ġcreate Ġour Ġown Ġembed d ings Ġinstead Ġof Ġusing Ġpre - trained Ġembed d ings , Ġbecause Ġthis Ġway Ġthe Ġembed d ings Ġwould Ġresemble Ġthe Ġprovided Ġdata Ġset : Ġboth Ġare Ġbased Ġon ĠTwitter Ġdata . ĠAdded Ġto Ġthis Ġset Ġwas Ġthe ĠAffect Ġin ĠTwe ets ĠDist ant ĠSuper vision ĠCorpus Ġ( DIS C ) Ġmade Ġavailable Ġby Ġthe Ġorganizers ĠB IB REF 0 Ġand Ġa Ġset Ġof Ġ4 . 1 Ġmillion Ġtweets Ġfrom Ġ2015 , Ġobtained Ġfrom ĠB IB REF 2 Ġ. ĠAfter Ġremoving Ġduplicate Ġtweets Ġand Ġtweets Ġwith Ġfewer Ġthan Ġten Ġtokens , Ġthis Ġresulted Ġin Ġa Ġset Ġof Ġ58 . 7 Ġmillion Ġtweets , Ġcontaining Ġ1 . 1 Ġbillion Ġtokens . ĠThe Ġtweets Ġwere Ġpre process ed Ġusing Ġthe Ġmethod Ġdescribed Ġin ĠSection ĠSEC REF 6 Ġ. ĠThe Ġword Ġembed d ings Ġwere Ġcreated Ġusing Ġword 2 vec Ġin Ġthe Ġg ens im Ġlibrary ĠB IB REF 3 Ġ, Ġusing ĠCB OW , Ġa Ġwindow Ġsize Ġof Ġ40 Ġand Ġa Ġminimum Ġcount Ġof Ġ5 . ĠThe Ġfeature Ġvectors Ġfor Ġeach Ġtweet Ġwere Ġthen Ġcreated Ġby Ġusing Ġthe ĠAffect ive Twe ets ĠWE KA Ġpackage ĠB IB REF 4 Ġ. </s> Trans l ating ĠLex icons </s> Most Ġlex ical Ġresources Ġfor Ġsentiment Ġanalysis Ġare Ġin ĠEnglish . ĠTo Ġstill Ġbe Ġable Ġto Ġbenefit Ġfrom Ġthese Ġsources , Ġthe Ġlex icons Ġin Ġthe ĠAffect ive Twe ets Ġpackage Ġwere Ġtranslated Ġto ĠSpanish , Ġusing Ġthe Ġmachine Ġtranslation Ġplatform ĠA pert ium ĠB IB REF 5 Ġ. </s> All Ġlex icons Ġfrom Ġthe ĠAffect ive Twe ets Ġpackage Ġwere Ġtranslated , Ġexcept Ġfor ĠSent i Strength . ĠInstead Ġof Ġtranslating Ġthis Ġlex icon , Ġthe ĠEnglish Ġversion Ġwas Ġreplaced Ġby Ġthe ĠSpanish Ġvariant Ġmade Ġavailable Ġby ĠB IB REF 6 Ġ. </s> For Ġeach Ġsubt ask , Ġthe Ġoptimal Ġcombination Ġof Ġlex icons Ġwas Ġdetermined . ĠThis Ġwas Ġdone Ġby Ġfirst Ġcalculating Ġthe Ġbenefits Ġof Ġadding Ġeach Ġlex icon Ġindividually , Ġafter Ġwhich Ġonly Ġbeneficial Ġlex icons Ġwere Ġadded Ġuntil Ġthe Ġscore Ġdid Ġnot Ġincrease Ġanymore Ġ( e . g . Ġafter Ġadding Ġthe Ġbest Ġfour Ġlex icons Ġthe Ġfifth Ġone Ġdid Ġnot Ġhelp Ġanymore , Ġso Ġonly Ġfour Ġwere Ġadded ). ĠThe Ġtests Ġwere Ġperformed Ġusing Ġa Ġdefault ĠS VM Ġmodel , Ġwith Ġthe Ġset Ġof Ġword Ġembed d ings Ġdescribed Ġin Ġthe Ġprevious Ġsection . ĠEach Ġsubt ask Ġthus Ġuses Ġa Ġdifferent Ġset Ġof Ġlex icons Ġ( see ĠTable ĠT AB REF 1 Ġfor Ġan Ġoverview Ġof Ġthe Ġlex icons Ġused Ġin Ġour Ġfinal Ġensemble ). ĠFor Ġeach Ġsubt ask , Ġthis Ġresulted Ġin Ġa Ġ( mod est ) Ġincrease Ġon Ġthe Ġdevelopment Ġset , Ġbetween Ġ0 . 01 Ġand Ġ0 . 05 . </s> Trans l ating ĠData </s> The Ġtraining Ġset Ġprovided Ġby ĠB IB REF 0 Ġis Ġnot Ġvery Ġlarge , Ġso Ġit Ġwas Ġinteresting Ġto Ġfind Ġa Ġway Ġto Ġaugment Ġthe Ġtraining Ġset . ĠA Ġpossible Ġmethod Ġis Ġto Ġsimply Ġtranslate Ġthe Ġdatasets Ġinto Ġother Ġlanguages , Ġleaving Ġthe Ġlabels Ġintact . ĠSince Ġthe Ġpresent Ġstudy Ġfocuses Ġon ĠSpanish Ġtweets , Ġall Ġtweets Ġfrom Ġthe ĠEnglish Ġdatasets Ġwere Ġtranslated Ġinto ĠSpanish . ĠThis Ġnew Ġset Ġof ĠâĢ ľ Spanish âĢ Ŀ Ġdata Ġwas Ġthen Ġadded Ġto Ġour Ġoriginal Ġtraining Ġset . ĠAgain , Ġthe Ġmachine Ġtranslation Ġplatform ĠA pert ium ĠB IB REF 5 Ġwas Ġused Ġfor Ġthe Ġtranslation Ġof Ġthe Ġdatasets . </s> Al gorith ms ĠUsed </s> Three Ġtypes Ġof Ġmodels Ġwere Ġused Ġin Ġour Ġsystem , Ġa Ġfeed - forward Ġneural Ġnetwork , Ġan ĠL ST M Ġnetwork Ġand Ġan ĠS VM Ġreg ressor . ĠThe Ġneural Ġnets Ġwere Ġinspired Ġby Ġthe Ġwork Ġof ĠPr ay as ĠB IB REF 7 Ġin Ġthe Ġprevious Ġshared Ġtask . ĠDifferent Ġregression Ġalgorithms Ġ( e . g . ĠAda Boost , ĠX GB o ost ) Ġwere Ġalso Ġtried Ġdue Ġto Ġthe Ġsuccess Ġof ĠSe er Net ĠB IB REF 8 Ġ, Ġbut Ġour Ġstudy Ġwas Ġnot Ġable Ġto Ġreproduce Ġtheir Ġresults Ġfor ĠSpanish . </s> For Ġboth Ġthe ĠL ST M Ġnetwork Ġand Ġthe Ġfeed - forward Ġnetwork , Ġa Ġparameter Ġsearch Ġwas Ġdone Ġfor Ġthe Ġnumber Ġof Ġlayers , Ġthe Ġnumber Ġof Ġnodes Ġand Ġdrop out Ġused . ĠThis Ġwas Ġdone Ġfor Ġeach Ġsubt ask , Ġi . e . Ġdifferent Ġtasks Ġcan Ġhave Ġa Ġdifferent Ġnumber Ġof Ġlayers . ĠAll Ġmodels Ġwere Ġimplemented Ġusing ĠKer as ĠB IB REF 9 Ġ. ĠAfter Ġthe Ġbest Ġparameter Ġsettings Ġwere Ġfound , Ġthe Ġresults Ġof Ġ10 Ġsystem Ġruns Ġto Ġproduce Ġour Ġpredictions Ġwere Ġaveraged Ġ( note Ġthat Ġthis Ġis Ġdifferent Ġfrom Ġaveraging Ġour Ġdifferent Ġtype Ġof Ġmodels Ġin ĠSection ĠSEC REF 16 Ġ). ĠFor Ġthe ĠS VM Ġ( im ple mented Ġin Ġsc ik it - learn ĠB IB REF 10 Ġ), Ġthe ĠRB F Ġkernel Ġwas Ġused Ġand Ġa Ġparameter Ġsearch Ġwas Ġconducted Ġfor Ġe ps ilon . ĠDet ailed Ġparameter Ġsettings Ġfor Ġeach Ġsubt ask Ġare Ġshown Ġin ĠTable ĠT AB REF 12 Ġ. ĠEach Ġparameter Ġsearch Ġwas Ġperformed Ġusing Ġ10 - fold Ġcross Ġvalidation , Ġas Ġto Ġnot Ġover fit Ġon Ġthe Ġdevelopment Ġset . </s> Sem i - super vised ĠLearning </s> One Ġof Ġthe Ġaims Ġof Ġthis Ġstudy Ġwas Ġto Ġsee Ġif Ġusing Ġsemi - super vised Ġlearning Ġis Ġbeneficial Ġfor Ġemotion Ġintensity Ġtasks . ĠFor Ġthis Ġpurpose , Ġthe ĠDIS C ĠB IB REF 0 Ġcorpus Ġwas Ġused . ĠThis Ġcorpus Ġwas Ġcreated Ġby Ġquer ying Ġcertain Ġemotion - related Ġwords , Ġwhich Ġmakes Ġit Ġvery Ġsuitable Ġas Ġa Ġsemi - super vised Ġcorpus . ĠHowever , Ġthe Ġspecific Ġemotion Ġthe Ġtweet Ġbelonged Ġto Ġwas Ġnot Ġmade Ġpublic . ĠTherefore , Ġa Ġmethod Ġwas Ġapplied Ġto Ġautomatically Ġassign Ġthe Ġtweets Ġto Ġan Ġemotion Ġby Ġcomparing Ġour Ġscrap ed Ġtweets Ġto Ġthis Ġnew Ġdata Ġset . </s> First , Ġin Ġan Ġattempt Ġto Ġobtain Ġthe Ġquery - terms , Ġwe Ġselected Ġthe Ġ100 Ġwords Ġwhich Ġoccurred Ġmost Ġfrequently Ġin Ġthe ĠDIS C Ġcorpus , Ġin Ġcomparison Ġwith Ġtheir Ġfrequencies Ġin Ġour Ġown Ġscrap ed Ġtweets Ġcorpus . ĠWords Ġthat Ġwere Ġclearly Ġnot Ġindicators Ġof Ġemotion Ġwere Ġremoved . ĠThe Ġrest Ġwas Ġannot ated Ġper Ġemotion Ġor Ġremoved Ġif Ġit Ġwas Ġunclear Ġto Ġwhich Ġemotion Ġthe Ġword Ġbelonged . ĠThis Ġallowed Ġus Ġto Ġcreate Ġsilver Ġdatasets Ġper Ġemotion , Ġassigning Ġtweets Ġto Ġan Ġemotion Ġif Ġan Ġannot ated Ġemotion - word Ġoccurred Ġin Ġthe Ġtweet . </s> Our Ġsemi - super vised Ġapproach Ġis Ġquite Ġstraightforward : Ġfirst Ġa Ġmodel Ġis Ġtrained Ġon Ġthe Ġtraining Ġset Ġand Ġthen Ġthis Ġmodel Ġis Ġused Ġto Ġpredict Ġthe Ġlabels Ġof Ġthe Ġsilver Ġdata . ĠThis Ġsilver Ġdata Ġis Ġthen Ġsimply Ġadded Ġto Ġour Ġtraining Ġset , Ġafter Ġwhich Ġthe Ġmodel Ġis Ġret rained . ĠHowever , Ġan Ġextra Ġstep Ġis Ġapplied Ġto Ġensure Ġthat Ġthe Ġsilver Ġdata Ġis Ġof Ġreasonable Ġquality . ĠInstead Ġof Ġtraining Ġa Ġsingle Ġmodel Ġinitially , Ġten Ġdifferent Ġmodels Ġwere Ġtrained Ġwhich Ġpredict Ġthe Ġlabels Ġof Ġthe Ġsilver Ġinstances . ĠIf Ġthe Ġhighest Ġand Ġlowest Ġprediction Ġdo Ġnot Ġdiffer Ġmore Ġthan Ġa Ġcertain Ġthreshold Ġthe Ġsilver Ġinstance Ġis Ġmaintained , Ġotherwise Ġit Ġis Ġdiscarded . </s> This Ġresults Ġin Ġtwo Ġparameters Ġthat Ġcould Ġbe Ġoptimized : Ġthe Ġthreshold Ġand Ġthe Ġnumber Ġof Ġsilver Ġinstances Ġthat Ġwould Ġbe Ġadded . ĠThis Ġmethod Ġcan Ġbe Ġapplied Ġto Ġboth Ġthe ĠL ST M Ġand Ġfeed - forward Ġnetworks Ġthat Ġwere Ġused . ĠAn Ġoverview Ġof Ġthe Ġcharacteristics Ġof Ġour Ġdata Ġset Ġwith Ġthe Ġfinal Ġparameter Ġsettings Ġis Ġshown Ġin ĠTable ĠT AB REF 14 Ġ. ĠUsually , Ġonly Ġa Ġsmall Ġsubset Ġof Ġdata Ġwas Ġadded Ġto Ġour Ġtraining Ġset , Ġmeaning Ġthat Ġmost Ġof Ġthe Ġsilver Ġdata Ġis Ġnot Ġused Ġin Ġthe Ġexperiments . ĠNote Ġthat Ġsince Ġonly Ġthe Ġemotions Ġwere Ġannot ated , Ġthis Ġmethod Ġis Ġonly Ġapplicable Ġto Ġthe ĠE I Ġtasks . </s> En semb ling </s> To Ġboost Ġperformance , Ġthe ĠS VM , ĠL ST M , Ġand Ġfeed - forward Ġmodels Ġwere Ġcombined Ġinto Ġan Ġensemble . ĠFor Ġboth Ġthe ĠL ST M Ġand Ġfeed - forward Ġapproach , Ġthree Ġdifferent Ġmodels Ġwere Ġtrained . ĠThe Ġfirst Ġmodel Ġwas Ġtrained Ġon Ġthe Ġtraining Ġdata Ġ( regular ), Ġthe Ġsecond Ġmodel Ġwas Ġtrained Ġon Ġboth Ġthe Ġtraining Ġand Ġtranslated Ġtraining Ġdata Ġ( trans lated ) Ġand Ġthe Ġthird Ġone Ġwas Ġtrained Ġon Ġboth Ġthe Ġtraining Ġdata Ġand Ġthe Ġsemi - super vised Ġdata Ġ( silver ). ĠDue Ġto Ġthe Ġnature Ġof Ġthe ĠS VM Ġalgorithm , Ġsemi - super vised Ġlearning Ġdoes Ġnot Ġhelp , Ġso Ġonly Ġthe Ġregular Ġand Ġtranslated Ġmodel Ġwere Ġtrained Ġin Ġthis Ġcase . ĠThis Ġresults Ġin Ġ8 Ġdifferent Ġmodels Ġper Ġsubt ask . ĠNote Ġthat Ġfor Ġthe Ġval ence Ġtasks Ġno Ġsilver Ġtraining Ġdata Ġwas Ġobtained , Ġmeaning Ġthat Ġfor Ġthose Ġtasks Ġthe Ġsemi - super vised Ġmodels Ġcould Ġnot Ġbe Ġused . </s> Per Ġtask , Ġthe ĠL ST M Ġand Ġfeed - forward Ġmodel 's Ġpredictions Ġwere Ġaveraged Ġover Ġ10 Ġprediction Ġruns . ĠSub sequently , Ġthe Ġpredictions Ġof Ġall Ġindividual Ġmodels Ġwere Ġcombined Ġinto Ġan Ġaverage . ĠFinally , Ġmodels Ġwere Ġremoved Ġfrom Ġthe Ġensemble Ġin Ġa Ġstep wise Ġmanner Ġif Ġthe Ġremoval Ġincreased Ġthe Ġaverage Ġscore . ĠThis Ġwas Ġdone Ġbased Ġon Ġtheir Ġoriginal Ġscores , Ġi . e . Ġstarting Ġout Ġby Ġtrying Ġto Ġremove Ġthe Ġworst Ġindividual Ġmodel Ġand Ġworking Ġour Ġway Ġup Ġto Ġthe Ġbest Ġmodel . ĠWe Ġonly Ġconsider Ġit Ġan Ġincrease Ġin Ġscore Ġif Ġthe Ġdifference Ġis Ġlarger Ġthan Ġ0 . 002 Ġ( i . e . Ġthe Ġdifference Ġbetween Ġ0 . 7 16 Ġand Ġ0 . 718 ). ĠIf Ġat Ġsome Ġpoint Ġthe Ġscore Ġdoes Ġnot Ġincrease Ġand Ġwe Ġare Ġtherefore Ġunable Ġto Ġremove Ġa Ġmodel , Ġthe Ġprocess Ġis Ġstopped Ġand Ġour Ġbest Ġensemble Ġof Ġmodels Ġhas Ġbeen Ġfound . ĠThis Ġprocess Ġuses Ġthe Ġscores Ġon Ġthe Ġdevelopment Ġset Ġof Ġdifferent Ġcombinations Ġof Ġmodels . ĠNote Ġthat Ġthis Ġmeans Ġthat Ġthe Ġen semb les Ġfor Ġdifferent Ġsubt asks Ġcan Ġcontain Ġdifferent Ġsets Ġof Ġmodels . ĠThe Ġfinal Ġmodel Ġselections Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 17 Ġ. </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 18 Ġshows Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġof Ġall Ġindividuals Ġmodels , Ġdistinguishing Ġthe Ġthree Ġtypes Ġof Ġtraining : Ġregular Ġ( r ), Ġtranslated Ġ( t ) Ġand Ġsemi - super vised Ġ( s ). ĠIn ĠTables ĠT AB REF 17 Ġand ĠT AB REF 18 Ġ, Ġthe Ġletter Ġbehind Ġeach Ġmodel Ġ( e . g . ĠS VM - r , ĠL ST M - r ) Ġcorresponds Ġto Ġthe Ġtype Ġof Ġtraining Ġused . ĠComp aring Ġthe Ġregular Ġand Ġtranslated Ġcolumns Ġfor Ġthe Ġthree Ġalgorithms , Ġit Ġshows Ġthat Ġin Ġ22 Ġout Ġof Ġ30 Ġcases , Ġusing Ġtranslated Ġinstances Ġas Ġextra Ġtraining Ġdata Ġresulted Ġin Ġan Ġimprovement . ĠFor Ġthe Ġsemi - super vised Ġlearning Ġapproach , Ġan Ġimprovement Ġis Ġfound Ġin Ġ15 Ġout Ġof Ġ16 Ġcases . ĠMoreover , Ġour Ġbest Ġindividual Ġmodel Ġfor Ġeach Ġsubt ask Ġ( bold ed Ġscores Ġin ĠTable ĠT AB REF 18 Ġ) Ġis Ġalways Ġeither Ġa Ġtranslated Ġor Ġsemi - super vised Ġmodel . ĠTable ĠT AB REF 18 Ġalso Ġshows Ġthat , Ġin Ġgeneral , Ġour Ġfeed - forward Ġnetwork Ġobtained Ġthe Ġbest Ġresults , Ġhaving Ġthe Ġhighest ĠF - score Ġfor Ġ8 Ġout Ġof Ġ10 Ġsubt asks . </s> However , ĠTable ĠT AB REF 19 Ġshows Ġthat Ġthese Ġscores Ġcan Ġstill Ġbe Ġimproved Ġby Ġaveraging Ġor Ġen semb ling Ġthe Ġindividual Ġmodels . ĠOn Ġthe Ġdev Ġset , Ġaveraging Ġour Ġ8 Ġindividual Ġmodels Ġresults Ġin Ġa Ġbetter Ġscore Ġfor Ġ8 Ġout Ġof Ġ10 Ġsubt asks , Ġwhile Ġcreating Ġan Ġensemble Ġbeats Ġall Ġof Ġthe Ġindividual Ġmodels Ġas Ġwell Ġas Ġthe Ġaverage Ġfor Ġeach Ġsubt ask . ĠOn Ġthe Ġtest Ġset , Ġhowever , Ġonly Ġa Ġsmall Ġincrease Ġin Ġscore Ġ( if Ġany ) Ġis Ġfound Ġfor Ġstep wise Ġen semb ling , Ġcompared Ġto Ġaveraging . ĠEven Ġthough Ġthe Ġresults Ġdo Ġnot Ġget Ġworse , Ġwe Ġcannot Ġconclude Ġthat Ġstep wise Ġen semb ling Ġis Ġa Ġbetter Ġmethod Ġthan Ġsimply Ġaveraging . </s> Our Ġofficial Ġscores Ġ( column ĠEns ĠTest Ġin ĠTable ĠT AB REF 19 Ġ) Ġhave Ġplaced Ġus Ġsecond Ġ( E I - Reg , ĠE I - O c ), Ġfourth Ġ( V - Reg ) Ġand Ġfifth Ġ( V - O c ) Ġon Ġthe ĠSem E val ĠA IT - 2018 Ġleader board . ĠHowever , Ġit Ġis Ġevident Ġthat Ġthe Ġresults Ġobtained Ġon Ġthe Ġtest Ġset Ġare Ġnot Ġalways Ġin Ġline Ġwith Ġthose Ġachieved Ġon Ġthe Ġdevelopment Ġset . ĠEspecially Ġon Ġthe Ġanger Ġsubt ask Ġfor Ġboth ĠE I - Reg Ġand ĠE I - O c , Ġthe Ġscores Ġare Ġconsiderably Ġlower Ġon Ġthe Ġtest Ġset Ġin Ġcomparison Ġwith Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset . ĠTherefore , Ġa Ġsmall Ġerror Ġanalysis Ġwas Ġperformed Ġon Ġthe Ġinstances Ġwhere Ġour Ġfinal Ġmodel Ġmade Ġthe Ġlargest Ġerrors . </s> Error ĠAnalysis </s> Due Ġto Ġsome Ġlarge Ġdifferences Ġbetween Ġour Ġresults Ġon Ġthe Ġdev Ġand Ġtest Ġset Ġof Ġthis Ġtask , Ġwe Ġperformed Ġa Ġsmall Ġerror Ġanalysis Ġin Ġorder Ġto Ġsee Ġwhat Ġcaused Ġthese Ġdifferences . ĠFor ĠE I - Reg - anger , Ġthe Ġgold Ġlabels Ġwere Ġcompared Ġto Ġour Ġown Ġpredictions , Ġand Ġwe Ġmanually Ġchecked Ġ50 Ġinstances Ġfor Ġwhich Ġour Ġsystem Ġmade Ġthe Ġlargest Ġerrors . </s> Some Ġexamples Ġthat Ġwere Ġindicative Ġof Ġthe Ġshortcomings Ġof Ġour Ġsystem Ġare Ġshown Ġin ĠTable ĠT AB REF 20 Ġ. ĠFirst Ġof Ġall , Ġour Ġsystem Ġdid Ġnot Ġtake Ġinto Ġaccount Ġcapital ization . ĠThe Ġimplications Ġof Ġthis Ġare Ġshown Ġin Ġthe Ġfirst Ġsentence , Ġwhere Ġcapital ization Ġintens ifies Ġthe Ġemotion Ġused Ġin Ġthe Ġsentence . ĠIn Ġthe Ġsecond Ġsentence , Ġthe Ġname ĠImper ator ĠFur ios a Ġis Ġnot Ġunderstood . ĠSince Ġour Ġtexts Ġwere Ġlower c ased , Ġour Ġsystem Ġwas Ġunable Ġto Ġcapture Ġthe Ġnamed Ġentity Ġand Ġthought Ġthe Ġsentence Ġwas Ġabout Ġan Ġangry Ġemperor Ġinstead . ĠIn Ġthe Ġthird Ġsentence , Ġour Ġsystem Ġfails Ġto Ġcapture Ġthat Ġwhen Ġyou Ġare Ġso Ġangry Ġthat Ġit Ġmakes Ġyou Ġlaugh , Ġit Ġresults Ġin Ġa Ġreduced Ġintensity Ġof Ġthe Ġan gr iness . ĠFinally , Ġin Ġthe Ġfourth Ġsentence , Ġit Ġis Ġthe Ġfigur ative Ġlanguage Ġme Ġinf la Ġla Ġv ena Ġ( it Ġinfl ates Ġmy Ġvein ) Ġthat Ġthe Ġsystem Ġis Ġnot Ġable Ġto Ġunderstand . </s> The Ġfirst Ġtwo Ġerror - c ategories Ġmight Ġbe Ġsolved Ġby Ġincluding Ġsmart Ġfeatures Ġregarding Ġcapital ization Ġand Ġnamed Ġentity Ġrecognition . ĠHowever , Ġthe Ġlast Ġtwo Ġcategories Ġare Ġproblems Ġof Ġnatural Ġlanguage Ġunderstanding Ġand Ġwill Ġbe Ġvery Ġdifficult Ġto Ġfix . </s> Conclusion </s> To Ġconclude , Ġthe Ġpresent Ġstudy Ġdescribed Ġour Ġsubmission Ġfor Ġthe ĠSem eval Ġ2018 ĠShared ĠTask Ġon ĠAffect Ġin ĠTwe ets . ĠWe Ġparticipated Ġin Ġfour ĠSpanish Ġsubt asks Ġand Ġour Ġsubmissions Ġranked Ġsecond , Ġsecond , Ġfourth Ġand Ġfifth Ġplace . ĠOur Ġstudy Ġaimed Ġto Ġinvestigate Ġwhether Ġthe Ġautomatic Ġgeneration Ġof Ġadditional Ġtraining Ġdata Ġthrough Ġtranslation Ġand Ġsemi - super vised Ġlearning , Ġas Ġwell Ġas Ġthe Ġcreation Ġof Ġstep wise Ġen semb les , Ġincrease Ġthe Ġperformance Ġof Ġour ĠSpanish - language Ġmodels . ĠStrong Ġsupport Ġwas Ġfound Ġfor Ġthe Ġtranslation Ġand Ġsemi - super vised Ġlearning Ġapproaches ; Ġour Ġbest Ġmodels Ġfor Ġall Ġsubt asks Ġuse Ġeither Ġone Ġof Ġthese Ġapproaches . ĠThese Ġresults Ġsuggest Ġthat Ġboth Ġof Ġthese Ġadditional Ġdata Ġresources Ġare Ġbeneficial Ġwhen Ġdetermining Ġemotion Ġintensity Ġ( for ĠSpanish ). ĠHowever , Ġthe Ġcreation Ġof Ġa Ġstep wise Ġensemble Ġfrom Ġthe Ġbest Ġmodels Ġdid Ġnot Ġresult Ġin Ġbetter Ġperformance Ġcompared Ġto Ġsimply Ġaveraging Ġthe Ġmodels . ĠIn Ġaddition , Ġsome Ġsigns Ġof Ġover fitting Ġon Ġthe Ġdev Ġset Ġwere Ġfound . ĠIn Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġapply Ġthe Ġmethods Ġ( translation Ġand Ġsemi - super vised Ġlearning ) Ġused Ġon ĠSpanish Ġon Ġother Ġlow - resource Ġlanguages Ġand Ġpotentially Ġalso Ġon Ġother Ġtasks .