the model with multi-attention mechanism and a projected layer
ABSTRACTIVE
Projected Layer
EXTRACTIVE
We have evaluated our models considering the F1 Score, which is the harmonic mean of precision and recall. We have run ten times the experiment for each model and considered the average F1 Score. The results are mentioned in Table TABREF11. Considering F1 Macro the models that include the multi-attention mechanism outperform the others and particularly the one with the Projected Layer has the highest performance. In three out of four pairs of models, the ones with the Projected Layer achieved better performance, so in most cases the addition of the Projected Layer had a significant enhancement.
We have evaluated our models considering the F1 Score, which is the harmonic mean of precision and recall. We have run ten times the experiment for each model and considered the average F1 Score. The results are mentioned in Table TABREF11. Considering F1 Macro the models that include the multi-attention mechanism outperform the others and particularly the one with the Projected Layer has the highest performance. In three out of four pairs of models, the ones with the Projected Layer achieved better performance, so in most cases the addition of the Projected Layer had a significant enhancement.
1909.13104
Introduction </s> In Ġthe Ġera Ġof Ġsocial Ġmedia Ġand Ġnetworking Ġplatforms , ĠTwitter Ġhas Ġbeen Ġdoomed Ġfor Ġabuse Ġand Ġharassment Ġtoward Ġusers Ġspecifically Ġwomen . ĠIn Ġfact , Ġonline Ġharassment Ġbecomes Ġvery Ġcommon Ġin ĠTwitter Ġand Ġthere Ġhave Ġbeen Ġa Ġlot Ġof Ġcritics Ġthat ĠTwitter Ġhas Ġbecome Ġthe Ġplatform Ġfor Ġmany Ġracists , Ġmisogyn ists Ġand Ġhate Ġgroups Ġwhich Ġcan Ġexpress Ġthemselves Ġopenly . ĠOnline Ġharassment Ġis Ġusually Ġin Ġthe Ġform Ġof Ġverbal Ġor Ġgraphical Ġformats Ġand Ġis Ġconsidered Ġharassment , Ġbecause Ġit Ġis Ġneither Ġinvited Ġnor Ġhas Ġthe Ġconsent Ġof Ġthe Ġreceipt . ĠMonitoring Ġthe Ġcontents Ġincluding Ġsexism Ġand Ġsexual Ġharassment Ġin Ġtraditional Ġmedia Ġis Ġeasier Ġthan Ġmonitoring Ġon Ġthe Ġonline Ġsocial Ġmedia Ġplatforms Ġlike ĠTwitter . ĠThe Ġmain Ġreason Ġis Ġbecause Ġof Ġthe Ġlarge Ġamount Ġof Ġuser Ġgenerated Ġcontent Ġin Ġthese Ġmedia . ĠSo , Ġthe Ġresearch Ġabout Ġthe Ġautomated Ġdetection Ġof Ġcontent Ġcontaining Ġsexual Ġharassment Ġis Ġan Ġimportant Ġissue Ġand Ġcould Ġbe Ġthe Ġbasis Ġfor Ġremoving Ġthat Ġcontent Ġor Ġflag ging Ġit Ġfor Ġhuman Ġevaluation . ĠThe Ġbasic Ġgoal Ġof Ġthis Ġautomatic Ġclassification Ġis Ġthat Ġit Ġwill Ġsignificantly Ġimprove Ġthe Ġprocess Ġof Ġdetecting Ġthese Ġtypes Ġof Ġhate Ġspeech Ġon Ġsocial Ġmedia Ġby Ġreducing Ġthe Ġtime Ġand Ġeffort Ġrequired Ġby Ġhuman Ġbeings . </s> Previous Ġstudies Ġhave Ġbeen Ġfocused Ġon Ġcollecting Ġdata Ġabout Ġsexism Ġand Ġracism Ġin Ġvery Ġbroad Ġterms Ġor Ġhave Ġproposed Ġtwo Ġcategories Ġof Ġsexism Ġas Ġbenevolent Ġor Ġhostile Ġsexism ĠB IB REF 0 , Ġwhich Ġundermines Ġother Ġtypes Ġof Ġonline Ġharassment . ĠHowever , Ġthere Ġis Ġno Ġmuch Ġstudy Ġfocusing Ġon Ġdifferent Ġtypes Ġonline Ġharassment Ġalone Ġattracting Ġnatural Ġlanguage Ġprocessing Ġtechniques . </s> In Ġthis Ġpaper Ġwe Ġpresent Ġour Ġwork , Ġwhich Ġis Ġa Ġpart Ġof Ġthe ĠSoc ia L ĠMedia ĠAnd ĠHar assment ĠCompetition Ġof Ġthe ĠEC ML ĠPK DD Ġ2019 ĠConference . ĠThe Ġtopic Ġof Ġthe Ġcompetition Ġis Ġthe Ġclassification Ġof Ġdifferent Ġtypes Ġof Ġharassment Ġand Ġit Ġis Ġdivided Ġin Ġtwo Ġtasks . ĠThe Ġfirst Ġone Ġis Ġthe Ġclassification Ġof Ġthe Ġtweets Ġin Ġharassment Ġand Ġnon - har assment Ġcategories , Ġwhile Ġthe Ġsecond Ġone Ġis Ġthe Ġclassification Ġin Ġspecific Ġharassment Ġcategories Ġlike Ġindirect Ġharassment , Ġphysical Ġand Ġsexual Ġharassment Ġas Ġwell . ĠWe Ġare Ġusing Ġthe Ġdataset Ġof Ġthe Ġcompetition , Ġwhich Ġincludes Ġtext Ġfrom Ġtweets Ġhaving Ġthe Ġaforementioned Ġcategories . ĠOur Ġapproach Ġis Ġbased Ġon Ġthe ĠRec urrent ĠNeural ĠNetworks Ġand Ġparticularly Ġwe Ġare Ġusing Ġa Ġdeep , Ġclass i ï ¬ ģ cation Ġspecific Ġattention Ġmechanism . ĠMoreover , Ġwe Ġpresent Ġa Ġcomparison Ġbetween Ġdifferent Ġvariations Ġof Ġthis Ġattention - based Ġapproach Ġlike Ġmulti - att ention Ġand Ġsingle Ġattention Ġmodels . ĠThe Ġnext ĠSection Ġincludes Ġa Ġshort Ġdescription Ġof Ġthe Ġrelated Ġwork , Ġwhile Ġthe Ġthird ĠSection Ġincludes Ġa Ġdescription Ġof Ġthe Ġdataset . ĠAfter Ġthat , Ġwe Ġdescribe Ġour Ġmethodology . ĠFinally , Ġwe Ġdescribe Ġthe Ġexperiments Ġand Ġwe Ġpresent Ġthe Ġresults Ġand Ġour Ġconclusion . </s> Related ĠWork </s> W ase em Ġet Ġal . ĠB IB REF 1 Ġwere Ġthe Ġfirst Ġwho Ġcollected Ġhateful Ġtweets Ġand Ġcategorized Ġthem Ġinto Ġbeing Ġsexist , Ġracist Ġor Ġneither . ĠHowever , Ġthey Ġdid Ġnot Ġprovide Ġspecific Ġdefinitions Ġfor Ġeach Ġcategory . ĠJ ha Ġand ĠMam idi ĠB IB REF 0 Ġfocused Ġon Ġjust Ġsexist Ġtweets Ġand Ġproposed Ġtwo Ġcategories Ġof Ġhostile Ġand Ġbenevolent Ġsexism . ĠHowever , Ġthese Ġcategories Ġwere Ġgeneral Ġas Ġthey Ġignored Ġother Ġtypes Ġof Ġsexism Ġhappening Ġin Ġsocial Ġmedia . ĠSharif ir ad ĠS . Ġand ĠMat win ĠS . ĠB IB REF 2 Ġproposed Ġcomplimentary Ġcategories Ġof Ġsexist Ġlanguage Ġinspired Ġfrom Ġsocial Ġscience Ġwork . ĠThey Ġcategorized Ġthe Ġsexist Ġtweets Ġinto Ġthe Ġcategories Ġof Ġindirect Ġharassment , Ġinformation Ġthreat , Ġsexual Ġharassment Ġand Ġphysical Ġharassment . ĠIn Ġthe Ġnext Ġyear Ġthe Ġsame Ġauthors Ġproposed ĠB IB REF 3 Ġa Ġmore Ġcomprehensive Ġcategor ization Ġof Ġonline Ġharassment Ġin Ġsocial Ġmedia Ġe . g . Ġtwitter Ġinto Ġthe Ġfollowing Ġcategories , Ġindirect Ġharassment , Ġinformation Ġthreat , Ġsexual Ġharassment , Ġphysical Ġharassment Ġand Ġnot Ġsexist . </s> For Ġthe Ġdetection Ġof Ġhate Ġspeech Ġin Ġsocial Ġmedia Ġlike Ġtwitter , Ġmany Ġapproaches Ġhave Ġbeen Ġproposed . ĠJ ha Ġand ĠMam idi ĠB IB REF 0 Ġtested Ġsupport Ġvector Ġmachine , Ġbi - direction al ĠR NN Ġenc oder - dec oder Ġand ĠFast Text Ġon Ġhostile Ġand Ġbenevolent Ġsexist Ġtweets . ĠThey Ġalso Ġused ĠSent i Word Net Ġand Ġsubject ivity Ġlex icon Ġon Ġthe Ġextracted Ġphrases Ġto Ġshow Ġthe Ġpol arity Ġof Ġthe Ġtweets . ĠSharif ir ad Ġet Ġal . ĠB IB REF 4 Ġtrained , Ġtested Ġand Ġevaluated Ġdifferent Ġclassification Ġmethods Ġon Ġthe ĠSem E val 2018 Ġdataset Ġand Ġchose Ġthe Ġclass ifier Ġwith Ġthe Ġhighest Ġaccuracy Ġfor Ġtesting Ġon Ġeach Ġcategory Ġof Ġsexist Ġtweets Ġto Ġknow Ġthe Ġmental Ġstate Ġand Ġthe Ġaffect ual Ġstate Ġof Ġthe Ġuser Ġwho Ġtweets Ġin Ġeach Ġcategory . ĠTo Ġovercome Ġthe Ġlimitations Ġof Ġsmall Ġdata Ġsets Ġon Ġsexist Ġspeech Ġdetection , ĠSharif ir ad ĠS . Ġet Ġal . ĠB IB REF 5 Ġhave Ġapplied Ġtext Ġaug mentation Ġand Ġtext Ġgeneration Ġwith Ġcertain Ġsuccess . ĠThey Ġhave Ġgenerated Ġnew Ġtweets Ġby Ġreplacing Ġwords Ġin Ġorder Ġto Ġincrease Ġthe Ġsize Ġof Ġour Ġtraining Ġset . ĠMoreover , Ġin Ġthe Ġpresented Ġtext Ġaug mentation Ġapproach , Ġthe Ġnumber Ġof Ġtweets Ġin Ġeach Ġclass Ġremains Ġthe Ġsame , Ġbut Ġtheir Ġwords Ġare Ġaugmented Ġwith Ġwords Ġextracted Ġfrom Ġtheir ĠConcept Net Ġrelations Ġand Ġtheir Ġdescription Ġextracted Ġfrom ĠWik id ata . ĠZhang Ġet Ġal . ĠB IB REF 6 Ġcombined Ġconv olution al Ġand Ġg ated Ġrecurrent Ġnetworks Ġto Ġdetect Ġhate Ġspeech Ġin Ġtweets . ĠOthers Ġhave Ġproposed Ġdifferent Ġmethods , Ġwhich Ġare Ġnot Ġbased Ġon Ġdeep Ġlearning . ĠBurn ap Ġand ĠWilliams ĠB IB REF 7 Ġused ĠSupport ĠVector ĠMachines , ĠRandom ĠFore sts Ġand Ġa Ġmeta - class ifier Ġto Ġdistinguish Ġbetween Ġhateful Ġand Ġnon - h ateful Ġmessages . ĠA Ġsurvey Ġof Ġrecent Ġresearch Ġin Ġthe Ġfield Ġis Ġpresented Ġin ĠB IB REF 8 . ĠFor Ġthe Ġproblem Ġof Ġthe Ġhate Ġspeech Ġdetection Ġa Ġfew Ġapproaches Ġhave Ġbeen Ġproposed Ġthat Ġare Ġbased Ġon Ġthe ĠAttention Ġmechanism . ĠPav l opoulos Ġet Ġal . ĠB IB REF 9 Ġhave Ġproposed Ġa Ġnovel , Ġclassification - specific Ġattention Ġmechanism Ġthat Ġimproves Ġthe Ġperformance Ġof Ġthe ĠR NN Ġfurther Ġfor Ġthe Ġdetection Ġof Ġabusive Ġcontent Ġin Ġthe Ġweb . ĠXie Ġet Ġal . ĠB IB REF 10 Ġfor Ġemotion Ġintensity Ġprediction , Ġwhich Ġis Ġa Ġsimilar Ġproblem Ġto Ġours , Ġhave Ġproposed Ġa Ġnovel Ġattention Ġmechanism Ġfor ĠCNN Ġmodel Ġthat Ġassociates Ġattention - based Ġweights Ġfor Ġevery Ġconv olution Ġwindow . ĠPark Ġand ĠF ung ĠB IB REF 11 Ġtransformed Ġthe Ġclass i ï ¬ ģ cation Ġinto Ġa Ġ2 - step Ġproblem , Ġwhere Ġabusive Ġtext Ġï ¬ ģ r st Ġis Ġdistinguished Ġfrom Ġthe Ġnon - ab usive , Ġand Ġthen Ġthe Ġclass Ġof Ġabuse Ġ( Sex ism Ġor ĠRac ism ) Ġis Ġdetermined . ĠHowever , Ġwhile Ġthe Ġfirst Ġpart Ġof Ġthe Ġtwo Ġstep Ġclass i ï ¬ ģ cation Ġperforms Ġquite Ġwell , Ġit Ġfalls Ġshort Ġin Ġdetecting Ġthe Ġparticular Ġclass Ġthe Ġabusive Ġtext Ġbelongs Ġto . ĠP its il is Ġet Ġal . ĠB IB REF 12 Ġhave Ġproposed Ġa Ġdetection Ġscheme Ġthat Ġis Ġan Ġensemble Ġof ĠR NN Ġclass i ï ¬ ģ ers , Ġwhich Ġincorporates Ġvarious Ġfeatures Ġassociated Ġwith Ġuser Ġrelated Ġinformation , Ġsuch Ġas Ġthe Ġusers âĢ Ļ Ġtendency Ġtowards Ġracism Ġor Ġsexism </s> Dat as et Ġdescription </s> The Ġdataset Ġfrom ĠTwitter Ġthat Ġwe Ġare Ġusing Ġin Ġour Ġwork , Ġconsists Ġof Ġa Ġtrain Ġset , Ġa Ġvalidation Ġset Ġand Ġa Ġtest Ġset . ĠIt Ġwas Ġpublished Ġfor Ġthe Ġ" First Ġworkshop Ġon Ġcategor izing Ġdifferent Ġtypes Ġof Ġonline Ġharassment Ġlanguages Ġin Ġsocial Ġmedia ". ĠThe Ġwhole Ġdataset Ġis Ġdivided Ġinto Ġtwo Ġcategories , Ġwhich Ġare Ġharassment Ġand Ġnon - har assment Ġtweets . ĠMoreover , Ġconsidering Ġthe Ġtype Ġof Ġthe Ġharassment , Ġthe Ġtweets Ġare Ġdivided Ġinto Ġthree Ġsub - c ategories Ġwhich Ġare Ġindirect Ġharassment , Ġsexual Ġand Ġphysical Ġharassment . ĠWe Ġcan Ġsee Ġin ĠTable ĠT AB REF 1 Ġthe Ġclass Ġdistribution Ġof Ġour Ġdataset . ĠOne Ġimportant Ġissue Ġhere Ġis Ġthat Ġthe Ġcategories Ġof Ġindirect Ġand Ġphysical Ġharassment Ġseem Ġto Ġbe Ġmore Ġrare Ġin Ġthe Ġtrain Ġset Ġthan Ġin Ġthe Ġvalidation Ġand Ġtest Ġsets . ĠTo Ġtackle Ġthis Ġissue , Ġas Ġwe Ġdescribe Ġin Ġthe Ġnext Ġsection , Ġwe Ġare Ġperforming Ġdata Ġaug mentation Ġtechniques . ĠHowever , Ġthe Ġdataset Ġis Ġim balanced Ġand Ġthis Ġhas Ġa Ġsignificant Ġimpact Ġin Ġour Ġresults . </s> Prop osed Ġmethodology Ġ: :: ĠData Ġaug mentation </s> As Ġdescribed Ġbefore Ġone Ġcrucial Ġissue Ġthat Ġwe Ġare Ġtrying Ġto Ġtackle Ġin Ġthis Ġwork Ġis Ġthat Ġthe Ġgiven Ġdataset Ġis Ġim balanced . ĠParticularly , Ġthere Ġare Ġonly Ġa Ġfew Ġinstances Ġfrom Ġindirect Ġand Ġphysical Ġharassment Ġcategories Ġrespectively Ġin Ġthe Ġtrain Ġset , Ġwhile Ġthere Ġare Ġmuch Ġmore Ġin Ġthe Ġvalidation Ġand Ġtest Ġsets Ġfor Ġthese Ġcategories . ĠTo Ġtackle Ġthis Ġissue Ġwe Ġapplying Ġa Ġback - translation Ġmethod ĠB IB REF 13 , Ġwhere Ġwe Ġtranslate Ġindirect Ġand Ġphysical Ġharassment Ġtweets Ġof Ġthe Ġtrain Ġset Ġfrom Ġenglish Ġto Ġg erman , Ġfrench Ġand Ġg reek . ĠAfter Ġthat , Ġwe Ġtranslate Ġthem Ġback Ġto Ġenglish Ġin Ġorder Ġto Ġachieve Ġdata Ġaug mentation . ĠThese Ġ" no isy " Ġdata Ġthat Ġhave Ġbeen Ġtranslated Ġback , Ġincrease Ġthe Ġnumber Ġof Ġindirect Ġand Ġphysical Ġharassment Ġtweets Ġand Ġboost Ġsignificantly Ġthe Ġperformance Ġof Ġour Ġmodels . </s> Another Ġway Ġto Ġenrich Ġour Ġmodels Ġis Ġthe Ġuse Ġof Ġpre - trained Ġword Ġembed d ings Ġfrom Ġ2 B ĠTwitter Ġdata ĠB IB REF 14 Ġhaving Ġ27 B Ġtokens , Ġfor Ġthe Ġinitialization Ġof Ġthe Ġembed ding Ġlayer . </s> Prop osed Ġmethodology Ġ: :: ĠText Ġprocessing </s> Before Ġtraining Ġour Ġmodels Ġwe Ġare Ġprocessing Ġthe Ġgiven Ġtweets Ġusing Ġa Ġtweet Ġpre - processor . ĠThe Ġscope Ġhere Ġis Ġthe Ġcleaning Ġand Ġtoken ization Ġof Ġthe Ġdataset . </s> Prop osed Ġmethodology Ġ: :: ĠR NN ĠModel Ġand ĠAttention ĠMechan ism </s> We Ġare Ġpresenting Ġan Ġattention - based Ġapproach Ġfor Ġthe Ġproblem Ġof Ġthe Ġharassment Ġdetection Ġin Ġtweets . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġbasic Ġapproach Ġof Ġour Ġwork . ĠWe Ġare Ġusing ĠR NN Ġmodels Ġbecause Ġof Ġtheir Ġability Ġto Ġdeal Ġwith Ġsequence Ġinformation . ĠThe ĠR NN Ġmodel Ġis Ġa Ġchain Ġof ĠGR U Ġcells ĠB IB REF 15 Ġthat Ġtransforms Ġthe Ġtokens Ġ$ w _{ 1 }, Ġw _{ 2 }, ... , Ġw _{ k } $ Ġof Ġeach Ġtweet Ġto Ġthe Ġhidden Ġstates Ġ$ h _{ 1 }, Ġh _{ 2 }, ... , Ġh _{ k } $, Ġfollowed Ġby Ġan ĠLR ĠLayer Ġthat Ġuses Ġ$ h _{ k } $ Ġto Ġclassify Ġthe Ġtweet Ġas Ġharassment Ġor Ġnon - har assment Ġ( similar ly Ġfor Ġthe Ġother Ġcategories ). ĠGiven Ġthe Ġvocabulary ĠV Ġand Ġa Ġmatrix ĠE Ġ$\ in Ġ$ Ġ$ R ^{ d Ġ\ times Ġ\ vert ĠV Ġ\ vert Ġ} $ Ġcontaining Ġd - dimensional Ġword Ġembed d ings , Ġan Ġinitial Ġ$ h _{ 0 } $ Ġand Ġa Ġtweet Ġ$ w Ġ= Ġ< w _{ 1 }, .. , Ġw _{ k } > $, Ġthe ĠR NN Ġcomp utes Ġ$ h _{ 1 }, Ġh _{ 2 }, ... , Ġh _{ k } $, Ġwith Ġ$ h _{ t } Ġ\ in ĠR ^{ m } $, Ġas Ġfollows : </s> where Ġ$ h ^{ ^ {\ prime Ġ}} _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġproposed Ġhidden Ġstate Ġat Ġposition Ġt , Ġobtained Ġusing Ġthe Ġword Ġembed ding Ġ$ x _{ t } $ Ġof Ġtoken Ġ$ w _{ t } $ Ġand Ġthe Ġprevious Ġhidden Ġstate Ġ$ h _{ t - 1 } $, Ġ$\ od ot Ġ$ Ġrepresents Ġthe Ġelement - wise Ġmultiplication , Ġ$ r _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġreset Ġgate , Ġ$ z _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġupdate Ġgate , Ġ$\ s igma Ġ$ Ġis Ġthe Ġs igm oid Ġfunction . ĠAlso Ġ$ W _{ h }, ĠW _{ z }, ĠW _{ r } Ġ\ in ĠR ^{ m Ġ\ times Ġd } $ Ġand Ġ$ U _{ h }, ĠU _{ z }, ĠU _{ r } Ġ\ in ĠR ^{ m Ġ\ times Ġm } $, Ġ$ b _{ h }, Ġb _{ z }, Ġb _{ r } Ġ\ in ĠR ^{ m } $. ĠAfter Ġthe Ġcomputation Ġof Ġstate Ġ$ h _{ k } $ Ġthe ĠLR ĠLayer Ġestimates Ġthe Ġprobability Ġthat Ġtweet Ġw Ġshould Ġbe Ġconsidered Ġas Ġharassment , Ġwith Ġ$ W _{ p } Ġ\ in ĠR ^{ 1 Ġ\ times Ġm }, Ġb _{ p } Ġ\ in ĠR $ : </s> We Ġwould Ġlike Ġto Ġadd Ġan Ġattention Ġmechanism Ġsimilar Ġto Ġthe Ġone Ġpresented Ġin ĠB IB REF 9 , Ġso Ġthat Ġthe ĠLR ĠLayer Ġwill Ġconsider Ġthe Ġweighted Ġsum Ġ$ h _{ sum } $ Ġof Ġall Ġthe Ġhidden Ġstates Ġinstead Ġof Ġ$ h _{ k } $ : </s> $ h _{ sum } Ġ= Ġ\ sum Ġ_ { t = 1 } ^{ k } Ġ\ alpha Ġ_ { t } h _{ t } $ </s> $ P _{ att ention R NN } Ġ= Ġ\ s igma Ġ( W _{ p } h _{ sum } Ġ+ Ġb _{ p }) $ </s> Alternatively , Ġwe Ġcould Ġpass Ġ$ h _{ sum } $ Ġthrough Ġan ĠML P Ġwith Ġk Ġlayers Ġand Ġthen Ġthe ĠLR Ġlayer Ġwill Ġestimate Ġthe Ġcorresponding Ġprobability . ĠMore Ġformally , </s> $ P _{ att ention R NN } Ġ= Ġ\ s igma Ġ( W _{ p } h _{ * } Ġ+ Ġb _{ p }) $ </s> where Ġ$ h _{ * } $ Ġis Ġthe Ġstate Ġthat Ġcomes Ġout Ġfrom Ġthe ĠML P . ĠThe Ġweights Ġ$\ alpha Ġ_ { t } $ Ġare Ġproduced Ġby Ġan Ġattention Ġmechanism Ġpresented Ġin ĠB IB REF 9 Ġ( see ĠFig . ĠFIG REF 7 ), Ġwhich Ġis Ġan ĠML P Ġwith Ġl Ġlayers . ĠThis Ġattention Ġmechanism Ġdiffers Ġfrom Ġmost Ġprevious Ġones ĠB IB REF 16 , ĠB IB REF 17 , Ġbecause Ġit Ġis Ġused Ġin Ġa Ġclassification Ġsetting , Ġwhere Ġthere Ġis Ġno Ġpreviously Ġgenerated Ġoutput Ġsub - sequence Ġto Ġdrive Ġthe Ġattention . ĠIt Ġassigns Ġlarger Ġweights Ġ$\ alpha Ġ_ { t } $ Ġto Ġhidden Ġstates Ġ$ h _{ t } $ Ġcorresponding Ġto Ġpositions , Ġwhere Ġthere Ġis Ġmore Ġevidence Ġthat Ġthe Ġtweet Ġshould Ġbe Ġharassment Ġ( or Ġany Ġother Ġspecific Ġtype Ġof Ġharassment ) Ġor Ġnot . ĠIn Ġour Ġwork Ġwe Ġare Ġusing Ġfour Ġattention Ġmechanisms Ġinstead Ġof Ġone Ġthat Ġis Ġpresented Ġin ĠB IB REF 9 . ĠParticularly , Ġwe Ġare Ġusing Ġone Ġattention Ġmechanism Ġper Ġcategory . ĠAnother Ġelement Ġthat Ġdifferent iates Ġour Ġapproach Ġfrom ĠPav l opoulos Ġet Ġal . ĠB IB REF 9 Ġis Ġthat Ġwe Ġare Ġusing Ġa Ġprojection Ġlayer Ġfor Ġthe Ġword Ġembed d ings Ġ( see ĠFig . ĠFIG REF 2 ). ĠIn Ġthe Ġnext Ġsubsection Ġwe Ġdescribe Ġthe ĠModel ĠArchitecture Ġof Ġour Ġapproach . </s> Prop osed Ġmethodology Ġ: :: ĠModel ĠArchitecture </s> The ĠEmb edd ing ĠLayer Ġis Ġinitialized Ġusing Ġpre - trained Ġword Ġembed d ings Ġof Ġdimension Ġ200 Ġfrom ĠTwitter Ġdata Ġthat Ġhave Ġbeen Ġdescribed Ġin Ġa Ġprevious Ġsub - section . ĠAfter Ġthe ĠEmb edd ing ĠLayer , Ġwe Ġare Ġapplying Ġa ĠSp atial ĠDrop out ĠLayer , Ġwhich Ġdrops Ġa Ġcertain Ġpercentage Ġof Ġdimensions Ġfrom Ġeach Ġword Ġvector Ġin Ġthe Ġtraining Ġsample . ĠThe Ġrole Ġof ĠDrop out Ġis Ġto Ġimprove Ġgeneral ization Ġperformance Ġby Ġpreventing Ġactiv ations Ġfrom Ġbecoming Ġstrongly Ġcorrelated ĠB IB REF 18 . ĠSp atial ĠDrop out , Ġwhich Ġhas Ġbeen Ġproposed Ġin ĠB IB REF 19 , Ġis Ġan Ġalternative Ġway Ġto Ġuse Ġdrop out Ġwith Ġconv olution al Ġneural Ġnetworks Ġas Ġit Ġis Ġable Ġto Ġdrop out Ġentire Ġfeature Ġmaps Ġfrom Ġthe Ġconv olution al Ġlayer Ġwhich Ġare Ġthen Ġnot Ġused Ġduring Ġpool ing . ĠAfter Ġthat , Ġthe Ġword Ġembed d ings Ġare Ġpassing Ġthrough Ġa Ġone - layer ĠML P , Ġwhich Ġhas Ġtan h Ġas Ġactivation Ġfunction Ġand Ġ128 Ġhidden Ġunits , Ġin Ġorder Ġto Ġproject Ġthem Ġin Ġthe Ġvector Ġspace Ġof Ġour Ġproblem Ġconsidering Ġthat Ġthey Ġhave Ġbeen Ġpre - trained Ġusing Ġtext Ġthat Ġhas Ġa Ġdifferent Ġsubject . ĠIn Ġthe Ġnext Ġstep Ġthe Ġembed d ings Ġare Ġfed Ġin Ġa Ġun id irection al ĠGR U Ġhaving Ġ1 ĠSt acked ĠLayer Ġand Ġsize Ġ128 . ĠWe Ġprefer ĠGR U Ġthan ĠL ST M , Ġbecause Ġit Ġis Ġmore Ġefficient Ġcomput ationally . ĠAlso Ġthe Ġbasic Ġadvantage Ġof ĠL ST M Ġwhich Ġis Ġthe Ġability Ġto Ġkeep Ġin Ġmemory Ġlarge Ġtext Ġdocuments , Ġdoes Ġnot Ġhold Ġhere , Ġbecause Ġtweets Ġsupposed Ġto Ġbe Ġnot Ġtoo Ġlarge Ġtext Ġdocuments . ĠThe Ġoutput Ġstates Ġof Ġthe ĠGR U Ġare Ġpassing Ġthrough Ġfour Ġself - att ent ions Ġlike Ġthe Ġone Ġdescribed Ġabove ĠB IB REF 9 , Ġbecause Ġwe Ġare Ġusing Ġone Ġattention Ġper Ġcategory Ġ( see ĠFig . ĠFIG REF 7 ). ĠFinally , Ġa Ġone - layer ĠML P Ġhaving Ġ128 Ġnodes Ġand ĠRe LU Ġas Ġactivation Ġfunction Ġcomp utes Ġthe Ġfinal Ġscore Ġfor Ġeach Ġcategory . ĠAt Ġthis Ġfinal Ġstage Ġwe Ġhave Ġavoided Ġusing Ġa Ġsoft max Ġfunction Ġto Ġdecide Ġthe Ġharassment Ġtype Ġconsidering Ġthat Ġthe Ġtweet Ġis Ġa Ġharassment , Ġotherwise Ġwe Ġhad Ġto Ġtrain Ġour Ġmodels Ġtaking Ġinto Ġaccount Ġonly Ġthe Ġharassment Ġtweets Ġand Ġthis Ġmight Ġhave Ġbeen Ġa Ġproblem Ġas Ġthe Ġdataset Ġis Ġnot Ġlarge Ġenough . </s> Exper iments Ġ: :: ĠTraining ĠModels </s> In Ġthis Ġsubsection Ġwe Ġare Ġgiving Ġthe Ġdetails Ġof Ġthe Ġtraining Ġprocess Ġof Ġour Ġmodels . ĠMoreover , Ġwe Ġare Ġdescribing Ġthe Ġdifferent Ġmodels Ġthat Ġwe Ġcompare Ġin Ġour Ġexperiments . </s> B atch Ġsize Ġwhich Ġper tains Ġto Ġthe Ġamount Ġof Ġtraining Ġsamples Ġto Ġconsider Ġat Ġa Ġtime Ġfor Ġupdating Ġour Ġnetwork Ġweights , Ġis Ġset Ġto Ġ32 , Ġbecause Ġour Ġdataset Ġis Ġnot Ġlarge Ġand Ġsmall Ġbatches Ġmight Ġhelp Ġto Ġgeneral ize Ġbetter . ĠAlso , Ġwe Ġset Ġother Ġhyper param eters Ġas : Ġepoch s Ġ= Ġ20 , Ġpatience Ġ= Ġ10 . ĠAs Ġearly Ġstopping Ġcriterion Ġwe Ġchoose Ġthe Ġaverage ĠA UC , Ġbecause Ġour Ġdataset Ġis Ġim balanced . </s> The Ġtraining Ġprocess Ġis Ġbased Ġon Ġthe Ġoptimization Ġof Ġthe Ġloss Ġfunction Ġmentioned Ġbelow Ġand Ġit Ġis Ġcarried Ġout Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 20 , Ġwhich Ġis Ġknown Ġfor Ġyielding Ġquicker Ġconvergence . ĠWe Ġset Ġthe Ġlearning Ġrate Ġequal Ġto Ġ0 . 001 : </s> $ L Ġ= Ġ\ frac { 1 }{ 2 } BC E ( har assment ) Ġ+ Ġ\ frac { 1 }{ 2 } (\ frac { 1 }{ 5 } BC E ( sexual H ) Ġ+ Ġ\ frac { 2 }{ 5 } BC E ( ind irect H )+ \ frac { 2 }{ 5 } BC E ( physical H )) $ </s> where ĠBCE Ġis Ġthe Ġbinary Ġcross - ent ropy Ġloss Ġfunction , </s> $ BC E Ġ= Ġ- \ frac { 1 }{ n }\ sum Ġ_ { i = 1 } ^{ n } [ y _{ i } log ( y ^{ ^ {\ prime Ġ}} _{ i }) Ġ+ Ġ( 1 Ġ- Ġy _{ i }) log ( 1 Ġ- Ġy ^{ ^ {\ prime Ġ}} _{ i } )) ] $ </s> $ i $ Ġdenotes Ġthe Ġ$ i $ th Ġtraining Ġsample , Ġ$ y $ Ġis Ġthe Ġbinary Ġrepresentation Ġof Ġtrue Ġharassment Ġlabel , Ġand Ġ$ y ^{ ^ {\ prime Ġ}} $ Ġis Ġthe Ġpredicted Ġprobability . ĠIn Ġthe Ġloss Ġfunction Ġwe Ġhave Ġapplied Ġequal Ġweight Ġto Ġboth Ġtasks . ĠHowever , Ġin Ġthe Ġsecond Ġtask Ġ( type Ġof Ġharassment Ġclassification ) Ġwe Ġhave Ġapplied Ġhigher Ġweight Ġin Ġthe Ġcategories Ġthat Ġit Ġis Ġharder Ġto Ġpredict Ġdue Ġto Ġthe Ġproblem Ġof Ġthe Ġclass Ġimbalance Ġbetween Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġrespectively . </s> Exper iments Ġ: :: ĠEvaluation Ġand ĠResults </s> Each Ġmodel Ġproduces Ġfour Ġscores Ġand Ġeach Ġscore Ġis Ġthe Ġprobability Ġthat Ġa Ġtweet Ġincludes Ġharassment Ġlanguage , Ġindirect , Ġphysical Ġand Ġsexual Ġharassment Ġlanguage Ġrespectively . ĠFor Ġany Ġtweet , Ġwe Ġfirst Ġcheck Ġthe Ġscore Ġof Ġthe Ġharassment Ġlanguage Ġand Ġif Ġit Ġis Ġless Ġthan Ġa Ġspecified Ġthreshold , Ġthen Ġthe Ġharassment Ġlabel Ġis Ġzero , Ġso Ġthe Ġother Ġthree Ġlabels Ġare Ġzero Ġas Ġwell . ĠIf Ġit Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġthat Ġthreshold , Ġthen Ġthe Ġharassment Ġlabel Ġis Ġone Ġand Ġthe Ġtype Ġof Ġharassment Ġis Ġthe Ġone Ġamong Ġthese Ġthree Ġhaving Ġthat Ġhas Ġthe Ġgreatest Ġscore Ġ( highest Ġprobability ). ĠWe Ġset Ġthis Ġthreshold Ġequal Ġto Ġ0 . 33 . </s> We Ġcompare Ġeight Ġdifferent Ġmodels Ġin Ġour Ġexperiments . ĠFour Ġof Ġthem Ġhave Ġa ĠProject ed ĠLayer Ġ( see ĠFig . ĠFIG REF 2 ), Ġwhile Ġthe Ġothers Ġdo Ġnot Ġhave , Ġand Ġthis Ġis Ġthe Ġonly Ġdifference Ġbetween Ġthese Ġtwo Ġgroups Ġof Ġour Ġmodels . ĠSo , Ġwe Ġactually Ġinclude Ġfour Ġmodels Ġin Ġour Ġexperiments Ġ( having Ġa Ġprojected Ġlayer Ġor Ġnot ). ĠFirstly , ĠLast State R NN Ġis Ġthe Ġclassic ĠR NN Ġmodel , Ġwhere Ġthe Ġlast Ġstate Ġpasses Ġthrough Ġan ĠML P Ġand Ġthen Ġthe ĠLR ĠLayer Ġestimates Ġthe Ġcorresponding Ġprobability . ĠIn Ġcontrast , Ġin Ġthe ĠAvg R NN Ġmodel Ġwe Ġconsider Ġthe Ġaverage Ġvector Ġof Ġall Ġstates Ġthat Ġcome Ġout Ġof Ġthe Ġcells . ĠThe ĠAttention R NN Ġmodel Ġis Ġthe Ġone Ġthat Ġit Ġhas Ġbeen Ġpresented Ġin ĠB IB REF 9 . ĠMoreover , Ġwe Ġintroduce Ġthe ĠMulti Att ention R NN Ġmodel Ġfor Ġthe Ġharassment Ġlanguage Ġdetection , Ġwhich Ġinstead Ġof Ġone Ġattention , Ġit Ġincludes Ġfour Ġattent ions , Ġone Ġfor Ġeach Ġcategory . </s> We Ġhave Ġevaluated Ġour Ġmodels Ġconsidering Ġthe ĠF 1 ĠScore , Ġwhich Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall . ĠWe Ġhave Ġrun Ġten Ġtimes Ġthe Ġexperiment Ġfor Ġeach Ġmodel Ġand Ġconsidered Ġthe Ġaverage ĠF 1 ĠScore . ĠThe Ġresults Ġare Ġmentioned Ġin ĠTable ĠT AB REF 11 . ĠConsidering ĠF 1 ĠMacro Ġthe Ġmodels Ġthat Ġinclude Ġthe Ġmulti - att ention Ġmechanism Ġoutper form Ġthe Ġothers Ġand Ġparticularly Ġthe Ġone Ġwith Ġthe ĠProject ed ĠLayer Ġhas Ġthe Ġhighest Ġperformance . ĠIn Ġthree Ġout Ġof Ġfour Ġpairs Ġof Ġmodels , Ġthe Ġones Ġwith Ġthe ĠProject ed ĠLayer Ġachieved Ġbetter Ġperformance , Ġso Ġin Ġmost Ġcases Ġthe Ġaddition Ġof Ġthe ĠProject ed ĠLayer Ġhad Ġa Ġsignificant Ġenhancement . </s> Conclusion Ġ- ĠFuture Ġwork </s> We Ġpresent Ġan Ġattention - based Ġapproach Ġfor Ġthe Ġdetection Ġof Ġharassment Ġlanguage Ġin Ġtweets Ġand Ġthe Ġdetection Ġof Ġdifferent Ġtypes Ġof Ġharassment Ġas Ġwell . ĠOur Ġapproach Ġis Ġbased Ġon Ġthe ĠRec urrent ĠNeural ĠNetworks Ġand Ġparticularly Ġwe Ġare Ġusing Ġa Ġdeep , Ġclass i ï ¬ ģ cation Ġspecific Ġattention Ġmechanism . ĠMoreover , Ġwe Ġpresent Ġa Ġcomparison Ġbetween Ġdifferent Ġvariations Ġof Ġthis Ġattention - based Ġapproach Ġand Ġa Ġfew Ġbaseline Ġmethods . ĠAccording Ġto Ġthe Ġresults Ġof Ġour Ġexperiments Ġand Ġconsidering Ġthe ĠF 1 ĠScore , Ġthe Ġmulti - att ention Ġmethod Ġhaving Ġa Ġprojected Ġlayer , Ġachieved Ġthe Ġhighest Ġperformance . ĠAlso , Ġwe Ġtackled Ġthe Ġproblem Ġof Ġthe Ġimbalance Ġbetween Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġperforming Ġthe Ġtechnique Ġof Ġback - translation . </s> In Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġperform Ġmore Ġexperiments Ġwith Ġthis Ġdataset Ġapplying Ġdifferent Ġmodels Ġusing ĠB ERT ĠB IB REF 21 . ĠAlso , Ġwe Ġwould Ġlike Ġto Ġapply Ġthe Ġmodels Ġpresented Ġin Ġthis Ġwork , Ġin Ġother Ġdatasets Ġabout Ġhate Ġspeech Ġin Ġsocial Ġmedia .
Introduction In the era of social media and networking platforms, Twitter has been doomed for abuse and harassment toward users specifically women. In fact, online harassment becomes very common in Twitter and there have been a lot of critics that Twitter has become the platform for many racists, misogynists and hate groups which can express themselves openly. Online harassment is usually in the form of verbal or graphical formats and is considered harassment, because it is neither invited nor has the consent of the receipt. Monitoring the contents including sexism and sexual harassment in traditional media is easier than monitoring on the online social media platforms like Twitter. The main reason is because of the large amount of user generated content in these media. So, the research about the automated detection of content containing sexual harassment is an important issue and could be the basis for removing that content or flagging it for human evaluation. The basic goal of this automatic classification is that it will significantly improve the process of detecting these types of hate speech on social media by reducing the time and effort required by human beings. Previous studies have been focused on collecting data about sexism and racism in very broad terms or have proposed two categories of sexism as benevolent or hostile sexism BIBREF0, which undermines other types of online harassment. However, there is no much study focusing on different types online harassment alone attracting natural language processing techniques. In this paper we present our work, which is a part of the SociaL Media And Harassment Competition of the ECML PKDD 2019 Conference. The topic of the competition is the classification of different types of harassment and it is divided in two tasks. The first one is the classification of the tweets in harassment and non-harassment categories, while the second one is the classification in specific harassment categories like indirect harassment, physical and sexual harassment as well. We are using the dataset of the competition, which includes text from tweets having the aforementioned categories. Our approach is based on the Recurrent Neural Networks and particularly we are using a deep, classiﬁcation specific attention mechanism. Moreover, we present a comparison between different variations of this attention-based approach like multi-attention and single attention models. The next Section includes a short description of the related work, while the third Section includes a description of the dataset. After that, we describe our methodology. Finally, we describe the experiments and we present the results and our conclusion. Related Work Waseem et al. BIBREF1 were the first who collected hateful tweets and categorized them into being sexist, racist or neither. However, they did not provide specific definitions for each category. Jha and Mamidi BIBREF0 focused on just sexist tweets and proposed two categories of hostile and benevolent sexism. However, these categories were general as they ignored other types of sexism happening in social media. Sharifirad S. and Matwin S. BIBREF2 proposed complimentary categories of sexist language inspired from social science work. They categorized the sexist tweets into the categories of indirect harassment, information threat, sexual harassment and physical harassment. In the next year the same authors proposed BIBREF3 a more comprehensive categorization of online harassment in social media e.g. twitter into the following categories, indirect harassment, information threat, sexual harassment, physical harassment and not sexist. For the detection of hate speech in social media like twitter, many approaches have been proposed. Jha and Mamidi BIBREF0 tested support vector machine, bi-directional RNN encoder-decoder and FastText on hostile and benevolent sexist tweets. They also used SentiWordNet and subjectivity lexicon on the extracted phrases to show the polarity of the tweets. Sharifirad et al. BIBREF4 trained, tested and evaluated different classification methods on the SemEval2018 dataset and chose the classifier with the highest accuracy for testing on each category of sexist tweets to know the mental state and the affectual state of the user who tweets in each category. To overcome the limitations of small data sets on sexist speech detection, Sharifirad S. et al. BIBREF5 have applied text augmentation and text generation with certain success. They have generated new tweets by replacing words in order to increase the size of our training set. Moreover, in the presented text augmentation approach, the number of tweets in each class remains the same, but their words are augmented with words extracted from their ConceptNet relations and their description extracted from Wikidata. Zhang et al. BIBREF6 combined convolutional and gated recurrent networks to detect hate speech in tweets. Others have proposed different methods, which are not based on deep learning. Burnap and Williams BIBREF7 used Support Vector Machines, Random Forests and a meta-classifier to distinguish between hateful and non-hateful messages. A survey of recent research in the field is presented in BIBREF8. For the problem of the hate speech detection a few approaches have been proposed that are based on the Attention mechanism. Pavlopoulos et al. BIBREF9 have proposed a novel, classification-specific attention mechanism that improves the performance of the RNN further for the detection of abusive content in the web. Xie et al. BIBREF10 for emotion intensity prediction, which is a similar problem to ours, have proposed a novel attention mechanism for CNN model that associates attention-based weights for every convolution window. Park and Fung BIBREF11 transformed the classiﬁcation into a 2-step problem, where abusive text ﬁrst is distinguished from the non-abusive, and then the class of abuse (Sexism or Racism) is determined. However, while the first part of the two step classiﬁcation performs quite well, it falls short in detecting the particular class the abusive text belongs to. Pitsilis et al. BIBREF12 have proposed a detection scheme that is an ensemble of RNN classiﬁers, which incorporates various features associated with user related information, such as the users’ tendency towards racism or sexism Dataset description The dataset from Twitter that we are using in our work, consists of a train set, a validation set and a test set. It was published for the "First workshop on categorizing different types of online harassment languages in social media". The whole dataset is divided into two categories, which are harassment and non-harassment tweets. Moreover, considering the type of the harassment, the tweets are divided into three sub-categories which are indirect harassment, sexual and physical harassment. We can see in Table TABREF1 the class distribution of our dataset. One important issue here is that the categories of indirect and physical harassment seem to be more rare in the train set than in the validation and test sets. To tackle this issue, as we describe in the next section, we are performing data augmentation techniques. However, the dataset is imbalanced and this has a significant impact in our results. Proposed methodology ::: Data augmentation As described before one crucial issue that we are trying to tackle in this work is that the given dataset is imbalanced. Particularly, there are only a few instances from indirect and physical harassment categories respectively in the train set, while there are much more in the validation and test sets for these categories. To tackle this issue we applying a back-translation method BIBREF13, where we translate indirect and physical harassment tweets of the train set from english to german, french and greek. After that, we translate them back to english in order to achieve data augmentation. These "noisy" data that have been translated back, increase the number of indirect and physical harassment tweets and boost significantly the performance of our models. Another way to enrich our models is the use of pre-trained word embeddings from 2B Twitter data BIBREF14 having 27B tokens, for the initialization of the embedding layer. Proposed methodology ::: Text processing Before training our models we are processing the given tweets using a tweet pre-processor. The scope here is the cleaning and tokenization of the dataset. Proposed methodology ::: RNN Model and Attention Mechanism We are presenting an attention-based approach for the problem of the harassment detection in tweets. In this section, we describe the basic approach of our work. We are using RNN models because of their ability to deal with sequence information. The RNN model is a chain of GRU cells BIBREF15 that transforms the tokens $w_{1}, w_{2},..., w_{k}$ of each tweet to the hidden states $h_{1}, h_{2},..., h_{k}$, followed by an LR Layer that uses $h_{k}$ to classify the tweet as harassment or non-harassment (similarly for the other categories). Given the vocabulary V and a matrix E $\in $ $R^{d \times \vert V \vert }$ containing d-dimensional word embeddings, an initial $h_{0}$ and a tweet $w = <w_{1},.., w_{k}>$, the RNN computes $h_{1}, h_{2},..., h_{k}$, with $h_{t} \in R^{m}$, as follows: where $h^{^{\prime }}_{t} \in R^{m}$ is the proposed hidden state at position t, obtained using the word embedding $x_{t}$ of token $w_{t}$ and the previous hidden state $h_{t-1}$, $\odot $ represents the element-wise multiplication, $r_{t} \in R^{m}$ is the reset gate, $z_{t} \in R^{m}$ is the update gate, $\sigma $ is the sigmoid function. Also $W_{h}, W_{z}, W_{r} \in R^{m \times d}$ and $U_{h}, U_{z}, U_{r} \in R^{m \times m}$, $b_{h}, b_{z}, b_{r} \in R^{m}$. After the computation of state $h_{k}$ the LR Layer estimates the probability that tweet w should be considered as harassment, with $W_{p} \in R^{1 \times m}, b_{p} \in R$: We would like to add an attention mechanism similar to the one presented in BIBREF9, so that the LR Layer will consider the weighted sum $h_{sum}$ of all the hidden states instead of $h_{k}$: $h_{sum} = \sum _{t=1}^{k} \alpha _{t}h_{t}$ $P_{attentionRNN} = \sigma (W_{p}h_{sum} + b_{p})$ Alternatively, we could pass $h_{sum}$ through an MLP with k layers and then the LR layer will estimate the corresponding probability. More formally, $P_{attentionRNN} = \sigma (W_{p}h_{*} + b_{p})$ where $h_{*}$ is the state that comes out from the MLP. The weights $\alpha _{t}$ are produced by an attention mechanism presented in BIBREF9 (see Fig. FIGREF7), which is an MLP with l layers. This attention mechanism differs from most previous ones BIBREF16, BIBREF17, because it is used in a classification setting, where there is no previously generated output sub-sequence to drive the attention. It assigns larger weights $\alpha _{t}$ to hidden states $h_{t}$ corresponding to positions, where there is more evidence that the tweet should be harassment (or any other specific type of harassment) or not. In our work we are using four attention mechanisms instead of one that is presented in BIBREF9. Particularly, we are using one attention mechanism per category. Another element that differentiates our approach from Pavlopoulos et al. BIBREF9 is that we are using a projection layer for the word embeddings (see Fig. FIGREF2). In the next subsection we describe the Model Architecture of our approach. Proposed methodology ::: Model Architecture The Embedding Layer is initialized using pre-trained word embeddings of dimension 200 from Twitter data that have been described in a previous sub-section. After the Embedding Layer, we are applying a Spatial Dropout Layer, which drops a certain percentage of dimensions from each word vector in the training sample. The role of Dropout is to improve generalization performance by preventing activations from becoming strongly correlated BIBREF18. Spatial Dropout, which has been proposed in BIBREF19, is an alternative way to use dropout with convolutional neural networks as it is able to dropout entire feature maps from the convolutional layer which are then not used during pooling. After that, the word embeddings are passing through a one-layer MLP, which has tanh as activation function and 128 hidden units, in order to project them in the vector space of our problem considering that they have been pre-trained using text that has a different subject. In the next step the embeddings are fed in a unidirectional GRU having 1 Stacked Layer and size 128. We prefer GRU than LSTM, because it is more efficient computationally. Also the basic advantage of LSTM which is the ability to keep in memory large text documents, does not hold here, because tweets supposed to be not too large text documents. The output states of the GRU are passing through four self-attentions like the one described above BIBREF9, because we are using one attention per category (see Fig. FIGREF7). Finally, a one-layer MLP having 128 nodes and ReLU as activation function computes the final score for each category. At this final stage we have avoided using a softmax function to decide the harassment type considering that the tweet is a harassment, otherwise we had to train our models taking into account only the harassment tweets and this might have been a problem as the dataset is not large enough. Experiments ::: Training Models In this subsection we are giving the details of the training process of our models. Moreover, we are describing the different models that we compare in our experiments. Batch size which pertains to the amount of training samples to consider at a time for updating our network weights, is set to 32, because our dataset is not large and small batches might help to generalize better. Also, we set other hyperparameters as: epochs = 20, patience = 10. As early stopping criterion we choose the average AUC, because our dataset is imbalanced. The training process is based on the optimization of the loss function mentioned below and it is carried out with the Adam optimizer BIBREF20, which is known for yielding quicker convergence. We set the learning rate equal to 0.001: $L = \frac{1}{2}BCE(harassment) + \frac{1}{2}(\frac{1}{5}BCE(sexualH) + \frac{2}{5}BCE(indirectH)+\frac{2}{5}BCE(physicalH))$ where BCE is the binary cross-entropy loss function, $BCE = -\frac{1}{n}\sum _{i=1}^{n}[y_{i}log(y^{^{\prime }}_{i}) + (1 - y_{i})log(1 - y^{^{\prime }}_{i}))]$ $i$ denotes the $i$th training sample, $y$ is the binary representation of true harassment label, and $y^{^{\prime }}$ is the predicted probability. In the loss function we have applied equal weight to both tasks. However, in the second task (type of harassment classification) we have applied higher weight in the categories that it is harder to predict due to the problem of the class imbalance between the training, validation and test sets respectively. Experiments ::: Evaluation and Results Each model produces four scores and each score is the probability that a tweet includes harassment language, indirect, physical and sexual harassment language respectively. For any tweet, we first check the score of the harassment language and if it is less than a specified threshold, then the harassment label is zero, so the other three labels are zero as well. If it is greater than or equal to that threshold, then the harassment label is one and the type of harassment is the one among these three having that has the greatest score (highest probability). We set this threshold equal to 0.33. We compare eight different models in our experiments. Four of them have a Projected Layer (see Fig. FIGREF2), while the others do not have, and this is the only difference between these two groups of our models. So, we actually include four models in our experiments (having a projected layer or not). Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. Moreover, we introduce the MultiAttentionRNN model for the harassment language detection, which instead of one attention, it includes four attentions, one for each category. We have evaluated our models considering the F1 Score, which is the harmonic mean of precision and recall. We have run ten times the experiment for each model and considered the average F1 Score. The results are mentioned in Table TABREF11. Considering F1 Macro the models that include the multi-attention mechanism outperform the others and particularly the one with the Projected Layer has the highest performance. In three out of four pairs of models, the ones with the Projected Layer achieved better performance, so in most cases the addition of the Projected Layer had a significant enhancement. Conclusion - Future work We present an attention-based approach for the detection of harassment language in tweets and the detection of different types of harassment as well. Our approach is based on the Recurrent Neural Networks and particularly we are using a deep, classiﬁcation specific attention mechanism. Moreover, we present a comparison between different variations of this attention-based approach and a few baseline methods. According to the results of our experiments and considering the F1 Score, the multi-attention method having a projected layer, achieved the highest performance. Also, we tackled the problem of the imbalance between the training, validation and test sets performing the technique of back-translation. In the future, we would like to perform more experiments with this dataset applying different models using BERT BIBREF21. Also, we would like to apply the models presented in this work, in other datasets about hate speech in social media.
Which variation provides the best results on this dataset?
203337c15bd1ee05763c748391d295a1f6415b9b
Which Ġvariation Ġprovides Ġthe Ġbest Ġresults Ġon Ġthis Ġdataset ?
the Ġmodel Ġwith Ġmulti - att ention Ġmechanism Ġand Ġa Ġprojected Ġlayer
<s> Which Ġvariation Ġprovides Ġthe Ġbest Ġresults Ġon Ġthis Ġdataset ? </s> Introduction </s> In Ġthe Ġera Ġof Ġsocial Ġmedia Ġand Ġnetworking Ġplatforms , ĠTwitter Ġhas Ġbeen Ġdoomed Ġfor Ġabuse Ġand Ġharassment Ġtoward Ġusers Ġspecifically Ġwomen . ĠIn Ġfact , Ġonline Ġharassment Ġbecomes Ġvery Ġcommon Ġin ĠTwitter Ġand Ġthere Ġhave Ġbeen Ġa Ġlot Ġof Ġcritics Ġthat ĠTwitter Ġhas Ġbecome Ġthe Ġplatform Ġfor Ġmany Ġracists , Ġmisogyn ists Ġand Ġhate Ġgroups Ġwhich Ġcan Ġexpress Ġthemselves Ġopenly . ĠOnline Ġharassment Ġis Ġusually Ġin Ġthe Ġform Ġof Ġverbal Ġor Ġgraphical Ġformats Ġand Ġis Ġconsidered Ġharassment , Ġbecause Ġit Ġis Ġneither Ġinvited Ġnor Ġhas Ġthe Ġconsent Ġof Ġthe Ġreceipt . ĠMonitoring Ġthe Ġcontents Ġincluding Ġsexism Ġand Ġsexual Ġharassment Ġin Ġtraditional Ġmedia Ġis Ġeasier Ġthan Ġmonitoring Ġon Ġthe Ġonline Ġsocial Ġmedia Ġplatforms Ġlike ĠTwitter . ĠThe Ġmain Ġreason Ġis Ġbecause Ġof Ġthe Ġlarge Ġamount Ġof Ġuser Ġgenerated Ġcontent Ġin Ġthese Ġmedia . ĠSo , Ġthe Ġresearch Ġabout Ġthe Ġautomated Ġdetection Ġof Ġcontent Ġcontaining Ġsexual Ġharassment Ġis Ġan Ġimportant Ġissue Ġand Ġcould Ġbe Ġthe Ġbasis Ġfor Ġremoving Ġthat Ġcontent Ġor Ġflag ging Ġit Ġfor Ġhuman Ġevaluation . ĠThe Ġbasic Ġgoal Ġof Ġthis Ġautomatic Ġclassification Ġis Ġthat Ġit Ġwill Ġsignificantly Ġimprove Ġthe Ġprocess Ġof Ġdetecting Ġthese Ġtypes Ġof Ġhate Ġspeech Ġon Ġsocial Ġmedia Ġby Ġreducing Ġthe Ġtime Ġand Ġeffort Ġrequired Ġby Ġhuman Ġbeings . </s> Previous Ġstudies Ġhave Ġbeen Ġfocused Ġon Ġcollecting Ġdata Ġabout Ġsexism Ġand Ġracism Ġin Ġvery Ġbroad Ġterms Ġor Ġhave Ġproposed Ġtwo Ġcategories Ġof Ġsexism Ġas Ġbenevolent Ġor Ġhostile Ġsexism ĠB IB REF 0 , Ġwhich Ġundermines Ġother Ġtypes Ġof Ġonline Ġharassment . ĠHowever , Ġthere Ġis Ġno Ġmuch Ġstudy Ġfocusing Ġon Ġdifferent Ġtypes Ġonline Ġharassment Ġalone Ġattracting Ġnatural Ġlanguage Ġprocessing Ġtechniques . </s> In Ġthis Ġpaper Ġwe Ġpresent Ġour Ġwork , Ġwhich Ġis Ġa Ġpart Ġof Ġthe ĠSoc ia L ĠMedia ĠAnd ĠHar assment ĠCompetition Ġof Ġthe ĠEC ML ĠPK DD Ġ2019 ĠConference . ĠThe Ġtopic Ġof Ġthe Ġcompetition Ġis Ġthe Ġclassification Ġof Ġdifferent Ġtypes Ġof Ġharassment Ġand Ġit Ġis Ġdivided Ġin Ġtwo Ġtasks . ĠThe Ġfirst Ġone Ġis Ġthe Ġclassification Ġof Ġthe Ġtweets Ġin Ġharassment Ġand Ġnon - har assment Ġcategories , Ġwhile Ġthe Ġsecond Ġone Ġis Ġthe Ġclassification Ġin Ġspecific Ġharassment Ġcategories Ġlike Ġindirect Ġharassment , Ġphysical Ġand Ġsexual Ġharassment Ġas Ġwell . ĠWe Ġare Ġusing Ġthe Ġdataset Ġof Ġthe Ġcompetition , Ġwhich Ġincludes Ġtext Ġfrom Ġtweets Ġhaving Ġthe Ġaforementioned Ġcategories . ĠOur Ġapproach Ġis Ġbased Ġon Ġthe ĠRec urrent ĠNeural ĠNetworks Ġand Ġparticularly Ġwe Ġare Ġusing Ġa Ġdeep , Ġclass i ï ¬ ģ cation Ġspecific Ġattention Ġmechanism . ĠMoreover , Ġwe Ġpresent Ġa Ġcomparison Ġbetween Ġdifferent Ġvariations Ġof Ġthis Ġattention - based Ġapproach Ġlike Ġmulti - att ention Ġand Ġsingle Ġattention Ġmodels . ĠThe Ġnext ĠSection Ġincludes Ġa Ġshort Ġdescription Ġof Ġthe Ġrelated Ġwork , Ġwhile Ġthe Ġthird ĠSection Ġincludes Ġa Ġdescription Ġof Ġthe Ġdataset . ĠAfter Ġthat , Ġwe Ġdescribe Ġour Ġmethodology . ĠFinally , Ġwe Ġdescribe Ġthe Ġexperiments Ġand Ġwe Ġpresent Ġthe Ġresults Ġand Ġour Ġconclusion . </s> Related ĠWork </s> W ase em Ġet Ġal . ĠB IB REF 1 Ġwere Ġthe Ġfirst Ġwho Ġcollected Ġhateful Ġtweets Ġand Ġcategorized Ġthem Ġinto Ġbeing Ġsexist , Ġracist Ġor Ġneither . ĠHowever , Ġthey Ġdid Ġnot Ġprovide Ġspecific Ġdefinitions Ġfor Ġeach Ġcategory . ĠJ ha Ġand ĠMam idi ĠB IB REF 0 Ġfocused Ġon Ġjust Ġsexist Ġtweets Ġand Ġproposed Ġtwo Ġcategories Ġof Ġhostile Ġand Ġbenevolent Ġsexism . ĠHowever , Ġthese Ġcategories Ġwere Ġgeneral Ġas Ġthey Ġignored Ġother Ġtypes Ġof Ġsexism Ġhappening Ġin Ġsocial Ġmedia . ĠSharif ir ad ĠS . Ġand ĠMat win ĠS . ĠB IB REF 2 Ġproposed Ġcomplimentary Ġcategories Ġof Ġsexist Ġlanguage Ġinspired Ġfrom Ġsocial Ġscience Ġwork . ĠThey Ġcategorized Ġthe Ġsexist Ġtweets Ġinto Ġthe Ġcategories Ġof Ġindirect Ġharassment , Ġinformation Ġthreat , Ġsexual Ġharassment Ġand Ġphysical Ġharassment . ĠIn Ġthe Ġnext Ġyear Ġthe Ġsame Ġauthors Ġproposed ĠB IB REF 3 Ġa Ġmore Ġcomprehensive Ġcategor ization Ġof Ġonline Ġharassment Ġin Ġsocial Ġmedia Ġe . g . Ġtwitter Ġinto Ġthe Ġfollowing Ġcategories , Ġindirect Ġharassment , Ġinformation Ġthreat , Ġsexual Ġharassment , Ġphysical Ġharassment Ġand Ġnot Ġsexist . </s> For Ġthe Ġdetection Ġof Ġhate Ġspeech Ġin Ġsocial Ġmedia Ġlike Ġtwitter , Ġmany Ġapproaches Ġhave Ġbeen Ġproposed . ĠJ ha Ġand ĠMam idi ĠB IB REF 0 Ġtested Ġsupport Ġvector Ġmachine , Ġbi - direction al ĠR NN Ġenc oder - dec oder Ġand ĠFast Text Ġon Ġhostile Ġand Ġbenevolent Ġsexist Ġtweets . ĠThey Ġalso Ġused ĠSent i Word Net Ġand Ġsubject ivity Ġlex icon Ġon Ġthe Ġextracted Ġphrases Ġto Ġshow Ġthe Ġpol arity Ġof Ġthe Ġtweets . ĠSharif ir ad Ġet Ġal . ĠB IB REF 4 Ġtrained , Ġtested Ġand Ġevaluated Ġdifferent Ġclassification Ġmethods Ġon Ġthe ĠSem E val 2018 Ġdataset Ġand Ġchose Ġthe Ġclass ifier Ġwith Ġthe Ġhighest Ġaccuracy Ġfor Ġtesting Ġon Ġeach Ġcategory Ġof Ġsexist Ġtweets Ġto Ġknow Ġthe Ġmental Ġstate Ġand Ġthe Ġaffect ual Ġstate Ġof Ġthe Ġuser Ġwho Ġtweets Ġin Ġeach Ġcategory . ĠTo Ġovercome Ġthe Ġlimitations Ġof Ġsmall Ġdata Ġsets Ġon Ġsexist Ġspeech Ġdetection , ĠSharif ir ad ĠS . Ġet Ġal . ĠB IB REF 5 Ġhave Ġapplied Ġtext Ġaug mentation Ġand Ġtext Ġgeneration Ġwith Ġcertain Ġsuccess . ĠThey Ġhave Ġgenerated Ġnew Ġtweets Ġby Ġreplacing Ġwords Ġin Ġorder Ġto Ġincrease Ġthe Ġsize Ġof Ġour Ġtraining Ġset . ĠMoreover , Ġin Ġthe Ġpresented Ġtext Ġaug mentation Ġapproach , Ġthe Ġnumber Ġof Ġtweets Ġin Ġeach Ġclass Ġremains Ġthe Ġsame , Ġbut Ġtheir Ġwords Ġare Ġaugmented Ġwith Ġwords Ġextracted Ġfrom Ġtheir ĠConcept Net Ġrelations Ġand Ġtheir Ġdescription Ġextracted Ġfrom ĠWik id ata . ĠZhang Ġet Ġal . ĠB IB REF 6 Ġcombined Ġconv olution al Ġand Ġg ated Ġrecurrent Ġnetworks Ġto Ġdetect Ġhate Ġspeech Ġin Ġtweets . ĠOthers Ġhave Ġproposed Ġdifferent Ġmethods , Ġwhich Ġare Ġnot Ġbased Ġon Ġdeep Ġlearning . ĠBurn ap Ġand ĠWilliams ĠB IB REF 7 Ġused ĠSupport ĠVector ĠMachines , ĠRandom ĠFore sts Ġand Ġa Ġmeta - class ifier Ġto Ġdistinguish Ġbetween Ġhateful Ġand Ġnon - h ateful Ġmessages . ĠA Ġsurvey Ġof Ġrecent Ġresearch Ġin Ġthe Ġfield Ġis Ġpresented Ġin ĠB IB REF 8 . ĠFor Ġthe Ġproblem Ġof Ġthe Ġhate Ġspeech Ġdetection Ġa Ġfew Ġapproaches Ġhave Ġbeen Ġproposed Ġthat Ġare Ġbased Ġon Ġthe ĠAttention Ġmechanism . ĠPav l opoulos Ġet Ġal . ĠB IB REF 9 Ġhave Ġproposed Ġa Ġnovel , Ġclassification - specific Ġattention Ġmechanism Ġthat Ġimproves Ġthe Ġperformance Ġof Ġthe ĠR NN Ġfurther Ġfor Ġthe Ġdetection Ġof Ġabusive Ġcontent Ġin Ġthe Ġweb . ĠXie Ġet Ġal . ĠB IB REF 10 Ġfor Ġemotion Ġintensity Ġprediction , Ġwhich Ġis Ġa Ġsimilar Ġproblem Ġto Ġours , Ġhave Ġproposed Ġa Ġnovel Ġattention Ġmechanism Ġfor ĠCNN Ġmodel Ġthat Ġassociates Ġattention - based Ġweights Ġfor Ġevery Ġconv olution Ġwindow . ĠPark Ġand ĠF ung ĠB IB REF 11 Ġtransformed Ġthe Ġclass i ï ¬ ģ cation Ġinto Ġa Ġ2 - step Ġproblem , Ġwhere Ġabusive Ġtext Ġï ¬ ģ r st Ġis Ġdistinguished Ġfrom Ġthe Ġnon - ab usive , Ġand Ġthen Ġthe Ġclass Ġof Ġabuse Ġ( Sex ism Ġor ĠRac ism ) Ġis Ġdetermined . ĠHowever , Ġwhile Ġthe Ġfirst Ġpart Ġof Ġthe Ġtwo Ġstep Ġclass i ï ¬ ģ cation Ġperforms Ġquite Ġwell , Ġit Ġfalls Ġshort Ġin Ġdetecting Ġthe Ġparticular Ġclass Ġthe Ġabusive Ġtext Ġbelongs Ġto . ĠP its il is Ġet Ġal . ĠB IB REF 12 Ġhave Ġproposed Ġa Ġdetection Ġscheme Ġthat Ġis Ġan Ġensemble Ġof ĠR NN Ġclass i ï ¬ ģ ers , Ġwhich Ġincorporates Ġvarious Ġfeatures Ġassociated Ġwith Ġuser Ġrelated Ġinformation , Ġsuch Ġas Ġthe Ġusers âĢ Ļ Ġtendency Ġtowards Ġracism Ġor Ġsexism </s> Dat as et Ġdescription </s> The Ġdataset Ġfrom ĠTwitter Ġthat Ġwe Ġare Ġusing Ġin Ġour Ġwork , Ġconsists Ġof Ġa Ġtrain Ġset , Ġa Ġvalidation Ġset Ġand Ġa Ġtest Ġset . ĠIt Ġwas Ġpublished Ġfor Ġthe Ġ" First Ġworkshop Ġon Ġcategor izing Ġdifferent Ġtypes Ġof Ġonline Ġharassment Ġlanguages Ġin Ġsocial Ġmedia ". ĠThe Ġwhole Ġdataset Ġis Ġdivided Ġinto Ġtwo Ġcategories , Ġwhich Ġare Ġharassment Ġand Ġnon - har assment Ġtweets . ĠMoreover , Ġconsidering Ġthe Ġtype Ġof Ġthe Ġharassment , Ġthe Ġtweets Ġare Ġdivided Ġinto Ġthree Ġsub - c ategories Ġwhich Ġare Ġindirect Ġharassment , Ġsexual Ġand Ġphysical Ġharassment . ĠWe Ġcan Ġsee Ġin ĠTable ĠT AB REF 1 Ġthe Ġclass Ġdistribution Ġof Ġour Ġdataset . ĠOne Ġimportant Ġissue Ġhere Ġis Ġthat Ġthe Ġcategories Ġof Ġindirect Ġand Ġphysical Ġharassment Ġseem Ġto Ġbe Ġmore Ġrare Ġin Ġthe Ġtrain Ġset Ġthan Ġin Ġthe Ġvalidation Ġand Ġtest Ġsets . ĠTo Ġtackle Ġthis Ġissue , Ġas Ġwe Ġdescribe Ġin Ġthe Ġnext Ġsection , Ġwe Ġare Ġperforming Ġdata Ġaug mentation Ġtechniques . ĠHowever , Ġthe Ġdataset Ġis Ġim balanced Ġand Ġthis Ġhas Ġa Ġsignificant Ġimpact Ġin Ġour Ġresults . </s> Prop osed Ġmethodology Ġ: :: ĠData Ġaug mentation </s> As Ġdescribed Ġbefore Ġone Ġcrucial Ġissue Ġthat Ġwe Ġare Ġtrying Ġto Ġtackle Ġin Ġthis Ġwork Ġis Ġthat Ġthe Ġgiven Ġdataset Ġis Ġim balanced . ĠParticularly , Ġthere Ġare Ġonly Ġa Ġfew Ġinstances Ġfrom Ġindirect Ġand Ġphysical Ġharassment Ġcategories Ġrespectively Ġin Ġthe Ġtrain Ġset , Ġwhile Ġthere Ġare Ġmuch Ġmore Ġin Ġthe Ġvalidation Ġand Ġtest Ġsets Ġfor Ġthese Ġcategories . ĠTo Ġtackle Ġthis Ġissue Ġwe Ġapplying Ġa Ġback - translation Ġmethod ĠB IB REF 13 , Ġwhere Ġwe Ġtranslate Ġindirect Ġand Ġphysical Ġharassment Ġtweets Ġof Ġthe Ġtrain Ġset Ġfrom Ġenglish Ġto Ġg erman , Ġfrench Ġand Ġg reek . ĠAfter Ġthat , Ġwe Ġtranslate Ġthem Ġback Ġto Ġenglish Ġin Ġorder Ġto Ġachieve Ġdata Ġaug mentation . ĠThese Ġ" no isy " Ġdata Ġthat Ġhave Ġbeen Ġtranslated Ġback , Ġincrease Ġthe Ġnumber Ġof Ġindirect Ġand Ġphysical Ġharassment Ġtweets Ġand Ġboost Ġsignificantly Ġthe Ġperformance Ġof Ġour Ġmodels . </s> Another Ġway Ġto Ġenrich Ġour Ġmodels Ġis Ġthe Ġuse Ġof Ġpre - trained Ġword Ġembed d ings Ġfrom Ġ2 B ĠTwitter Ġdata ĠB IB REF 14 Ġhaving Ġ27 B Ġtokens , Ġfor Ġthe Ġinitialization Ġof Ġthe Ġembed ding Ġlayer . </s> Prop osed Ġmethodology Ġ: :: ĠText Ġprocessing </s> Before Ġtraining Ġour Ġmodels Ġwe Ġare Ġprocessing Ġthe Ġgiven Ġtweets Ġusing Ġa Ġtweet Ġpre - processor . ĠThe Ġscope Ġhere Ġis Ġthe Ġcleaning Ġand Ġtoken ization Ġof Ġthe Ġdataset . </s> Prop osed Ġmethodology Ġ: :: ĠR NN ĠModel Ġand ĠAttention ĠMechan ism </s> We Ġare Ġpresenting Ġan Ġattention - based Ġapproach Ġfor Ġthe Ġproblem Ġof Ġthe Ġharassment Ġdetection Ġin Ġtweets . ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġbasic Ġapproach Ġof Ġour Ġwork . ĠWe Ġare Ġusing ĠR NN Ġmodels Ġbecause Ġof Ġtheir Ġability Ġto Ġdeal Ġwith Ġsequence Ġinformation . ĠThe ĠR NN Ġmodel Ġis Ġa Ġchain Ġof ĠGR U Ġcells ĠB IB REF 15 Ġthat Ġtransforms Ġthe Ġtokens Ġ$ w _{ 1 }, Ġw _{ 2 }, ... , Ġw _{ k } $ Ġof Ġeach Ġtweet Ġto Ġthe Ġhidden Ġstates Ġ$ h _{ 1 }, Ġh _{ 2 }, ... , Ġh _{ k } $, Ġfollowed Ġby Ġan ĠLR ĠLayer Ġthat Ġuses Ġ$ h _{ k } $ Ġto Ġclassify Ġthe Ġtweet Ġas Ġharassment Ġor Ġnon - har assment Ġ( similar ly Ġfor Ġthe Ġother Ġcategories ). ĠGiven Ġthe Ġvocabulary ĠV Ġand Ġa Ġmatrix ĠE Ġ$\ in Ġ$ Ġ$ R ^{ d Ġ\ times Ġ\ vert ĠV Ġ\ vert Ġ} $ Ġcontaining Ġd - dimensional Ġword Ġembed d ings , Ġan Ġinitial Ġ$ h _{ 0 } $ Ġand Ġa Ġtweet Ġ$ w Ġ= Ġ< w _{ 1 }, .. , Ġw _{ k } > $, Ġthe ĠR NN Ġcomp utes Ġ$ h _{ 1 }, Ġh _{ 2 }, ... , Ġh _{ k } $, Ġwith Ġ$ h _{ t } Ġ\ in ĠR ^{ m } $, Ġas Ġfollows : </s> where Ġ$ h ^{ ^ {\ prime Ġ}} _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġproposed Ġhidden Ġstate Ġat Ġposition Ġt , Ġobtained Ġusing Ġthe Ġword Ġembed ding Ġ$ x _{ t } $ Ġof Ġtoken Ġ$ w _{ t } $ Ġand Ġthe Ġprevious Ġhidden Ġstate Ġ$ h _{ t - 1 } $, Ġ$\ od ot Ġ$ Ġrepresents Ġthe Ġelement - wise Ġmultiplication , Ġ$ r _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġreset Ġgate , Ġ$ z _{ t } Ġ\ in ĠR ^{ m } $ Ġis Ġthe Ġupdate Ġgate , Ġ$\ s igma Ġ$ Ġis Ġthe Ġs igm oid Ġfunction . ĠAlso Ġ$ W _{ h }, ĠW _{ z }, ĠW _{ r } Ġ\ in ĠR ^{ m Ġ\ times Ġd } $ Ġand Ġ$ U _{ h }, ĠU _{ z }, ĠU _{ r } Ġ\ in ĠR ^{ m Ġ\ times Ġm } $, Ġ$ b _{ h }, Ġb _{ z }, Ġb _{ r } Ġ\ in ĠR ^{ m } $. ĠAfter Ġthe Ġcomputation Ġof Ġstate Ġ$ h _{ k } $ Ġthe ĠLR ĠLayer Ġestimates Ġthe Ġprobability Ġthat Ġtweet Ġw Ġshould Ġbe Ġconsidered Ġas Ġharassment , Ġwith Ġ$ W _{ p } Ġ\ in ĠR ^{ 1 Ġ\ times Ġm }, Ġb _{ p } Ġ\ in ĠR $ : </s> We Ġwould Ġlike Ġto Ġadd Ġan Ġattention Ġmechanism Ġsimilar Ġto Ġthe Ġone Ġpresented Ġin ĠB IB REF 9 , Ġso Ġthat Ġthe ĠLR ĠLayer Ġwill Ġconsider Ġthe Ġweighted Ġsum Ġ$ h _{ sum } $ Ġof Ġall Ġthe Ġhidden Ġstates Ġinstead Ġof Ġ$ h _{ k } $ : </s> $ h _{ sum } Ġ= Ġ\ sum Ġ_ { t = 1 } ^{ k } Ġ\ alpha Ġ_ { t } h _{ t } $ </s> $ P _{ att ention R NN } Ġ= Ġ\ s igma Ġ( W _{ p } h _{ sum } Ġ+ Ġb _{ p }) $ </s> Alternatively , Ġwe Ġcould Ġpass Ġ$ h _{ sum } $ Ġthrough Ġan ĠML P Ġwith Ġk Ġlayers Ġand Ġthen Ġthe ĠLR Ġlayer Ġwill Ġestimate Ġthe Ġcorresponding Ġprobability . ĠMore Ġformally , </s> $ P _{ att ention R NN } Ġ= Ġ\ s igma Ġ( W _{ p } h _{ * } Ġ+ Ġb _{ p }) $ </s> where Ġ$ h _{ * } $ Ġis Ġthe Ġstate Ġthat Ġcomes Ġout Ġfrom Ġthe ĠML P . ĠThe Ġweights Ġ$\ alpha Ġ_ { t } $ Ġare Ġproduced Ġby Ġan Ġattention Ġmechanism Ġpresented Ġin ĠB IB REF 9 Ġ( see ĠFig . ĠFIG REF 7 ), Ġwhich Ġis Ġan ĠML P Ġwith Ġl Ġlayers . ĠThis Ġattention Ġmechanism Ġdiffers Ġfrom Ġmost Ġprevious Ġones ĠB IB REF 16 , ĠB IB REF 17 , Ġbecause Ġit Ġis Ġused Ġin Ġa Ġclassification Ġsetting , Ġwhere Ġthere Ġis Ġno Ġpreviously Ġgenerated Ġoutput Ġsub - sequence Ġto Ġdrive Ġthe Ġattention . ĠIt Ġassigns Ġlarger Ġweights Ġ$\ alpha Ġ_ { t } $ Ġto Ġhidden Ġstates Ġ$ h _{ t } $ Ġcorresponding Ġto Ġpositions , Ġwhere Ġthere Ġis Ġmore Ġevidence Ġthat Ġthe Ġtweet Ġshould Ġbe Ġharassment Ġ( or Ġany Ġother Ġspecific Ġtype Ġof Ġharassment ) Ġor Ġnot . ĠIn Ġour Ġwork Ġwe Ġare Ġusing Ġfour Ġattention Ġmechanisms Ġinstead Ġof Ġone Ġthat Ġis Ġpresented Ġin ĠB IB REF 9 . ĠParticularly , Ġwe Ġare Ġusing Ġone Ġattention Ġmechanism Ġper Ġcategory . ĠAnother Ġelement Ġthat Ġdifferent iates Ġour Ġapproach Ġfrom ĠPav l opoulos Ġet Ġal . ĠB IB REF 9 Ġis Ġthat Ġwe Ġare Ġusing Ġa Ġprojection Ġlayer Ġfor Ġthe Ġword Ġembed d ings Ġ( see ĠFig . ĠFIG REF 2 ). ĠIn Ġthe Ġnext Ġsubsection Ġwe Ġdescribe Ġthe ĠModel ĠArchitecture Ġof Ġour Ġapproach . </s> Prop osed Ġmethodology Ġ: :: ĠModel ĠArchitecture </s> The ĠEmb edd ing ĠLayer Ġis Ġinitialized Ġusing Ġpre - trained Ġword Ġembed d ings Ġof Ġdimension Ġ200 Ġfrom ĠTwitter Ġdata Ġthat Ġhave Ġbeen Ġdescribed Ġin Ġa Ġprevious Ġsub - section . ĠAfter Ġthe ĠEmb edd ing ĠLayer , Ġwe Ġare Ġapplying Ġa ĠSp atial ĠDrop out ĠLayer , Ġwhich Ġdrops Ġa Ġcertain Ġpercentage Ġof Ġdimensions Ġfrom Ġeach Ġword Ġvector Ġin Ġthe Ġtraining Ġsample . ĠThe Ġrole Ġof ĠDrop out Ġis Ġto Ġimprove Ġgeneral ization Ġperformance Ġby Ġpreventing Ġactiv ations Ġfrom Ġbecoming Ġstrongly Ġcorrelated ĠB IB REF 18 . ĠSp atial ĠDrop out , Ġwhich Ġhas Ġbeen Ġproposed Ġin ĠB IB REF 19 , Ġis Ġan Ġalternative Ġway Ġto Ġuse Ġdrop out Ġwith Ġconv olution al Ġneural Ġnetworks Ġas Ġit Ġis Ġable Ġto Ġdrop out Ġentire Ġfeature Ġmaps Ġfrom Ġthe Ġconv olution al Ġlayer Ġwhich Ġare Ġthen Ġnot Ġused Ġduring Ġpool ing . ĠAfter Ġthat , Ġthe Ġword Ġembed d ings Ġare Ġpassing Ġthrough Ġa Ġone - layer ĠML P , Ġwhich Ġhas Ġtan h Ġas Ġactivation Ġfunction Ġand Ġ128 Ġhidden Ġunits , Ġin Ġorder Ġto Ġproject Ġthem Ġin Ġthe Ġvector Ġspace Ġof Ġour Ġproblem Ġconsidering Ġthat Ġthey Ġhave Ġbeen Ġpre - trained Ġusing Ġtext Ġthat Ġhas Ġa Ġdifferent Ġsubject . ĠIn Ġthe Ġnext Ġstep Ġthe Ġembed d ings Ġare Ġfed Ġin Ġa Ġun id irection al ĠGR U Ġhaving Ġ1 ĠSt acked ĠLayer Ġand Ġsize Ġ128 . ĠWe Ġprefer ĠGR U Ġthan ĠL ST M , Ġbecause Ġit Ġis Ġmore Ġefficient Ġcomput ationally . ĠAlso Ġthe Ġbasic Ġadvantage Ġof ĠL ST M Ġwhich Ġis Ġthe Ġability Ġto Ġkeep Ġin Ġmemory Ġlarge Ġtext Ġdocuments , Ġdoes Ġnot Ġhold Ġhere , Ġbecause Ġtweets Ġsupposed Ġto Ġbe Ġnot Ġtoo Ġlarge Ġtext Ġdocuments . ĠThe Ġoutput Ġstates Ġof Ġthe ĠGR U Ġare Ġpassing Ġthrough Ġfour Ġself - att ent ions Ġlike Ġthe Ġone Ġdescribed Ġabove ĠB IB REF 9 , Ġbecause Ġwe Ġare Ġusing Ġone Ġattention Ġper Ġcategory Ġ( see ĠFig . ĠFIG REF 7 ). ĠFinally , Ġa Ġone - layer ĠML P Ġhaving Ġ128 Ġnodes Ġand ĠRe LU Ġas Ġactivation Ġfunction Ġcomp utes Ġthe Ġfinal Ġscore Ġfor Ġeach Ġcategory . ĠAt Ġthis Ġfinal Ġstage Ġwe Ġhave Ġavoided Ġusing Ġa Ġsoft max Ġfunction Ġto Ġdecide Ġthe Ġharassment Ġtype Ġconsidering Ġthat Ġthe Ġtweet Ġis Ġa Ġharassment , Ġotherwise Ġwe Ġhad Ġto Ġtrain Ġour Ġmodels Ġtaking Ġinto Ġaccount Ġonly Ġthe Ġharassment Ġtweets Ġand Ġthis Ġmight Ġhave Ġbeen Ġa Ġproblem Ġas Ġthe Ġdataset Ġis Ġnot Ġlarge Ġenough . </s> Exper iments Ġ: :: ĠTraining ĠModels </s> In Ġthis Ġsubsection Ġwe Ġare Ġgiving Ġthe Ġdetails Ġof Ġthe Ġtraining Ġprocess Ġof Ġour Ġmodels . ĠMoreover , Ġwe Ġare Ġdescribing Ġthe Ġdifferent Ġmodels Ġthat Ġwe Ġcompare Ġin Ġour Ġexperiments . </s> B atch Ġsize Ġwhich Ġper tains Ġto Ġthe Ġamount Ġof Ġtraining Ġsamples Ġto Ġconsider Ġat Ġa Ġtime Ġfor Ġupdating Ġour Ġnetwork Ġweights , Ġis Ġset Ġto Ġ32 , Ġbecause Ġour Ġdataset Ġis Ġnot Ġlarge Ġand Ġsmall Ġbatches Ġmight Ġhelp Ġto Ġgeneral ize Ġbetter . ĠAlso , Ġwe Ġset Ġother Ġhyper param eters Ġas : Ġepoch s Ġ= Ġ20 , Ġpatience Ġ= Ġ10 . ĠAs Ġearly Ġstopping Ġcriterion Ġwe Ġchoose Ġthe Ġaverage ĠA UC , Ġbecause Ġour Ġdataset Ġis Ġim balanced . </s> The Ġtraining Ġprocess Ġis Ġbased Ġon Ġthe Ġoptimization Ġof Ġthe Ġloss Ġfunction Ġmentioned Ġbelow Ġand Ġit Ġis Ġcarried Ġout Ġwith Ġthe ĠAdam Ġoptim izer ĠB IB REF 20 , Ġwhich Ġis Ġknown Ġfor Ġyielding Ġquicker Ġconvergence . ĠWe Ġset Ġthe Ġlearning Ġrate Ġequal Ġto Ġ0 . 001 : </s> $ L Ġ= Ġ\ frac { 1 }{ 2 } BC E ( har assment ) Ġ+ Ġ\ frac { 1 }{ 2 } (\ frac { 1 }{ 5 } BC E ( sexual H ) Ġ+ Ġ\ frac { 2 }{ 5 } BC E ( ind irect H )+ \ frac { 2 }{ 5 } BC E ( physical H )) $ </s> where ĠBCE Ġis Ġthe Ġbinary Ġcross - ent ropy Ġloss Ġfunction , </s> $ BC E Ġ= Ġ- \ frac { 1 }{ n }\ sum Ġ_ { i = 1 } ^{ n } [ y _{ i } log ( y ^{ ^ {\ prime Ġ}} _{ i }) Ġ+ Ġ( 1 Ġ- Ġy _{ i }) log ( 1 Ġ- Ġy ^{ ^ {\ prime Ġ}} _{ i } )) ] $ </s> $ i $ Ġdenotes Ġthe Ġ$ i $ th Ġtraining Ġsample , Ġ$ y $ Ġis Ġthe Ġbinary Ġrepresentation Ġof Ġtrue Ġharassment Ġlabel , Ġand Ġ$ y ^{ ^ {\ prime Ġ}} $ Ġis Ġthe Ġpredicted Ġprobability . ĠIn Ġthe Ġloss Ġfunction Ġwe Ġhave Ġapplied Ġequal Ġweight Ġto Ġboth Ġtasks . ĠHowever , Ġin Ġthe Ġsecond Ġtask Ġ( type Ġof Ġharassment Ġclassification ) Ġwe Ġhave Ġapplied Ġhigher Ġweight Ġin Ġthe Ġcategories Ġthat Ġit Ġis Ġharder Ġto Ġpredict Ġdue Ġto Ġthe Ġproblem Ġof Ġthe Ġclass Ġimbalance Ġbetween Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġrespectively . </s> Exper iments Ġ: :: ĠEvaluation Ġand ĠResults </s> Each Ġmodel Ġproduces Ġfour Ġscores Ġand Ġeach Ġscore Ġis Ġthe Ġprobability Ġthat Ġa Ġtweet Ġincludes Ġharassment Ġlanguage , Ġindirect , Ġphysical Ġand Ġsexual Ġharassment Ġlanguage Ġrespectively . ĠFor Ġany Ġtweet , Ġwe Ġfirst Ġcheck Ġthe Ġscore Ġof Ġthe Ġharassment Ġlanguage Ġand Ġif Ġit Ġis Ġless Ġthan Ġa Ġspecified Ġthreshold , Ġthen Ġthe Ġharassment Ġlabel Ġis Ġzero , Ġso Ġthe Ġother Ġthree Ġlabels Ġare Ġzero Ġas Ġwell . ĠIf Ġit Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġthat Ġthreshold , Ġthen Ġthe Ġharassment Ġlabel Ġis Ġone Ġand Ġthe Ġtype Ġof Ġharassment Ġis Ġthe Ġone Ġamong Ġthese Ġthree Ġhaving Ġthat Ġhas Ġthe Ġgreatest Ġscore Ġ( highest Ġprobability ). ĠWe Ġset Ġthis Ġthreshold Ġequal Ġto Ġ0 . 33 . </s> We Ġcompare Ġeight Ġdifferent Ġmodels Ġin Ġour Ġexperiments . ĠFour Ġof Ġthem Ġhave Ġa ĠProject ed ĠLayer Ġ( see ĠFig . ĠFIG REF 2 ), Ġwhile Ġthe Ġothers Ġdo Ġnot Ġhave , Ġand Ġthis Ġis Ġthe Ġonly Ġdifference Ġbetween Ġthese Ġtwo Ġgroups Ġof Ġour Ġmodels . ĠSo , Ġwe Ġactually Ġinclude Ġfour Ġmodels Ġin Ġour Ġexperiments Ġ( having Ġa Ġprojected Ġlayer Ġor Ġnot ). ĠFirstly , ĠLast State R NN Ġis Ġthe Ġclassic ĠR NN Ġmodel , Ġwhere Ġthe Ġlast Ġstate Ġpasses Ġthrough Ġan ĠML P Ġand Ġthen Ġthe ĠLR ĠLayer Ġestimates Ġthe Ġcorresponding Ġprobability . ĠIn Ġcontrast , Ġin Ġthe ĠAvg R NN Ġmodel Ġwe Ġconsider Ġthe Ġaverage Ġvector Ġof Ġall Ġstates Ġthat Ġcome Ġout Ġof Ġthe Ġcells . ĠThe ĠAttention R NN Ġmodel Ġis Ġthe Ġone Ġthat Ġit Ġhas Ġbeen Ġpresented Ġin ĠB IB REF 9 . ĠMoreover , Ġwe Ġintroduce Ġthe ĠMulti Att ention R NN Ġmodel Ġfor Ġthe Ġharassment Ġlanguage Ġdetection , Ġwhich Ġinstead Ġof Ġone Ġattention , Ġit Ġincludes Ġfour Ġattent ions , Ġone Ġfor Ġeach Ġcategory . </s> We Ġhave Ġevaluated Ġour Ġmodels Ġconsidering Ġthe ĠF 1 ĠScore , Ġwhich Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall . ĠWe Ġhave Ġrun Ġten Ġtimes Ġthe Ġexperiment Ġfor Ġeach Ġmodel Ġand Ġconsidered Ġthe Ġaverage ĠF 1 ĠScore . ĠThe Ġresults Ġare Ġmentioned Ġin ĠTable ĠT AB REF 11 . ĠConsidering ĠF 1 ĠMacro Ġthe Ġmodels Ġthat Ġinclude Ġthe Ġmulti - att ention Ġmechanism Ġoutper form Ġthe Ġothers Ġand Ġparticularly Ġthe Ġone Ġwith Ġthe ĠProject ed ĠLayer Ġhas Ġthe Ġhighest Ġperformance . ĠIn Ġthree Ġout Ġof Ġfour Ġpairs Ġof Ġmodels , Ġthe Ġones Ġwith Ġthe ĠProject ed ĠLayer Ġachieved Ġbetter Ġperformance , Ġso Ġin Ġmost Ġcases Ġthe Ġaddition Ġof Ġthe ĠProject ed ĠLayer Ġhad Ġa Ġsignificant Ġenhancement . </s> Conclusion Ġ- ĠFuture Ġwork </s> We Ġpresent Ġan Ġattention - based Ġapproach Ġfor Ġthe Ġdetection Ġof Ġharassment Ġlanguage Ġin Ġtweets Ġand Ġthe Ġdetection Ġof Ġdifferent Ġtypes Ġof Ġharassment Ġas Ġwell . ĠOur Ġapproach Ġis Ġbased Ġon Ġthe ĠRec urrent ĠNeural ĠNetworks Ġand Ġparticularly Ġwe Ġare Ġusing Ġa Ġdeep , Ġclass i ï ¬ ģ cation Ġspecific Ġattention Ġmechanism . ĠMoreover , Ġwe Ġpresent Ġa Ġcomparison Ġbetween Ġdifferent Ġvariations Ġof Ġthis Ġattention - based Ġapproach Ġand Ġa Ġfew Ġbaseline Ġmethods . ĠAccording Ġto Ġthe Ġresults Ġof Ġour Ġexperiments Ġand Ġconsidering Ġthe ĠF 1 ĠScore , Ġthe Ġmulti - att ention Ġmethod Ġhaving Ġa Ġprojected Ġlayer , Ġachieved Ġthe Ġhighest Ġperformance . ĠAlso , Ġwe Ġtackled Ġthe Ġproblem Ġof Ġthe Ġimbalance Ġbetween Ġthe Ġtraining , Ġvalidation Ġand Ġtest Ġsets Ġperforming Ġthe Ġtechnique Ġof Ġback - translation . </s> In Ġthe Ġfuture , Ġwe Ġwould Ġlike Ġto Ġperform Ġmore Ġexperiments Ġwith Ġthis Ġdataset Ġapplying Ġdifferent Ġmodels Ġusing ĠB ERT ĠB IB REF 21 . ĠAlso , Ġwe Ġwould Ġlike Ġto Ġapply Ġthe Ġmodels Ġpresented Ġin Ġthis Ġwork , Ġin Ġother Ġdatasets Ġabout Ġhate Ġspeech Ġin Ġsocial Ġmedia .