Naïve Bayes (NB), Logistic Regression (LR), Support Vector Machine (SVM), Random Forests (RF), Gradient Boosted Trees (GBT),  Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN)
EXTRACTIVE
Naïve Bayes (NB), Logistic Regression (LR), Support Vector Machine (SVM), Random Forests (RF), Gradient Boosted Trees (GBT), CNN, RNN
EXTRACTIVE
We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection. In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values. We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams. Each classifier is implemented with the following specifications: Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1 Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset. In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models. A pre-trained GloVe BIBREF17 representation is used for word-level features.
We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection. In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values. We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams. Each classifier is implemented with the following specifications: Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1 Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset. In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models. A pre-trained GloVe BIBREF17 representation is used for word-level features. CNN: We adopt Kim's kim2014convolutional implementation as the baseline. The word-level CNN models have 3 convolutional filters of different sizes [1,2,3] with ReLU activation, and a max-pooling layer. For the character-level CNN, we use 6 convolutional filters of various sizes [3,4,5,6,7,8], then add max-pooling layers followed by 1 fully-connected layer with a dimension of 1024. Park and Fung park2017one proposed a HybridCNN model which outperformed both word-level and character-level CNNs in abusive language detection. In order to evaluate the HybridCNN for this dataset, we concatenate the output of max-pooled layers from word-level and character-level CNN, and feed this vector to a fully-connected layer in order to predict the output. All three CNN models (word-level, character-level, and hybrid) use cross entropy with softmax as their loss function and Adam BIBREF18 as the optimizer. RNN: We use bidirectional RNN BIBREF19 as the baseline, implementing a GRU BIBREF20 cell for each recurrent unit. From extensive parameter-search experiments, we chose 1 encoding layer with 50 dimensional hidden states and an input dropout probability of 0.3. The RNN models use cross entropy with sigmoid as their loss function and Adam as the optimizer. For a possible improvement, we apply a self-matching attention mechanism on RNN baseline models BIBREF21 so that they may better understand the data by retrieving text sequences twice. We also investigate a recently introduced method, Latent Topic Clustering (LTC) BIBREF22 . The LTC method extracts latent topic information from the hidden states of RNN, and uses it for additional information in classifying the text data.
1808.10245
Introduction </s> Ab usive Ġlanguage Ġrefers Ġto Ġany Ġtype Ġof Ġinsult , Ġvulgar ity , Ġor Ġprof anity Ġthat Ġdeb ases Ġthe Ġtarget ; Ġit Ġalso Ġcan Ġbe Ġanything Ġthat Ġcauses Ġaggrav ation ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠAb usive Ġlanguage Ġis Ġoften Ġref ram ed Ġas , Ġbut Ġnot Ġlimited Ġto , Ġoffensive Ġlanguage ĠB IB REF 2 Ġ, Ġcyber bull ying ĠB IB REF 3 Ġ, Ġother ing Ġlanguage ĠB IB REF 4 Ġ, Ġand Ġhate Ġspeech ĠB IB REF 5 Ġ. </s> Recently , Ġan Ġincreasing Ġnumber Ġof Ġusers Ġhave Ġbeen Ġsubjected Ġto Ġharassment , Ġor Ġhave Ġwitnessed Ġoffensive Ġbehaviors Ġonline ĠB IB REF 6 Ġ. ĠMajor Ġsocial Ġmedia Ġcompanies Ġ( i . e . ĠFacebook , ĠTwitter ) Ġhave Ġutilized Ġmultiple Ġresources âĢĶ art ificial Ġintelligence , Ġhuman Ġreviewers , Ġuser Ġreporting Ġprocesses , Ġetc .âĢĶ in Ġeffort Ġto Ġcensor Ġoffensive Ġlanguage , Ġyet Ġit Ġseems Ġnearly Ġimpossible Ġto Ġsuccessfully Ġresolve Ġthe Ġissue ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> The Ġmajor Ġreason Ġof Ġthe Ġfailure Ġin Ġabusive Ġlanguage Ġdetection Ġcomes Ġfrom Ġits Ġsubject ivity Ġand Ġcontext - dependent Ġcharacteristics ĠB IB REF 9 Ġ. ĠFor Ġinstance , Ġa Ġmessage Ġcan Ġbe Ġregarded Ġas Ġharmless Ġon Ġits Ġown , Ġbut Ġwhen Ġtaking Ġprevious Ġthreads Ġinto Ġaccount Ġit Ġmay Ġbe Ġseen Ġas Ġabusive , Ġand Ġvice Ġversa . ĠThis Ġaspect Ġmakes Ġdetecting Ġabusive Ġlanguage Ġextremely Ġlabor ious Ġeven Ġfor Ġhuman Ġannot ators ; Ġtherefore Ġit Ġis Ġdifficult Ġto Ġbuild Ġa Ġlarge Ġand Ġreliable Ġdataset ĠB IB REF 10 Ġ. </s> Previously , Ġdatasets Ġopenly Ġavailable Ġin Ġabusive Ġlanguage Ġdetection Ġresearch Ġon ĠTwitter Ġranged Ġfrom Ġ10 K Ġto Ġ35 K Ġin Ġsize ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ. ĠThis Ġquantity Ġis Ġnot Ġsufficient Ġto Ġtrain Ġthe Ġsignificant Ġnumber Ġof Ġparameters Ġin Ġdeep Ġlearning Ġmodels . ĠDue Ġto Ġthis Ġreason , Ġthese Ġdatasets Ġhave Ġbeen Ġmainly Ġstudied Ġby Ġtraditional Ġmachine Ġlearning Ġmethods . ĠMost Ġrecently , ĠF ount a Ġet Ġal . Ġf ount a 2018 large Ġintroduced ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter , Ġa Ġdataset Ġcontaining Ġ100 K Ġtweets Ġwith Ġcross - valid ated Ġlabels . ĠAlthough Ġthis Ġcorpus Ġhas Ġgreat Ġpotential Ġin Ġtraining Ġdeep Ġmodels Ġwith Ġits Ġsignificant Ġsize , Ġthere Ġare Ġno Ġbaseline Ġreports Ġto Ġdate . </s> This Ġpaper Ġinvestigates Ġthe Ġefficacy Ġof Ġdifferent Ġlearning Ġmodels Ġin Ġdetecting Ġabusive Ġlanguage . ĠWe Ġcompare Ġaccuracy Ġusing Ġthe Ġmost Ġfrequently Ġstudied Ġmachine Ġlearning Ġclass ifiers Ġas Ġwell Ġas Ġrecent Ġneural Ġnetwork Ġmodels . ĠRel iable Ġbaseline Ġresults Ġare Ġpresented Ġwith Ġthe Ġfirst Ġcomparative Ġstudy Ġon Ġthis Ġdataset . ĠAdditionally , Ġwe Ġdemonstrate Ġthe Ġeffect Ġof Ġdifferent Ġfeatures Ġand Ġvariants , Ġand Ġdescribe Ġthe Ġpossibility Ġfor Ġfurther Ġimprovements Ġwith Ġthe Ġuse Ġof Ġensemble Ġmodels . </s> Related ĠWork </s> The Ġresearch Ġcommunity Ġintroduced Ġvarious Ġapproaches Ġon Ġabusive Ġlanguage Ġdetection . ĠRaz avi Ġet Ġal . Ġr az avi 2010 offensive Ġapplied ĠNa Ã¯ve ĠBay es , Ġand ĠWarner Ġand ĠH irsch berg Ġwar ner 2012 det ect ing Ġused ĠSupport ĠVector ĠMachine Ġ( S VM ), Ġboth Ġwith Ġword - level Ġfeatures Ġto Ġclassify Ġoffensive Ġlanguage . ĠXiang Ġet Ġal . Ġx iang 2012 det ect ing Ġgenerated Ġtopic Ġdistributions Ġwith ĠLat ent ĠDir ich let ĠAll ocation ĠB IB REF 12 Ġ, Ġalso Ġusing Ġword - level Ġfeatures Ġin Ġorder Ġto Ġclassify Ġoffensive Ġtweets . </s> More Ġrecently , Ġdistributed Ġword Ġrepresentations Ġand Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġwidely Ġapplied Ġfor Ġabusive Ġlanguage Ġdetection . ĠDj ur ic Ġet Ġal . Ġdj ur ic 2015 hate Ġused Ġthe ĠContinuous ĠBag ĠOf ĠWords Ġmodel Ġwith Ġparagraph 2 vec Ġalgorithm ĠB IB REF 13 Ġto Ġmore Ġaccurately Ġdetect Ġhate Ġspeech Ġthan Ġthat Ġof Ġthe Ġplain ĠBag ĠOf ĠWords Ġmodels . ĠBad j ati ya Ġet Ġal . Ġbad j ati ya 2017 deep Ġimplemented ĠGrad ient ĠBoost ed ĠDecision ĠTrees Ġclass ifiers Ġusing Ġword Ġrepresentations Ġtrained Ġby Ġdeep Ġlearning Ġmodels . ĠOther Ġresearchers Ġhave Ġinvestigated Ġcharacter - level Ġrepresentations Ġand Ġtheir Ġeffectiveness Ġcompared Ġto Ġword - level Ġrepresentations ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. </s> As Ġtraditional Ġmachine Ġlearning Ġmethods Ġhave Ġrelied Ġon Ġfeature Ġengineering , Ġ( i . e . Ġn - gram s , ĠPOS Ġtags , Ġuser Ġinformation ) ĠB IB REF 1 Ġ, Ġresearchers Ġhave Ġproposed Ġneural - based Ġmodels Ġwith Ġthe Ġadvent Ġof Ġlarger Ġdatasets . ĠConv olution al ĠNeural ĠNetworks Ġand ĠRec urrent ĠNeural ĠNetworks Ġhave Ġbeen Ġapplied Ġto Ġdetect Ġabusive Ġlanguage , Ġand Ġthey Ġhave Ġoutper formed Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġsuch Ġas ĠLog istic ĠReg ression Ġand ĠS VM ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠHowever , Ġthere Ġare Ġno Ġstudies Ġinvestigating Ġthe Ġefficiency Ġof Ġneural Ġmodels Ġwith Ġlarge - scale Ġdatasets Ġover Ġ100 K . </s> Method ology </s> This Ġsection Ġillustrates Ġour Ġimplementations Ġon Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġand Ġneural Ġnetwork Ġbased Ġmodels Ġin Ġdetail . ĠFurthermore , Ġwe Ġdescribe Ġadditional Ġfeatures Ġand Ġvariant Ġmodels Ġinvestigated . </s> Traditional ĠMachine ĠLearning ĠModels </s> We Ġimplement Ġfive Ġfeature Ġengineering Ġbased Ġmachine Ġlearning Ġclass ifiers Ġthat Ġare Ġmost Ġoften Ġused Ġfor Ġabusive Ġlanguage Ġdetection . ĠIn Ġdata Ġpre processing , Ġtext Ġsequences Ġare Ġconverted Ġinto ĠBag ĠOf ĠWords Ġ( B OW ) Ġrepresentations , Ġand Ġnormalized Ġwith ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency Ġ( TF - ID F ) Ġvalues . ĠWe Ġexperiment Ġwith Ġword - level Ġfeatures Ġusing Ġn - gram s Ġranging Ġfrom Ġ1 Ġto Ġ3 , Ġand Ġcharacter - level Ġfeatures Ġfrom Ġ3 Ġto Ġ8 - gram s . ĠEach Ġclass ifier Ġis Ġimplemented Ġwith Ġthe Ġfollowing Ġspecifications : </s> Na Ã¯ve ĠBay es Ġ( NB ): ĠMult in omial ĠNB Ġwith Ġadditive Ġsmoot hing Ġconstant Ġ1 </s> Log istic ĠReg ression Ġ( LR ): ĠLinear ĠLR Ġwith ĠL 2 Ġregular ization Ġconstant Ġ1 Ġand Ġlimited - memory ĠBF GS Ġoptimization </s> Support ĠVector ĠMachine Ġ( S VM ): ĠLinear ĠS VM Ġwith ĠL 2 Ġregular ization Ġconstant Ġ1 Ġand Ġlog istic Ġloss Ġfunction </s> Random ĠFore sts Ġ( RF ): ĠA ver aging Ġprob abil istic Ġpredictions Ġof Ġ10 Ġrandomized Ġdecision Ġtrees </s> Grad ient ĠBoost ed ĠTrees Ġ( GBT ): ĠTree Ġboosting Ġwith Ġlearning Ġrate Ġ1 Ġand Ġlog istic Ġloss Ġfunction </s> Ne ural ĠNetwork Ġbased ĠModels </s> Along Ġwith Ġtraditional Ġmachine Ġlearning Ġapproaches , Ġwe Ġinvestigate Ġneural Ġnetwork Ġbased Ġmodels Ġto Ġevaluate Ġtheir Ġefficacy Ġwithin Ġa Ġlarger Ġdataset . ĠIn Ġparticular , Ġwe Ġexplore ĠConv olution al ĠNeural ĠNetworks Ġ( CNN ), ĠRec urrent ĠNeural ĠNetworks Ġ( R NN ), Ġand Ġtheir Ġvariant Ġmodels . ĠA Ġpre - trained ĠGl o Ve ĠB IB REF 17 Ġrepresentation Ġis Ġused Ġfor Ġword - level Ġfeatures . </s> CNN : ĠWe Ġadopt ĠKim 's Ġk im 2014 conv olution al Ġimplementation Ġas Ġthe Ġbaseline . ĠThe Ġword - level ĠCNN Ġmodels Ġhave Ġ3 Ġconv olution al Ġfilters Ġof Ġdifferent Ġsizes Ġ[ 1 , 2 , 3 ] Ġwith ĠRe LU Ġactivation , Ġand Ġa Ġmax - pool ing Ġlayer . ĠFor Ġthe Ġcharacter - level ĠCNN , Ġwe Ġuse Ġ6 Ġconv olution al Ġfilters Ġof Ġvarious Ġsizes Ġ[ 3 , 4 , 5 , 6 , 7 , 8 ], Ġthen Ġadd Ġmax - pool ing Ġlayers Ġfollowed Ġby Ġ1 Ġfully - connected Ġlayer Ġwith Ġa Ġdimension Ġof Ġ1024 . </s> Park Ġand ĠF ung Ġpark 2017 one Ġproposed Ġa ĠHybrid CNN Ġmodel Ġwhich Ġoutper formed Ġboth Ġword - level Ġand Ġcharacter - level ĠCNN s Ġin Ġabusive Ġlanguage Ġdetection . ĠIn Ġorder Ġto Ġevaluate Ġthe ĠHybrid CNN Ġfor Ġthis Ġdataset , Ġwe Ġconc aten ate Ġthe Ġoutput Ġof Ġmax - pool ed Ġlayers Ġfrom Ġword - level Ġand Ġcharacter - level ĠCNN , Ġand Ġfeed Ġthis Ġvector Ġto Ġa Ġfully - connected Ġlayer Ġin Ġorder Ġto Ġpredict Ġthe Ġoutput . </s> All Ġthree ĠCNN Ġmodels Ġ( word - level , Ġcharacter - level , Ġand Ġhybrid ) Ġuse Ġcross Ġentropy Ġwith Ġsoft max Ġas Ġtheir Ġloss Ġfunction Ġand ĠAdam ĠB IB REF 18 Ġas Ġthe Ġoptim izer . </s> R NN : ĠWe Ġuse Ġbid irection al ĠR NN ĠB IB REF 19 Ġas Ġthe Ġbaseline , Ġimplementing Ġa ĠGR U ĠB IB REF 20 Ġcell Ġfor Ġeach Ġrecurrent Ġunit . ĠFrom Ġextensive Ġparameter - search Ġexperiments , Ġwe Ġchose Ġ1 Ġencoding Ġlayer Ġwith Ġ50 Ġdimensional Ġhidden Ġstates Ġand Ġan Ġinput Ġdrop out Ġprobability Ġof Ġ0 . 3 . ĠThe ĠR NN Ġmodels Ġuse Ġcross Ġentropy Ġwith Ġs igm oid Ġas Ġtheir Ġloss Ġfunction Ġand ĠAdam Ġas Ġthe Ġoptim izer . </s> For Ġa Ġpossible Ġimprovement , Ġwe Ġapply Ġa Ġself - match ing Ġattention Ġmechanism Ġon ĠR NN Ġbaseline Ġmodels ĠB IB REF 21 Ġso Ġthat Ġthey Ġmay Ġbetter Ġunderstand Ġthe Ġdata Ġby Ġretrieving Ġtext Ġsequences Ġtwice . ĠWe Ġalso Ġinvestigate Ġa Ġrecently Ġintroduced Ġmethod , ĠLat ent ĠTopic ĠCl ust ering Ġ( L TC ) ĠB IB REF 22 Ġ. ĠThe ĠL TC Ġmethod Ġextracts Ġlatent Ġtopic Ġinformation Ġfrom Ġthe Ġhidden Ġstates Ġof ĠR NN , Ġand Ġuses Ġit Ġfor Ġadditional Ġinformation Ġin Ġclass ifying Ġthe Ġtext Ġdata . </s> Feature ĠExtension </s> While Ġmanually Ġanalyzing Ġthe Ġraw Ġdataset , Ġwe Ġnoticed Ġthat Ġlooking Ġat Ġthe Ġtweet Ġone Ġhas Ġreplied Ġto Ġor Ġhas Ġquoted , Ġprovides Ġsignificant Ġcontextual Ġinformation . ĠWe Ġcall Ġthese , ĠâĢ ľ context Ġtweets ". ĠAs Ġhumans Ġcan Ġbetter Ġunderstand Ġa Ġtweet Ġwith Ġthe Ġreference Ġof Ġits Ġcontext , Ġour Ġassumption Ġis Ġthat Ġcomputers Ġalso Ġbenefit Ġfrom Ġtaking Ġcontext Ġtweets Ġinto Ġaccount Ġin Ġdetecting Ġabusive Ġlanguage . </s> As Ġshown Ġin Ġthe Ġexamples Ġbelow , Ġ( 2 ) Ġis Ġlabeled Ġabusive Ġdue Ġto Ġthe Ġuse Ġof Ġvulgar Ġlanguage . ĠHowever , Ġthe Ġintention Ġof Ġthe Ġuser Ġcan Ġbe Ġbetter Ġunderstood Ġwith Ġits Ġcontext Ġtweet Ġ( 1 ). </s> ( 1 ) ĠI Ġhate Ġwhen ĠI 'm Ġsitting Ġin Ġfront Ġof Ġthe Ġbus Ġand Ġsomebody Ġwith Ġa Ġwheelchair Ġget Ġon . </s> IN LINE FORM 0 Ġ( 2 ) ĠI Ġhate Ġit Ġwhen ĠI 'm Ġtrying Ġto Ġboard Ġa Ġbus Ġand Ġthere 's Ġalready Ġan Ġas ** ole Ġon Ġit . </s> Similarly , Ġcontext Ġtweet Ġ( 3 ) Ġis Ġimportant Ġin Ġunderstanding Ġthe Ġabusive Ġtweet Ġ( 4 ), Ġespecially Ġin Ġidentifying Ġthe Ġtarget Ġof Ġthe Ġmalice . </s> ( 3 ) ĠSurvivors Ġof Ġ# Syria ĠGas ĠAttack ĠRec ount Ġ` a ĠCruel ĠScene '. </s> IN LINE FORM 0 Ġ( 4 ) ĠWho Ġthe ĠHELL Ġis ĠâĢ ľ LI KE " ĠIN G Ġthis Ġpost ? ĠSick Ġpeople .... </s> Hu ang Ġet Ġal . Ġh u ang 2016 mod eling Ġused Ġseveral Ġattributes Ġof Ġcontext Ġtweets Ġfor Ġsentiment Ġanalysis Ġin Ġorder Ġto Ġimprove Ġthe Ġbaseline ĠL ST M Ġmodel . ĠHowever , Ġtheir Ġapproach Ġwas Ġlimited Ġbecause Ġthe Ġmeta - information Ġthey Ġfocused Ġon âĢĶ author Ġinformation , Ġconversation Ġtype , Ġuse Ġof Ġthe Ġsame Ġhasht ags Ġor Ġem oj is âĢĶ are Ġall Ġhighly Ġdependent Ġon Ġdata . </s> In Ġorder Ġto Ġavoid Ġdata Ġdependency , Ġtext Ġsequences Ġof Ġcontext Ġtweets Ġare Ġdirectly Ġused Ġas Ġan Ġadditional Ġfeature Ġof Ġneural Ġnetwork Ġmodels . ĠWe Ġuse Ġthe Ġsame Ġbaseline Ġmodel Ġto Ġconvert Ġcontext Ġtweets Ġto Ġvectors , Ġthen Ġconc aten ate Ġthese Ġvectors Ġwith Ġoutputs Ġof Ġtheir Ġcorresponding Ġlabeled Ġtweets . ĠMore Ġspecifically , Ġwe Ġconc aten ate Ġmax - pool ed Ġlayers Ġof Ġcontext Ġand Ġlabeled Ġtweets Ġfor Ġthe ĠCNN Ġbaseline Ġmodel . ĠAs Ġfor ĠR NN , Ġthe Ġlast Ġhidden Ġstates Ġof Ġcontext Ġand Ġlabeled Ġtweets Ġare Ġconc aten ated . </s> Dat as et </s> H ate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter ĠB IB REF 10 Ġclass ifies Ġtweets Ġinto Ġ4 Ġlabels , ĠâĢ ľ normal ", ĠâĢ ľ sp am ", ĠâĢ ľ h ateful " Ġand ĠâĢ ľ ab usive ". ĠWe Ġwere Ġonly Ġable Ġto Ġcrawl Ġ70 , 9 04 Ġtweets Ġout Ġof Ġ99 , 996 Ġtweet ĠIDs , Ġmainly Ġbecause Ġthe Ġtweet Ġwas Ġdeleted Ġor Ġthe Ġuser Ġaccount Ġhad Ġbeen Ġsuspended . ĠTable Ġshows Ġthe Ġdistribution Ġof Ġlabels Ġof Ġthe Ġcrawled Ġdata . </s> Data ĠPre processing </s> In Ġthe Ġdata Ġpre processing Ġsteps , Ġuser ĠIDs , ĠURLs , Ġand Ġfrequently Ġused Ġem oj is Ġare Ġreplaced Ġas Ġspecial Ġtokens . ĠSince Ġhasht ags Ġtend Ġto Ġhave Ġa Ġhigh Ġcorrelation Ġwith Ġthe Ġcontent Ġof Ġthe Ġtweet ĠB IB REF 23 Ġ, Ġwe Ġuse Ġa Ġsegment ation Ġlibrary ĠB IB REF 24 Ġfor Ġhasht ags Ġto Ġextract Ġmore Ġinformation . </s> For Ġcharacter - level Ġrepresentations , Ġwe Ġapply Ġthe Ġmethod ĠZhang Ġet Ġal . Ġz hang 2015 character Ġproposed . ĠTwe ets Ġare Ġtransformed Ġinto Ġone - hot Ġencoded Ġvectors Ġusing Ġ70 Ġcharacter Ġdimensions âĢĶ 26 Ġlower - c ased Ġal ph ab ets , Ġ10 Ġdigits , Ġand Ġ34 Ġspecial Ġcharacters Ġincluding Ġwhites pace . </s> Training Ġand ĠEvaluation </s> In Ġtraining Ġthe Ġfeature Ġengineering Ġbased Ġmachine Ġlearning Ġclass ifiers , Ġwe Ġtrunc ate Ġvector Ġrepresentations Ġaccording Ġto Ġthe ĠTF - ID F Ġvalues Ġ( the Ġtop Ġ14 , 000 Ġand Ġ53 , 000 Ġfor Ġword - level Ġand Ġcharacter - level Ġrepresentations , Ġrespectively ) Ġto Ġavoid Ġover fitting . ĠFor Ġneural Ġnetwork Ġmodels , Ġwords Ġthat Ġappear Ġonly Ġonce Ġare Ġreplaced Ġas Ġunknown Ġtokens . </s> Since Ġthe Ġdataset Ġused Ġis Ġnot Ġsplit Ġinto Ġtrain , Ġdevelopment , Ġand Ġtest Ġsets , Ġwe Ġperform Ġ10 - fold Ġcross Ġvalidation , Ġobtaining Ġthe Ġaverage Ġof Ġ5 Ġtries ; Ġwe Ġdivide Ġthe Ġdataset Ġrandomly Ġby Ġa Ġratio Ġof Ġ85 : 5 : 10 , Ġrespectively . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġoverall Ġperformance , Ġwe Ġcalculate Ġthe Ġweighted Ġaverage Ġof Ġprecision , Ġrecall , Ġand ĠF 1 Ġscores Ġof Ġall Ġfour Ġlabels , ĠâĢ ľ normal âĢ Ŀ , ĠâĢ ľ sp am âĢ Ŀ , ĠâĢ ľ h ateful âĢ Ŀ , Ġand ĠâĢ ľ ab usive âĢ Ŀ . </s> E mp ir ical ĠResults </s> As Ġshown Ġin ĠTable Ġ, Ġneural Ġnetwork Ġmodels Ġare Ġmore Ġaccurate Ġthan Ġfeature Ġengineering Ġbased Ġmodels Ġ( i . e . ĠNB , ĠS VM , Ġetc .) Ġexcept Ġfor Ġthe ĠLR Ġmodel âĢĶ the Ġbest ĠLR Ġmodel Ġhas Ġthe Ġsame ĠF 1 Ġscore Ġas Ġthe Ġbest ĠCNN Ġmodel . </s> Among Ġtraditional Ġmachine Ġlearning Ġmodels , Ġthe Ġmost Ġaccurate Ġin Ġclass ifying Ġabusive Ġlanguage Ġis Ġthe ĠLR Ġmodel Ġfollowed Ġby Ġensemble Ġmodels Ġsuch Ġas ĠGB T Ġand ĠRF . ĠCharacter - level Ġrepresentations Ġimprove ĠF 1 Ġscores Ġof ĠS VM Ġand ĠRF Ġclass ifiers , Ġbut Ġthey Ġhave Ġno Ġpositive Ġeffect Ġon Ġother Ġmodels . </s> For Ġneural Ġnetwork Ġmodels , ĠR NN Ġwith ĠL TC Ġmodules Ġhave Ġthe Ġhighest Ġaccuracy Ġscore , Ġbut Ġthere Ġare Ġno Ġsignificant Ġimprovements Ġfrom Ġits Ġbaseline Ġmodel Ġand Ġits Ġattention - added Ġmodel . ĠSimilarly , ĠHybrid CNN Ġdoes Ġnot Ġimprove Ġthe Ġbaseline ĠCNN Ġmodel . ĠFor Ġboth ĠCNN Ġand ĠR NN Ġmodels , Ġcharacter - level Ġfeatures Ġsignificantly Ġdecrease Ġthe Ġaccuracy Ġof Ġclassification . </s> The Ġuse Ġof Ġcontext Ġtweets Ġgenerally Ġhave Ġlittle Ġeffect Ġon Ġbaseline Ġmodels , Ġhowever Ġthey Ġnoticeably Ġimprove Ġthe Ġscores Ġof Ġseveral Ġmetrics . ĠFor Ġinstance , ĠCNN Ġwith Ġcontext Ġtweets Ġscore Ġthe Ġhighest Ġrecall Ġand ĠF 1 Ġfor ĠâĢ ľ h ateful " Ġlabels , Ġand ĠR NN Ġmodels Ġwith Ġcontext Ġtweets Ġhave Ġthe Ġhighest Ġrecall Ġfor ĠâĢ ľ ab usive " Ġtweets . </s> Discussion Ġand ĠConclusion </s> While Ġcharacter - level Ġfeatures Ġare Ġknown Ġto Ġimprove Ġthe Ġaccuracy Ġof Ġneural Ġnetwork Ġmodels ĠB IB REF 16 Ġ, Ġthey Ġreduce Ġclassification Ġaccuracy Ġfor ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter . ĠWe Ġconclude Ġthis Ġis Ġbecause Ġof Ġthe Ġlack Ġof Ġlabeled Ġdata Ġas Ġwell Ġas Ġthe Ġsignificant Ġimbalance Ġamong Ġthe Ġdifferent Ġlabels . ĠUnlike Ġneural Ġnetwork Ġmodels , Ġcharacter - level Ġfeatures Ġin Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġhave Ġpositive Ġresults Ġbecause Ġwe Ġhave Ġtrained Ġthe Ġmodels Ġonly Ġwith Ġthe Ġmost Ġsignificant Ġcharacter Ġelements Ġusing ĠTF - ID F Ġvalues . </s> Vari ants Ġof Ġneural Ġnetwork Ġmodels Ġalso Ġsuffer Ġfrom Ġdata Ġins uff iciency . ĠHowever , Ġthese Ġmodels Ġshow Ġpositive Ġperformances Ġon ĠâĢ ľ sp am " Ġ( 14 %) Ġand ĠâĢ ľ h ateful " Ġ( 4 %) Ġtweets âĢĶ the Ġlower Ġdistributed Ġlabels . ĠThe Ġhighest ĠF 1 Ġscore Ġfor ĠâĢ ľ sp am " Ġis Ġfrom Ġthe ĠR NN - L TC Ġmodel Ġ( 0 . 551 ), Ġand Ġthe Ġhighest Ġfor ĠâĢ ľ h ateful " Ġis ĠCNN Ġwith Ġcontext Ġtweets Ġ( 0 . 309 ). ĠSince Ġeach Ġvariant Ġmodel Ġexcel s Ġin Ġdifferent Ġmetrics , Ġwe Ġexpect Ġto Ġsee Ġadditional Ġimprovements Ġwith Ġthe Ġuse Ġof Ġensemble Ġmodels Ġof Ġthese Ġvariants Ġin Ġfuture Ġworks . </s> In Ġthis Ġpaper , Ġwe Ġreport Ġthe Ġbaseline Ġaccuracy Ġof Ġdifferent Ġlearning Ġmodels Ġas Ġwell Ġas Ġtheir Ġvariants Ġon Ġthe Ġrecently Ġintroduced Ġdataset , ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter . ĠExperimental Ġresults Ġshow Ġthat Ġbid irection al ĠGR U Ġnetworks Ġwith ĠL TC Ġprovide Ġthe Ġmost Ġaccurate Ġresults Ġin Ġdetecting Ġabusive Ġlanguage . ĠAdditionally , Ġwe Ġpresent Ġthe Ġpossibility Ġof Ġusing Ġensemble Ġmodels Ġof Ġvariant Ġmodels Ġand Ġfeatures Ġfor Ġfurther Ġimprovements . </s> Acknowled gments </s> K . ĠJung Ġis Ġwith Ġthe ĠDepartment Ġof ĠElectrical Ġand ĠComputer ĠEngineering , ĠAS RI , ĠSeoul ĠNational ĠUniversity , ĠSeoul , ĠKorea . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠResearch ĠFoundation Ġof ĠKorea Ġ( NR F ) Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( No . Ġ2016 M 3 C 4 A 795 26 32 ), Ġthe ĠTechnology ĠInnovation ĠProgram Ġ( 100 73 144 ) Ġfunded Ġby Ġthe ĠMinistry Ġof ĠTrade , ĠIndustry Ġ& ĠEnergy Ġ( M OT IE , ĠKorea ). </s> We Ġwould Ġalso Ġlike Ġto Ġthank ĠYong ke un ĠH wang Ġand ĠJi ĠHo ĠPark Ġfor Ġhelpful Ġdiscussions Ġand Ġtheir Ġvaluable Ġinsights .
Introduction Abusive language refers to any type of insult, vulgarity, or profanity that debases the target; it also can be anything that causes aggravation BIBREF0 , BIBREF1 . Abusive language is often reframed as, but not limited to, offensive language BIBREF2 , cyberbullying BIBREF3 , othering language BIBREF4 , and hate speech BIBREF5 . Recently, an increasing number of users have been subjected to harassment, or have witnessed offensive behaviors online BIBREF6 . Major social media companies (i.e. Facebook, Twitter) have utilized multiple resources—artificial intelligence, human reviewers, user reporting processes, etc.—in effort to censor offensive language, yet it seems nearly impossible to successfully resolve the issue BIBREF7 , BIBREF8 . The major reason of the failure in abusive language detection comes from its subjectivity and context-dependent characteristics BIBREF9 . For instance, a message can be regarded as harmless on its own, but when taking previous threads into account it may be seen as abusive, and vice versa. This aspect makes detecting abusive language extremely laborious even for human annotators; therefore it is difficult to build a large and reliable dataset BIBREF10 . Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11 . This quantity is not sufficient to train the significant number of parameters in deep learning models. Due to this reason, these datasets have been mainly studied by traditional machine learning methods. Most recently, Founta et al. founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels. Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date. This paper investigates the efficacy of different learning models in detecting abusive language. We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models. Reliable baseline results are presented with the first comparative study on this dataset. Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models. Related Work The research community introduced various approaches on abusive language detection. Razavi et al. razavi2010offensive applied Naïve Bayes, and Warner and Hirschberg warner2012detecting used Support Vector Machine (SVM), both with word-level features to classify offensive language. Xiang et al. xiang2012detecting generated topic distributions with Latent Dirichlet Allocation BIBREF12 , also using word-level features in order to classify offensive tweets. More recently, distributed word representations and neural network models have been widely applied for abusive language detection. Djuric et al. djuric2015hate used the Continuous Bag Of Words model with paragraph2vec algorithm BIBREF13 to more accurately detect hate speech than that of the plain Bag Of Words models. Badjatiya et al. badjatiya2017deep implemented Gradient Boosted Decision Trees classifiers using word representations trained by deep learning models. Other researchers have investigated character-level representations and their effectiveness compared to word-level representations BIBREF14 , BIBREF15 . As traditional machine learning methods have relied on feature engineering, (i.e. n-grams, POS tags, user information) BIBREF1 , researchers have proposed neural-based models with the advent of larger datasets. Convolutional Neural Networks and Recurrent Neural Networks have been applied to detect abusive language, and they have outperformed traditional machine learning classifiers such as Logistic Regression and SVM BIBREF15 , BIBREF16 . However, there are no studies investigating the efficiency of neural models with large-scale datasets over 100K. Methodology This section illustrates our implementations on traditional machine learning classifiers and neural network based models in detail. Furthermore, we describe additional features and variant models investigated. Traditional Machine Learning Models We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection. In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values. We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams. Each classifier is implemented with the following specifications: Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1 Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function Neural Network based Models Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset. In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models. A pre-trained GloVe BIBREF17 representation is used for word-level features. CNN: We adopt Kim's kim2014convolutional implementation as the baseline. The word-level CNN models have 3 convolutional filters of different sizes [1,2,3] with ReLU activation, and a max-pooling layer. For the character-level CNN, we use 6 convolutional filters of various sizes [3,4,5,6,7,8], then add max-pooling layers followed by 1 fully-connected layer with a dimension of 1024. Park and Fung park2017one proposed a HybridCNN model which outperformed both word-level and character-level CNNs in abusive language detection. In order to evaluate the HybridCNN for this dataset, we concatenate the output of max-pooled layers from word-level and character-level CNN, and feed this vector to a fully-connected layer in order to predict the output. All three CNN models (word-level, character-level, and hybrid) use cross entropy with softmax as their loss function and Adam BIBREF18 as the optimizer. RNN: We use bidirectional RNN BIBREF19 as the baseline, implementing a GRU BIBREF20 cell for each recurrent unit. From extensive parameter-search experiments, we chose 1 encoding layer with 50 dimensional hidden states and an input dropout probability of 0.3. The RNN models use cross entropy with sigmoid as their loss function and Adam as the optimizer. For a possible improvement, we apply a self-matching attention mechanism on RNN baseline models BIBREF21 so that they may better understand the data by retrieving text sequences twice. We also investigate a recently introduced method, Latent Topic Clustering (LTC) BIBREF22 . The LTC method extracts latent topic information from the hidden states of RNN, and uses it for additional information in classifying the text data. Feature Extension While manually analyzing the raw dataset, we noticed that looking at the tweet one has replied to or has quoted, provides significant contextual information. We call these, “context tweets". As humans can better understand a tweet with the reference of its context, our assumption is that computers also benefit from taking context tweets into account in detecting abusive language. As shown in the examples below, (2) is labeled abusive due to the use of vulgar language. However, the intention of the user can be better understood with its context tweet (1). (1) I hate when I'm sitting in front of the bus and somebody with a wheelchair get on. INLINEFORM0 (2) I hate it when I'm trying to board a bus and there's already an as**ole on it. Similarly, context tweet (3) is important in understanding the abusive tweet (4), especially in identifying the target of the malice. (3) Survivors of #Syria Gas Attack Recount `a Cruel Scene'. INLINEFORM0 (4) Who the HELL is “LIKE" ING this post? Sick people.... Huang et al. huang2016modeling used several attributes of context tweets for sentiment analysis in order to improve the baseline LSTM model. However, their approach was limited because the meta-information they focused on—author information, conversation type, use of the same hashtags or emojis—are all highly dependent on data. In order to avoid data dependency, text sequences of context tweets are directly used as an additional feature of neural network models. We use the same baseline model to convert context tweets to vectors, then concatenate these vectors with outputs of their corresponding labeled tweets. More specifically, we concatenate max-pooled layers of context and labeled tweets for the CNN baseline model. As for RNN, the last hidden states of context and labeled tweets are concatenated. Dataset Hate and Abusive Speech on Twitter BIBREF10 classifies tweets into 4 labels, “normal", “spam", “hateful" and “abusive". We were only able to crawl 70,904 tweets out of 99,996 tweet IDs, mainly because the tweet was deleted or the user account had been suspended. Table shows the distribution of labels of the crawled data. Data Preprocessing In the data preprocessing steps, user IDs, URLs, and frequently used emojis are replaced as special tokens. Since hashtags tend to have a high correlation with the content of the tweet BIBREF23 , we use a segmentation library BIBREF24 for hashtags to extract more information. For character-level representations, we apply the method Zhang et al. zhang2015character proposed. Tweets are transformed into one-hot encoded vectors using 70 character dimensions—26 lower-cased alphabets, 10 digits, and 34 special characters including whitespace. Training and Evaluation In training the feature engineering based machine learning classifiers, we truncate vector representations according to the TF-IDF values (the top 14,000 and 53,000 for word-level and character-level representations, respectively) to avoid overfitting. For neural network models, words that appear only once are replaced as unknown tokens. Since the dataset used is not split into train, development, and test sets, we perform 10-fold cross validation, obtaining the average of 5 tries; we divide the dataset randomly by a ratio of 85:5:10, respectively. In order to evaluate the overall performance, we calculate the weighted average of precision, recall, and F1 scores of all four labels, “normal”, “spam”, “hateful”, and “abusive”. Empirical Results As shown in Table , neural network models are more accurate than feature engineering based models (i.e. NB, SVM, etc.) except for the LR model—the best LR model has the same F1 score as the best CNN model. Among traditional machine learning models, the most accurate in classifying abusive language is the LR model followed by ensemble models such as GBT and RF. Character-level representations improve F1 scores of SVM and RF classifiers, but they have no positive effect on other models. For neural network models, RNN with LTC modules have the highest accuracy score, but there are no significant improvements from its baseline model and its attention-added model. Similarly, HybridCNN does not improve the baseline CNN model. For both CNN and RNN models, character-level features significantly decrease the accuracy of classification. The use of context tweets generally have little effect on baseline models, however they noticeably improve the scores of several metrics. For instance, CNN with context tweets score the highest recall and F1 for “hateful" labels, and RNN models with context tweets have the highest recall for “abusive" tweets. Discussion and Conclusion While character-level features are known to improve the accuracy of neural network models BIBREF16 , they reduce classification accuracy for Hate and Abusive Speech on Twitter. We conclude this is because of the lack of labeled data as well as the significant imbalance among the different labels. Unlike neural network models, character-level features in traditional machine learning classifiers have positive results because we have trained the models only with the most significant character elements using TF-IDF values. Variants of neural network models also suffer from data insufficiency. However, these models show positive performances on “spam" (14%) and “hateful" (4%) tweets—the lower distributed labels. The highest F1 score for “spam" is from the RNN-LTC model (0.551), and the highest for “hateful" is CNN with context tweets (0.309). Since each variant model excels in different metrics, we expect to see additional improvements with the use of ensemble models of these variants in future works. In this paper, we report the baseline accuracy of different learning models as well as their variants on the recently introduced dataset, Hate and Abusive Speech on Twitter. Experimental results show that bidirectional GRU networks with LTC provide the most accurate results in detecting abusive language. Additionally, we present the possibility of using ensemble models of variant models and features for further improvements. Acknowledgments K. Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea. This work was supported by the National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (No. 2016M3C4A7952632), the Technology Innovation Program (10073144) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea). We would also like to thank Yongkeun Hwang and Ji Ho Park for helpful discussions and their valuable insights.
What learning models are used on the dataset?
48088a842f7a433d3290eb45eb0d4c6ab1d8f13c
What Ġlearning Ġmodels Ġare Ġused Ġon Ġthe Ġdataset ?
Na Ã¯ve ĠBay es Ġ( NB ), ĠLog istic ĠReg ression Ġ( LR ), ĠSupport ĠVector ĠMachine Ġ( S VM ), ĠRandom ĠFore sts Ġ( RF ), ĠGrad ient ĠBoost ed ĠTrees Ġ( GBT ), Ġ ĠConv olution al ĠNeural ĠNetworks Ġ( CNN ), ĠRec urrent ĠNeural ĠNetworks Ġ( R NN )
<s> What Ġlearning Ġmodels Ġare Ġused Ġon Ġthe Ġdataset ? </s> Introduction </s> Ab usive Ġlanguage Ġrefers Ġto Ġany Ġtype Ġof Ġinsult , Ġvulgar ity , Ġor Ġprof anity Ġthat Ġdeb ases Ġthe Ġtarget ; Ġit Ġalso Ġcan Ġbe Ġanything Ġthat Ġcauses Ġaggrav ation ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠAb usive Ġlanguage Ġis Ġoften Ġref ram ed Ġas , Ġbut Ġnot Ġlimited Ġto , Ġoffensive Ġlanguage ĠB IB REF 2 Ġ, Ġcyber bull ying ĠB IB REF 3 Ġ, Ġother ing Ġlanguage ĠB IB REF 4 Ġ, Ġand Ġhate Ġspeech ĠB IB REF 5 Ġ. </s> Recently , Ġan Ġincreasing Ġnumber Ġof Ġusers Ġhave Ġbeen Ġsubjected Ġto Ġharassment , Ġor Ġhave Ġwitnessed Ġoffensive Ġbehaviors Ġonline ĠB IB REF 6 Ġ. ĠMajor Ġsocial Ġmedia Ġcompanies Ġ( i . e . ĠFacebook , ĠTwitter ) Ġhave Ġutilized Ġmultiple Ġresources âĢĶ art ificial Ġintelligence , Ġhuman Ġreviewers , Ġuser Ġreporting Ġprocesses , Ġetc .âĢĶ in Ġeffort Ġto Ġcensor Ġoffensive Ġlanguage , Ġyet Ġit Ġseems Ġnearly Ġimpossible Ġto Ġsuccessfully Ġresolve Ġthe Ġissue ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> The Ġmajor Ġreason Ġof Ġthe Ġfailure Ġin Ġabusive Ġlanguage Ġdetection Ġcomes Ġfrom Ġits Ġsubject ivity Ġand Ġcontext - dependent Ġcharacteristics ĠB IB REF 9 Ġ. ĠFor Ġinstance , Ġa Ġmessage Ġcan Ġbe Ġregarded Ġas Ġharmless Ġon Ġits Ġown , Ġbut Ġwhen Ġtaking Ġprevious Ġthreads Ġinto Ġaccount Ġit Ġmay Ġbe Ġseen Ġas Ġabusive , Ġand Ġvice Ġversa . ĠThis Ġaspect Ġmakes Ġdetecting Ġabusive Ġlanguage Ġextremely Ġlabor ious Ġeven Ġfor Ġhuman Ġannot ators ; Ġtherefore Ġit Ġis Ġdifficult Ġto Ġbuild Ġa Ġlarge Ġand Ġreliable Ġdataset ĠB IB REF 10 Ġ. </s> Previously , Ġdatasets Ġopenly Ġavailable Ġin Ġabusive Ġlanguage Ġdetection Ġresearch Ġon ĠTwitter Ġranged Ġfrom Ġ10 K Ġto Ġ35 K Ġin Ġsize ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ. ĠThis Ġquantity Ġis Ġnot Ġsufficient Ġto Ġtrain Ġthe Ġsignificant Ġnumber Ġof Ġparameters Ġin Ġdeep Ġlearning Ġmodels . ĠDue Ġto Ġthis Ġreason , Ġthese Ġdatasets Ġhave Ġbeen Ġmainly Ġstudied Ġby Ġtraditional Ġmachine Ġlearning Ġmethods . ĠMost Ġrecently , ĠF ount a Ġet Ġal . Ġf ount a 2018 large Ġintroduced ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter , Ġa Ġdataset Ġcontaining Ġ100 K Ġtweets Ġwith Ġcross - valid ated Ġlabels . ĠAlthough Ġthis Ġcorpus Ġhas Ġgreat Ġpotential Ġin Ġtraining Ġdeep Ġmodels Ġwith Ġits Ġsignificant Ġsize , Ġthere Ġare Ġno Ġbaseline Ġreports Ġto Ġdate . </s> This Ġpaper Ġinvestigates Ġthe Ġefficacy Ġof Ġdifferent Ġlearning Ġmodels Ġin Ġdetecting Ġabusive Ġlanguage . ĠWe Ġcompare Ġaccuracy Ġusing Ġthe Ġmost Ġfrequently Ġstudied Ġmachine Ġlearning Ġclass ifiers Ġas Ġwell Ġas Ġrecent Ġneural Ġnetwork Ġmodels . ĠRel iable Ġbaseline Ġresults Ġare Ġpresented Ġwith Ġthe Ġfirst Ġcomparative Ġstudy Ġon Ġthis Ġdataset . ĠAdditionally , Ġwe Ġdemonstrate Ġthe Ġeffect Ġof Ġdifferent Ġfeatures Ġand Ġvariants , Ġand Ġdescribe Ġthe Ġpossibility Ġfor Ġfurther Ġimprovements Ġwith Ġthe Ġuse Ġof Ġensemble Ġmodels . </s> Related ĠWork </s> The Ġresearch Ġcommunity Ġintroduced Ġvarious Ġapproaches Ġon Ġabusive Ġlanguage Ġdetection . ĠRaz avi Ġet Ġal . Ġr az avi 2010 offensive Ġapplied ĠNa Ã¯ve ĠBay es , Ġand ĠWarner Ġand ĠH irsch berg Ġwar ner 2012 det ect ing Ġused ĠSupport ĠVector ĠMachine Ġ( S VM ), Ġboth Ġwith Ġword - level Ġfeatures Ġto Ġclassify Ġoffensive Ġlanguage . ĠXiang Ġet Ġal . Ġx iang 2012 det ect ing Ġgenerated Ġtopic Ġdistributions Ġwith ĠLat ent ĠDir ich let ĠAll ocation ĠB IB REF 12 Ġ, Ġalso Ġusing Ġword - level Ġfeatures Ġin Ġorder Ġto Ġclassify Ġoffensive Ġtweets . </s> More Ġrecently , Ġdistributed Ġword Ġrepresentations Ġand Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġwidely Ġapplied Ġfor Ġabusive Ġlanguage Ġdetection . ĠDj ur ic Ġet Ġal . Ġdj ur ic 2015 hate Ġused Ġthe ĠContinuous ĠBag ĠOf ĠWords Ġmodel Ġwith Ġparagraph 2 vec Ġalgorithm ĠB IB REF 13 Ġto Ġmore Ġaccurately Ġdetect Ġhate Ġspeech Ġthan Ġthat Ġof Ġthe Ġplain ĠBag ĠOf ĠWords Ġmodels . ĠBad j ati ya Ġet Ġal . Ġbad j ati ya 2017 deep Ġimplemented ĠGrad ient ĠBoost ed ĠDecision ĠTrees Ġclass ifiers Ġusing Ġword Ġrepresentations Ġtrained Ġby Ġdeep Ġlearning Ġmodels . ĠOther Ġresearchers Ġhave Ġinvestigated Ġcharacter - level Ġrepresentations Ġand Ġtheir Ġeffectiveness Ġcompared Ġto Ġword - level Ġrepresentations ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. </s> As Ġtraditional Ġmachine Ġlearning Ġmethods Ġhave Ġrelied Ġon Ġfeature Ġengineering , Ġ( i . e . Ġn - gram s , ĠPOS Ġtags , Ġuser Ġinformation ) ĠB IB REF 1 Ġ, Ġresearchers Ġhave Ġproposed Ġneural - based Ġmodels Ġwith Ġthe Ġadvent Ġof Ġlarger Ġdatasets . ĠConv olution al ĠNeural ĠNetworks Ġand ĠRec urrent ĠNeural ĠNetworks Ġhave Ġbeen Ġapplied Ġto Ġdetect Ġabusive Ġlanguage , Ġand Ġthey Ġhave Ġoutper formed Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġsuch Ġas ĠLog istic ĠReg ression Ġand ĠS VM ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠHowever , Ġthere Ġare Ġno Ġstudies Ġinvestigating Ġthe Ġefficiency Ġof Ġneural Ġmodels Ġwith Ġlarge - scale Ġdatasets Ġover Ġ100 K . </s> Method ology </s> This Ġsection Ġillustrates Ġour Ġimplementations Ġon Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġand Ġneural Ġnetwork Ġbased Ġmodels Ġin Ġdetail . ĠFurthermore , Ġwe Ġdescribe Ġadditional Ġfeatures Ġand Ġvariant Ġmodels Ġinvestigated . </s> Traditional ĠMachine ĠLearning ĠModels </s> We Ġimplement Ġfive Ġfeature Ġengineering Ġbased Ġmachine Ġlearning Ġclass ifiers Ġthat Ġare Ġmost Ġoften Ġused Ġfor Ġabusive Ġlanguage Ġdetection . ĠIn Ġdata Ġpre processing , Ġtext Ġsequences Ġare Ġconverted Ġinto ĠBag ĠOf ĠWords Ġ( B OW ) Ġrepresentations , Ġand Ġnormalized Ġwith ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency Ġ( TF - ID F ) Ġvalues . ĠWe Ġexperiment Ġwith Ġword - level Ġfeatures Ġusing Ġn - gram s Ġranging Ġfrom Ġ1 Ġto Ġ3 , Ġand Ġcharacter - level Ġfeatures Ġfrom Ġ3 Ġto Ġ8 - gram s . ĠEach Ġclass ifier Ġis Ġimplemented Ġwith Ġthe Ġfollowing Ġspecifications : </s> Na Ã¯ve ĠBay es Ġ( NB ): ĠMult in omial ĠNB Ġwith Ġadditive Ġsmoot hing Ġconstant Ġ1 </s> Log istic ĠReg ression Ġ( LR ): ĠLinear ĠLR Ġwith ĠL 2 Ġregular ization Ġconstant Ġ1 Ġand Ġlimited - memory ĠBF GS Ġoptimization </s> Support ĠVector ĠMachine Ġ( S VM ): ĠLinear ĠS VM Ġwith ĠL 2 Ġregular ization Ġconstant Ġ1 Ġand Ġlog istic Ġloss Ġfunction </s> Random ĠFore sts Ġ( RF ): ĠA ver aging Ġprob abil istic Ġpredictions Ġof Ġ10 Ġrandomized Ġdecision Ġtrees </s> Grad ient ĠBoost ed ĠTrees Ġ( GBT ): ĠTree Ġboosting Ġwith Ġlearning Ġrate Ġ1 Ġand Ġlog istic Ġloss Ġfunction </s> Ne ural ĠNetwork Ġbased ĠModels </s> Along Ġwith Ġtraditional Ġmachine Ġlearning Ġapproaches , Ġwe Ġinvestigate Ġneural Ġnetwork Ġbased Ġmodels Ġto Ġevaluate Ġtheir Ġefficacy Ġwithin Ġa Ġlarger Ġdataset . ĠIn Ġparticular , Ġwe Ġexplore ĠConv olution al ĠNeural ĠNetworks Ġ( CNN ), ĠRec urrent ĠNeural ĠNetworks Ġ( R NN ), Ġand Ġtheir Ġvariant Ġmodels . ĠA Ġpre - trained ĠGl o Ve ĠB IB REF 17 Ġrepresentation Ġis Ġused Ġfor Ġword - level Ġfeatures . </s> CNN : ĠWe Ġadopt ĠKim 's Ġk im 2014 conv olution al Ġimplementation Ġas Ġthe Ġbaseline . ĠThe Ġword - level ĠCNN Ġmodels Ġhave Ġ3 Ġconv olution al Ġfilters Ġof Ġdifferent Ġsizes Ġ[ 1 , 2 , 3 ] Ġwith ĠRe LU Ġactivation , Ġand Ġa Ġmax - pool ing Ġlayer . ĠFor Ġthe Ġcharacter - level ĠCNN , Ġwe Ġuse Ġ6 Ġconv olution al Ġfilters Ġof Ġvarious Ġsizes Ġ[ 3 , 4 , 5 , 6 , 7 , 8 ], Ġthen Ġadd Ġmax - pool ing Ġlayers Ġfollowed Ġby Ġ1 Ġfully - connected Ġlayer Ġwith Ġa Ġdimension Ġof Ġ1024 . </s> Park Ġand ĠF ung Ġpark 2017 one Ġproposed Ġa ĠHybrid CNN Ġmodel Ġwhich Ġoutper formed Ġboth Ġword - level Ġand Ġcharacter - level ĠCNN s Ġin Ġabusive Ġlanguage Ġdetection . ĠIn Ġorder Ġto Ġevaluate Ġthe ĠHybrid CNN Ġfor Ġthis Ġdataset , Ġwe Ġconc aten ate Ġthe Ġoutput Ġof Ġmax - pool ed Ġlayers Ġfrom Ġword - level Ġand Ġcharacter - level ĠCNN , Ġand Ġfeed Ġthis Ġvector Ġto Ġa Ġfully - connected Ġlayer Ġin Ġorder Ġto Ġpredict Ġthe Ġoutput . </s> All Ġthree ĠCNN Ġmodels Ġ( word - level , Ġcharacter - level , Ġand Ġhybrid ) Ġuse Ġcross Ġentropy Ġwith Ġsoft max Ġas Ġtheir Ġloss Ġfunction Ġand ĠAdam ĠB IB REF 18 Ġas Ġthe Ġoptim izer . </s> R NN : ĠWe Ġuse Ġbid irection al ĠR NN ĠB IB REF 19 Ġas Ġthe Ġbaseline , Ġimplementing Ġa ĠGR U ĠB IB REF 20 Ġcell Ġfor Ġeach Ġrecurrent Ġunit . ĠFrom Ġextensive Ġparameter - search Ġexperiments , Ġwe Ġchose Ġ1 Ġencoding Ġlayer Ġwith Ġ50 Ġdimensional Ġhidden Ġstates Ġand Ġan Ġinput Ġdrop out Ġprobability Ġof Ġ0 . 3 . ĠThe ĠR NN Ġmodels Ġuse Ġcross Ġentropy Ġwith Ġs igm oid Ġas Ġtheir Ġloss Ġfunction Ġand ĠAdam Ġas Ġthe Ġoptim izer . </s> For Ġa Ġpossible Ġimprovement , Ġwe Ġapply Ġa Ġself - match ing Ġattention Ġmechanism Ġon ĠR NN Ġbaseline Ġmodels ĠB IB REF 21 Ġso Ġthat Ġthey Ġmay Ġbetter Ġunderstand Ġthe Ġdata Ġby Ġretrieving Ġtext Ġsequences Ġtwice . ĠWe Ġalso Ġinvestigate Ġa Ġrecently Ġintroduced Ġmethod , ĠLat ent ĠTopic ĠCl ust ering Ġ( L TC ) ĠB IB REF 22 Ġ. ĠThe ĠL TC Ġmethod Ġextracts Ġlatent Ġtopic Ġinformation Ġfrom Ġthe Ġhidden Ġstates Ġof ĠR NN , Ġand Ġuses Ġit Ġfor Ġadditional Ġinformation Ġin Ġclass ifying Ġthe Ġtext Ġdata . </s> Feature ĠExtension </s> While Ġmanually Ġanalyzing Ġthe Ġraw Ġdataset , Ġwe Ġnoticed Ġthat Ġlooking Ġat Ġthe Ġtweet Ġone Ġhas Ġreplied Ġto Ġor Ġhas Ġquoted , Ġprovides Ġsignificant Ġcontextual Ġinformation . ĠWe Ġcall Ġthese , ĠâĢ ľ context Ġtweets ". ĠAs Ġhumans Ġcan Ġbetter Ġunderstand Ġa Ġtweet Ġwith Ġthe Ġreference Ġof Ġits Ġcontext , Ġour Ġassumption Ġis Ġthat Ġcomputers Ġalso Ġbenefit Ġfrom Ġtaking Ġcontext Ġtweets Ġinto Ġaccount Ġin Ġdetecting Ġabusive Ġlanguage . </s> As Ġshown Ġin Ġthe Ġexamples Ġbelow , Ġ( 2 ) Ġis Ġlabeled Ġabusive Ġdue Ġto Ġthe Ġuse Ġof Ġvulgar Ġlanguage . ĠHowever , Ġthe Ġintention Ġof Ġthe Ġuser Ġcan Ġbe Ġbetter Ġunderstood Ġwith Ġits Ġcontext Ġtweet Ġ( 1 ). </s> ( 1 ) ĠI Ġhate Ġwhen ĠI 'm Ġsitting Ġin Ġfront Ġof Ġthe Ġbus Ġand Ġsomebody Ġwith Ġa Ġwheelchair Ġget Ġon . </s> IN LINE FORM 0 Ġ( 2 ) ĠI Ġhate Ġit Ġwhen ĠI 'm Ġtrying Ġto Ġboard Ġa Ġbus Ġand Ġthere 's Ġalready Ġan Ġas ** ole Ġon Ġit . </s> Similarly , Ġcontext Ġtweet Ġ( 3 ) Ġis Ġimportant Ġin Ġunderstanding Ġthe Ġabusive Ġtweet Ġ( 4 ), Ġespecially Ġin Ġidentifying Ġthe Ġtarget Ġof Ġthe Ġmalice . </s> ( 3 ) ĠSurvivors Ġof Ġ# Syria ĠGas ĠAttack ĠRec ount Ġ` a ĠCruel ĠScene '. </s> IN LINE FORM 0 Ġ( 4 ) ĠWho Ġthe ĠHELL Ġis ĠâĢ ľ LI KE " ĠIN G Ġthis Ġpost ? ĠSick Ġpeople .... </s> Hu ang Ġet Ġal . Ġh u ang 2016 mod eling Ġused Ġseveral Ġattributes Ġof Ġcontext Ġtweets Ġfor Ġsentiment Ġanalysis Ġin Ġorder Ġto Ġimprove Ġthe Ġbaseline ĠL ST M Ġmodel . ĠHowever , Ġtheir Ġapproach Ġwas Ġlimited Ġbecause Ġthe Ġmeta - information Ġthey Ġfocused Ġon âĢĶ author Ġinformation , Ġconversation Ġtype , Ġuse Ġof Ġthe Ġsame Ġhasht ags Ġor Ġem oj is âĢĶ are Ġall Ġhighly Ġdependent Ġon Ġdata . </s> In Ġorder Ġto Ġavoid Ġdata Ġdependency , Ġtext Ġsequences Ġof Ġcontext Ġtweets Ġare Ġdirectly Ġused Ġas Ġan Ġadditional Ġfeature Ġof Ġneural Ġnetwork Ġmodels . ĠWe Ġuse Ġthe Ġsame Ġbaseline Ġmodel Ġto Ġconvert Ġcontext Ġtweets Ġto Ġvectors , Ġthen Ġconc aten ate Ġthese Ġvectors Ġwith Ġoutputs Ġof Ġtheir Ġcorresponding Ġlabeled Ġtweets . ĠMore Ġspecifically , Ġwe Ġconc aten ate Ġmax - pool ed Ġlayers Ġof Ġcontext Ġand Ġlabeled Ġtweets Ġfor Ġthe ĠCNN Ġbaseline Ġmodel . ĠAs Ġfor ĠR NN , Ġthe Ġlast Ġhidden Ġstates Ġof Ġcontext Ġand Ġlabeled Ġtweets Ġare Ġconc aten ated . </s> Dat as et </s> H ate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter ĠB IB REF 10 Ġclass ifies Ġtweets Ġinto Ġ4 Ġlabels , ĠâĢ ľ normal ", ĠâĢ ľ sp am ", ĠâĢ ľ h ateful " Ġand ĠâĢ ľ ab usive ". ĠWe Ġwere Ġonly Ġable Ġto Ġcrawl Ġ70 , 9 04 Ġtweets Ġout Ġof Ġ99 , 996 Ġtweet ĠIDs , Ġmainly Ġbecause Ġthe Ġtweet Ġwas Ġdeleted Ġor Ġthe Ġuser Ġaccount Ġhad Ġbeen Ġsuspended . ĠTable Ġshows Ġthe Ġdistribution Ġof Ġlabels Ġof Ġthe Ġcrawled Ġdata . </s> Data ĠPre processing </s> In Ġthe Ġdata Ġpre processing Ġsteps , Ġuser ĠIDs , ĠURLs , Ġand Ġfrequently Ġused Ġem oj is Ġare Ġreplaced Ġas Ġspecial Ġtokens . ĠSince Ġhasht ags Ġtend Ġto Ġhave Ġa Ġhigh Ġcorrelation Ġwith Ġthe Ġcontent Ġof Ġthe Ġtweet ĠB IB REF 23 Ġ, Ġwe Ġuse Ġa Ġsegment ation Ġlibrary ĠB IB REF 24 Ġfor Ġhasht ags Ġto Ġextract Ġmore Ġinformation . </s> For Ġcharacter - level Ġrepresentations , Ġwe Ġapply Ġthe Ġmethod ĠZhang Ġet Ġal . Ġz hang 2015 character Ġproposed . ĠTwe ets Ġare Ġtransformed Ġinto Ġone - hot Ġencoded Ġvectors Ġusing Ġ70 Ġcharacter Ġdimensions âĢĶ 26 Ġlower - c ased Ġal ph ab ets , Ġ10 Ġdigits , Ġand Ġ34 Ġspecial Ġcharacters Ġincluding Ġwhites pace . </s> Training Ġand ĠEvaluation </s> In Ġtraining Ġthe Ġfeature Ġengineering Ġbased Ġmachine Ġlearning Ġclass ifiers , Ġwe Ġtrunc ate Ġvector Ġrepresentations Ġaccording Ġto Ġthe ĠTF - ID F Ġvalues Ġ( the Ġtop Ġ14 , 000 Ġand Ġ53 , 000 Ġfor Ġword - level Ġand Ġcharacter - level Ġrepresentations , Ġrespectively ) Ġto Ġavoid Ġover fitting . ĠFor Ġneural Ġnetwork Ġmodels , Ġwords Ġthat Ġappear Ġonly Ġonce Ġare Ġreplaced Ġas Ġunknown Ġtokens . </s> Since Ġthe Ġdataset Ġused Ġis Ġnot Ġsplit Ġinto Ġtrain , Ġdevelopment , Ġand Ġtest Ġsets , Ġwe Ġperform Ġ10 - fold Ġcross Ġvalidation , Ġobtaining Ġthe Ġaverage Ġof Ġ5 Ġtries ; Ġwe Ġdivide Ġthe Ġdataset Ġrandomly Ġby Ġa Ġratio Ġof Ġ85 : 5 : 10 , Ġrespectively . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġoverall Ġperformance , Ġwe Ġcalculate Ġthe Ġweighted Ġaverage Ġof Ġprecision , Ġrecall , Ġand ĠF 1 Ġscores Ġof Ġall Ġfour Ġlabels , ĠâĢ ľ normal âĢ Ŀ , ĠâĢ ľ sp am âĢ Ŀ , ĠâĢ ľ h ateful âĢ Ŀ , Ġand ĠâĢ ľ ab usive âĢ Ŀ . </s> E mp ir ical ĠResults </s> As Ġshown Ġin ĠTable Ġ, Ġneural Ġnetwork Ġmodels Ġare Ġmore Ġaccurate Ġthan Ġfeature Ġengineering Ġbased Ġmodels Ġ( i . e . ĠNB , ĠS VM , Ġetc .) Ġexcept Ġfor Ġthe ĠLR Ġmodel âĢĶ the Ġbest ĠLR Ġmodel Ġhas Ġthe Ġsame ĠF 1 Ġscore Ġas Ġthe Ġbest ĠCNN Ġmodel . </s> Among Ġtraditional Ġmachine Ġlearning Ġmodels , Ġthe Ġmost Ġaccurate Ġin Ġclass ifying Ġabusive Ġlanguage Ġis Ġthe ĠLR Ġmodel Ġfollowed Ġby Ġensemble Ġmodels Ġsuch Ġas ĠGB T Ġand ĠRF . ĠCharacter - level Ġrepresentations Ġimprove ĠF 1 Ġscores Ġof ĠS VM Ġand ĠRF Ġclass ifiers , Ġbut Ġthey Ġhave Ġno Ġpositive Ġeffect Ġon Ġother Ġmodels . </s> For Ġneural Ġnetwork Ġmodels , ĠR NN Ġwith ĠL TC Ġmodules Ġhave Ġthe Ġhighest Ġaccuracy Ġscore , Ġbut Ġthere Ġare Ġno Ġsignificant Ġimprovements Ġfrom Ġits Ġbaseline Ġmodel Ġand Ġits Ġattention - added Ġmodel . ĠSimilarly , ĠHybrid CNN Ġdoes Ġnot Ġimprove Ġthe Ġbaseline ĠCNN Ġmodel . ĠFor Ġboth ĠCNN Ġand ĠR NN Ġmodels , Ġcharacter - level Ġfeatures Ġsignificantly Ġdecrease Ġthe Ġaccuracy Ġof Ġclassification . </s> The Ġuse Ġof Ġcontext Ġtweets Ġgenerally Ġhave Ġlittle Ġeffect Ġon Ġbaseline Ġmodels , Ġhowever Ġthey Ġnoticeably Ġimprove Ġthe Ġscores Ġof Ġseveral Ġmetrics . ĠFor Ġinstance , ĠCNN Ġwith Ġcontext Ġtweets Ġscore Ġthe Ġhighest Ġrecall Ġand ĠF 1 Ġfor ĠâĢ ľ h ateful " Ġlabels , Ġand ĠR NN Ġmodels Ġwith Ġcontext Ġtweets Ġhave Ġthe Ġhighest Ġrecall Ġfor ĠâĢ ľ ab usive " Ġtweets . </s> Discussion Ġand ĠConclusion </s> While Ġcharacter - level Ġfeatures Ġare Ġknown Ġto Ġimprove Ġthe Ġaccuracy Ġof Ġneural Ġnetwork Ġmodels ĠB IB REF 16 Ġ, Ġthey Ġreduce Ġclassification Ġaccuracy Ġfor ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter . ĠWe Ġconclude Ġthis Ġis Ġbecause Ġof Ġthe Ġlack Ġof Ġlabeled Ġdata Ġas Ġwell Ġas Ġthe Ġsignificant Ġimbalance Ġamong Ġthe Ġdifferent Ġlabels . ĠUnlike Ġneural Ġnetwork Ġmodels , Ġcharacter - level Ġfeatures Ġin Ġtraditional Ġmachine Ġlearning Ġclass ifiers Ġhave Ġpositive Ġresults Ġbecause Ġwe Ġhave Ġtrained Ġthe Ġmodels Ġonly Ġwith Ġthe Ġmost Ġsignificant Ġcharacter Ġelements Ġusing ĠTF - ID F Ġvalues . </s> Vari ants Ġof Ġneural Ġnetwork Ġmodels Ġalso Ġsuffer Ġfrom Ġdata Ġins uff iciency . ĠHowever , Ġthese Ġmodels Ġshow Ġpositive Ġperformances Ġon ĠâĢ ľ sp am " Ġ( 14 %) Ġand ĠâĢ ľ h ateful " Ġ( 4 %) Ġtweets âĢĶ the Ġlower Ġdistributed Ġlabels . ĠThe Ġhighest ĠF 1 Ġscore Ġfor ĠâĢ ľ sp am " Ġis Ġfrom Ġthe ĠR NN - L TC Ġmodel Ġ( 0 . 551 ), Ġand Ġthe Ġhighest Ġfor ĠâĢ ľ h ateful " Ġis ĠCNN Ġwith Ġcontext Ġtweets Ġ( 0 . 309 ). ĠSince Ġeach Ġvariant Ġmodel Ġexcel s Ġin Ġdifferent Ġmetrics , Ġwe Ġexpect Ġto Ġsee Ġadditional Ġimprovements Ġwith Ġthe Ġuse Ġof Ġensemble Ġmodels Ġof Ġthese Ġvariants Ġin Ġfuture Ġworks . </s> In Ġthis Ġpaper , Ġwe Ġreport Ġthe Ġbaseline Ġaccuracy Ġof Ġdifferent Ġlearning Ġmodels Ġas Ġwell Ġas Ġtheir Ġvariants Ġon Ġthe Ġrecently Ġintroduced Ġdataset , ĠHate Ġand ĠAb usive ĠSpeech Ġon ĠTwitter . ĠExperimental Ġresults Ġshow Ġthat Ġbid irection al ĠGR U Ġnetworks Ġwith ĠL TC Ġprovide Ġthe Ġmost Ġaccurate Ġresults Ġin Ġdetecting Ġabusive Ġlanguage . ĠAdditionally , Ġwe Ġpresent Ġthe Ġpossibility Ġof Ġusing Ġensemble Ġmodels Ġof Ġvariant Ġmodels Ġand Ġfeatures Ġfor Ġfurther Ġimprovements . </s> Acknowled gments </s> K . ĠJung Ġis Ġwith Ġthe ĠDepartment Ġof ĠElectrical Ġand ĠComputer ĠEngineering , ĠAS RI , ĠSeoul ĠNational ĠUniversity , ĠSeoul , ĠKorea . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠResearch ĠFoundation Ġof ĠKorea Ġ( NR F ) Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( No . Ġ2016 M 3 C 4 A 795 26 32 ), Ġthe ĠTechnology ĠInnovation ĠProgram Ġ( 100 73 144 ) Ġfunded Ġby Ġthe ĠMinistry Ġof ĠTrade , ĠIndustry Ġ& ĠEnergy Ġ( M OT IE , ĠKorea ). </s> We Ġwould Ġalso Ġlike Ġto Ġthank ĠYong ke un ĠH wang Ġand ĠJi ĠHo ĠPark Ġfor Ġhelpful Ġdiscussions Ġand Ġtheir Ġvaluable Ġinsights .