By multiplying crowd-annotated document-emotion matrix with emotion-word matrix. 
ABSTRACTIVE
researchers asked subjects to report their emotions after reading each article, multiplied the document-emotion matrix and word-document matrix to derive emotion-word matrix for these words, Depechemood simply creates dictionaries of words where each word has scores between 0 and 1 for all of these 8 emotion categories
EXTRACTIVE
Depechemood is a lexicon-based emotion detection method gathered from crowd-annotated news BIBREF24. Drawing on approximately 23.5K documents with average of 500 words per document from rappler.com, researchers asked subjects to report their emotions after reading each article. They then multiplied the document-emotion matrix and word-document matrix to derive emotion-word matrix for these words. Due to limitations of their experiment setup, the emotion categories that they present does not exactly match the emotions from the Plutchik wheel categories. However, they still provide a good sense of the general feeling of an individual after reading an article. The emotion categories of Depechemood are: AFRAID, AMUSED, ANGRY, ANNOYED, DON'T CARE, HAPPY, INSPIRED, SAD. Depechemood simply creates dictionaries of words where each word has scores between 0 and 1 for all of these 8 emotion categories. We present our finding using this approach in the result section.
Depechemood is a lexicon-based emotion detection method gathered from crowd-annotated news BIBREF24. Drawing on approximately 23.5K documents with average of 500 words per document from rappler.com, researchers asked subjects to report their emotions after reading each article. They then multiplied the document-emotion matrix and word-document matrix to derive emotion-word matrix for these words. Due to limitations of their experiment setup, the emotion categories that they present does not exactly match the emotions from the Plutchik wheel categories. However, they still provide a good sense of the general feeling of an individual after reading an article. The emotion categories of Depechemood are: AFRAID, AMUSED, ANGRY, ANNOYED, DON'T CARE, HAPPY, INSPIRED, SAD. Depechemood simply creates dictionaries of words where each word has scores between 0 and 1 for all of these 8 emotion categories. We present our finding using this approach in the result section.
1912.03804
Introduction </s> Since Ġits Ġrise Ġin Ġ2013 , Ġthe ĠIslamic ĠState Ġof ĠIraq Ġand ĠSyria Ġ( ISIS ) Ġhas Ġutilized Ġthe ĠInternet Ġto Ġspread Ġits Ġideology , Ġradical ize Ġindividuals , Ġand Ġrecruit Ġthem Ġto Ġtheir Ġcause . ĠIn Ġcomparison Ġto Ġother ĠIslamic Ġextremist Ġgroups , ĠISIS ' Ġuse Ġof Ġtechnology Ġwas Ġmore Ġsophisticated , Ġvol umin ous , Ġand Ġtargeted . ĠFor Ġexample , Ġduring ĠISIS ' Ġadvance Ġtoward ĠMosul , ĠISIS Ġrelated Ġaccounts Ġtweeted Ġsome Ġ40 , 000 Ġtweets Ġin Ġone Ġday ĠB IB REF 0 . However , Ġthis Ġheavy Ġengagement Ġforced Ġsocial Ġmedia Ġplatforms Ġto Ġinstitute Ġpolicies Ġto Ġprevent Ġunchecked Ġdissemination Ġof Ġterrorist Ġpropaganda Ġto Ġtheir Ġusers , Ġforcing ĠISIS Ġto Ġadapt Ġto Ġother Ġmeans Ġto Ġreach Ġtheir Ġtarget Ġaudience . </s> One Ġsuch Ġapproach Ġwas Ġthe Ġpublication Ġof Ġonline Ġmagazines Ġin Ġdifferent Ġlanguages Ġincluding ĠEnglish . ĠAlthough Ġdiscontinued Ġnow , Ġthese Ġonline Ġresources Ġprovided Ġa Ġwindow Ġinto ĠISIS Ġideology , Ġrecruitment , Ġand Ġhow Ġthey Ġwanted Ġthe Ġworld Ġto Ġperceive Ġthem . ĠFor Ġexample , Ġafter Ġpredominantly Ġrecruiting Ġmen , ĠISIS Ġbegan Ġto Ġalso Ġinclude Ġarticles Ġin Ġtheir Ġmagazines Ġthat Ġspecifically Ġaddressed Ġwomen . ĠISIS Ġencouraged Ġwomen Ġto Ġjoin Ġthe Ġgroup Ġby Ġeither Ġtraveling Ġto Ġthe Ġcaliphate Ġor Ġby Ġcarrying Ġout Ġdomestic Ġattacks Ġon Ġbehalf Ġof ĠISIS Ġin Ġtheir Ġrespective Ġcountries . ĠThis Ġtactical Ġchange Ġconcerned Ġboth Ġpractitioners Ġand Ġresearchers Ġin Ġthe Ġcounterterrorism Ġcommunity . ĠNew Ġadvancements Ġin Ġdata Ġscience Ġcan Ġshed Ġlight Ġon Ġexactly Ġhow Ġthe Ġtargeting Ġof Ġwomen Ġin Ġextremist Ġpropaganda Ġworks Ġand Ġwhether Ġit Ġdiffers Ġsignificantly Ġfrom Ġmainstream Ġreligious Ġrhetoric . </s> We Ġutilize Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġanswer Ġthree Ġquestions : </s> What Ġare Ġthe Ġmain Ġtopics Ġin Ġwomen - related Ġarticles Ġin ĠISIS ' Ġonline Ġmagazines ? </s> What Ġsimilarities Ġand / or Ġdifferences Ġdo Ġthese Ġtopics Ġhave Ġwith Ġnon - violent , Ġnon - Islamic Ġreligious Ġmaterial Ġaddressed Ġspecifically Ġto Ġwomen ? </s> What Ġkind Ġof Ġemotions Ġdo Ġthese Ġarticles Ġevoke Ġin Ġtheir Ġreaders Ġand Ġare Ġthere Ġsimilarities Ġin Ġthe Ġemotions Ġev oked Ġfrom Ġboth ĠISIS Ġand Ġnon - violent Ġreligious Ġmaterials ? </s> As Ġthese Ġquestions Ġsuggest , Ġto Ġunderstand Ġwhat , Ġif Ġanything , Ġmakes Ġextremist Ġappeals Ġdistinctive , Ġwe Ġneed Ġa Ġpoint Ġof Ġcomparison Ġin Ġterms Ġof Ġthe Ġoutreach Ġefforts Ġto Ġwomen Ġfrom Ġa Ġmainstream , Ġnon - violent Ġreligious Ġgroup . ĠFor Ġthis Ġpurpose , Ġwe Ġrely Ġon Ġan Ġonline ĠCatholic Ġwomen 's Ġforum . ĠComparison Ġbetween ĠCatholic Ġmaterial Ġand Ġthe Ġcontent Ġof ĠISIS ' Ġonline Ġmagazines Ġallows Ġfor Ġnovel Ġinsight Ġinto Ġthe Ġdistinct iveness Ġof Ġextremist Ġrhetoric Ġwhen Ġtargeted Ġtowards Ġthe Ġfemale Ġpopulation . ĠTo Ġaccomplish Ġthis Ġtask , Ġwe Ġemploy Ġtopic Ġmodeling Ġand Ġan Ġun super vised Ġemotion Ġdetection Ġmethod . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : Ġin ĠSection ĠSEC REF 2 , Ġwe Ġreview Ġrelated Ġworks Ġon ĠISIS Ġpropaganda Ġand Ġapplications Ġof Ġnatural Ġlanguage Ġmethods . ĠSection ĠSEC REF 3 Ġdescribes Ġdata Ġcollection Ġand Ġpre - processing . ĠSection ĠSEC REF 4 Ġdescribes Ġin Ġdetail Ġthe Ġapproach . ĠSection ĠSEC REF 5 Ġreports Ġthe Ġresults , Ġand Ġfinally , ĠSection ĠSEC REF 6 Ġpresents Ġthe Ġconclusion . </s> Related ĠWork </s> Soon Ġafter ĠISIS Ġemerged Ġand Ġdeclared Ġits Ġcaliphate , Ġcounterterrorism Ġpractitioners Ġand Ġpolitical Ġscience Ġresearchers Ġstarted Ġto Ġturn Ġtheir Ġattention Ġtowards Ġunderstanding Ġhow Ġthe Ġgroup Ġoperated . ĠResearchers Ġinvestigated Ġthe Ġorigins Ġof ĠISIS , Ġits Ġleadership , Ġfunding , Ġand Ġhow Ġthey Ġrose Ġbecame Ġa Ġglobally Ġdominant Ġnon - state Ġactor ĠB IB REF 1 . ĠThis Ġinterest Ġin Ġthe Ġorganization 's Ġdistinct iveness Ġimmediately Ġled Ġto Ġinquiries Ġinto ĠISIS ' Ġrhetoric , Ġparticularly Ġtheir Ġuse Ġof Ġsocial Ġmedia Ġand Ġonline Ġresources Ġin Ġrecruitment Ġand Ġideological Ġdissemination . ĠFor Ġexample , ĠAl - Tam imi Ġexamines Ġhow ĠISIS Ġdifferentiated Ġitself Ġfrom Ġother Ġjihadist Ġmovements Ġby Ġusing Ġsocial Ġmedia Ġwith Ġunprecedented Ġefficiency Ġto Ġimprove Ġits Ġimage Ġwith Ġlocals ĠB IB REF 2 . ĠOne Ġof ĠISIS ' Ġmost Ġimpressive Ġapplications Ġof Ġits Ġonline Ġprowess Ġwas Ġin Ġthe Ġrecruitment Ġprocess . ĠThe Ġorganization Ġhas Ġused Ġa Ġvariety Ġof Ġmaterials , Ġespecially Ġvideos , Ġto Ġrecruit Ġboth Ġforeign Ġand Ġlocal Ġfighters . ĠResearch Ġshows Ġthat ĠISIS Ġpropaganda Ġis Ġdesigned Ġto Ġportray Ġthe Ġorganization Ġas Ġa Ġprovider Ġof Ġjustice , Ġgovernance , Ġand Ġdevelopment Ġin Ġa Ġfashion Ġthat Ġreson ates Ġwith Ġyoung Ġwestern ers ĠB IB REF 3 . ĠThis Ġpropaganda Ġmachine Ġhas Ġbecome Ġa Ġsignificant Ġarea Ġof Ġresearch , Ġwith Ġscholars Ġsuch Ġas ĠWinter Ġidentifying Ġkey Ġthemes Ġin Ġit Ġsuch Ġas Ġbrutality , Ġmercy , Ġvictim hood , Ġwar , Ġbelonging Ġand Ġutopian ism . ĠB IB REF 4 . ĠHowever , Ġthere Ġhas Ġbeen Ġinsufficient Ġattention Ġfocused Ġon Ġhow Ġthese Ġapproaches Ġhave Ġparticularly Ġtargeted Ġand Ġimpacted Ġwomen . ĠThis Ġis Ġsignificant Ġgiven Ġthat Ġscholars Ġhave Ġidentified Ġthe Ġdistinct iveness Ġof Ġthis Ġpopulation Ġwhen Ġit Ġcomes Ġto Ġnearly Ġall Ġfacets Ġof Ġterrorism . </s> ISIS Ġused Ġdifferent Ġtypes Ġof Ġmedia Ġto Ġpropagate Ġits Ġmessages , Ġsuch Ġas Ġvideos , Ġimages , Ġtexts , Ġand Ġeven Ġmusic . ĠTwitter Ġwas Ġparticularly Ġeffective Ġand Ġthe ĠArabic ĠTwitter Ġapp Ġallowed ĠISIS Ġto Ġtweet Ġextensively Ġwithout Ġtriggering Ġspam - det ection Ġmechanisms Ġthe Ġplatform Ġuses ĠB IB REF 0 . ĠScholars Ġfollowed Ġthe Ġresulting Ġtrove Ġof Ġdata Ġand Ġthis Ġbecame Ġthe Ġpre em inent Ġway Ġin Ġwhich Ġthey Ġassess ĠISIS Ġmessages . ĠFor Ġexample , Ġin ĠB IB REF 5 Ġthey Ġuse Ġboth Ġlex ical Ġanalysis Ġof Ġtweets Ġas Ġwell Ġas Ġsocial Ġnetwork Ġanalysis Ġto Ġexamine ĠISIS Ġsupport Ġor Ġopposition Ġon ĠTwitter . ĠOther Ġresearchers Ġused Ġdata Ġmining Ġtechniques Ġto Ġdetect Ġpro - ISIS Ġuser Ġdivergence Ġbehavior Ġat Ġvarious Ġpoints Ġin Ġtime ĠB IB REF 6 . ĠBy Ġlooking Ġat Ġthese Ġworks , Ġthe Ġimpact Ġof Ġusing Ġtext Ġmining Ġand Ġlex ical Ġanalysis Ġto Ġaddress Ġimportant Ġquestions Ġbecomes Ġobvious . ĠProper Ġusage Ġof Ġthese Ġtools Ġallows Ġthe Ġresearch Ġcommunity Ġto Ġanalyze Ġbig Ġchunks Ġof Ġun struct ured Ġdata . ĠThis Ġapproach , Ġhowever , Ġbecame Ġless Ġproductive Ġas Ġthe Ġsocial Ġmedia Ġnetworks Ġbegan Ġcracking Ġdown Ġand ĠISIS Ġrecruit ers Ġmoved Ġoff Ġof Ġthem . </s> With Ġtheir Ġability Ġto Ġoperate Ġfreely Ġon Ġsocial Ġmedia Ġnow Ġcur tailed , ĠISIS Ġrecruit ers Ġand Ġpropag and ists Ġincreased Ġtheir Ġattent iveness Ġto Ġanother Ġlongstanding Ġtool âĢĵ English Ġlanguage Ġonline Ġmagazines Ġtargeting Ġwestern Ġaudiences . ĠAl ĠHay at , Ġthe Ġmedia Ġwing Ġof ĠISIS , Ġpublished Ġmultiple Ġonline Ġmagazines Ġin Ġdifferent Ġlanguages Ġincluding ĠEnglish . ĠThe ĠEnglish Ġonline Ġmagazine Ġof ĠISIS Ġwas Ġnamed ĠD abi q Ġand Ġfirst Ġappeared Ġon Ġthe Ġdark Ġweb Ġon ĠJuly Ġ2014 Ġand Ġcontinued Ġpublishing Ġfor Ġ15 Ġissues . ĠThis Ġpublication Ġwas Ġfollowed Ġby ĠRum iyah Ġwhich Ġproduced Ġ13 ĠEnglish Ġlanguage Ġissues Ġthrough ĠSeptember Ġ2017 . ĠThe Ġcontent Ġof Ġthese Ġmagazines Ġprovides Ġa Ġvaluable Ġbut Ġunder util ized Ġresource Ġfor Ġunderstanding ĠISIS Ġstrategies Ġand Ġhow Ġthey Ġappeal Ġto Ġrecruits , Ġspecifically ĠEnglish - speaking Ġaudiences . ĠThey Ġalso Ġprovide Ġa Ġway Ġto Ġcompare ĠISIS ' Ġapproach Ġwith Ġother Ġradical Ġgroups . ĠIngram Ġcompared ĠD abi q Ġcontents Ġwith ĠInsp ire Ġ( Al ĠQaeda Ġpublication ) Ġand Ġsuggested Ġthat ĠAl ĠQaeda Ġheavily Ġemphasized Ġidentity - choice , Ġwhile ĠISIS ' Ġmessages Ġwere Ġmore Ġbalanced Ġbetween Ġidentity - choice Ġand Ġrational - choice ĠB IB REF 7 . ĠIn Ġanother Ġresearch Ġpaper , ĠW ign ell Ġet Ġal . ĠB IB REF 8 Ġcompared ĠD abi q Ġand ĠRum iah Ġby Ġexamining Ġtheir Ġstyle Ġand Ġwhat Ġboth Ġmagazine Ġmessages Ġemphasized . ĠDespite Ġthe Ġvolume Ġof Ġresearch Ġon Ġthese Ġmagazines , Ġonly Ġa Ġfew Ġresearchers Ġused Ġlex ical Ġanalysis Ġand Ġmostly Ġrelied Ġon Ġexperts ' Ġopinions . ĠB IB REF 9 Ġis Ġone Ġexception Ġto Ġthis Ġapproach Ġwhere Ġthey Ġused Ġword Ġfrequency Ġon Ġ11 Ġissues Ġof ĠD abi q Ġpublications Ġand Ġcompared Ġattributes Ġsuch Ġas Ġanger , Ġanxiety , Ġpower , Ġmotive , Ġetc . </s> This Ġpaper Ġseeks Ġto Ġestablish Ġhow ĠISIS Ġspecifically Ġtailored Ġpropaganda Ġtargeting Ġwestern Ġwomen , Ġwho Ġbecame Ġa Ġparticular Ġtarget Ġfor Ġthe Ġorganization Ġas Ġthe ĠâĢ ľ cal iphate âĢ Ŀ Ġexpanded . ĠAlthough Ġthe Ġnumber Ġof Ġrecruits Ġis Ġunknown , Ġin Ġ2015 Ġit Ġwas Ġestimated Ġthat Ġaround Ġ10 Ġpercent Ġof Ġall Ġwestern Ġrecruits Ġwere Ġfemale ĠB IB REF 10 . ĠSome Ġresearchers Ġhave Ġattempted Ġto Ġunderstand Ġhow ĠISIS Ġpropaganda Ġtargets Ġwomen . ĠK ne ip , Ġfor Ġexample , Ġanalyzed Ġwomen 's Ġdesire Ġto Ġjoin Ġas Ġa Ġform Ġof Ġemancipation ĠB IB REF 11 . ĠWe Ġextend Ġthat Ġline Ġof Ġinquiry Ġby Ġleveraging Ġtechnology Ġto Ġanswer Ġkey Ġoutstanding Ġquestions Ġabout Ġthe Ġtargeting Ġof Ġwomen Ġin ĠISIS Ġpropaganda . </s> To Ġfurther Ġassess Ġhow ĠISIS Ġpropaganda Ġmight Ġaffect Ġwomen , Ġwe Ġused Ġemotion Ġdetection Ġmethods Ġon Ġthese Ġtexts . ĠEm otion Ġdetection Ġtechniques Ġare Ġmostly Ġdivided Ġinto Ġlex icon - base Ġor Ġmachine Ġlearning - base Ġmethods . ĠLex icon - base Ġmethods Ġrely Ġon Ġseveral Ġlex icons Ġwhile Ġmachine Ġlearning Ġ( ML ) Ġmethods Ġuse Ġalgorithm Ġto Ġdetect Ġthe Ġel ation Ġof Ġtexts Ġas Ġinputs Ġand Ġemotions Ġas Ġthe Ġtarget , Ġusually Ġtrained Ġon Ġa Ġlarge Ġcorpus . ĠUn super vised Ġmethods Ġusually Ġuse ĠNon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġand ĠLat ent ĠSem antic ĠAnalysis Ġ( L SA ) ĠB IB REF 12 Ġapproaches . ĠAn Ġimportant Ġdistinction Ġthat Ġshould Ġbe Ġmade Ġwhen Ġusing Ġtext Ġfor Ġemotion Ġdetection Ġis Ġthat Ġemotion Ġdetected Ġin Ġthe Ġtext Ġand Ġthe Ġemotion Ġev oked Ġin Ġthe Ġreader Ġof Ġthat Ġtext Ġmight Ġdiffer . ĠIn Ġthe Ġcase Ġof Ġpropaganda , Ġit Ġis Ġmore Ġdesirable Ġto Ġdetect Ġpossible Ġemotions Ġthat Ġwill Ġbe Ġev oked Ġin Ġa Ġhypothetical Ġreader . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdescribe Ġmethods Ġto Ġanalyze Ġcontent Ġand Ġtechnique Ġto Ġfind Ġev oked Ġemotions Ġin Ġa Ġpotential Ġreader Ġusing Ġavailable Ġnatural Ġlanguage Ġprocessing Ġtools . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠData Ġcollection </s> Finding Ġuseful Ġcollections Ġof Ġtexts Ġwhere ĠISIS Ġtargets Ġwomen Ġis Ġa Ġchallenging Ġtask . ĠMost Ġof Ġthe Ġavailable Ġmaterial Ġare Ġnot Ġreflecting ĠISIS ' Ġofficial Ġpoint Ġof Ġview Ġor Ġthey Ġdo Ġnot Ġtalk Ġspecifically Ġabout Ġwomen . ĠHowever , ĠISIS ' Ġonline Ġmagazines Ġare Ġvaluable Ġresources Ġfor Ġunderstanding Ġhow Ġthe Ġorganization Ġattempts Ġto Ġappeal Ġto Ġwestern Ġaudiences , Ġparticularly Ġwomen . ĠLooking Ġthrough Ġboth ĠD abi q Ġand ĠRum iyah , Ġmany Ġissues Ġof Ġthe Ġmagazines Ġcontain Ġarticles Ġspecifically Ġaddressing Ġwomen , Ġusually Ġwith ĠâĢ ľ Ġto Ġour Ġsisters ĠâĢ Ŀ Ġincorporated Ġinto Ġthe Ġtitle . ĠSeven Ġout Ġof Ġfifteen ĠD abi q Ġissues Ġand Ġall Ġthirteen Ġissues Ġof ĠRum iyah Ġcontain Ġarticles Ġtargeting Ġwomen , Ġclearly Ġsuggesting Ġan Ġincrease Ġin Ġattention Ġto Ġwomen Ġover Ġtime . </s> We Ġconverted Ġall Ġthe ĠISIS Ġmagazines Ġto Ġtexts Ġusing Ġpdf Ġreaders Ġand Ġall Ġarticles Ġthat Ġaddressed Ġwomen Ġin Ġboth Ġmagazines Ġ( 20 Ġarticles ) Ġwere Ġselected Ġfor Ġour Ġanalysis . ĠTo Ġfacilitate Ġcomparison Ġwith Ġa Ġmainstream , Ġnon - violent Ġreligious Ġgroup , Ġwe Ġcollected Ġarticles Ġfrom Ġcath olic w om ens forum . org , Ġan Ġonline Ġresource Ġcatering Ġto ĠCatholic Ġwomen . ĠWe Ġscrapped Ġ132 Ġarticles Ġfrom Ġthis Ġdomain . ĠWhile Ġthis Ġnumber Ġis Ġlarge , Ġthe Ġarticles Ġthemselves Ġare Ġmuch Ġshorter Ġthan Ġthose Ġpublished Ġby ĠISIS . ĠThese Ġtexts Ġwere Ġpre - process ed Ġby Ġtoken izing Ġthe Ġsentences Ġand Ġeliminating Ġnon - word Ġtokens Ġand Ġpunct uation Ġmarks . ĠAlso , Ġall Ġwords Ġturned Ġinto Ġlower Ġcase Ġand Ġnumbers Ġand ĠEnglish Ġstop Ġwords Ġsuch Ġas ĠâĢ ľ our , Ġis , Ġdid , Ġcan , Ġetc . ĠâĢ Ŀ Ġhave Ġbeen Ġremoved Ġfrom Ġthe Ġproduced Ġtokens . ĠFor Ġthe Ġemotion Ġanalysis Ġpart , Ġwe Ġused Ġa Ġsp acy Ġlibrary Ġas Ġpart Ġof Ġspeech Ġtagging Ġto Ġidentify Ġthe Ġexact Ġrole Ġof Ġwords Ġin Ġthe Ġsentence . ĠA Ġword Ġand Ġits Ġrole Ġhave Ġbeen Ġused Ġto Ġlook Ġfor Ġemotional Ġvalues Ġof Ġthat Ġword Ġin Ġthe Ġsame Ġrole Ġin Ġthe Ġsentence . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠText ĠClean ing Ġand ĠPre - processing </s> Most Ġtext Ġand Ġdocument Ġdatasets Ġcontain Ġmany Ġunnecessary Ġwords Ġsuch Ġas Ġstop words , Ġmiss pelling , Ġslang , Ġetc . ĠIn Ġmany Ġalgorithms , Ġespecially Ġstatistical Ġand Ġprob abil istic Ġlearning Ġalgorithms , Ġnoise Ġand Ġunnecessary Ġfeatures Ġcan Ġhave Ġadverse Ġeffects Ġon Ġsystem Ġperformance . ĠIn Ġthis Ġsection , Ġwe Ġbriefly Ġexplain Ġsome Ġtechniques Ġand Ġmethods Ġfor Ġtext Ġcleaning Ġand Ġpre - processing Ġtext Ġdatasets ĠB IB REF 13 . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠToken ization </s> Token ization Ġis Ġa Ġpre - processing Ġmethod Ġwhich Ġbreaks Ġa Ġstream Ġof Ġtext Ġinto Ġwords , Ġphrases , Ġsymbols , Ġor Ġother Ġmeaningful Ġelements Ġcalled Ġtokens ĠB IB REF 14 . ĠThe Ġmain Ġgoal Ġof Ġthis Ġstep Ġis Ġto Ġinvestigate Ġthe Ġwords Ġin Ġa Ġsentence ĠB IB REF 14 . ĠBoth Ġtext Ġclassification Ġand Ġtext Ġmining Ġrequires Ġa Ġparser Ġwhich Ġprocesses Ġthe Ġtoken ization Ġof Ġthe Ġdocuments ; Ġfor Ġexample : </s> sent ence ĠB IB REF 15 Ġ: </s> After Ġsleeping Ġfor Ġfour Ġhours , Ġhe Ġdecided Ġto Ġsleep Ġfor Ġanother Ġfour . </s> In Ġthis Ġcase , Ġthe Ġtokens Ġare Ġas Ġfollows : </s> { âĢ ľ After âĢ Ŀ ĠâĢ ľ sle eping âĢ Ŀ ĠâĢ ľ for âĢ Ŀ ĠâĢ ľ four âĢ Ŀ ĠâĢ ľ hours âĢ Ŀ ĠâĢ ľ he âĢ Ŀ ĠâĢ ľ dec ided âĢ Ŀ ĠâĢ ľ to âĢ Ŀ ĠâĢ ľ sleep âĢ Ŀ ĠâĢ ľ for âĢ Ŀ ĠâĢ ľ another âĢ Ŀ ĠâĢ ľ four âĢ Ŀ }. </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠStop Ġwords </s> Text Ġand Ġdocument Ġclassification Ġincludes Ġmany Ġwords Ġwhich Ġdo Ġnot Ġhold Ġimportant Ġsignificance Ġto Ġbe Ġused Ġin Ġclassification Ġalgorithms Ġsuch Ġas Ġ{ âĢ ľ a âĢ Ŀ , ĠâĢ ľ about âĢ Ŀ , ĠâĢ ľ above âĢ Ŀ , ĠâĢ ľ ac ross âĢ Ŀ , ĠâĢ ľ after âĢ Ŀ , ĠâĢ ľ after wards âĢ Ŀ , ĠâĢ ľ again âĢ Ŀ , $ \ hd ots Ġ$ }. ĠThe Ġmost Ġcommon Ġtechnique Ġto Ġdeal Ġwith Ġthese Ġwords Ġis Ġto Ġremove Ġthem Ġfrom Ġthe Ġtexts Ġand Ġdocuments ĠB IB REF 16 . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency </s> K ĠSp ar ck ĠJones ĠB IB REF 17 Ġproposed Ġinverse Ġdocument Ġfrequency Ġ( ID F ) Ġas Ġa Ġmethod Ġto Ġbe Ġused Ġin Ġconjunction Ġwith Ġterm Ġfrequency Ġin Ġorder Ġto Ġlessen Ġthe Ġeffect Ġof Ġimplicitly Ġcommon Ġwords Ġin Ġthe Ġcorpus . ĠIDF Ġassigns Ġa Ġhigher Ġweight Ġto Ġwords Ġwith Ġeither Ġhigh Ġfrequency Ġor Ġlow Ġfrequency Ġterm Ġin Ġthe Ġdocument . ĠThis Ġcombination Ġof ĠTF Ġand ĠIDF Ġis Ġwell Ġknown Ġas Ġterm Ġfrequency - in verse Ġdocument Ġfrequency Ġ( tf - id f ). ĠThe Ġmathematical Ġrepresentation Ġof Ġthe Ġweight Ġof Ġa Ġterm Ġin Ġa Ġdocument Ġby Ġtf - id f Ġis Ġgiven Ġin ĠEqu ation ĠDIS PLAY _ FORM 10 . </s> Here ĠN Ġis Ġthe Ġnumber Ġof Ġdocuments Ġand Ġ$ df ( t ) $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġcontaining Ġthe Ġterm Ġt Ġin Ġthe Ġcorpus . ĠThe Ġfirst Ġterm Ġin ĠEqu ation ĠDIS PLAY _ FORM 10 Ġimproves Ġthe Ġrecall Ġwhile Ġthe Ġsecond Ġterm Ġimproves Ġthe Ġprecision Ġof Ġthe Ġword Ġembed ding ĠB IB REF 18 . ĠAlthough Ġtf - id f Ġtries Ġto Ġovercome Ġthe Ġproblem Ġof Ġcommon Ġterms Ġin Ġthe Ġdocument , Ġit Ġstill Ġsuffers Ġfrom Ġsome Ġother Ġdescriptive Ġlimitations . ĠName ly , Ġtf - id f Ġcannot Ġaccount Ġfor Ġthe Ġsimilarity Ġbetween Ġthe Ġwords Ġin Ġthe Ġdocument Ġsince Ġeach Ġword Ġis Ġindependently Ġpresented Ġas Ġan Ġindex . ĠHowever , Ġwith Ġthe Ġdevelopment Ġof Ġmore Ġcomplex Ġmodels Ġin Ġrecent Ġyears , Ġnew Ġmethods , Ġsuch Ġas Ġword Ġembed ding , Ġhave Ġbeen Ġpresented Ġthat Ġcan Ġincorporate Ġconcepts Ġsuch Ġas Ġsimilarity Ġof Ġwords Ġand Ġpart Ġof Ġspeech Ġtagging . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġmethods Ġused Ġfor Ġcomparing Ġtopics Ġand Ġev oked Ġemotions Ġin Ġboth ĠISIS Ġand Ġnon - violent Ġreligious Ġmaterials . </s> Method Ġ: :: ĠContent ĠAnalysis </s> The Ġkey Ġtask Ġin Ġcomparing ĠISIS Ġmaterial Ġwith Ġthat Ġof Ġa Ġnon - violent Ġgroup Ġinvolves Ġanalyzing Ġthe Ġcontent Ġof Ġthese Ġtwo Ġcorpor a Ġto Ġidentify Ġthe Ġtopics . ĠFor Ġour Ġanalysis , Ġwe Ġconsidered Ġa Ġsimple Ġun i - gram Ġmodel Ġwhere Ġeach Ġword Ġis Ġconsidered Ġas Ġa Ġsingle Ġunit . ĠUnderstanding Ġwhat Ġwords Ġappear Ġmost Ġfrequently Ġprovides Ġa Ġsimple Ġmetric Ġfor Ġcomparison . ĠTo Ġdo Ġso Ġwe Ġnormalized Ġthe Ġcount Ġof Ġwords Ġwith Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġcorpor a Ġto Ġaccount Ġfor Ġthe Ġsize Ġof Ġeach Ġcorpus . ĠIt Ġshould Ġbe Ġnoted , Ġhowever , Ġthat Ġa Ġdrawback Ġof Ġword Ġfrequencies Ġis Ġthat Ġthere Ġmight Ġbe Ġsome Ġdominant Ġwords Ġthat Ġwill Ġovercome Ġall Ġthe Ġother Ġcontents Ġwithout Ġconve ying Ġmuch Ġinformation . </s> Topic Ġmodeling Ġmethods Ġare Ġthe Ġmore Ġpowerful Ġtechnique Ġfor Ġunderstanding Ġthe Ġcontents Ġof Ġa Ġcorpus . ĠThese Ġmethods Ġtry Ġto Ġdiscover Ġabstract Ġtopics Ġin Ġa Ġcorpus Ġand Ġreveal Ġhidden Ġsemantic Ġstructures Ġin Ġa Ġcollection Ġof Ġdocuments . ĠThe Ġmost Ġpopular Ġtopic Ġmodeling Ġmethods Ġuse Ġprob abil istic Ġapproaches Ġsuch Ġas Ġprob abil istic Ġlatent Ġsemantic Ġanalysis Ġ( PL SA ) Ġand Ġlatent ĠDir ich let Ġallocation Ġ( L DA ). ĠL DA Ġis Ġa Ġgeneral ization Ġof Ġp L SA Ġwhere Ġdocuments Ġare Ġconsidered Ġas Ġa Ġmixture Ġof Ġtopics Ġand Ġthe Ġdistribution Ġof Ġtopics Ġis Ġgoverned Ġby Ġa ĠDir ich let Ġprior Ġ($ \ alpha Ġ$ ). ĠFigure ĠFIG REF 12 Ġshows Ġplate Ġnotation Ġof Ġgeneral ĠL DA Ġstructure Ġwhere Ġ$\ beta Ġ$ Ġrepresents Ġprior Ġof Ġword Ġdistribution Ġper Ġtopic Ġand Ġ$\ the ta Ġ$ Ġrefers Ġto Ġtopics Ġdistribution Ġfor Ġdocuments ĠB IB REF 19 . ĠSince ĠL DA Ġis Ġamong Ġthe Ġmost Ġwidely Ġutilized Ġalgorithms Ġfor Ġtopic Ġmodeling , Ġwe Ġapplied Ġit Ġto Ġour Ġdata . ĠHowever , Ġthe Ġco herence Ġof Ġthe Ġtopics Ġproduced Ġby ĠL DA Ġis Ġpoorer Ġthan Ġexpected . </s> To Ġaddress Ġthis Ġlack Ġof Ġco herence , Ġwe Ġapplied Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ). ĠThis Ġmethod Ġdecom poses Ġthe Ġterm - document Ġmatrix Ġinto Ġtwo Ġnon - negative Ġmat rices Ġas Ġshown Ġin ĠFigure ĠFIG REF 13 . ĠThe Ġresulting Ġnon - negative Ġmat rices Ġare Ġsuch Ġthat Ġtheir Ġproduct Ġclosely Ġapproximate Ġthe Ġoriginal Ġdata . ĠMat hematically Ġspeaking , Ġgiven Ġan Ġinput Ġmatrix Ġof Ġdocument - terms Ġ$ V $, ĠNM F Ġfinds Ġtwo Ġmat rices Ġby Ġsolving Ġthe Ġfollowing Ġequation ĠB IB REF 20 : </s> Where ĠW Ġis Ġtopic - word Ġmatrix Ġand ĠH Ġrepresents Ġtopic - document Ġmatrix . </s> NM F Ġappears Ġto Ġprovide Ġmore Ġcoherent Ġtopic Ġon Ġspecific Ġcorpor a . ĠO ' Call aghan Ġet Ġal . Ġcompared ĠL DA Ġwith ĠNM F Ġand Ġconcluded Ġthat ĠNM F Ġperforms Ġbetter Ġin Ġcorpor as Ġwith Ġspecific Ġand Ġnon - main stream Ġareas ĠB IB REF 21 . ĠOur Ġfindings Ġalign Ġwith Ġthis Ġassessment Ġand Ġthus Ġour Ġcomparison Ġof Ġtopics Ġis Ġbased Ġon ĠNM F . </s> Method Ġ: :: ĠEm otion Ġdetection </s> Prop aganda Ġeffectiveness Ġhinges Ġon Ġthe Ġemotions Ġthat Ġit Ġelic its . ĠBut Ġdetecting Ġemotion Ġin Ġtext Ġrequires Ġthat Ġtwo Ġessential Ġchallenges Ġare Ġovercome . </s> First , Ġemotions Ġare Ġgenerally Ġcomplex Ġand Ġemotional Ġrepresentation Ġmodels Ġare Ġcorrespond ingly Ġcontested . ĠDespite Ġthis , Ġsome Ġmodels Ġproposed Ġby Ġpsychologists Ġhave Ġgained Ġwide - spread Ġusage Ġthat Ġextends Ġto Ġtext - em otion Ġanalysis . ĠRobert ĠPl utch ik Ġpresented Ġa Ġmodel Ġthat Ġarranged Ġemotions Ġfrom Ġbasic Ġto Ġcomplex Ġin Ġa Ġcirc ump lex Ġas Ġshown Ġin ĠFigure ĠFIG REF 15 . ĠThe Ġmodel Ġcategor izes Ġemotions Ġinto Ġ8 Ġmain Ġsubs ets Ġand Ġwith Ġaddition Ġof Ġintensity Ġand Ġinteractions Ġit Ġwill Ġclassify Ġemotions Ġinto Ġ24 Ġclasses ĠB IB REF 23 . ĠOther Ġmodels Ġhave Ġbeen Ġdeveloped Ġto Ġcapture Ġall Ġemotions Ġby Ġdefining Ġa Ġ3 - dimensional Ġmodel Ġof Ġpleasure , Ġarousal , Ġand Ġdominance . </s> The Ġsecond Ġchallenge Ġlies Ġin Ġusing Ġtext Ġfor Ġdetecting Ġemotion Ġev oked Ġin Ġa Ġpotential Ġreader . ĠCommon Ġapproaches Ġuse Ġeither Ġlex icon - base Ġmethods Ġ( such Ġas Ġkeyword - based Ġor Ġont ology - based Ġmodel ) Ġor Ġmachine Ġlearning - base Ġmodels Ġ( usually Ġusing Ġlarge Ġcorpus Ġwith Ġlabeled Ġemotions ) ĠB IB REF 12 . ĠThese Ġmethods Ġare Ġsuited Ġto Ġaddressing Ġthe Ġemotion Ġthat Ġexist Ġin Ġthe Ġtext , Ġbut Ġin Ġthe Ġcase Ġof Ġpropaganda Ġwe Ġare Ġmore Ġinterested Ġin Ġemotions Ġthat Ġare Ġelic ited Ġin Ġthe Ġreader Ġof Ġsuch Ġmaterials . ĠThe Ġclosest Ġanalogy Ġto Ġthis Ġproblem Ġcan Ġbe Ġfound Ġin Ġresearch Ġthat Ġseek Ġto Ġmodel Ġfeelings Ġof Ġpeople Ġafter Ġreading Ġa Ġnews Ġarticle . ĠOne Ġsolution Ġfor Ġthis Ġtype Ġof Ġproblem Ġis Ġto Ġuse Ġan Ġapproach Ġcalled ĠDe pe chem ood . </s> De pe chem ood Ġis Ġa Ġlex icon - based Ġemotion Ġdetection Ġmethod Ġgathered Ġfrom Ġcrowd - annot ated Ġnews ĠB IB REF 24 . ĠDrawing Ġon Ġapproximately Ġ23 . 5 K Ġdocuments Ġwith Ġaverage Ġof Ġ500 Ġwords Ġper Ġdocument Ġfrom Ġrapp ler . com , Ġresearchers Ġasked Ġsubjects Ġto Ġreport Ġtheir Ġemotions Ġafter Ġreading Ġeach Ġarticle . ĠThey Ġthen Ġmultiplied Ġthe Ġdocument - em otion Ġmatrix Ġand Ġword - document Ġmatrix Ġto Ġderive Ġemotion - word Ġmatrix Ġfor Ġthese Ġwords . ĠDue Ġto Ġlimitations Ġof Ġtheir Ġexperiment Ġsetup , Ġthe Ġemotion Ġcategories Ġthat Ġthey Ġpresent Ġdoes Ġnot Ġexactly Ġmatch Ġthe Ġemotions Ġfrom Ġthe ĠPl utch ik Ġwheel Ġcategories . ĠHowever , Ġthey Ġstill Ġprovide Ġa Ġgood Ġsense Ġof Ġthe Ġgeneral Ġfeeling Ġof Ġan Ġindividual Ġafter Ġreading Ġan Ġarticle . ĠThe Ġemotion Ġcategories Ġof ĠDe pe chem ood Ġare : ĠAF RA ID , ĠAM US ED , ĠAN GR Y , ĠAN NO Y ED , ĠDON ' T ĠCARE , ĠH APP Y , ĠIN SP IRED , ĠS AD . ĠDe pe chem ood Ġsimply Ġcreates Ġdiction aries Ġof Ġwords Ġwhere Ġeach Ġword Ġhas Ġscores Ġbetween Ġ0 Ġand Ġ1 Ġfor Ġall Ġof Ġthese Ġ8 Ġemotion Ġcategories . ĠWe Ġpresent Ġour Ġfinding Ġusing Ġthis Ġapproach Ġin Ġthe Ġresult Ġsection . </s> Results </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġresults Ġof Ġour Ġanalysis Ġbased Ġon Ġthe Ġcontents Ġof ĠISIS Ġpropaganda Ġmaterials Ġas Ġcompared Ġto Ġarticles Ġfrom Ġthe ĠCatholic Ġwomen Ġforum . ĠWe Ġthen Ġpresent Ġthe Ġresults Ġof Ġemotion Ġanalysis Ġconducted Ġon Ġboth Ġcorpor a . </s> Results Ġ: :: ĠContent ĠAnalysis </s> After Ġpre - processing Ġthe Ġtext , Ġboth Ġcorpor a Ġwere Ġanalyzed Ġfor Ġword Ġfrequencies . ĠThese Ġword Ġfrequencies Ġhave Ġbeen Ġnormalized Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġcorpus . ĠFigure ĠFIG REF 17 Ġshows Ġthe Ġmost Ġcommon Ġwords Ġin Ġeach Ġof Ġthese Ġcorpor a . </s> A Ġcomparison Ġof Ġcommon Ġwords Ġsuggests Ġthat Ġthose Ġrelated Ġto Ġmarital Ġrelationships Ġ( Ġhusband , Ġwife , Ġetc .) Ġappear Ġin Ġboth Ġcorpor a , Ġbut Ġthe Ġreligious Ġtheme Ġof ĠISIS Ġmaterial Ġappears Ġto Ġbe Ġstronger . ĠA Ġstronger Ġcomparison Ġcan Ġbe Ġmade Ġusing Ġtopic Ġmodeling Ġtechniques Ġto Ġdiscover Ġmain Ġtopics Ġof Ġthese Ġdocuments . ĠAlthough Ġwe Ġused ĠL DA , Ġour Ġresults Ġby Ġusing ĠNM F Ġoutper form ĠL DA Ġtopics , Ġdue Ġto Ġthe Ġnature Ġof Ġthese Ġcorpor a . ĠAlso , Ġfewer Ġnumbers Ġof ĠISIS Ġdocuments Ġmight Ġcontribute Ġto Ġthe Ġcomparatively Ġworse Ġperformance . ĠTherefore , Ġwe Ġpresent Ġonly ĠNM F Ġresults . ĠBased Ġon Ġtheir Ġco herence , Ġwe Ġselected Ġ10 Ġtopics Ġfor Ġanalyzing Ġwithin Ġboth Ġcorpor as . ĠTable ĠT AB REF 18 Ġand ĠTable ĠT AB REF 19 Ġshow Ġthe Ġmost Ġimportant Ġwords Ġin Ġeach Ġtopic Ġwith Ġa Ġgeneral Ġlabel Ġthat Ġwe Ġassigned Ġto Ġthe Ġtopic Ġmanually . ĠBased Ġon Ġthe ĠNM F Ġoutput , ĠISIS Ġarticles Ġthat Ġaddress Ġwomen Ġinclude Ġtopics Ġmainly Ġabout ĠIslam , Ġwomen 's Ġrole Ġin Ġearly ĠIslam , Ġhij rah Ġ( moving Ġto Ġanother Ġland ), Ġsp ous al Ġrelations , Ġmarriage , Ġand Ġmother hood . </s> The Ġtopics Ġgenerated Ġfrom Ġthe ĠCatholic Ġwomen Ġforum Ġare Ġclearly Ġquite Ġdifferent . ĠSome , Ġhowever , Ġexist Ġin Ġboth Ġcontexts . ĠMore Ġspecifically , Ġmarriage / div orce , Ġmother hood , Ġand Ġto Ġsome Ġextent Ġsp ous al Ġrelations Ġappeared Ġin Ġboth Ġgenerated Ġtopics . ĠThis Ġsuggests Ġthat Ġwhen Ġaddressing Ġwomen Ġin Ġa Ġreligious Ġcontext , Ġthese Ġmay Ġbe Ġvery Ġbroadly Ġeffective Ġand Ġappeal Ġto Ġthe Ġfeminine Ġaudience . ĠMore Ġimportantly , Ġsuitable Ġtopic Ġmodeling Ġmethods Ġwill Ġbe Ġable Ġto Ġidentify Ġthese Ġsimilarities Ġno Ġmatter Ġthe Ġsize Ġof Ġthe Ġcorpus Ġwe Ġare Ġworking Ġwith . ĠAlthough , Ġfinding Ġthe Ġsimilarities / diff erences Ġbetween Ġtopics Ġin Ġthese Ġtwo Ġgroups Ġof Ġarticles Ġmight Ġprovide Ġsome Ġnew Ġinsights , Ġwe Ġturn Ġto Ġemotional Ġanalysis Ġto Ġalso Ġcompare Ġthe Ġemotions Ġev oked Ġin Ġthe Ġaudience . </s> Results Ġ: :: ĠEm otion ĠAnalysis </s> We Ġrely Ġon ĠDe pe chem ood Ġdiction aries Ġto Ġanalyze Ġemotions Ġin Ġboth Ġcorpor a . ĠThese Ġdiction aries Ġare Ġfreely Ġavailable Ġand Ġcome Ġin Ġmultiple Ġarrangements . ĠWe Ġused Ġa Ġversion Ġthat Ġincludes Ġwords Ġwith Ġtheir Ġpart Ġof Ġspeech Ġ( POS ) Ġtags . ĠOnly Ġwords Ġthat Ġexist Ġin Ġthe ĠDe pe chem ood Ġdictionary Ġwith Ġthe Ġsame ĠPOS Ġtag Ġare Ġconsidered Ġfor Ġour Ġanalysis . ĠWe Ġaggreg ated Ġthe Ġscore Ġfor Ġeach Ġword Ġand Ġnormalized Ġeach Ġarticle Ġby Ġemotions . ĠTo Ġbetter Ġcompare Ġthe Ġresult , Ġwe Ġadded Ġa Ġbaseline Ġof Ġ100 Ġrandom Ġarticles Ġfrom Ġa ĠReuters Ġnews Ġdataset Ġas Ġa Ġnon - religious Ġgeneral Ġresource Ġwhich Ġis Ġavailable Ġin Ġan ĠNL T K Ġpython Ġlibrary . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġaggreg ated Ġscore Ġfor Ġdifferent Ġfeelings Ġin Ġour Ġcorpor a . </s> Both ĠCatholic Ġand ĠISIS Ġrelated Ġmaterials Ġscore Ġthe Ġhighest Ġin ĠâĢ ľ inspired âĢ Ŀ Ġcategory . ĠFurthermore , Ġin Ġboth Ġcases , Ġbeing Ġafraid Ġhas Ġthe Ġlowest Ġscore . ĠHowever , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġrandom Ġnews Ġmaterial Ġsuch Ġas Ġthe ĠReuters Ġcorpus , Ġwhich Ġare Ġnot Ġthat Ġinspiring Ġand , Ġaccording Ġto Ġthis Ġmethod , Ġseems Ġto Ġcause Ġmore Ġfear Ġin Ġtheir Ġaudience . ĠWe Ġinvestigate Ġthese Ġresults Ġfurther Ġby Ġlooking Ġat Ġthe Ġmost Ġinspiring Ġwords Ġdetected Ġin Ġthese Ġtwo Ġcorpor a . ĠTable ĠT AB REF 24 Ġpresents Ġ10 Ġwords Ġthat Ġare Ġamong Ġthe Ġmost Ġinspiring Ġin Ġboth Ġcorpor a . ĠThe Ġcomparison Ġof Ġthe Ġtwo Ġlists Ġindicate Ġthat Ġthe Ġmethod Ġpicks Ġvery Ġdifferent Ġwords Ġin Ġeach Ġcorpus Ġto Ġreach Ġto Ġthe Ġsame Ġconclusion . ĠAlso , Ġwe Ġlooked Ġat Ġseparate Ġarticles Ġin Ġeach Ġof Ġthe Ġissues Ġof ĠISIS Ġmaterial Ġaddressing Ġwomen . ĠFigure ĠFIG REF 23 Ġshows Ġemotion Ġscores Ġin Ġeach Ġof Ġthe Ġ20 Ġissues Ġof ĠISIS Ġpropaganda . ĠAs Ġdemonstrated , Ġin Ġevery Ġseparate Ġarticle , Ġthis Ġmethod Ġgives Ġthe Ġhighest Ġscore Ġto Ġev oking Ġinspir ations Ġin Ġthe Ġreader . ĠAlso , Ġin Ġmost Ġof Ġthese Ġissues Ġthe Ġmethod Ġscored ĠâĢ ľ being Ġafraid âĢ Ŀ Ġas Ġthe Ġlowest Ġscore Ġin Ġeach Ġissue . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġhave Ġapplied Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto ĠISIS Ġpropaganda Ġmaterials Ġin Ġan Ġattempt Ġto Ġunderstand Ġthese Ġmaterials Ġusing Ġavailable Ġtechnologies . ĠWe Ġalso Ġcompared Ġthese Ġtexts Ġwith Ġa Ġnon - violent Ġreligious Ġgroups ' Ġ( both Ġfocusing Ġon Ġwomen Ġrelated Ġarticles ) Ġto Ġexamine Ġpossible Ġsimilarities Ġor Ġdifferences Ġin Ġtheir Ġapproaches . ĠTo Ġcompare Ġthe Ġcontents , Ġwe Ġused Ġword Ġfrequency Ġand Ġtopic Ġmodeling Ġwith ĠNM F . ĠAlso , Ġour Ġresults Ġshowed Ġthat ĠNM F Ġoutper forms ĠL DA Ġdue Ġto Ġthe Ġniche Ġdomain Ġand Ġrelatively Ġsmall Ġnumber Ġof Ġdocuments . </s> The Ġresults Ġsuggest Ġthat Ġcertain Ġtopics Ġplay Ġa Ġparticularly Ġimportant Ġroles Ġin ĠISIS Ġpropaganda Ġtargeting Ġwomen . ĠThese Ġrelate Ġto Ġthe Ġrole Ġof Ġwomen Ġin Ġearly ĠIslam , ĠIslamic Ġideology , Ġmarriage / div orce , Ġmother hood , Ġsp ous al Ġrelationships , Ġand Ġhij rah Ġ( moving Ġto Ġa Ġnew Ġland ). </s> Comp aring Ġthese Ġtopics Ġwith Ġthose Ġthat Ġappeared Ġon Ġa ĠCatholic Ġwomen Ġforum , Ġit Ġseems Ġthat Ġboth ĠISIS Ġand Ġnon - violent Ġgroups Ġuse Ġtopics Ġabout Ġmother hood , Ġsp ous al Ġrelationship , Ġand Ġmarriage / div orce Ġwhen Ġthey Ġaddress Ġwomen . ĠMoreover , Ġwe Ġused ĠDe pe chem ood Ġmethods Ġto Ġanalyze Ġthe Ġemotions Ġthat Ġthese Ġmaterials Ġare Ġlikely Ġto Ġelicit Ġin Ġreaders . ĠThe Ġresult Ġof Ġour Ġemotion Ġanalysis Ġsuggests Ġthat Ġboth Ġcorp uses Ġused Ġwords Ġthat Ġaim Ġto Ġinspire Ġreaders Ġwhile Ġavoiding Ġfear . ĠHowever , Ġthe Ġactual Ġwords Ġthat Ġlead Ġto Ġthese Ġeffects Ġare Ġvery Ġdifferent Ġin Ġthe Ġtwo Ġcontexts . ĠOverall , Ġour Ġfindings Ġindicate Ġthat , Ġusing Ġproper Ġmethods , Ġautomated Ġanalysis Ġof Ġlarge Ġbodies Ġof Ġtextual Ġdata Ġcan Ġprovide Ġnovel Ġinsight Ġinsight Ġinto Ġextremist Ġpropaganda Ġthat Ġcan Ġassist Ġthe Ġcounterterrorism Ġcommunity .
Introduction Since its rise in 2013, the Islamic State of Iraq and Syria (ISIS) has utilized the Internet to spread its ideology, radicalize individuals, and recruit them to their cause. In comparison to other Islamic extremist groups, ISIS' use of technology was more sophisticated, voluminous, and targeted. For example, during ISIS' advance toward Mosul, ISIS related accounts tweeted some 40,000 tweets in one day BIBREF0.However, this heavy engagement forced social media platforms to institute policies to prevent unchecked dissemination of terrorist propaganda to their users, forcing ISIS to adapt to other means to reach their target audience. One such approach was the publication of online magazines in different languages including English. Although discontinued now, these online resources provided a window into ISIS ideology, recruitment, and how they wanted the world to perceive them. For example, after predominantly recruiting men, ISIS began to also include articles in their magazines that specifically addressed women. ISIS encouraged women to join the group by either traveling to the caliphate or by carrying out domestic attacks on behalf of ISIS in their respective countries. This tactical change concerned both practitioners and researchers in the counterterrorism community. New advancements in data science can shed light on exactly how the targeting of women in extremist propaganda works and whether it differs significantly from mainstream religious rhetoric. We utilize natural language processing methods to answer three questions: What are the main topics in women-related articles in ISIS' online magazines? What similarities and/or differences do these topics have with non-violent, non-Islamic religious material addressed specifically to women? What kind of emotions do these articles evoke in their readers and are there similarities in the emotions evoked from both ISIS and non-violent religious materials? As these questions suggest, to understand what, if anything, makes extremist appeals distinctive, we need a point of comparison in terms of the outreach efforts to women from a mainstream, non-violent religious group. For this purpose, we rely on an online Catholic women's forum. Comparison between Catholic material and the content of ISIS' online magazines allows for novel insight into the distinctiveness of extremist rhetoric when targeted towards the female population. To accomplish this task, we employ topic modeling and an unsupervised emotion detection method. The rest of the paper is organized as follows: in Section SECREF2, we review related works on ISIS propaganda and applications of natural language methods. Section SECREF3 describes data collection and pre-processing. Section SECREF4 describes in detail the approach. Section SECREF5 reports the results, and finally, Section SECREF6 presents the conclusion. Related Work Soon after ISIS emerged and declared its caliphate, counterterrorism practitioners and political science researchers started to turn their attention towards understanding how the group operated. Researchers investigated the origins of ISIS, its leadership, funding, and how they rose became a globally dominant non-state actor BIBREF1. This interest in the organization's distinctiveness immediately led to inquiries into ISIS' rhetoric, particularly their use of social media and online resources in recruitment and ideological dissemination. For example, Al-Tamimi examines how ISIS differentiated itself from other jihadist movements by using social media with unprecedented efficiency to improve its image with locals BIBREF2. One of ISIS' most impressive applications of its online prowess was in the recruitment process. The organization has used a variety of materials, especially videos, to recruit both foreign and local fighters. Research shows that ISIS propaganda is designed to portray the organization as a provider of justice, governance, and development in a fashion that resonates with young westerners BIBREF3. This propaganda machine has become a significant area of research, with scholars such as Winter identifying key themes in it such as brutality, mercy, victimhood, war, belonging and utopianism. BIBREF4. However, there has been insufficient attention focused on how these approaches have particularly targeted and impacted women. This is significant given that scholars have identified the distinctiveness of this population when it comes to nearly all facets of terrorism. ISIS used different types of media to propagate its messages, such as videos, images, texts, and even music. Twitter was particularly effective and the Arabic Twitter app allowed ISIS to tweet extensively without triggering spam-detection mechanisms the platform uses BIBREF0. Scholars followed the resulting trove of data and this became the preeminent way in which they assess ISIS messages. For example, in BIBREF5 they use both lexical analysis of tweets as well as social network analysis to examine ISIS support or opposition on Twitter. Other researchers used data mining techniques to detect pro-ISIS user divergence behavior at various points in time BIBREF6. By looking at these works, the impact of using text mining and lexical analysis to address important questions becomes obvious. Proper usage of these tools allows the research community to analyze big chunks of unstructured data. This approach, however, became less productive as the social media networks began cracking down and ISIS recruiters moved off of them. With their ability to operate freely on social media now curtailed, ISIS recruiters and propagandists increased their attentiveness to another longstanding tool–English language online magazines targeting western audiences. Al Hayat, the media wing of ISIS, published multiple online magazines in different languages including English. The English online magazine of ISIS was named Dabiq and first appeared on the dark web on July 2014 and continued publishing for 15 issues. This publication was followed by Rumiyah which produced 13 English language issues through September 2017. The content of these magazines provides a valuable but underutilized resource for understanding ISIS strategies and how they appeal to recruits, specifically English-speaking audiences. They also provide a way to compare ISIS' approach with other radical groups. Ingram compared Dabiq contents with Inspire (Al Qaeda publication) and suggested that Al Qaeda heavily emphasized identity-choice, while ISIS' messages were more balanced between identity-choice and rational-choice BIBREF7. In another research paper, Wignell et al. BIBREF8 compared Dabiq and Rumiah by examining their style and what both magazine messages emphasized. Despite the volume of research on these magazines, only a few researchers used lexical analysis and mostly relied on experts' opinions. BIBREF9 is one exception to this approach where they used word frequency on 11 issues of Dabiq publications and compared attributes such as anger, anxiety, power, motive, etc. This paper seeks to establish how ISIS specifically tailored propaganda targeting western women, who became a particular target for the organization as the “caliphate” expanded. Although the number of recruits is unknown, in 2015 it was estimated that around 10 percent of all western recruits were female BIBREF10. Some researchers have attempted to understand how ISIS propaganda targets women. Kneip, for example, analyzed women's desire to join as a form of emancipation BIBREF11. We extend that line of inquiry by leveraging technology to answer key outstanding questions about the targeting of women in ISIS propaganda. To further assess how ISIS propaganda might affect women, we used emotion detection methods on these texts. Emotion detection techniques are mostly divided into lexicon-base or machine learning-base methods. Lexicon-base methods rely on several lexicons while machine learning (ML) methods use algorithm to detect the elation of texts as inputs and emotions as the target, usually trained on a large corpus. Unsupervised methods usually use Non-negative matrix factorization (NMF) and Latent Semantic Analysis (LSA) BIBREF12 approaches. An important distinction that should be made when using text for emotion detection is that emotion detected in the text and the emotion evoked in the reader of that text might differ. In the case of propaganda, it is more desirable to detect possible emotions that will be evoked in a hypothetical reader. In the next section, we describe methods to analyze content and technique to find evoked emotions in a potential reader using available natural language processing tools. Data Collection & Pre-Processing ::: Data collection Finding useful collections of texts where ISIS targets women is a challenging task. Most of the available material are not reflecting ISIS' official point of view or they do not talk specifically about women. However, ISIS' online magazines are valuable resources for understanding how the organization attempts to appeal to western audiences, particularly women. Looking through both Dabiq and Rumiyah, many issues of the magazines contain articles specifically addressing women, usually with “ to our sisters ” incorporated into the title. Seven out of fifteen Dabiq issues and all thirteen issues of Rumiyah contain articles targeting women, clearly suggesting an increase in attention to women over time. We converted all the ISIS magazines to texts using pdf readers and all articles that addressed women in both magazines (20 articles) were selected for our analysis. To facilitate comparison with a mainstream, non-violent religious group, we collected articles from catholicwomensforum.org, an online resource catering to Catholic women. We scrapped 132 articles from this domain. While this number is large, the articles themselves are much shorter than those published by ISIS. These texts were pre-processed by tokenizing the sentences and eliminating non-word tokens and punctuation marks. Also, all words turned into lower case and numbers and English stop words such as “our, is, did, can, etc. ” have been removed from the produced tokens. For the emotion analysis part, we used a spacy library as part of speech tagging to identify the exact role of words in the sentence. A word and its role have been used to look for emotional values of that word in the same role in the sentence. Data Collection & Pre-Processing ::: Pre-Processing ::: Text Cleaning and Pre-processing Most text and document datasets contain many unnecessary words such as stopwords, misspelling, slang, etc. In many algorithms, especially statistical and probabilistic learning algorithms, noise and unnecessary features can have adverse effects on system performance. In this section, we briefly explain some techniques and methods for text cleaning and pre-processing text datasets BIBREF13. Data Collection & Pre-Processing ::: Pre-Processing ::: Tokenization Tokenization is a pre-processing method which breaks a stream of text into words, phrases, symbols, or other meaningful elements called tokens BIBREF14. The main goal of this step is to investigate the words in a sentence BIBREF14. Both text classification and text mining requires a parser which processes the tokenization of the documents; for example: sentence BIBREF15 : After sleeping for four hours, he decided to sleep for another four. In this case, the tokens are as follows: {“After” “sleeping” “for” “four” “hours” “he” “decided” “to” “sleep” “for” “another” “four”}. Data Collection & Pre-Processing ::: Pre-Processing ::: Stop words Text and document classification includes many words which do not hold important significance to be used in classification algorithms such as {“a”, “about”, “above”, “across”, “after”, “afterwards”, “again”,$\hdots $}. The most common technique to deal with these words is to remove them from the texts and documents BIBREF16. Data Collection & Pre-Processing ::: Pre-Processing ::: Term Frequency-Inverse Document Frequency K Sparck Jones BIBREF17 proposed inverse document frequency (IDF) as a method to be used in conjunction with term frequency in order to lessen the effect of implicitly common words in the corpus. IDF assigns a higher weight to words with either high frequency or low frequency term in the document. This combination of TF and IDF is well known as term frequency-inverse document frequency (tf-idf). The mathematical representation of the weight of a term in a document by tf-idf is given in Equation DISPLAY_FORM10. Here N is the number of documents and $df(t)$ is the number of documents containing the term t in the corpus. The first term in Equation DISPLAY_FORM10 improves the recall while the second term improves the precision of the word embedding BIBREF18. Although tf-idf tries to overcome the problem of common terms in the document, it still suffers from some other descriptive limitations. Namely, tf-idf cannot account for the similarity between the words in the document since each word is independently presented as an index. However, with the development of more complex models in recent years, new methods, such as word embedding, have been presented that can incorporate concepts such as similarity of words and part of speech tagging. Method In this section, we describe our methods used for comparing topics and evoked emotions in both ISIS and non-violent religious materials. Method ::: Content Analysis The key task in comparing ISIS material with that of a non-violent group involves analyzing the content of these two corpora to identify the topics. For our analysis, we considered a simple uni-gram model where each word is considered as a single unit. Understanding what words appear most frequently provides a simple metric for comparison. To do so we normalized the count of words with the number of words in each corpora to account for the size of each corpus. It should be noted, however, that a drawback of word frequencies is that there might be some dominant words that will overcome all the other contents without conveying much information. Topic modeling methods are the more powerful technique for understanding the contents of a corpus. These methods try to discover abstract topics in a corpus and reveal hidden semantic structures in a collection of documents. The most popular topic modeling methods use probabilistic approaches such as probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA). LDA is a generalization of pLSA where documents are considered as a mixture of topics and the distribution of topics is governed by a Dirichlet prior ($\alpha $). Figure FIGREF12 shows plate notation of general LDA structure where $\beta $ represents prior of word distribution per topic and $\theta $ refers to topics distribution for documents BIBREF19. Since LDA is among the most widely utilized algorithms for topic modeling, we applied it to our data. However, the coherence of the topics produced by LDA is poorer than expected. To address this lack of coherence, we applied non-negative matrix factorization (NMF). This method decomposes the term-document matrix into two non-negative matrices as shown in Figure FIGREF13. The resulting non-negative matrices are such that their product closely approximate the original data. Mathematically speaking, given an input matrix of document-terms $V$, NMF finds two matrices by solving the following equation BIBREF20: Where W is topic-word matrix and H represents topic-document matrix. NMF appears to provide more coherent topic on specific corpora. O'Callaghan et al. compared LDA with NMF and concluded that NMF performs better in corporas with specific and non-mainstream areas BIBREF21. Our findings align with this assessment and thus our comparison of topics is based on NMF. Method ::: Emotion detection Propaganda effectiveness hinges on the emotions that it elicits. But detecting emotion in text requires that two essential challenges are overcome. First, emotions are generally complex and emotional representation models are correspondingly contested. Despite this, some models proposed by psychologists have gained wide-spread usage that extends to text-emotion analysis. Robert Plutchik presented a model that arranged emotions from basic to complex in a circumplex as shown in Figure FIGREF15. The model categorizes emotions into 8 main subsets and with addition of intensity and interactions it will classify emotions into 24 classes BIBREF23. Other models have been developed to capture all emotions by defining a 3-dimensional model of pleasure, arousal, and dominance. The second challenge lies in using text for detecting emotion evoked in a potential reader. Common approaches use either lexicon-base methods (such as keyword-based or ontology-based model) or machine learning-base models (usually using large corpus with labeled emotions) BIBREF12. These methods are suited to addressing the emotion that exist in the text, but in the case of propaganda we are more interested in emotions that are elicited in the reader of such materials. The closest analogy to this problem can be found in research that seek to model feelings of people after reading a news article. One solution for this type of problem is to use an approach called Depechemood. Depechemood is a lexicon-based emotion detection method gathered from crowd-annotated news BIBREF24. Drawing on approximately 23.5K documents with average of 500 words per document from rappler.com, researchers asked subjects to report their emotions after reading each article. They then multiplied the document-emotion matrix and word-document matrix to derive emotion-word matrix for these words. Due to limitations of their experiment setup, the emotion categories that they present does not exactly match the emotions from the Plutchik wheel categories. However, they still provide a good sense of the general feeling of an individual after reading an article. The emotion categories of Depechemood are: AFRAID, AMUSED, ANGRY, ANNOYED, DON'T CARE, HAPPY, INSPIRED, SAD. Depechemood simply creates dictionaries of words where each word has scores between 0 and 1 for all of these 8 emotion categories. We present our finding using this approach in the result section. Results In this section, we present the results of our analysis based on the contents of ISIS propaganda materials as compared to articles from the Catholic women forum. We then present the results of emotion analysis conducted on both corpora. Results ::: Content Analysis After pre-processing the text, both corpora were analyzed for word frequencies. These word frequencies have been normalized by the number of words in each corpus. Figure FIGREF17 shows the most common words in each of these corpora. A comparison of common words suggests that those related to marital relationships ( husband, wife, etc.) appear in both corpora, but the religious theme of ISIS material appears to be stronger. A stronger comparison can be made using topic modeling techniques to discover main topics of these documents. Although we used LDA, our results by using NMF outperform LDA topics, due to the nature of these corpora. Also, fewer numbers of ISIS documents might contribute to the comparatively worse performance. Therefore, we present only NMF results. Based on their coherence, we selected 10 topics for analyzing within both corporas. Table TABREF18 and Table TABREF19 show the most important words in each topic with a general label that we assigned to the topic manually. Based on the NMF output, ISIS articles that address women include topics mainly about Islam, women's role in early Islam, hijrah (moving to another land), spousal relations, marriage, and motherhood. The topics generated from the Catholic women forum are clearly quite different. Some, however, exist in both contexts. More specifically, marriage/divorce, motherhood, and to some extent spousal relations appeared in both generated topics. This suggests that when addressing women in a religious context, these may be very broadly effective and appeal to the feminine audience. More importantly, suitable topic modeling methods will be able to identify these similarities no matter the size of the corpus we are working with. Although, finding the similarities/differences between topics in these two groups of articles might provide some new insights, we turn to emotional analysis to also compare the emotions evoked in the audience. Results ::: Emotion Analysis We rely on Depechemood dictionaries to analyze emotions in both corpora. These dictionaries are freely available and come in multiple arrangements. We used a version that includes words with their part of speech (POS) tags. Only words that exist in the Depechemood dictionary with the same POS tag are considered for our analysis. We aggregated the score for each word and normalized each article by emotions. To better compare the result, we added a baseline of 100 random articles from a Reuters news dataset as a non-religious general resource which is available in an NLTK python library. Figure FIGREF22 shows the aggregated score for different feelings in our corpora. Both Catholic and ISIS related materials score the highest in “inspired” category. Furthermore, in both cases, being afraid has the lowest score. However, this is not the case for random news material such as the Reuters corpus, which are not that inspiring and, according to this method, seems to cause more fear in their audience. We investigate these results further by looking at the most inspiring words detected in these two corpora. Table TABREF24 presents 10 words that are among the most inspiring in both corpora. The comparison of the two lists indicate that the method picks very different words in each corpus to reach to the same conclusion. Also, we looked at separate articles in each of the issues of ISIS material addressing women. Figure FIGREF23 shows emotion scores in each of the 20 issues of ISIS propaganda. As demonstrated, in every separate article, this method gives the highest score to evoking inspirations in the reader. Also, in most of these issues the method scored “being afraid” as the lowest score in each issue. Conclusion and Future Work In this paper, we have applied natural language processing methods to ISIS propaganda materials in an attempt to understand these materials using available technologies. We also compared these texts with a non-violent religious groups' (both focusing on women related articles) to examine possible similarities or differences in their approaches. To compare the contents, we used word frequency and topic modeling with NMF. Also, our results showed that NMF outperforms LDA due to the niche domain and relatively small number of documents. The results suggest that certain topics play a particularly important roles in ISIS propaganda targeting women. These relate to the role of women in early Islam, Islamic ideology, marriage/divorce, motherhood, spousal relationships, and hijrah (moving to a new land). Comparing these topics with those that appeared on a Catholic women forum, it seems that both ISIS and non-violent groups use topics about motherhood, spousal relationship, and marriage/divorce when they address women. Moreover, we used Depechemood methods to analyze the emotions that these materials are likely to elicit in readers. The result of our emotion analysis suggests that both corpuses used words that aim to inspire readers while avoiding fear. However, the actual words that lead to these effects are very different in the two contexts. Overall, our findings indicate that, using proper methods, automated analysis of large bodies of textual data can provide novel insight insight into extremist propaganda that can assist the counterterrorism community.
How id Depechemood trained?
6bbbb9933aab97ce2342200447c6322527427061
How Ġid ĠDe pe chem ood Ġtrained ?
By Ġmultiplying Ġcrowd - annot ated Ġdocument - em otion Ġmatrix Ġwith Ġemotion - word Ġmatrix . Ġ
<s> How Ġid ĠDe pe chem ood Ġtrained ? </s> Introduction </s> Since Ġits Ġrise Ġin Ġ2013 , Ġthe ĠIslamic ĠState Ġof ĠIraq Ġand ĠSyria Ġ( ISIS ) Ġhas Ġutilized Ġthe ĠInternet Ġto Ġspread Ġits Ġideology , Ġradical ize Ġindividuals , Ġand Ġrecruit Ġthem Ġto Ġtheir Ġcause . ĠIn Ġcomparison Ġto Ġother ĠIslamic Ġextremist Ġgroups , ĠISIS ' Ġuse Ġof Ġtechnology Ġwas Ġmore Ġsophisticated , Ġvol umin ous , Ġand Ġtargeted . ĠFor Ġexample , Ġduring ĠISIS ' Ġadvance Ġtoward ĠMosul , ĠISIS Ġrelated Ġaccounts Ġtweeted Ġsome Ġ40 , 000 Ġtweets Ġin Ġone Ġday ĠB IB REF 0 . However , Ġthis Ġheavy Ġengagement Ġforced Ġsocial Ġmedia Ġplatforms Ġto Ġinstitute Ġpolicies Ġto Ġprevent Ġunchecked Ġdissemination Ġof Ġterrorist Ġpropaganda Ġto Ġtheir Ġusers , Ġforcing ĠISIS Ġto Ġadapt Ġto Ġother Ġmeans Ġto Ġreach Ġtheir Ġtarget Ġaudience . </s> One Ġsuch Ġapproach Ġwas Ġthe Ġpublication Ġof Ġonline Ġmagazines Ġin Ġdifferent Ġlanguages Ġincluding ĠEnglish . ĠAlthough Ġdiscontinued Ġnow , Ġthese Ġonline Ġresources Ġprovided Ġa Ġwindow Ġinto ĠISIS Ġideology , Ġrecruitment , Ġand Ġhow Ġthey Ġwanted Ġthe Ġworld Ġto Ġperceive Ġthem . ĠFor Ġexample , Ġafter Ġpredominantly Ġrecruiting Ġmen , ĠISIS Ġbegan Ġto Ġalso Ġinclude Ġarticles Ġin Ġtheir Ġmagazines Ġthat Ġspecifically Ġaddressed Ġwomen . ĠISIS Ġencouraged Ġwomen Ġto Ġjoin Ġthe Ġgroup Ġby Ġeither Ġtraveling Ġto Ġthe Ġcaliphate Ġor Ġby Ġcarrying Ġout Ġdomestic Ġattacks Ġon Ġbehalf Ġof ĠISIS Ġin Ġtheir Ġrespective Ġcountries . ĠThis Ġtactical Ġchange Ġconcerned Ġboth Ġpractitioners Ġand Ġresearchers Ġin Ġthe Ġcounterterrorism Ġcommunity . ĠNew Ġadvancements Ġin Ġdata Ġscience Ġcan Ġshed Ġlight Ġon Ġexactly Ġhow Ġthe Ġtargeting Ġof Ġwomen Ġin Ġextremist Ġpropaganda Ġworks Ġand Ġwhether Ġit Ġdiffers Ġsignificantly Ġfrom Ġmainstream Ġreligious Ġrhetoric . </s> We Ġutilize Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto Ġanswer Ġthree Ġquestions : </s> What Ġare Ġthe Ġmain Ġtopics Ġin Ġwomen - related Ġarticles Ġin ĠISIS ' Ġonline Ġmagazines ? </s> What Ġsimilarities Ġand / or Ġdifferences Ġdo Ġthese Ġtopics Ġhave Ġwith Ġnon - violent , Ġnon - Islamic Ġreligious Ġmaterial Ġaddressed Ġspecifically Ġto Ġwomen ? </s> What Ġkind Ġof Ġemotions Ġdo Ġthese Ġarticles Ġevoke Ġin Ġtheir Ġreaders Ġand Ġare Ġthere Ġsimilarities Ġin Ġthe Ġemotions Ġev oked Ġfrom Ġboth ĠISIS Ġand Ġnon - violent Ġreligious Ġmaterials ? </s> As Ġthese Ġquestions Ġsuggest , Ġto Ġunderstand Ġwhat , Ġif Ġanything , Ġmakes Ġextremist Ġappeals Ġdistinctive , Ġwe Ġneed Ġa Ġpoint Ġof Ġcomparison Ġin Ġterms Ġof Ġthe Ġoutreach Ġefforts Ġto Ġwomen Ġfrom Ġa Ġmainstream , Ġnon - violent Ġreligious Ġgroup . ĠFor Ġthis Ġpurpose , Ġwe Ġrely Ġon Ġan Ġonline ĠCatholic Ġwomen 's Ġforum . ĠComparison Ġbetween ĠCatholic Ġmaterial Ġand Ġthe Ġcontent Ġof ĠISIS ' Ġonline Ġmagazines Ġallows Ġfor Ġnovel Ġinsight Ġinto Ġthe Ġdistinct iveness Ġof Ġextremist Ġrhetoric Ġwhen Ġtargeted Ġtowards Ġthe Ġfemale Ġpopulation . ĠTo Ġaccomplish Ġthis Ġtask , Ġwe Ġemploy Ġtopic Ġmodeling Ġand Ġan Ġun super vised Ġemotion Ġdetection Ġmethod . </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows : Ġin ĠSection ĠSEC REF 2 , Ġwe Ġreview Ġrelated Ġworks Ġon ĠISIS Ġpropaganda Ġand Ġapplications Ġof Ġnatural Ġlanguage Ġmethods . ĠSection ĠSEC REF 3 Ġdescribes Ġdata Ġcollection Ġand Ġpre - processing . ĠSection ĠSEC REF 4 Ġdescribes Ġin Ġdetail Ġthe Ġapproach . ĠSection ĠSEC REF 5 Ġreports Ġthe Ġresults , Ġand Ġfinally , ĠSection ĠSEC REF 6 Ġpresents Ġthe Ġconclusion . </s> Related ĠWork </s> Soon Ġafter ĠISIS Ġemerged Ġand Ġdeclared Ġits Ġcaliphate , Ġcounterterrorism Ġpractitioners Ġand Ġpolitical Ġscience Ġresearchers Ġstarted Ġto Ġturn Ġtheir Ġattention Ġtowards Ġunderstanding Ġhow Ġthe Ġgroup Ġoperated . ĠResearchers Ġinvestigated Ġthe Ġorigins Ġof ĠISIS , Ġits Ġleadership , Ġfunding , Ġand Ġhow Ġthey Ġrose Ġbecame Ġa Ġglobally Ġdominant Ġnon - state Ġactor ĠB IB REF 1 . ĠThis Ġinterest Ġin Ġthe Ġorganization 's Ġdistinct iveness Ġimmediately Ġled Ġto Ġinquiries Ġinto ĠISIS ' Ġrhetoric , Ġparticularly Ġtheir Ġuse Ġof Ġsocial Ġmedia Ġand Ġonline Ġresources Ġin Ġrecruitment Ġand Ġideological Ġdissemination . ĠFor Ġexample , ĠAl - Tam imi Ġexamines Ġhow ĠISIS Ġdifferentiated Ġitself Ġfrom Ġother Ġjihadist Ġmovements Ġby Ġusing Ġsocial Ġmedia Ġwith Ġunprecedented Ġefficiency Ġto Ġimprove Ġits Ġimage Ġwith Ġlocals ĠB IB REF 2 . ĠOne Ġof ĠISIS ' Ġmost Ġimpressive Ġapplications Ġof Ġits Ġonline Ġprowess Ġwas Ġin Ġthe Ġrecruitment Ġprocess . ĠThe Ġorganization Ġhas Ġused Ġa Ġvariety Ġof Ġmaterials , Ġespecially Ġvideos , Ġto Ġrecruit Ġboth Ġforeign Ġand Ġlocal Ġfighters . ĠResearch Ġshows Ġthat ĠISIS Ġpropaganda Ġis Ġdesigned Ġto Ġportray Ġthe Ġorganization Ġas Ġa Ġprovider Ġof Ġjustice , Ġgovernance , Ġand Ġdevelopment Ġin Ġa Ġfashion Ġthat Ġreson ates Ġwith Ġyoung Ġwestern ers ĠB IB REF 3 . ĠThis Ġpropaganda Ġmachine Ġhas Ġbecome Ġa Ġsignificant Ġarea Ġof Ġresearch , Ġwith Ġscholars Ġsuch Ġas ĠWinter Ġidentifying Ġkey Ġthemes Ġin Ġit Ġsuch Ġas Ġbrutality , Ġmercy , Ġvictim hood , Ġwar , Ġbelonging Ġand Ġutopian ism . ĠB IB REF 4 . ĠHowever , Ġthere Ġhas Ġbeen Ġinsufficient Ġattention Ġfocused Ġon Ġhow Ġthese Ġapproaches Ġhave Ġparticularly Ġtargeted Ġand Ġimpacted Ġwomen . ĠThis Ġis Ġsignificant Ġgiven Ġthat Ġscholars Ġhave Ġidentified Ġthe Ġdistinct iveness Ġof Ġthis Ġpopulation Ġwhen Ġit Ġcomes Ġto Ġnearly Ġall Ġfacets Ġof Ġterrorism . </s> ISIS Ġused Ġdifferent Ġtypes Ġof Ġmedia Ġto Ġpropagate Ġits Ġmessages , Ġsuch Ġas Ġvideos , Ġimages , Ġtexts , Ġand Ġeven Ġmusic . ĠTwitter Ġwas Ġparticularly Ġeffective Ġand Ġthe ĠArabic ĠTwitter Ġapp Ġallowed ĠISIS Ġto Ġtweet Ġextensively Ġwithout Ġtriggering Ġspam - det ection Ġmechanisms Ġthe Ġplatform Ġuses ĠB IB REF 0 . ĠScholars Ġfollowed Ġthe Ġresulting Ġtrove Ġof Ġdata Ġand Ġthis Ġbecame Ġthe Ġpre em inent Ġway Ġin Ġwhich Ġthey Ġassess ĠISIS Ġmessages . ĠFor Ġexample , Ġin ĠB IB REF 5 Ġthey Ġuse Ġboth Ġlex ical Ġanalysis Ġof Ġtweets Ġas Ġwell Ġas Ġsocial Ġnetwork Ġanalysis Ġto Ġexamine ĠISIS Ġsupport Ġor Ġopposition Ġon ĠTwitter . ĠOther Ġresearchers Ġused Ġdata Ġmining Ġtechniques Ġto Ġdetect Ġpro - ISIS Ġuser Ġdivergence Ġbehavior Ġat Ġvarious Ġpoints Ġin Ġtime ĠB IB REF 6 . ĠBy Ġlooking Ġat Ġthese Ġworks , Ġthe Ġimpact Ġof Ġusing Ġtext Ġmining Ġand Ġlex ical Ġanalysis Ġto Ġaddress Ġimportant Ġquestions Ġbecomes Ġobvious . ĠProper Ġusage Ġof Ġthese Ġtools Ġallows Ġthe Ġresearch Ġcommunity Ġto Ġanalyze Ġbig Ġchunks Ġof Ġun struct ured Ġdata . ĠThis Ġapproach , Ġhowever , Ġbecame Ġless Ġproductive Ġas Ġthe Ġsocial Ġmedia Ġnetworks Ġbegan Ġcracking Ġdown Ġand ĠISIS Ġrecruit ers Ġmoved Ġoff Ġof Ġthem . </s> With Ġtheir Ġability Ġto Ġoperate Ġfreely Ġon Ġsocial Ġmedia Ġnow Ġcur tailed , ĠISIS Ġrecruit ers Ġand Ġpropag and ists Ġincreased Ġtheir Ġattent iveness Ġto Ġanother Ġlongstanding Ġtool âĢĵ English Ġlanguage Ġonline Ġmagazines Ġtargeting Ġwestern Ġaudiences . ĠAl ĠHay at , Ġthe Ġmedia Ġwing Ġof ĠISIS , Ġpublished Ġmultiple Ġonline Ġmagazines Ġin Ġdifferent Ġlanguages Ġincluding ĠEnglish . ĠThe ĠEnglish Ġonline Ġmagazine Ġof ĠISIS Ġwas Ġnamed ĠD abi q Ġand Ġfirst Ġappeared Ġon Ġthe Ġdark Ġweb Ġon ĠJuly Ġ2014 Ġand Ġcontinued Ġpublishing Ġfor Ġ15 Ġissues . ĠThis Ġpublication Ġwas Ġfollowed Ġby ĠRum iyah Ġwhich Ġproduced Ġ13 ĠEnglish Ġlanguage Ġissues Ġthrough ĠSeptember Ġ2017 . ĠThe Ġcontent Ġof Ġthese Ġmagazines Ġprovides Ġa Ġvaluable Ġbut Ġunder util ized Ġresource Ġfor Ġunderstanding ĠISIS Ġstrategies Ġand Ġhow Ġthey Ġappeal Ġto Ġrecruits , Ġspecifically ĠEnglish - speaking Ġaudiences . ĠThey Ġalso Ġprovide Ġa Ġway Ġto Ġcompare ĠISIS ' Ġapproach Ġwith Ġother Ġradical Ġgroups . ĠIngram Ġcompared ĠD abi q Ġcontents Ġwith ĠInsp ire Ġ( Al ĠQaeda Ġpublication ) Ġand Ġsuggested Ġthat ĠAl ĠQaeda Ġheavily Ġemphasized Ġidentity - choice , Ġwhile ĠISIS ' Ġmessages Ġwere Ġmore Ġbalanced Ġbetween Ġidentity - choice Ġand Ġrational - choice ĠB IB REF 7 . ĠIn Ġanother Ġresearch Ġpaper , ĠW ign ell Ġet Ġal . ĠB IB REF 8 Ġcompared ĠD abi q Ġand ĠRum iah Ġby Ġexamining Ġtheir Ġstyle Ġand Ġwhat Ġboth Ġmagazine Ġmessages Ġemphasized . ĠDespite Ġthe Ġvolume Ġof Ġresearch Ġon Ġthese Ġmagazines , Ġonly Ġa Ġfew Ġresearchers Ġused Ġlex ical Ġanalysis Ġand Ġmostly Ġrelied Ġon Ġexperts ' Ġopinions . ĠB IB REF 9 Ġis Ġone Ġexception Ġto Ġthis Ġapproach Ġwhere Ġthey Ġused Ġword Ġfrequency Ġon Ġ11 Ġissues Ġof ĠD abi q Ġpublications Ġand Ġcompared Ġattributes Ġsuch Ġas Ġanger , Ġanxiety , Ġpower , Ġmotive , Ġetc . </s> This Ġpaper Ġseeks Ġto Ġestablish Ġhow ĠISIS Ġspecifically Ġtailored Ġpropaganda Ġtargeting Ġwestern Ġwomen , Ġwho Ġbecame Ġa Ġparticular Ġtarget Ġfor Ġthe Ġorganization Ġas Ġthe ĠâĢ ľ cal iphate âĢ Ŀ Ġexpanded . ĠAlthough Ġthe Ġnumber Ġof Ġrecruits Ġis Ġunknown , Ġin Ġ2015 Ġit Ġwas Ġestimated Ġthat Ġaround Ġ10 Ġpercent Ġof Ġall Ġwestern Ġrecruits Ġwere Ġfemale ĠB IB REF 10 . ĠSome Ġresearchers Ġhave Ġattempted Ġto Ġunderstand Ġhow ĠISIS Ġpropaganda Ġtargets Ġwomen . ĠK ne ip , Ġfor Ġexample , Ġanalyzed Ġwomen 's Ġdesire Ġto Ġjoin Ġas Ġa Ġform Ġof Ġemancipation ĠB IB REF 11 . ĠWe Ġextend Ġthat Ġline Ġof Ġinquiry Ġby Ġleveraging Ġtechnology Ġto Ġanswer Ġkey Ġoutstanding Ġquestions Ġabout Ġthe Ġtargeting Ġof Ġwomen Ġin ĠISIS Ġpropaganda . </s> To Ġfurther Ġassess Ġhow ĠISIS Ġpropaganda Ġmight Ġaffect Ġwomen , Ġwe Ġused Ġemotion Ġdetection Ġmethods Ġon Ġthese Ġtexts . ĠEm otion Ġdetection Ġtechniques Ġare Ġmostly Ġdivided Ġinto Ġlex icon - base Ġor Ġmachine Ġlearning - base Ġmethods . ĠLex icon - base Ġmethods Ġrely Ġon Ġseveral Ġlex icons Ġwhile Ġmachine Ġlearning Ġ( ML ) Ġmethods Ġuse Ġalgorithm Ġto Ġdetect Ġthe Ġel ation Ġof Ġtexts Ġas Ġinputs Ġand Ġemotions Ġas Ġthe Ġtarget , Ġusually Ġtrained Ġon Ġa Ġlarge Ġcorpus . ĠUn super vised Ġmethods Ġusually Ġuse ĠNon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġand ĠLat ent ĠSem antic ĠAnalysis Ġ( L SA ) ĠB IB REF 12 Ġapproaches . ĠAn Ġimportant Ġdistinction Ġthat Ġshould Ġbe Ġmade Ġwhen Ġusing Ġtext Ġfor Ġemotion Ġdetection Ġis Ġthat Ġemotion Ġdetected Ġin Ġthe Ġtext Ġand Ġthe Ġemotion Ġev oked Ġin Ġthe Ġreader Ġof Ġthat Ġtext Ġmight Ġdiffer . ĠIn Ġthe Ġcase Ġof Ġpropaganda , Ġit Ġis Ġmore Ġdesirable Ġto Ġdetect Ġpossible Ġemotions Ġthat Ġwill Ġbe Ġev oked Ġin Ġa Ġhypothetical Ġreader . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdescribe Ġmethods Ġto Ġanalyze Ġcontent Ġand Ġtechnique Ġto Ġfind Ġev oked Ġemotions Ġin Ġa Ġpotential Ġreader Ġusing Ġavailable Ġnatural Ġlanguage Ġprocessing Ġtools . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠData Ġcollection </s> Finding Ġuseful Ġcollections Ġof Ġtexts Ġwhere ĠISIS Ġtargets Ġwomen Ġis Ġa Ġchallenging Ġtask . ĠMost Ġof Ġthe Ġavailable Ġmaterial Ġare Ġnot Ġreflecting ĠISIS ' Ġofficial Ġpoint Ġof Ġview Ġor Ġthey Ġdo Ġnot Ġtalk Ġspecifically Ġabout Ġwomen . ĠHowever , ĠISIS ' Ġonline Ġmagazines Ġare Ġvaluable Ġresources Ġfor Ġunderstanding Ġhow Ġthe Ġorganization Ġattempts Ġto Ġappeal Ġto Ġwestern Ġaudiences , Ġparticularly Ġwomen . ĠLooking Ġthrough Ġboth ĠD abi q Ġand ĠRum iyah , Ġmany Ġissues Ġof Ġthe Ġmagazines Ġcontain Ġarticles Ġspecifically Ġaddressing Ġwomen , Ġusually Ġwith ĠâĢ ľ Ġto Ġour Ġsisters ĠâĢ Ŀ Ġincorporated Ġinto Ġthe Ġtitle . ĠSeven Ġout Ġof Ġfifteen ĠD abi q Ġissues Ġand Ġall Ġthirteen Ġissues Ġof ĠRum iyah Ġcontain Ġarticles Ġtargeting Ġwomen , Ġclearly Ġsuggesting Ġan Ġincrease Ġin Ġattention Ġto Ġwomen Ġover Ġtime . </s> We Ġconverted Ġall Ġthe ĠISIS Ġmagazines Ġto Ġtexts Ġusing Ġpdf Ġreaders Ġand Ġall Ġarticles Ġthat Ġaddressed Ġwomen Ġin Ġboth Ġmagazines Ġ( 20 Ġarticles ) Ġwere Ġselected Ġfor Ġour Ġanalysis . ĠTo Ġfacilitate Ġcomparison Ġwith Ġa Ġmainstream , Ġnon - violent Ġreligious Ġgroup , Ġwe Ġcollected Ġarticles Ġfrom Ġcath olic w om ens forum . org , Ġan Ġonline Ġresource Ġcatering Ġto ĠCatholic Ġwomen . ĠWe Ġscrapped Ġ132 Ġarticles Ġfrom Ġthis Ġdomain . ĠWhile Ġthis Ġnumber Ġis Ġlarge , Ġthe Ġarticles Ġthemselves Ġare Ġmuch Ġshorter Ġthan Ġthose Ġpublished Ġby ĠISIS . ĠThese Ġtexts Ġwere Ġpre - process ed Ġby Ġtoken izing Ġthe Ġsentences Ġand Ġeliminating Ġnon - word Ġtokens Ġand Ġpunct uation Ġmarks . ĠAlso , Ġall Ġwords Ġturned Ġinto Ġlower Ġcase Ġand Ġnumbers Ġand ĠEnglish Ġstop Ġwords Ġsuch Ġas ĠâĢ ľ our , Ġis , Ġdid , Ġcan , Ġetc . ĠâĢ Ŀ Ġhave Ġbeen Ġremoved Ġfrom Ġthe Ġproduced Ġtokens . ĠFor Ġthe Ġemotion Ġanalysis Ġpart , Ġwe Ġused Ġa Ġsp acy Ġlibrary Ġas Ġpart Ġof Ġspeech Ġtagging Ġto Ġidentify Ġthe Ġexact Ġrole Ġof Ġwords Ġin Ġthe Ġsentence . ĠA Ġword Ġand Ġits Ġrole Ġhave Ġbeen Ġused Ġto Ġlook Ġfor Ġemotional Ġvalues Ġof Ġthat Ġword Ġin Ġthe Ġsame Ġrole Ġin Ġthe Ġsentence . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠText ĠClean ing Ġand ĠPre - processing </s> Most Ġtext Ġand Ġdocument Ġdatasets Ġcontain Ġmany Ġunnecessary Ġwords Ġsuch Ġas Ġstop words , Ġmiss pelling , Ġslang , Ġetc . ĠIn Ġmany Ġalgorithms , Ġespecially Ġstatistical Ġand Ġprob abil istic Ġlearning Ġalgorithms , Ġnoise Ġand Ġunnecessary Ġfeatures Ġcan Ġhave Ġadverse Ġeffects Ġon Ġsystem Ġperformance . ĠIn Ġthis Ġsection , Ġwe Ġbriefly Ġexplain Ġsome Ġtechniques Ġand Ġmethods Ġfor Ġtext Ġcleaning Ġand Ġpre - processing Ġtext Ġdatasets ĠB IB REF 13 . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠToken ization </s> Token ization Ġis Ġa Ġpre - processing Ġmethod Ġwhich Ġbreaks Ġa Ġstream Ġof Ġtext Ġinto Ġwords , Ġphrases , Ġsymbols , Ġor Ġother Ġmeaningful Ġelements Ġcalled Ġtokens ĠB IB REF 14 . ĠThe Ġmain Ġgoal Ġof Ġthis Ġstep Ġis Ġto Ġinvestigate Ġthe Ġwords Ġin Ġa Ġsentence ĠB IB REF 14 . ĠBoth Ġtext Ġclassification Ġand Ġtext Ġmining Ġrequires Ġa Ġparser Ġwhich Ġprocesses Ġthe Ġtoken ization Ġof Ġthe Ġdocuments ; Ġfor Ġexample : </s> sent ence ĠB IB REF 15 Ġ: </s> After Ġsleeping Ġfor Ġfour Ġhours , Ġhe Ġdecided Ġto Ġsleep Ġfor Ġanother Ġfour . </s> In Ġthis Ġcase , Ġthe Ġtokens Ġare Ġas Ġfollows : </s> { âĢ ľ After âĢ Ŀ ĠâĢ ľ sle eping âĢ Ŀ ĠâĢ ľ for âĢ Ŀ ĠâĢ ľ four âĢ Ŀ ĠâĢ ľ hours âĢ Ŀ ĠâĢ ľ he âĢ Ŀ ĠâĢ ľ dec ided âĢ Ŀ ĠâĢ ľ to âĢ Ŀ ĠâĢ ľ sleep âĢ Ŀ ĠâĢ ľ for âĢ Ŀ ĠâĢ ľ another âĢ Ŀ ĠâĢ ľ four âĢ Ŀ }. </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠStop Ġwords </s> Text Ġand Ġdocument Ġclassification Ġincludes Ġmany Ġwords Ġwhich Ġdo Ġnot Ġhold Ġimportant Ġsignificance Ġto Ġbe Ġused Ġin Ġclassification Ġalgorithms Ġsuch Ġas Ġ{ âĢ ľ a âĢ Ŀ , ĠâĢ ľ about âĢ Ŀ , ĠâĢ ľ above âĢ Ŀ , ĠâĢ ľ ac ross âĢ Ŀ , ĠâĢ ľ after âĢ Ŀ , ĠâĢ ľ after wards âĢ Ŀ , ĠâĢ ľ again âĢ Ŀ , $ \ hd ots Ġ$ }. ĠThe Ġmost Ġcommon Ġtechnique Ġto Ġdeal Ġwith Ġthese Ġwords Ġis Ġto Ġremove Ġthem Ġfrom Ġthe Ġtexts Ġand Ġdocuments ĠB IB REF 16 . </s> Data ĠCollection Ġ& ĠPre - Process ing Ġ: :: ĠPre - Process ing Ġ: :: ĠTerm ĠFrequency - In verse ĠDocument ĠFrequency </s> K ĠSp ar ck ĠJones ĠB IB REF 17 Ġproposed Ġinverse Ġdocument Ġfrequency Ġ( ID F ) Ġas Ġa Ġmethod Ġto Ġbe Ġused Ġin Ġconjunction Ġwith Ġterm Ġfrequency Ġin Ġorder Ġto Ġlessen Ġthe Ġeffect Ġof Ġimplicitly Ġcommon Ġwords Ġin Ġthe Ġcorpus . ĠIDF Ġassigns Ġa Ġhigher Ġweight Ġto Ġwords Ġwith Ġeither Ġhigh Ġfrequency Ġor Ġlow Ġfrequency Ġterm Ġin Ġthe Ġdocument . ĠThis Ġcombination Ġof ĠTF Ġand ĠIDF Ġis Ġwell Ġknown Ġas Ġterm Ġfrequency - in verse Ġdocument Ġfrequency Ġ( tf - id f ). ĠThe Ġmathematical Ġrepresentation Ġof Ġthe Ġweight Ġof Ġa Ġterm Ġin Ġa Ġdocument Ġby Ġtf - id f Ġis Ġgiven Ġin ĠEqu ation ĠDIS PLAY _ FORM 10 . </s> Here ĠN Ġis Ġthe Ġnumber Ġof Ġdocuments Ġand Ġ$ df ( t ) $ Ġis Ġthe Ġnumber Ġof Ġdocuments Ġcontaining Ġthe Ġterm Ġt Ġin Ġthe Ġcorpus . ĠThe Ġfirst Ġterm Ġin ĠEqu ation ĠDIS PLAY _ FORM 10 Ġimproves Ġthe Ġrecall Ġwhile Ġthe Ġsecond Ġterm Ġimproves Ġthe Ġprecision Ġof Ġthe Ġword Ġembed ding ĠB IB REF 18 . ĠAlthough Ġtf - id f Ġtries Ġto Ġovercome Ġthe Ġproblem Ġof Ġcommon Ġterms Ġin Ġthe Ġdocument , Ġit Ġstill Ġsuffers Ġfrom Ġsome Ġother Ġdescriptive Ġlimitations . ĠName ly , Ġtf - id f Ġcannot Ġaccount Ġfor Ġthe Ġsimilarity Ġbetween Ġthe Ġwords Ġin Ġthe Ġdocument Ġsince Ġeach Ġword Ġis Ġindependently Ġpresented Ġas Ġan Ġindex . ĠHowever , Ġwith Ġthe Ġdevelopment Ġof Ġmore Ġcomplex Ġmodels Ġin Ġrecent Ġyears , Ġnew Ġmethods , Ġsuch Ġas Ġword Ġembed ding , Ġhave Ġbeen Ġpresented Ġthat Ġcan Ġincorporate Ġconcepts Ġsuch Ġas Ġsimilarity Ġof Ġwords Ġand Ġpart Ġof Ġspeech Ġtagging . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġmethods Ġused Ġfor Ġcomparing Ġtopics Ġand Ġev oked Ġemotions Ġin Ġboth ĠISIS Ġand Ġnon - violent Ġreligious Ġmaterials . </s> Method Ġ: :: ĠContent ĠAnalysis </s> The Ġkey Ġtask Ġin Ġcomparing ĠISIS Ġmaterial Ġwith Ġthat Ġof Ġa Ġnon - violent Ġgroup Ġinvolves Ġanalyzing Ġthe Ġcontent Ġof Ġthese Ġtwo Ġcorpor a Ġto Ġidentify Ġthe Ġtopics . ĠFor Ġour Ġanalysis , Ġwe Ġconsidered Ġa Ġsimple Ġun i - gram Ġmodel Ġwhere Ġeach Ġword Ġis Ġconsidered Ġas Ġa Ġsingle Ġunit . ĠUnderstanding Ġwhat Ġwords Ġappear Ġmost Ġfrequently Ġprovides Ġa Ġsimple Ġmetric Ġfor Ġcomparison . ĠTo Ġdo Ġso Ġwe Ġnormalized Ġthe Ġcount Ġof Ġwords Ġwith Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġcorpor a Ġto Ġaccount Ġfor Ġthe Ġsize Ġof Ġeach Ġcorpus . ĠIt Ġshould Ġbe Ġnoted , Ġhowever , Ġthat Ġa Ġdrawback Ġof Ġword Ġfrequencies Ġis Ġthat Ġthere Ġmight Ġbe Ġsome Ġdominant Ġwords Ġthat Ġwill Ġovercome Ġall Ġthe Ġother Ġcontents Ġwithout Ġconve ying Ġmuch Ġinformation . </s> Topic Ġmodeling Ġmethods Ġare Ġthe Ġmore Ġpowerful Ġtechnique Ġfor Ġunderstanding Ġthe Ġcontents Ġof Ġa Ġcorpus . ĠThese Ġmethods Ġtry Ġto Ġdiscover Ġabstract Ġtopics Ġin Ġa Ġcorpus Ġand Ġreveal Ġhidden Ġsemantic Ġstructures Ġin Ġa Ġcollection Ġof Ġdocuments . ĠThe Ġmost Ġpopular Ġtopic Ġmodeling Ġmethods Ġuse Ġprob abil istic Ġapproaches Ġsuch Ġas Ġprob abil istic Ġlatent Ġsemantic Ġanalysis Ġ( PL SA ) Ġand Ġlatent ĠDir ich let Ġallocation Ġ( L DA ). ĠL DA Ġis Ġa Ġgeneral ization Ġof Ġp L SA Ġwhere Ġdocuments Ġare Ġconsidered Ġas Ġa Ġmixture Ġof Ġtopics Ġand Ġthe Ġdistribution Ġof Ġtopics Ġis Ġgoverned Ġby Ġa ĠDir ich let Ġprior Ġ($ \ alpha Ġ$ ). ĠFigure ĠFIG REF 12 Ġshows Ġplate Ġnotation Ġof Ġgeneral ĠL DA Ġstructure Ġwhere Ġ$\ beta Ġ$ Ġrepresents Ġprior Ġof Ġword Ġdistribution Ġper Ġtopic Ġand Ġ$\ the ta Ġ$ Ġrefers Ġto Ġtopics Ġdistribution Ġfor Ġdocuments ĠB IB REF 19 . ĠSince ĠL DA Ġis Ġamong Ġthe Ġmost Ġwidely Ġutilized Ġalgorithms Ġfor Ġtopic Ġmodeling , Ġwe Ġapplied Ġit Ġto Ġour Ġdata . ĠHowever , Ġthe Ġco herence Ġof Ġthe Ġtopics Ġproduced Ġby ĠL DA Ġis Ġpoorer Ġthan Ġexpected . </s> To Ġaddress Ġthis Ġlack Ġof Ġco herence , Ġwe Ġapplied Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ). ĠThis Ġmethod Ġdecom poses Ġthe Ġterm - document Ġmatrix Ġinto Ġtwo Ġnon - negative Ġmat rices Ġas Ġshown Ġin ĠFigure ĠFIG REF 13 . ĠThe Ġresulting Ġnon - negative Ġmat rices Ġare Ġsuch Ġthat Ġtheir Ġproduct Ġclosely Ġapproximate Ġthe Ġoriginal Ġdata . ĠMat hematically Ġspeaking , Ġgiven Ġan Ġinput Ġmatrix Ġof Ġdocument - terms Ġ$ V $, ĠNM F Ġfinds Ġtwo Ġmat rices Ġby Ġsolving Ġthe Ġfollowing Ġequation ĠB IB REF 20 : </s> Where ĠW Ġis Ġtopic - word Ġmatrix Ġand ĠH Ġrepresents Ġtopic - document Ġmatrix . </s> NM F Ġappears Ġto Ġprovide Ġmore Ġcoherent Ġtopic Ġon Ġspecific Ġcorpor a . ĠO ' Call aghan Ġet Ġal . Ġcompared ĠL DA Ġwith ĠNM F Ġand Ġconcluded Ġthat ĠNM F Ġperforms Ġbetter Ġin Ġcorpor as Ġwith Ġspecific Ġand Ġnon - main stream Ġareas ĠB IB REF 21 . ĠOur Ġfindings Ġalign Ġwith Ġthis Ġassessment Ġand Ġthus Ġour Ġcomparison Ġof Ġtopics Ġis Ġbased Ġon ĠNM F . </s> Method Ġ: :: ĠEm otion Ġdetection </s> Prop aganda Ġeffectiveness Ġhinges Ġon Ġthe Ġemotions Ġthat Ġit Ġelic its . ĠBut Ġdetecting Ġemotion Ġin Ġtext Ġrequires Ġthat Ġtwo Ġessential Ġchallenges Ġare Ġovercome . </s> First , Ġemotions Ġare Ġgenerally Ġcomplex Ġand Ġemotional Ġrepresentation Ġmodels Ġare Ġcorrespond ingly Ġcontested . ĠDespite Ġthis , Ġsome Ġmodels Ġproposed Ġby Ġpsychologists Ġhave Ġgained Ġwide - spread Ġusage Ġthat Ġextends Ġto Ġtext - em otion Ġanalysis . ĠRobert ĠPl utch ik Ġpresented Ġa Ġmodel Ġthat Ġarranged Ġemotions Ġfrom Ġbasic Ġto Ġcomplex Ġin Ġa Ġcirc ump lex Ġas Ġshown Ġin ĠFigure ĠFIG REF 15 . ĠThe Ġmodel Ġcategor izes Ġemotions Ġinto Ġ8 Ġmain Ġsubs ets Ġand Ġwith Ġaddition Ġof Ġintensity Ġand Ġinteractions Ġit Ġwill Ġclassify Ġemotions Ġinto Ġ24 Ġclasses ĠB IB REF 23 . ĠOther Ġmodels Ġhave Ġbeen Ġdeveloped Ġto Ġcapture Ġall Ġemotions Ġby Ġdefining Ġa Ġ3 - dimensional Ġmodel Ġof Ġpleasure , Ġarousal , Ġand Ġdominance . </s> The Ġsecond Ġchallenge Ġlies Ġin Ġusing Ġtext Ġfor Ġdetecting Ġemotion Ġev oked Ġin Ġa Ġpotential Ġreader . ĠCommon Ġapproaches Ġuse Ġeither Ġlex icon - base Ġmethods Ġ( such Ġas Ġkeyword - based Ġor Ġont ology - based Ġmodel ) Ġor Ġmachine Ġlearning - base Ġmodels Ġ( usually Ġusing Ġlarge Ġcorpus Ġwith Ġlabeled Ġemotions ) ĠB IB REF 12 . ĠThese Ġmethods Ġare Ġsuited Ġto Ġaddressing Ġthe Ġemotion Ġthat Ġexist Ġin Ġthe Ġtext , Ġbut Ġin Ġthe Ġcase Ġof Ġpropaganda Ġwe Ġare Ġmore Ġinterested Ġin Ġemotions Ġthat Ġare Ġelic ited Ġin Ġthe Ġreader Ġof Ġsuch Ġmaterials . ĠThe Ġclosest Ġanalogy Ġto Ġthis Ġproblem Ġcan Ġbe Ġfound Ġin Ġresearch Ġthat Ġseek Ġto Ġmodel Ġfeelings Ġof Ġpeople Ġafter Ġreading Ġa Ġnews Ġarticle . ĠOne Ġsolution Ġfor Ġthis Ġtype Ġof Ġproblem Ġis Ġto Ġuse Ġan Ġapproach Ġcalled ĠDe pe chem ood . </s> De pe chem ood Ġis Ġa Ġlex icon - based Ġemotion Ġdetection Ġmethod Ġgathered Ġfrom Ġcrowd - annot ated Ġnews ĠB IB REF 24 . ĠDrawing Ġon Ġapproximately Ġ23 . 5 K Ġdocuments Ġwith Ġaverage Ġof Ġ500 Ġwords Ġper Ġdocument Ġfrom Ġrapp ler . com , Ġresearchers Ġasked Ġsubjects Ġto Ġreport Ġtheir Ġemotions Ġafter Ġreading Ġeach Ġarticle . ĠThey Ġthen Ġmultiplied Ġthe Ġdocument - em otion Ġmatrix Ġand Ġword - document Ġmatrix Ġto Ġderive Ġemotion - word Ġmatrix Ġfor Ġthese Ġwords . ĠDue Ġto Ġlimitations Ġof Ġtheir Ġexperiment Ġsetup , Ġthe Ġemotion Ġcategories Ġthat Ġthey Ġpresent Ġdoes Ġnot Ġexactly Ġmatch Ġthe Ġemotions Ġfrom Ġthe ĠPl utch ik Ġwheel Ġcategories . ĠHowever , Ġthey Ġstill Ġprovide Ġa Ġgood Ġsense Ġof Ġthe Ġgeneral Ġfeeling Ġof Ġan Ġindividual Ġafter Ġreading Ġan Ġarticle . ĠThe Ġemotion Ġcategories Ġof ĠDe pe chem ood Ġare : ĠAF RA ID , ĠAM US ED , ĠAN GR Y , ĠAN NO Y ED , ĠDON ' T ĠCARE , ĠH APP Y , ĠIN SP IRED , ĠS AD . ĠDe pe chem ood Ġsimply Ġcreates Ġdiction aries Ġof Ġwords Ġwhere Ġeach Ġword Ġhas Ġscores Ġbetween Ġ0 Ġand Ġ1 Ġfor Ġall Ġof Ġthese Ġ8 Ġemotion Ġcategories . ĠWe Ġpresent Ġour Ġfinding Ġusing Ġthis Ġapproach Ġin Ġthe Ġresult Ġsection . </s> Results </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġresults Ġof Ġour Ġanalysis Ġbased Ġon Ġthe Ġcontents Ġof ĠISIS Ġpropaganda Ġmaterials Ġas Ġcompared Ġto Ġarticles Ġfrom Ġthe ĠCatholic Ġwomen Ġforum . ĠWe Ġthen Ġpresent Ġthe Ġresults Ġof Ġemotion Ġanalysis Ġconducted Ġon Ġboth Ġcorpor a . </s> Results Ġ: :: ĠContent ĠAnalysis </s> After Ġpre - processing Ġthe Ġtext , Ġboth Ġcorpor a Ġwere Ġanalyzed Ġfor Ġword Ġfrequencies . ĠThese Ġword Ġfrequencies Ġhave Ġbeen Ġnormalized Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġeach Ġcorpus . ĠFigure ĠFIG REF 17 Ġshows Ġthe Ġmost Ġcommon Ġwords Ġin Ġeach Ġof Ġthese Ġcorpor a . </s> A Ġcomparison Ġof Ġcommon Ġwords Ġsuggests Ġthat Ġthose Ġrelated Ġto Ġmarital Ġrelationships Ġ( Ġhusband , Ġwife , Ġetc .) Ġappear Ġin Ġboth Ġcorpor a , Ġbut Ġthe Ġreligious Ġtheme Ġof ĠISIS Ġmaterial Ġappears Ġto Ġbe Ġstronger . ĠA Ġstronger Ġcomparison Ġcan Ġbe Ġmade Ġusing Ġtopic Ġmodeling Ġtechniques Ġto Ġdiscover Ġmain Ġtopics Ġof Ġthese Ġdocuments . ĠAlthough Ġwe Ġused ĠL DA , Ġour Ġresults Ġby Ġusing ĠNM F Ġoutper form ĠL DA Ġtopics , Ġdue Ġto Ġthe Ġnature Ġof Ġthese Ġcorpor a . ĠAlso , Ġfewer Ġnumbers Ġof ĠISIS Ġdocuments Ġmight Ġcontribute Ġto Ġthe Ġcomparatively Ġworse Ġperformance . ĠTherefore , Ġwe Ġpresent Ġonly ĠNM F Ġresults . ĠBased Ġon Ġtheir Ġco herence , Ġwe Ġselected Ġ10 Ġtopics Ġfor Ġanalyzing Ġwithin Ġboth Ġcorpor as . ĠTable ĠT AB REF 18 Ġand ĠTable ĠT AB REF 19 Ġshow Ġthe Ġmost Ġimportant Ġwords Ġin Ġeach Ġtopic Ġwith Ġa Ġgeneral Ġlabel Ġthat Ġwe Ġassigned Ġto Ġthe Ġtopic Ġmanually . ĠBased Ġon Ġthe ĠNM F Ġoutput , ĠISIS Ġarticles Ġthat Ġaddress Ġwomen Ġinclude Ġtopics Ġmainly Ġabout ĠIslam , Ġwomen 's Ġrole Ġin Ġearly ĠIslam , Ġhij rah Ġ( moving Ġto Ġanother Ġland ), Ġsp ous al Ġrelations , Ġmarriage , Ġand Ġmother hood . </s> The Ġtopics Ġgenerated Ġfrom Ġthe ĠCatholic Ġwomen Ġforum Ġare Ġclearly Ġquite Ġdifferent . ĠSome , Ġhowever , Ġexist Ġin Ġboth Ġcontexts . ĠMore Ġspecifically , Ġmarriage / div orce , Ġmother hood , Ġand Ġto Ġsome Ġextent Ġsp ous al Ġrelations Ġappeared Ġin Ġboth Ġgenerated Ġtopics . ĠThis Ġsuggests Ġthat Ġwhen Ġaddressing Ġwomen Ġin Ġa Ġreligious Ġcontext , Ġthese Ġmay Ġbe Ġvery Ġbroadly Ġeffective Ġand Ġappeal Ġto Ġthe Ġfeminine Ġaudience . ĠMore Ġimportantly , Ġsuitable Ġtopic Ġmodeling Ġmethods Ġwill Ġbe Ġable Ġto Ġidentify Ġthese Ġsimilarities Ġno Ġmatter Ġthe Ġsize Ġof Ġthe Ġcorpus Ġwe Ġare Ġworking Ġwith . ĠAlthough , Ġfinding Ġthe Ġsimilarities / diff erences Ġbetween Ġtopics Ġin Ġthese Ġtwo Ġgroups Ġof Ġarticles Ġmight Ġprovide Ġsome Ġnew Ġinsights , Ġwe Ġturn Ġto Ġemotional Ġanalysis Ġto Ġalso Ġcompare Ġthe Ġemotions Ġev oked Ġin Ġthe Ġaudience . </s> Results Ġ: :: ĠEm otion ĠAnalysis </s> We Ġrely Ġon ĠDe pe chem ood Ġdiction aries Ġto Ġanalyze Ġemotions Ġin Ġboth Ġcorpor a . ĠThese Ġdiction aries Ġare Ġfreely Ġavailable Ġand Ġcome Ġin Ġmultiple Ġarrangements . ĠWe Ġused Ġa Ġversion Ġthat Ġincludes Ġwords Ġwith Ġtheir Ġpart Ġof Ġspeech Ġ( POS ) Ġtags . ĠOnly Ġwords Ġthat Ġexist Ġin Ġthe ĠDe pe chem ood Ġdictionary Ġwith Ġthe Ġsame ĠPOS Ġtag Ġare Ġconsidered Ġfor Ġour Ġanalysis . ĠWe Ġaggreg ated Ġthe Ġscore Ġfor Ġeach Ġword Ġand Ġnormalized Ġeach Ġarticle Ġby Ġemotions . ĠTo Ġbetter Ġcompare Ġthe Ġresult , Ġwe Ġadded Ġa Ġbaseline Ġof Ġ100 Ġrandom Ġarticles Ġfrom Ġa ĠReuters Ġnews Ġdataset Ġas Ġa Ġnon - religious Ġgeneral Ġresource Ġwhich Ġis Ġavailable Ġin Ġan ĠNL T K Ġpython Ġlibrary . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġaggreg ated Ġscore Ġfor Ġdifferent Ġfeelings Ġin Ġour Ġcorpor a . </s> Both ĠCatholic Ġand ĠISIS Ġrelated Ġmaterials Ġscore Ġthe Ġhighest Ġin ĠâĢ ľ inspired âĢ Ŀ Ġcategory . ĠFurthermore , Ġin Ġboth Ġcases , Ġbeing Ġafraid Ġhas Ġthe Ġlowest Ġscore . ĠHowever , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġrandom Ġnews Ġmaterial Ġsuch Ġas Ġthe ĠReuters Ġcorpus , Ġwhich Ġare Ġnot Ġthat Ġinspiring Ġand , Ġaccording Ġto Ġthis Ġmethod , Ġseems Ġto Ġcause Ġmore Ġfear Ġin Ġtheir Ġaudience . ĠWe Ġinvestigate Ġthese Ġresults Ġfurther Ġby Ġlooking Ġat Ġthe Ġmost Ġinspiring Ġwords Ġdetected Ġin Ġthese Ġtwo Ġcorpor a . ĠTable ĠT AB REF 24 Ġpresents Ġ10 Ġwords Ġthat Ġare Ġamong Ġthe Ġmost Ġinspiring Ġin Ġboth Ġcorpor a . ĠThe Ġcomparison Ġof Ġthe Ġtwo Ġlists Ġindicate Ġthat Ġthe Ġmethod Ġpicks Ġvery Ġdifferent Ġwords Ġin Ġeach Ġcorpus Ġto Ġreach Ġto Ġthe Ġsame Ġconclusion . ĠAlso , Ġwe Ġlooked Ġat Ġseparate Ġarticles Ġin Ġeach Ġof Ġthe Ġissues Ġof ĠISIS Ġmaterial Ġaddressing Ġwomen . ĠFigure ĠFIG REF 23 Ġshows Ġemotion Ġscores Ġin Ġeach Ġof Ġthe Ġ20 Ġissues Ġof ĠISIS Ġpropaganda . ĠAs Ġdemonstrated , Ġin Ġevery Ġseparate Ġarticle , Ġthis Ġmethod Ġgives Ġthe Ġhighest Ġscore Ġto Ġev oking Ġinspir ations Ġin Ġthe Ġreader . ĠAlso , Ġin Ġmost Ġof Ġthese Ġissues Ġthe Ġmethod Ġscored ĠâĢ ľ being Ġafraid âĢ Ŀ Ġas Ġthe Ġlowest Ġscore Ġin Ġeach Ġissue . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġhave Ġapplied Ġnatural Ġlanguage Ġprocessing Ġmethods Ġto ĠISIS Ġpropaganda Ġmaterials Ġin Ġan Ġattempt Ġto Ġunderstand Ġthese Ġmaterials Ġusing Ġavailable Ġtechnologies . ĠWe Ġalso Ġcompared Ġthese Ġtexts Ġwith Ġa Ġnon - violent Ġreligious Ġgroups ' Ġ( both Ġfocusing Ġon Ġwomen Ġrelated Ġarticles ) Ġto Ġexamine Ġpossible Ġsimilarities Ġor Ġdifferences Ġin Ġtheir Ġapproaches . ĠTo Ġcompare Ġthe Ġcontents , Ġwe Ġused Ġword Ġfrequency Ġand Ġtopic Ġmodeling Ġwith ĠNM F . ĠAlso , Ġour Ġresults Ġshowed Ġthat ĠNM F Ġoutper forms ĠL DA Ġdue Ġto Ġthe Ġniche Ġdomain Ġand Ġrelatively Ġsmall Ġnumber Ġof Ġdocuments . </s> The Ġresults Ġsuggest Ġthat Ġcertain Ġtopics Ġplay Ġa Ġparticularly Ġimportant Ġroles Ġin ĠISIS Ġpropaganda Ġtargeting Ġwomen . ĠThese Ġrelate Ġto Ġthe Ġrole Ġof Ġwomen Ġin Ġearly ĠIslam , ĠIslamic Ġideology , Ġmarriage / div orce , Ġmother hood , Ġsp ous al Ġrelationships , Ġand Ġhij rah Ġ( moving Ġto Ġa Ġnew Ġland ). </s> Comp aring Ġthese Ġtopics Ġwith Ġthose Ġthat Ġappeared Ġon Ġa ĠCatholic Ġwomen Ġforum , Ġit Ġseems Ġthat Ġboth ĠISIS Ġand Ġnon - violent Ġgroups Ġuse Ġtopics Ġabout Ġmother hood , Ġsp ous al Ġrelationship , Ġand Ġmarriage / div orce Ġwhen Ġthey Ġaddress Ġwomen . ĠMoreover , Ġwe Ġused ĠDe pe chem ood Ġmethods Ġto Ġanalyze Ġthe Ġemotions Ġthat Ġthese Ġmaterials Ġare Ġlikely Ġto Ġelicit Ġin Ġreaders . ĠThe Ġresult Ġof Ġour Ġemotion Ġanalysis Ġsuggests Ġthat Ġboth Ġcorp uses Ġused Ġwords Ġthat Ġaim Ġto Ġinspire Ġreaders Ġwhile Ġavoiding Ġfear . ĠHowever , Ġthe Ġactual Ġwords Ġthat Ġlead Ġto Ġthese Ġeffects Ġare Ġvery Ġdifferent Ġin Ġthe Ġtwo Ġcontexts . ĠOverall , Ġour Ġfindings Ġindicate Ġthat , Ġusing Ġproper Ġmethods , Ġautomated Ġanalysis Ġof Ġlarge Ġbodies Ġof Ġtextual Ġdata Ġcan Ġprovide Ġnovel Ġinsight Ġinsight Ġinto Ġextremist Ġpropaganda Ġthat Ġcan Ġassist Ġthe Ġcounterterrorism Ġcommunity .