For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy
EXTRACTIVE
public resources where suspicious Twitter accounts were annotated, list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy
EXTRACTIVE
Data. We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. This list was created based on public resources where suspicious Twitter accounts were annotated with the main fake news types (clickbait, propaganda, satire, and hoax). We discard the satire labeled accounts since their intention is not to mislead or deceive. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. We discard some accounts that publish news in languages other than English (e.g., Russian or Arabic). Moreover, to ensure the quality of the data, we remove the duplicate, media-based, and link-only tweets. For each account, we collect the maximum amount of tweets allowed by Twitter API. Table TABREF13 presents statistics on our dataset.
Data. We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. This list was created based on public resources where suspicious Twitter accounts were annotated with the main fake news types (clickbait, propaganda, satire, and hoax). We discard the satire labeled accounts since their intention is not to mislead or deceive. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. We discard some accounts that publish news in languages other than English (e.g., Russian or Arabic). Moreover, to ensure the quality of the data, we remove the duplicate, media-based, and link-only tweets. For each account, we collect the maximum amount of tweets allowed by Twitter API. Table TABREF13 presents statistics on our dataset.
1910.06592
Introduction </s> Social Ġmedia Ġplatforms Ġhave Ġmade Ġthe Ġspreading Ġof Ġfake Ġnews Ġeasier , Ġfaster Ġas Ġwell Ġas Ġable Ġto Ġreach Ġa Ġwider Ġaudience . ĠSocial Ġmedia Ġoffer Ġanother Ġfeature Ġwhich Ġis Ġthe Ġanonymity Ġfor Ġthe Ġauthors , Ġand Ġthis Ġopens Ġthe Ġdoor Ġto Ġmany Ġsuspicious Ġindividuals Ġor Ġorganizations Ġto Ġutilize Ġthese Ġplatforms . ĠRecently , Ġthere Ġhas Ġbeen Ġan Ġincreased Ġnumber Ġof Ġspreading Ġfake Ġnews Ġand Ġrumors Ġover Ġthe Ġweb Ġand Ġsocial Ġmedia ĠB IB REF 0 . ĠFake Ġnews Ġin Ġsocial Ġmedia Ġvary Ġconsidering Ġthe Ġintention Ġto Ġmislead . ĠSome Ġof Ġthese Ġnews Ġare Ġspread Ġwith Ġthe Ġintention Ġto Ġbe Ġironic Ġor Ġto Ġdeliver Ġthe Ġnews Ġin Ġan Ġironic Ġway Ġ( sat ir ical Ġnews ). ĠOthers , Ġsuch Ġas Ġpropaganda , Ġhoax es , Ġand Ġclick ba its , Ġare Ġspread Ġto Ġmislead Ġthe Ġaudience Ġor Ġto Ġmanipulate Ġtheir Ġopinions . ĠIn Ġthe Ġcase Ġof ĠTwitter , Ġsuspicious Ġnews Ġannotations Ġshould Ġbe Ġdone Ġon Ġa Ġtweet Ġrather Ġthan Ġan Ġaccount Ġlevel , Ġsince Ġsome Ġaccounts Ġmix Ġfake Ġwith Ġreal Ġnews . ĠHowever , Ġthese Ġannotations Ġare Ġextremely Ġcostly Ġand Ġtime Ġconsuming ĠâĢĵ Ġi . e ., Ġdue Ġto Ġhigh Ġvolume Ġof Ġavailable Ġtweets ĠConsequently , Ġa Ġfirst Ġstep Ġin Ġthis Ġdirection , Ġe . g ., Ġas Ġa Ġpre - fil tering Ġstep , Ġcan Ġbe Ġviewed Ġas Ġthe Ġtask Ġof Ġdetecting Ġfake Ġnews Ġat Ġthe Ġaccount Ġlevel . </s> The Ġmain Ġobstacle Ġfor Ġdetecting Ġsuspicious ĠTwitter Ġaccounts Ġis Ġdue Ġto Ġthe Ġbehavior Ġof Ġmixing Ġsome Ġreal Ġnews Ġwith Ġthe Ġmisleading Ġones . ĠConsequently , Ġwe Ġinvestigate Ġways Ġto Ġdetect Ġsuspicious Ġaccounts Ġby Ġconsidering Ġtheir Ġtweets Ġin Ġgroups Ġ( ch unks ). ĠOur Ġhypothesis Ġis Ġthat Ġsuspicious Ġaccounts Ġhave Ġa Ġunique Ġpattern Ġin Ġposting Ġtweet Ġsequences . ĠSince Ġtheir Ġintention Ġis Ġto Ġmislead , Ġthe Ġway Ġthey Ġtransition Ġfrom Ġone Ġset Ġof Ġtweets Ġto Ġthe Ġnext Ġhas Ġa Ġhidden Ġsignature , Ġbiased Ġby Ġtheir Ġintentions . ĠTherefore , Ġreading Ġthese Ġtweets Ġin Ġchunks Ġhas Ġthe Ġpotential Ġto Ġimprove Ġthe Ġdetection Ġof Ġthe Ġfake Ġnews Ġaccounts . </s> In Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġproblem Ġof Ġdiscriminating Ġbetween Ġfactual Ġand Ġnon - fact ual Ġaccounts Ġin ĠTwitter . ĠTo Ġthis Ġend , Ġwe Ġcollect Ġa Ġlarge Ġdataset Ġof Ġtweets Ġusing Ġa Ġlist Ġof Ġpropaganda , Ġhoax Ġand Ġclick b ait Ġaccounts Ġand Ġcompare Ġdifferent Ġversions Ġof Ġsequential Ġchunk - based Ġapproaches Ġusing Ġa Ġvariety Ġof Ġfeature Ġsets Ġagainst Ġseveral Ġbas elines . ĠSeveral Ġapproaches Ġhave Ġbeen Ġproposed Ġfor Ġnews Ġverification , Ġwhether Ġin Ġsocial Ġmedia Ġ( rum ors Ġdetection ) ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , Ġor Ġin Ġnews Ġclaims ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 . ĠThe Ġmain Ġorientation Ġin Ġthe Ġprevious Ġworks Ġis Ġto Ġverify Ġthe Ġtextual Ġclaims / t we ets Ġbut Ġnot Ġtheir Ġsources . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġaiming Ġto Ġdetect Ġfact uality Ġat Ġthe Ġaccount Ġlevel , Ġand Ġespecially Ġfrom Ġa Ġtextual Ġperspective . ĠOur Ġcontributions Ġare : </s> [ left margin = 4 mm ] </s> We Ġpropose Ġan Ġapproach Ġto Ġdetect Ġnon - fact ual ĠTwitter Ġaccounts Ġby Ġtreating Ġpost Ġstreams Ġas Ġa Ġsequence Ġof Ġtweets ' Ġchunks . ĠWe Ġtest Ġseveral Ġsemantic Ġand Ġdictionary - based Ġfeatures Ġtogether Ġwith Ġa Ġneural Ġsequential Ġapproach , Ġand Ġapply Ġan Ġab lation Ġtest Ġto Ġinvestigate Ġtheir Ġcontribution . </s> We Ġbenchmark Ġour Ġapproach Ġagainst Ġother Ġapproaches Ġthat Ġdiscard Ġthe Ġchronological Ġorder Ġof Ġthe Ġtweets Ġor Ġread Ġthe Ġtweets Ġindividually . ĠThe Ġresults Ġshow Ġthat Ġour Ġapproach Ġproduces Ġsuperior Ġresults Ġat Ġdetecting Ġnon - fact ual Ġaccounts . </s> Method ology </s> Given Ġa Ġnews ĠTwitter Ġaccount , Ġwe Ġread Ġits Ġtweets Ġfrom Ġthe Ġaccount 's Ġtimeline . ĠThen Ġwe Ġsort Ġthe Ġtweets Ġby Ġthe Ġposting Ġdate Ġin Ġascending Ġway Ġand Ġwe Ġsplit Ġthem Ġinto Ġ$ N $ Ġchunks . ĠEach Ġchunk Ġconsists Ġof Ġa Ġsorted Ġsequence Ġof Ġtweets Ġlabeled Ġby Ġthe Ġlabel Ġof Ġits Ġcorresponding Ġaccount . ĠWe Ġextract Ġa Ġset Ġof Ġfeatures Ġfrom Ġeach Ġchunk Ġand Ġwe Ġfeed Ġthem Ġinto Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġmodel Ġthe Ġsequential Ġflow Ġof Ġthe Ġchunks ' Ġtweets . ĠWe Ġuse Ġan Ġattention Ġlayer Ġwith Ġdrop out Ġto Ġattend Ġover Ġthe Ġmost Ġimportant Ġtweets Ġin Ġeach Ġchunk . ĠFinally , Ġthe Ġrepresentation Ġis Ġfed Ġinto Ġa Ġsoft max Ġlayer Ġto Ġproduce Ġa Ġprobability Ġdistribution Ġover Ġthe Ġaccount Ġtypes Ġand Ġthus Ġpredict Ġthe Ġfact uality Ġof Ġthe Ġaccounts . ĠSince Ġwe Ġhave Ġmany Ġchunks Ġfor Ġeach Ġaccount , Ġthe Ġlabel Ġfor Ġan Ġaccount Ġis Ġobtained Ġby Ġtaking Ġthe Ġmajority Ġclass Ġof Ġthe Ġaccount 's Ġchunks . </s> Input ĠRepresent ation . ĠLet Ġ$ t $ Ġbe Ġa ĠTwitter Ġaccount Ġthat Ġcontains Ġ$ m $ Ġtweets . ĠThese Ġtweets Ġare Ġsorted Ġby Ġdate Ġand Ġsplit Ġinto Ġa Ġsequence Ġof Ġchunks Ġ$ ck Ġ= Ġ\ l angle Ġc k _ 1 , Ġ\ ld ots Ġ, Ġc k _ n Ġ\ r angle Ġ$ , Ġwhere Ġeach Ġ$ ck _ i $ Ġcontains Ġ$ s $ Ġtweets . ĠEach Ġtweet Ġin Ġ$ ck _ i $ Ġis Ġrepresented Ġby Ġa Ġvector Ġ$ v Ġ\ in Ġ{\ rm ĠI \ ! R } ^ d $ Ġ, Ġwhere Ġ$ v $ Ġis Ġthe Ġconc aten ation Ġof Ġa Ġset Ġof Ġfeatures ' Ġvectors , Ġthat Ġis Ġ$ v Ġ= Ġ\ l angle Ġf _ 1 , Ġ\ ld ots Ġ, Ġf _ n Ġ\ r angle Ġ$ . ĠEach Ġfeature Ġvector Ġ$ f _ i $ Ġis Ġbuilt Ġby Ġcounting Ġthe Ġpresence Ġof Ġtweet 's Ġwords Ġin Ġa Ġset Ġof Ġlex ical Ġlists . ĠThe Ġfinal Ġrepresentation Ġof Ġthe Ġtweet Ġis Ġbuilt Ġby Ġaveraging Ġthe Ġsingle Ġword Ġvectors . </s> Features . ĠWe Ġargue Ġthat Ġdifferent Ġkinds Ġof Ġfeatures Ġlike Ġthe Ġsentiment Ġof Ġthe Ġtext , Ġmorality , Ġand Ġother Ġtext - based Ġfeatures Ġare Ġcritical Ġto Ġdetect Ġthe Ġnon fact ual ĠTwitter Ġaccounts Ġby Ġutilizing Ġtheir Ġoccurrence Ġduring Ġreporting Ġthe Ġnews Ġin Ġan Ġaccount 's Ġtimeline . ĠWe Ġemploy Ġa Ġrich Ġset Ġof Ġfeatures Ġborrowed Ġfrom Ġprevious Ġworks Ġin Ġfake Ġnews , Ġbias , Ġand Ġrumors Ġdetection ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 8 , ĠB IB REF 9 . </s> [ left margin = 4 mm ] </s> Em otion : ĠWe Ġbuild Ġan Ġemotions Ġvector Ġusing Ġword Ġoccurrences Ġof Ġ15 Ġemotion Ġtypes Ġfrom Ġtwo Ġavailable Ġemotional Ġlex icons . ĠWe Ġuse Ġthe ĠN RC Ġlex icon ĠB IB REF 10 , Ġwhich Ġcontains Ġ$\ sim Ġ$ 14 K Ġwords Ġlabeled Ġusing Ġthe Ġeight ĠPl utch ik 's Ġemotions ĠB IB REF 11 . ĠThe Ġother Ġlex icon Ġis ĠSent i Sense ĠB IB REF 12 Ġwhich Ġis Ġa Ġconcept - based Ġaffect ive Ġlex icon Ġthat Ġattaches Ġemotional Ġmeanings Ġto Ġconcepts Ġfrom Ġthe ĠWord Net Ġlex ical Ġdatabase . ĠIt Ġhas Ġ$\ sim Ġ$ 5 . 5 Ġwords Ġlabeled Ġwith Ġemotions Ġfrom Ġa Ġset Ġof Ġ14 Ġemotional Ġcategories ĠWe Ġuse Ġthe Ġcategories Ġthat Ġdo Ġnot Ġexist Ġin Ġthe ĠN RC Ġlex icon . </s> Sent iment : ĠWe Ġextract Ġthe Ġsentiment Ġof Ġthe Ġtweets Ġby Ġemploying ĠEffect Word Net ĠB IB REF 13 , ĠSent ic Net ĠB IB REF 14 , ĠN RC ĠB IB REF 10 , Ġand Ġsub j _ lex icon ĠB IB REF 15 , Ġwhere Ġeach Ġhas Ġthe Ġtwo Ġsentiment Ġclasses , Ġpositive Ġand Ġnegative . </s> Mor ality : ĠFeatures Ġbased Ġon Ġmorality Ġfoundation Ġtheory ĠB IB REF 16 Ġwhere Ġwords Ġare Ġlabeled Ġin Ġone Ġof Ġthe Ġfollowing Ġ10 Ġcategories Ġ( care , Ġharm , Ġfairness , Ġcheating , Ġloyalty , Ġbetrayal , Ġauthority , Ġsub version , Ġsanct ity , Ġand Ġdegradation ). </s> Style : ĠWe Ġuse Ġcanonical Ġstyl istic Ġfeatures , Ġsuch Ġas Ġthe Ġcount Ġof Ġquestion Ġmarks , Ġex clamation Ġmarks , Ġconsecutive Ġcharacters Ġand Ġletters , Ġlinks , Ġhasht ags , Ġusers ' Ġmentions . ĠIn Ġaddition , Ġwe Ġextract Ġthe Ġu pperc ase Ġratio Ġand Ġthe Ġtweet Ġlength . </s> Words Ġembed d ings : ĠWe Ġextract Ġwords Ġembed d ings Ġof Ġthe Ġwords Ġof Ġthe Ġtweet Ġusing Ġ$ Gl ove \- 840 B - 300 d $ ĠB IB REF 17 Ġpret rained Ġmodel . ĠThe Ġtweet Ġfinal Ġrepresentation Ġis Ġobtained Ġby Ġaveraging Ġits Ġwords Ġembed d ings . </s> Model . ĠTo Ġaccount Ġfor Ġchunk Ġsequences Ġwe Ġmake Ġuse Ġof Ġa Ġde Ġfacto Ġstandard Ġapproach Ġand Ġopt Ġfor Ġa Ġrecurrent Ġneural Ġmodel Ġusing Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 18 . ĠIn Ġour Ġmodel , Ġthe Ġsequence Ġconsists Ġof Ġa Ġsequence Ġof Ġtweets Ġbelonging Ġto Ġone Ġchunk Ġ( Figure ĠFIG REF 11 ). ĠThe ĠL ST M Ġlearns Ġthe Ġhidden Ġstate Ġ$ h \ text sub script Ġ{ t } $ Ġby Ġcapturing Ġthe Ġprevious Ġtim est eps Ġ( past Ġtweets ). ĠThe Ġproduced Ġhidden Ġstate Ġ$ h \ text sub script Ġ{ t } $ Ġat Ġeach Ġtime Ġstep Ġis Ġpassed Ġto Ġthe Ġattention Ġlayer Ġwhich Ġcomp utes Ġa Ġ` context ' Ġvector Ġ$ c \ text sub script Ġ{ t } $ Ġas Ġthe Ġweighted Ġmean Ġof Ġthe Ġstate Ġsequence Ġ$ h $ Ġby : </s> Where Ġ$ T $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġtim est eps Ġin Ġthe Ġinput Ġsequence Ġand Ġ$\ alpha Ġ\ text sub script Ġ{ t j } $ Ġis Ġa Ġweight Ġcomputed Ġat Ġeach Ġtime Ġstep Ġ$ j $ Ġfor Ġeach Ġstate Ġh j . </s> Exper iments Ġand ĠResults </s> Data . ĠWe Ġbuild Ġa Ġdataset Ġof ĠTwitter Ġaccounts Ġbased Ġon Ġtwo Ġlists Ġannot ated Ġin Ġprevious Ġworks . ĠFor Ġthe Ġnon - fact ual Ġaccounts , Ġwe Ġrely Ġon Ġa Ġlist Ġof Ġ180 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 1 . ĠThis Ġlist Ġwas Ġcreated Ġbased Ġon Ġpublic Ġresources Ġwhere Ġsuspicious ĠTwitter Ġaccounts Ġwere Ġannot ated Ġwith Ġthe Ġmain Ġfake Ġnews Ġtypes Ġ( click b ait , Ġpropaganda , Ġsatire , Ġand Ġhoax ). ĠWe Ġdiscard Ġthe Ġsatire Ġlabeled Ġaccounts Ġsince Ġtheir Ġintention Ġis Ġnot Ġto Ġmislead Ġor Ġdeceive . ĠOn Ġthe Ġother Ġhand , Ġfor Ġthe Ġfactual Ġaccounts , Ġwe Ġuse Ġa Ġlist Ġwith Ġanother Ġ32 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 19 Ġthat Ġare Ġconsidered Ġtrustworthy Ġby Ġindependent Ġthird Ġparties . ĠWe Ġdiscard Ġsome Ġaccounts Ġthat Ġpublish Ġnews Ġin Ġlanguages Ġother Ġthan ĠEnglish Ġ( e . g ., ĠRussian Ġor ĠArabic ). ĠMoreover , Ġto Ġensure Ġthe Ġquality Ġof Ġthe Ġdata , Ġwe Ġremove Ġthe Ġduplicate , Ġmedia - based , Ġand Ġlink - only Ġtweets . ĠFor Ġeach Ġaccount , Ġwe Ġcollect Ġthe Ġmaximum Ġamount Ġof Ġtweets Ġallowed Ġby ĠTwitter ĠAPI . ĠTable ĠT AB REF 13 Ġpresents Ġstatistics Ġon Ġour Ġdataset . </s> Bas elines . ĠWe Ġcompare Ġour Ġapproach Ġ( Fac Tweet ) Ġto Ġthe Ġfollowing Ġset Ġof Ġbas elines : </s> [ left margin = 4 mm ] </s> LR Ġ+ ĠBag - of - words : ĠWe Ġaggregate Ġthe Ġtweets Ġof Ġa Ġfeed Ġand Ġwe Ġuse Ġa Ġbag - of - words Ġrepresentation Ġwith Ġa Ġlog istic Ġregression Ġ( LR ) Ġclass ifier . </s> Tweet 2 vec : ĠWe Ġuse Ġthe ĠBid irection al ĠG ated Ġrecurrent Ġneural Ġnetwork Ġmodel Ġproposed Ġin ĠB IB REF 20 . ĠWe Ġkeep Ġthe Ġdefault Ġparameters Ġthat Ġwere Ġprovided Ġwith Ġthe Ġimplementation . ĠTo Ġrepresent Ġthe Ġtweets , Ġwe Ġuse Ġthe Ġdec oded Ġembed ding Ġproduced Ġby Ġthe Ġmodel . ĠWith Ġthis Ġbaseline Ġwe Ġaim Ġat Ġassessing Ġif Ġthe Ġtweets ' Ġhasht ags Ġmay Ġhelp Ġdetecting Ġthe Ġnon - fact ual Ġaccounts . </s> LR Ġ+ ĠAll ĠFeatures Ġ( t weet - level ): ĠWe Ġextract Ġall Ġour Ġfeatures Ġfrom Ġeach Ġtweet Ġand Ġfeed Ġthem Ġinto Ġa ĠLR Ġclass ifier . ĠHere , Ġwe Ġdo Ġnot Ġaggregate Ġover Ġtweets Ġand Ġthus Ġview Ġeach Ġtweet Ġindependently . </s> LR Ġ+ ĠAll ĠFeatures Ġ( ch unk - level ): ĠWe Ġconc aten ate Ġthe Ġfeatures ' Ġvectors Ġof Ġthe Ġtweets Ġin Ġa Ġchunk Ġand Ġfeed Ġthem Ġinto Ġa ĠLR Ġclass ifier . </s> Fac Tweet Ġ( t weet - level ): ĠSimilar Ġto Ġthe ĠFac Tweet Ġapproach , Ġbut Ġat Ġtweet - level ; Ġthe Ġsequential Ġflow Ġof Ġthe Ġtweets Ġis Ġnot Ġutilized . ĠWe Ġaim Ġat Ġinvestigating Ġthe Ġimportance Ġof Ġthe Ġsequential Ġflow Ġof Ġtweets . </s> Top -$ k $ Ġreplies , Ġlikes , Ġor Ġre - t we ets : ĠSome Ġapproaches Ġin Ġrumors Ġdetection Ġuse Ġthe Ġnumber Ġof Ġreplies , Ġlikes , Ġand Ġre - t we ets Ġto Ġdetect Ġrumors ĠB IB REF 21 . ĠThus , Ġwe Ġextract Ġtop Ġ$ k $ Ġreplied , Ġliked Ġor Ġre - t weet ed Ġtweets Ġfrom Ġeach Ġaccount Ġto Ġassess Ġthe Ġaccounts Ġfact uality . ĠWe Ġtested Ġdifferent Ġ$ k $ Ġvalues Ġbetween Ġ10 Ġtweets Ġto Ġthe Ġmax Ġnumber Ġof Ġtweets Ġfrom Ġeach Ġaccount . ĠFigure ĠFIG REF 24 Ġshows Ġthe Ġmacro - F 1 Ġvalues Ġfor Ġdifferent Ġ$ k $ Ġvalues . ĠIt Ġseems Ġthat Ġ$ k = 500 $ Ġfor Ġthe Ġtop Ġreplied Ġtweets Ġachieves Ġthe Ġhighest Ġresult . ĠTherefore , Ġwe Ġconsider Ġthis Ġas Ġa Ġbaseline . </s> Exper imental ĠSetup . ĠWe Ġapply Ġa Ġ5 Ġcross - valid ation Ġon Ġthe Ġaccount 's Ġlevel . ĠFor Ġthe ĠFac Tweet Ġmodel , Ġwe Ġexperiment Ġwith Ġ25 % Ġof Ġthe Ġaccounts Ġfor Ġvalidation Ġand Ġparameters Ġselection . ĠWe Ġuse Ġhyper opt Ġlibrary Ġto Ġselect Ġthe Ġhyper - param eters Ġon Ġthe Ġfollowing Ġvalues : ĠL ST M Ġlayer Ġsize Ġ( 16 , Ġ32 , Ġ64 ), Ġdrop out Ġ($ 0 . 0 - 0 . 9 $ ), Ġactivation Ġfunction Ġ($ re lu $, Ġ$ sel u $, Ġ$ tan h $ ), Ġoptim izer Ġ($ s gd $, Ġ$ ad am $, Ġ$ r ms prop $ ) Ġwith Ġvarying Ġthe Ġvalue Ġof Ġthe Ġlearning Ġrate Ġ( 1 e - 1 , .. ,- 5 ), Ġand Ġbatch Ġsize Ġ( 4 , Ġ8 , Ġ16 ). ĠThe Ġvalidation Ġsplit Ġis Ġextracted Ġon Ġthe Ġclass Ġlevel Ġusing Ġstrat ified Ġsampling : Ġwe Ġtook Ġa Ġrandom Ġ25 % Ġof Ġthe Ġaccounts Ġfrom Ġeach Ġclass Ġsince Ġthe Ġdataset Ġis Ġun balanced . ĠDisc arding Ġthe Ġclasses ' Ġsize Ġin Ġthe Ġsplitting Ġprocess Ġmay Ġaffect Ġthe Ġminority Ġclasses Ġ( e . g . Ġhoax ). ĠFor Ġthe Ġbas elines ' Ġclass ifier , Ġwe Ġtested Ġmany Ġclass ifiers Ġand Ġthe ĠLR Ġshowed Ġthe Ġbest Ġoverall Ġperformance . </s> Results . ĠTable ĠT AB REF 25 Ġpresents Ġthe Ġresults . ĠWe Ġpresent Ġthe Ġresults Ġusing Ġa Ġchunk Ġsize Ġof Ġ20 , Ġwhich Ġwas Ġfound Ġto Ġbe Ġthe Ġbest Ġsize Ġon Ġthe Ġheld - out Ġdata . ĠFigure ĠFIG REF 24 Ġshows Ġthe Ġresults Ġof Ġdifferent Ġchunks Ġsizes . ĠFac Tweet Ġperforms Ġbetter Ġthan Ġthe Ġproposed Ġbas elines Ġand Ġob tains Ġthe Ġhighest Ġmacro - F 1 Ġvalue Ġof Ġ$ 0 . 565 $. ĠOur Ġresults Ġindicate Ġthe Ġimportance Ġof Ġtaking Ġinto Ġaccount Ġthe Ġsequence Ġof Ġthe Ġtweets Ġin Ġthe Ġaccounts ' Ġtimelines . ĠThe Ġsequence Ġof Ġthese Ġtweets Ġis Ġbetter Ġcaptured Ġby Ġour Ġproposed Ġmodel Ġsequence - agn ostic Ġor Ġnon - ne ural Ġclass ifiers . ĠMoreover , Ġthe Ġresults Ġdemonstrate Ġthat Ġthe Ġfeatures Ġat Ġtweet - level Ġdo Ġnot Ġperform Ġwell Ġto Ġdetect Ġthe ĠTwitter Ġaccounts Ġfact uality , Ġsince Ġthey Ġobtain Ġa Ġresult Ġnear Ġto Ġthe Ġmajority Ġclass Ġ($ 0 . 18 $ ). ĠAnother Ġfinding Ġfrom Ġour Ġexperiments Ġshows Ġthat Ġthe Ġperformance Ġof Ġthe ĠTweet 2 vec Ġis Ġweak . ĠThis Ġdemonstrates Ġthat Ġtweets ' Ġhasht ags Ġare Ġnot Ġinformative Ġto Ġdetect Ġnon - fact ual Ġaccounts . ĠIn ĠTable ĠT AB REF 25 , Ġwe Ġpresent Ġab lation Ġtests Ġso Ġas Ġto Ġquantify Ġthe Ġcontribution Ġof Ġsubset Ġof Ġfeatures . ĠThe Ġresults Ġindicate Ġthat Ġmost Ġperformance Ġgains Ġcome Ġfrom Ġwords Ġembed d ings , Ġstyle , Ġand Ġmorality Ġfeatures . ĠOther Ġfeatures Ġ( em otion Ġand Ġsentiment ) Ġshow Ġlower Ġimportance : Ġnevertheless , Ġthey Ġstill Ġimprove Ġthe Ġoverall Ġsystem Ġperformance Ġ( on Ġaverage Ġ0 . 35 % ĠMacro - F $ _ 1 $ Ġimprovement ). ĠThese Ġperformance Ġfigures Ġsuggest Ġthat Ġnon - fact ual Ġaccounts Ġuse Ġsemantic Ġand Ġstyl istic Ġhidden Ġsignatures Ġmostly Ġwhile Ġtweeting Ġnews , Ġso Ġas Ġto Ġbe Ġable Ġto Ġmislead Ġthe Ġreaders Ġand Ġbehave Ġas Ġreputable Ġ( i . e ., Ġfactual ) Ġsources . ĠWe Ġleave Ġa Ġmore Ġfine - gr ained , Ġdi ach ronic Ġanalysis Ġof Ġsemantic Ġand Ġstyl istic Ġfeatures ĠâĢĵ Ġhow Ġsemantic Ġand Ġstyl istic Ġsignature Ġevolve Ġacross Ġtime Ġand Ġchange Ġacross Ġthe Ġaccounts ' Ġtimelines ĠâĢĵ Ġfor Ġfuture Ġwork . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġmodel Ġthat Ġutilizes Ġchunk ed Ġtimelines Ġof Ġtweets Ġand Ġa Ġrecurrent Ġneural Ġmodel Ġin Ġorder Ġto Ġinfer Ġthe Ġfact uality Ġof Ġa ĠTwitter Ġnews Ġaccount . ĠOur Ġexperimental Ġresults Ġindicate Ġthe Ġimportance Ġof Ġanalyzing Ġtweet Ġstream Ġinto Ġchunks , Ġas Ġwell Ġas Ġthe Ġbenefits Ġof Ġheter ogeneous Ġknowledge Ġsource Ġ( i . e ., Ġlex ica Ġas Ġwell Ġas Ġtext ) Ġin Ġorder Ġto Ġcapture Ġfact uality . ĠIn Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġextend Ġthis Ġline Ġof Ġresearch Ġwith Ġfurther Ġin - depth Ġanalysis Ġto Ġunderstand Ġthe Ġflow Ġchange Ġof Ġthe Ġused Ġfeatures Ġin Ġthe Ġaccounts ' Ġstreams . ĠMoreover , Ġwe Ġwould Ġlike Ġto Ġtake Ġour Ġapproach Ġone Ġstep Ġfurther Ġincorporating Ġexplicit Ġtemporal Ġinformation , Ġe . g ., Ġusing Ġtim est amps . ĠCru cially , Ġwe Ġare Ġalso Ġinterested Ġin Ġdeveloping Ġa Ġmult ilingual Ġversion Ġof Ġour Ġapproach , Ġfor Ġinstance Ġby Ġleveraging Ġthe Ġnow Ġubiquitous Ġcross - ling ual Ġembed d ings ĠB IB REF 22 , ĠB IB REF 23 .
Introduction Social media platforms have made the spreading of fake news easier, faster as well as able to reach a wider audience. Social media offer another feature which is the anonymity for the authors, and this opens the door to many suspicious individuals or organizations to utilize these platforms. Recently, there has been an increased number of spreading fake news and rumors over the web and social media BIBREF0. Fake news in social media vary considering the intention to mislead. Some of these news are spread with the intention to be ironic or to deliver the news in an ironic way (satirical news). Others, such as propaganda, hoaxes, and clickbaits, are spread to mislead the audience or to manipulate their opinions. In the case of Twitter, suspicious news annotations should be done on a tweet rather than an account level, since some accounts mix fake with real news. However, these annotations are extremely costly and time consuming – i.e., due to high volume of available tweets Consequently, a first step in this direction, e.g., as a pre-filtering step, can be viewed as the task of detecting fake news at the account level. The main obstacle for detecting suspicious Twitter accounts is due to the behavior of mixing some real news with the misleading ones. Consequently, we investigate ways to detect suspicious accounts by considering their tweets in groups (chunks). Our hypothesis is that suspicious accounts have a unique pattern in posting tweet sequences. Since their intention is to mislead, the way they transition from one set of tweets to the next has a hidden signature, biased by their intentions. Therefore, reading these tweets in chunks has the potential to improve the detection of the fake news accounts. In this work, we investigate the problem of discriminating between factual and non-factual accounts in Twitter. To this end, we collect a large dataset of tweets using a list of propaganda, hoax and clickbait accounts and compare different versions of sequential chunk-based approaches using a variety of feature sets against several baselines. Several approaches have been proposed for news verification, whether in social media (rumors detection) BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, or in news claims BIBREF5, BIBREF6, BIBREF7, BIBREF8. The main orientation in the previous works is to verify the textual claims/tweets but not their sources. To the best of our knowledge, this is the first work aiming to detect factuality at the account level, and especially from a textual perspective. Our contributions are: [leftmargin=4mm] We propose an approach to detect non-factual Twitter accounts by treating post streams as a sequence of tweets' chunks. We test several semantic and dictionary-based features together with a neural sequential approach, and apply an ablation test to investigate their contribution. We benchmark our approach against other approaches that discard the chronological order of the tweets or read the tweets individually. The results show that our approach produces superior results at detecting non-factual accounts. Methodology Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. We extract a set of features from each chunk and we feed them into a recurrent neural network to model the sequential flow of the chunks' tweets. We use an attention layer with dropout to attend over the most important tweets in each chunk. Finally, the representation is fed into a softmax layer to produce a probability distribution over the account types and thus predict the factuality of the accounts. Since we have many chunks for each account, the label for an account is obtained by taking the majority class of the account's chunks. Input Representation. Let $t$ be a Twitter account that contains $m$ tweets. These tweets are sorted by date and split into a sequence of chunks $ck = \langle ck_1, \ldots , ck_n \rangle $, where each $ck_i$ contains $s$ tweets. Each tweet in $ck_i$ is represented by a vector $v \in {\rm I\!R}^d$ , where $v$ is the concatenation of a set of features' vectors, that is $v = \langle f_1, \ldots , f_n \rangle $. Each feature vector $f_i$ is built by counting the presence of tweet's words in a set of lexical lists. The final representation of the tweet is built by averaging the single word vectors. Features. We argue that different kinds of features like the sentiment of the text, morality, and other text-based features are critical to detect the nonfactual Twitter accounts by utilizing their occurrence during reporting the news in an account's timeline. We employ a rich set of features borrowed from previous works in fake news, bias, and rumors detection BIBREF0, BIBREF1, BIBREF8, BIBREF9. [leftmargin=4mm] Emotion: We build an emotions vector using word occurrences of 15 emotion types from two available emotional lexicons. We use the NRC lexicon BIBREF10, which contains $\sim $14K words labeled using the eight Plutchik's emotions BIBREF11. The other lexicon is SentiSense BIBREF12 which is a concept-based affective lexicon that attaches emotional meanings to concepts from the WordNet lexical database. It has $\sim $5.5 words labeled with emotions from a set of 14 emotional categories We use the categories that do not exist in the NRC lexicon. Sentiment: We extract the sentiment of the tweets by employing EffectWordNet BIBREF13, SenticNet BIBREF14, NRC BIBREF10, and subj_lexicon BIBREF15, where each has the two sentiment classes, positive and negative. Morality: Features based on morality foundation theory BIBREF16 where words are labeled in one of the following 10 categories (care, harm, fairness, cheating, loyalty, betrayal, authority, subversion, sanctity, and degradation). Style: We use canonical stylistic features, such as the count of question marks, exclamation marks, consecutive characters and letters, links, hashtags, users' mentions. In addition, we extract the uppercase ratio and the tweet length. Words embeddings: We extract words embeddings of the words of the tweet using $Glove\-840B-300d$ BIBREF17 pretrained model. The tweet final representation is obtained by averaging its words embeddings. Model. To account for chunk sequences we make use of a de facto standard approach and opt for a recurrent neural model using long short-term memory (LSTM) BIBREF18. In our model, the sequence consists of a sequence of tweets belonging to one chunk (Figure FIGREF11). The LSTM learns the hidden state $h\textsubscript {t}$ by capturing the previous timesteps (past tweets). The produced hidden state $h\textsubscript {t}$ at each time step is passed to the attention layer which computes a `context' vector $c\textsubscript {t}$ as the weighted mean of the state sequence $h$ by: Where $T$ is the total number of timesteps in the input sequence and $\alpha \textsubscript {tj}$ is a weight computed at each time step $j$ for each state hj. Experiments and Results Data. We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. This list was created based on public resources where suspicious Twitter accounts were annotated with the main fake news types (clickbait, propaganda, satire, and hoax). We discard the satire labeled accounts since their intention is not to mislead or deceive. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. We discard some accounts that publish news in languages other than English (e.g., Russian or Arabic). Moreover, to ensure the quality of the data, we remove the duplicate, media-based, and link-only tweets. For each account, we collect the maximum amount of tweets allowed by Twitter API. Table TABREF13 presents statistics on our dataset. Baselines. We compare our approach (FacTweet) to the following set of baselines: [leftmargin=4mm] LR + Bag-of-words: We aggregate the tweets of a feed and we use a bag-of-words representation with a logistic regression (LR) classifier. Tweet2vec: We use the Bidirectional Gated recurrent neural network model proposed in BIBREF20. We keep the default parameters that were provided with the implementation. To represent the tweets, we use the decoded embedding produced by the model. With this baseline we aim at assessing if the tweets' hashtags may help detecting the non-factual accounts. LR + All Features (tweet-level): We extract all our features from each tweet and feed them into a LR classifier. Here, we do not aggregate over tweets and thus view each tweet independently. LR + All Features (chunk-level): We concatenate the features' vectors of the tweets in a chunk and feed them into a LR classifier. FacTweet (tweet-level): Similar to the FacTweet approach, but at tweet-level; the sequential flow of the tweets is not utilized. We aim at investigating the importance of the sequential flow of tweets. Top-$k$ replies, likes, or re-tweets: Some approaches in rumors detection use the number of replies, likes, and re-tweets to detect rumors BIBREF21. Thus, we extract top $k$ replied, liked or re-tweeted tweets from each account to assess the accounts factuality. We tested different $k$ values between 10 tweets to the max number of tweets from each account. Figure FIGREF24 shows the macro-F1 values for different $k$ values. It seems that $k=500$ for the top replied tweets achieves the highest result. Therefore, we consider this as a baseline. Experimental Setup. We apply a 5 cross-validation on the account's level. For the FacTweet model, we experiment with 25% of the accounts for validation and parameters selection. We use hyperopt library to select the hyper-parameters on the following values: LSTM layer size (16, 32, 64), dropout ($0.0-0.9$), activation function ($relu$, $selu$, $tanh$), optimizer ($sgd$, $adam$, $rmsprop$) with varying the value of the learning rate (1e-1,..,-5), and batch size (4, 8, 16). The validation split is extracted on the class level using stratified sampling: we took a random 25% of the accounts from each class since the dataset is unbalanced. Discarding the classes' size in the splitting process may affect the minority classes (e.g. hoax). For the baselines' classifier, we tested many classifiers and the LR showed the best overall performance. Results. Table TABREF25 presents the results. We present the results using a chunk size of 20, which was found to be the best size on the held-out data. Figure FIGREF24 shows the results of different chunks sizes. FacTweet performs better than the proposed baselines and obtains the highest macro-F1 value of $0.565$. Our results indicate the importance of taking into account the sequence of the tweets in the accounts' timelines. The sequence of these tweets is better captured by our proposed model sequence-agnostic or non-neural classifiers. Moreover, the results demonstrate that the features at tweet-level do not perform well to detect the Twitter accounts factuality, since they obtain a result near to the majority class ($0.18$). Another finding from our experiments shows that the performance of the Tweet2vec is weak. This demonstrates that tweets' hashtags are not informative to detect non-factual accounts. In Table TABREF25, we present ablation tests so as to quantify the contribution of subset of features. The results indicate that most performance gains come from words embeddings, style, and morality features. Other features (emotion and sentiment) show lower importance: nevertheless, they still improve the overall system performance (on average 0.35% Macro-F$_1$ improvement). These performance figures suggest that non-factual accounts use semantic and stylistic hidden signatures mostly while tweeting news, so as to be able to mislead the readers and behave as reputable (i.e., factual) sources. We leave a more fine-grained, diachronic analysis of semantic and stylistic features – how semantic and stylistic signature evolve across time and change across the accounts' timelines – for future work. Conclusions In this paper, we proposed a model that utilizes chunked timelines of tweets and a recurrent neural model in order to infer the factuality of a Twitter news account. Our experimental results indicate the importance of analyzing tweet stream into chunks, as well as the benefits of heterogeneous knowledge source (i.e., lexica as well as text) in order to capture factuality. In future work, we would like to extend this line of research with further in-depth analysis to understand the flow change of the used features in the accounts' streams. Moreover, we would like to take our approach one step further incorporating explicit temporal information, e.g., using timestamps. Crucially, we are also interested in developing a multilingual version of our approach, for instance by leveraging the now ubiquitous cross-lingual embeddings BIBREF22, BIBREF23.
How did they obtain the dataset?
e35c2fa99d5c84d8cb5d83fca2b434dcd83f3851
How Ġdid Ġthey Ġobtain Ġthe Ġdataset ?
For Ġthe Ġnon - fact ual Ġaccounts , Ġwe Ġrely Ġon Ġa Ġlist Ġof Ġ180 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 1 , Ġwe Ġuse Ġa Ġlist Ġwith Ġanother Ġ32 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 19 Ġthat Ġare Ġconsidered Ġtrustworthy
<s> How Ġdid Ġthey Ġobtain Ġthe Ġdataset ? </s> Introduction </s> Social Ġmedia Ġplatforms Ġhave Ġmade Ġthe Ġspreading Ġof Ġfake Ġnews Ġeasier , Ġfaster Ġas Ġwell Ġas Ġable Ġto Ġreach Ġa Ġwider Ġaudience . ĠSocial Ġmedia Ġoffer Ġanother Ġfeature Ġwhich Ġis Ġthe Ġanonymity Ġfor Ġthe Ġauthors , Ġand Ġthis Ġopens Ġthe Ġdoor Ġto Ġmany Ġsuspicious Ġindividuals Ġor Ġorganizations Ġto Ġutilize Ġthese Ġplatforms . ĠRecently , Ġthere Ġhas Ġbeen Ġan Ġincreased Ġnumber Ġof Ġspreading Ġfake Ġnews Ġand Ġrumors Ġover Ġthe Ġweb Ġand Ġsocial Ġmedia ĠB IB REF 0 . ĠFake Ġnews Ġin Ġsocial Ġmedia Ġvary Ġconsidering Ġthe Ġintention Ġto Ġmislead . ĠSome Ġof Ġthese Ġnews Ġare Ġspread Ġwith Ġthe Ġintention Ġto Ġbe Ġironic Ġor Ġto Ġdeliver Ġthe Ġnews Ġin Ġan Ġironic Ġway Ġ( sat ir ical Ġnews ). ĠOthers , Ġsuch Ġas Ġpropaganda , Ġhoax es , Ġand Ġclick ba its , Ġare Ġspread Ġto Ġmislead Ġthe Ġaudience Ġor Ġto Ġmanipulate Ġtheir Ġopinions . ĠIn Ġthe Ġcase Ġof ĠTwitter , Ġsuspicious Ġnews Ġannotations Ġshould Ġbe Ġdone Ġon Ġa Ġtweet Ġrather Ġthan Ġan Ġaccount Ġlevel , Ġsince Ġsome Ġaccounts Ġmix Ġfake Ġwith Ġreal Ġnews . ĠHowever , Ġthese Ġannotations Ġare Ġextremely Ġcostly Ġand Ġtime Ġconsuming ĠâĢĵ Ġi . e ., Ġdue Ġto Ġhigh Ġvolume Ġof Ġavailable Ġtweets ĠConsequently , Ġa Ġfirst Ġstep Ġin Ġthis Ġdirection , Ġe . g ., Ġas Ġa Ġpre - fil tering Ġstep , Ġcan Ġbe Ġviewed Ġas Ġthe Ġtask Ġof Ġdetecting Ġfake Ġnews Ġat Ġthe Ġaccount Ġlevel . </s> The Ġmain Ġobstacle Ġfor Ġdetecting Ġsuspicious ĠTwitter Ġaccounts Ġis Ġdue Ġto Ġthe Ġbehavior Ġof Ġmixing Ġsome Ġreal Ġnews Ġwith Ġthe Ġmisleading Ġones . ĠConsequently , Ġwe Ġinvestigate Ġways Ġto Ġdetect Ġsuspicious Ġaccounts Ġby Ġconsidering Ġtheir Ġtweets Ġin Ġgroups Ġ( ch unks ). ĠOur Ġhypothesis Ġis Ġthat Ġsuspicious Ġaccounts Ġhave Ġa Ġunique Ġpattern Ġin Ġposting Ġtweet Ġsequences . ĠSince Ġtheir Ġintention Ġis Ġto Ġmislead , Ġthe Ġway Ġthey Ġtransition Ġfrom Ġone Ġset Ġof Ġtweets Ġto Ġthe Ġnext Ġhas Ġa Ġhidden Ġsignature , Ġbiased Ġby Ġtheir Ġintentions . ĠTherefore , Ġreading Ġthese Ġtweets Ġin Ġchunks Ġhas Ġthe Ġpotential Ġto Ġimprove Ġthe Ġdetection Ġof Ġthe Ġfake Ġnews Ġaccounts . </s> In Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġproblem Ġof Ġdiscriminating Ġbetween Ġfactual Ġand Ġnon - fact ual Ġaccounts Ġin ĠTwitter . ĠTo Ġthis Ġend , Ġwe Ġcollect Ġa Ġlarge Ġdataset Ġof Ġtweets Ġusing Ġa Ġlist Ġof Ġpropaganda , Ġhoax Ġand Ġclick b ait Ġaccounts Ġand Ġcompare Ġdifferent Ġversions Ġof Ġsequential Ġchunk - based Ġapproaches Ġusing Ġa Ġvariety Ġof Ġfeature Ġsets Ġagainst Ġseveral Ġbas elines . ĠSeveral Ġapproaches Ġhave Ġbeen Ġproposed Ġfor Ġnews Ġverification , Ġwhether Ġin Ġsocial Ġmedia Ġ( rum ors Ġdetection ) ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , Ġor Ġin Ġnews Ġclaims ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 . ĠThe Ġmain Ġorientation Ġin Ġthe Ġprevious Ġworks Ġis Ġto Ġverify Ġthe Ġtextual Ġclaims / t we ets Ġbut Ġnot Ġtheir Ġsources . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġaiming Ġto Ġdetect Ġfact uality Ġat Ġthe Ġaccount Ġlevel , Ġand Ġespecially Ġfrom Ġa Ġtextual Ġperspective . ĠOur Ġcontributions Ġare : </s> [ left margin = 4 mm ] </s> We Ġpropose Ġan Ġapproach Ġto Ġdetect Ġnon - fact ual ĠTwitter Ġaccounts Ġby Ġtreating Ġpost Ġstreams Ġas Ġa Ġsequence Ġof Ġtweets ' Ġchunks . ĠWe Ġtest Ġseveral Ġsemantic Ġand Ġdictionary - based Ġfeatures Ġtogether Ġwith Ġa Ġneural Ġsequential Ġapproach , Ġand Ġapply Ġan Ġab lation Ġtest Ġto Ġinvestigate Ġtheir Ġcontribution . </s> We Ġbenchmark Ġour Ġapproach Ġagainst Ġother Ġapproaches Ġthat Ġdiscard Ġthe Ġchronological Ġorder Ġof Ġthe Ġtweets Ġor Ġread Ġthe Ġtweets Ġindividually . ĠThe Ġresults Ġshow Ġthat Ġour Ġapproach Ġproduces Ġsuperior Ġresults Ġat Ġdetecting Ġnon - fact ual Ġaccounts . </s> Method ology </s> Given Ġa Ġnews ĠTwitter Ġaccount , Ġwe Ġread Ġits Ġtweets Ġfrom Ġthe Ġaccount 's Ġtimeline . ĠThen Ġwe Ġsort Ġthe Ġtweets Ġby Ġthe Ġposting Ġdate Ġin Ġascending Ġway Ġand Ġwe Ġsplit Ġthem Ġinto Ġ$ N $ Ġchunks . ĠEach Ġchunk Ġconsists Ġof Ġa Ġsorted Ġsequence Ġof Ġtweets Ġlabeled Ġby Ġthe Ġlabel Ġof Ġits Ġcorresponding Ġaccount . ĠWe Ġextract Ġa Ġset Ġof Ġfeatures Ġfrom Ġeach Ġchunk Ġand Ġwe Ġfeed Ġthem Ġinto Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġmodel Ġthe Ġsequential Ġflow Ġof Ġthe Ġchunks ' Ġtweets . ĠWe Ġuse Ġan Ġattention Ġlayer Ġwith Ġdrop out Ġto Ġattend Ġover Ġthe Ġmost Ġimportant Ġtweets Ġin Ġeach Ġchunk . ĠFinally , Ġthe Ġrepresentation Ġis Ġfed Ġinto Ġa Ġsoft max Ġlayer Ġto Ġproduce Ġa Ġprobability Ġdistribution Ġover Ġthe Ġaccount Ġtypes Ġand Ġthus Ġpredict Ġthe Ġfact uality Ġof Ġthe Ġaccounts . ĠSince Ġwe Ġhave Ġmany Ġchunks Ġfor Ġeach Ġaccount , Ġthe Ġlabel Ġfor Ġan Ġaccount Ġis Ġobtained Ġby Ġtaking Ġthe Ġmajority Ġclass Ġof Ġthe Ġaccount 's Ġchunks . </s> Input ĠRepresent ation . ĠLet Ġ$ t $ Ġbe Ġa ĠTwitter Ġaccount Ġthat Ġcontains Ġ$ m $ Ġtweets . ĠThese Ġtweets Ġare Ġsorted Ġby Ġdate Ġand Ġsplit Ġinto Ġa Ġsequence Ġof Ġchunks Ġ$ ck Ġ= Ġ\ l angle Ġc k _ 1 , Ġ\ ld ots Ġ, Ġc k _ n Ġ\ r angle Ġ$ , Ġwhere Ġeach Ġ$ ck _ i $ Ġcontains Ġ$ s $ Ġtweets . ĠEach Ġtweet Ġin Ġ$ ck _ i $ Ġis Ġrepresented Ġby Ġa Ġvector Ġ$ v Ġ\ in Ġ{\ rm ĠI \ ! R } ^ d $ Ġ, Ġwhere Ġ$ v $ Ġis Ġthe Ġconc aten ation Ġof Ġa Ġset Ġof Ġfeatures ' Ġvectors , Ġthat Ġis Ġ$ v Ġ= Ġ\ l angle Ġf _ 1 , Ġ\ ld ots Ġ, Ġf _ n Ġ\ r angle Ġ$ . ĠEach Ġfeature Ġvector Ġ$ f _ i $ Ġis Ġbuilt Ġby Ġcounting Ġthe Ġpresence Ġof Ġtweet 's Ġwords Ġin Ġa Ġset Ġof Ġlex ical Ġlists . ĠThe Ġfinal Ġrepresentation Ġof Ġthe Ġtweet Ġis Ġbuilt Ġby Ġaveraging Ġthe Ġsingle Ġword Ġvectors . </s> Features . ĠWe Ġargue Ġthat Ġdifferent Ġkinds Ġof Ġfeatures Ġlike Ġthe Ġsentiment Ġof Ġthe Ġtext , Ġmorality , Ġand Ġother Ġtext - based Ġfeatures Ġare Ġcritical Ġto Ġdetect Ġthe Ġnon fact ual ĠTwitter Ġaccounts Ġby Ġutilizing Ġtheir Ġoccurrence Ġduring Ġreporting Ġthe Ġnews Ġin Ġan Ġaccount 's Ġtimeline . ĠWe Ġemploy Ġa Ġrich Ġset Ġof Ġfeatures Ġborrowed Ġfrom Ġprevious Ġworks Ġin Ġfake Ġnews , Ġbias , Ġand Ġrumors Ġdetection ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 8 , ĠB IB REF 9 . </s> [ left margin = 4 mm ] </s> Em otion : ĠWe Ġbuild Ġan Ġemotions Ġvector Ġusing Ġword Ġoccurrences Ġof Ġ15 Ġemotion Ġtypes Ġfrom Ġtwo Ġavailable Ġemotional Ġlex icons . ĠWe Ġuse Ġthe ĠN RC Ġlex icon ĠB IB REF 10 , Ġwhich Ġcontains Ġ$\ sim Ġ$ 14 K Ġwords Ġlabeled Ġusing Ġthe Ġeight ĠPl utch ik 's Ġemotions ĠB IB REF 11 . ĠThe Ġother Ġlex icon Ġis ĠSent i Sense ĠB IB REF 12 Ġwhich Ġis Ġa Ġconcept - based Ġaffect ive Ġlex icon Ġthat Ġattaches Ġemotional Ġmeanings Ġto Ġconcepts Ġfrom Ġthe ĠWord Net Ġlex ical Ġdatabase . ĠIt Ġhas Ġ$\ sim Ġ$ 5 . 5 Ġwords Ġlabeled Ġwith Ġemotions Ġfrom Ġa Ġset Ġof Ġ14 Ġemotional Ġcategories ĠWe Ġuse Ġthe Ġcategories Ġthat Ġdo Ġnot Ġexist Ġin Ġthe ĠN RC Ġlex icon . </s> Sent iment : ĠWe Ġextract Ġthe Ġsentiment Ġof Ġthe Ġtweets Ġby Ġemploying ĠEffect Word Net ĠB IB REF 13 , ĠSent ic Net ĠB IB REF 14 , ĠN RC ĠB IB REF 10 , Ġand Ġsub j _ lex icon ĠB IB REF 15 , Ġwhere Ġeach Ġhas Ġthe Ġtwo Ġsentiment Ġclasses , Ġpositive Ġand Ġnegative . </s> Mor ality : ĠFeatures Ġbased Ġon Ġmorality Ġfoundation Ġtheory ĠB IB REF 16 Ġwhere Ġwords Ġare Ġlabeled Ġin Ġone Ġof Ġthe Ġfollowing Ġ10 Ġcategories Ġ( care , Ġharm , Ġfairness , Ġcheating , Ġloyalty , Ġbetrayal , Ġauthority , Ġsub version , Ġsanct ity , Ġand Ġdegradation ). </s> Style : ĠWe Ġuse Ġcanonical Ġstyl istic Ġfeatures , Ġsuch Ġas Ġthe Ġcount Ġof Ġquestion Ġmarks , Ġex clamation Ġmarks , Ġconsecutive Ġcharacters Ġand Ġletters , Ġlinks , Ġhasht ags , Ġusers ' Ġmentions . ĠIn Ġaddition , Ġwe Ġextract Ġthe Ġu pperc ase Ġratio Ġand Ġthe Ġtweet Ġlength . </s> Words Ġembed d ings : ĠWe Ġextract Ġwords Ġembed d ings Ġof Ġthe Ġwords Ġof Ġthe Ġtweet Ġusing Ġ$ Gl ove \- 840 B - 300 d $ ĠB IB REF 17 Ġpret rained Ġmodel . ĠThe Ġtweet Ġfinal Ġrepresentation Ġis Ġobtained Ġby Ġaveraging Ġits Ġwords Ġembed d ings . </s> Model . ĠTo Ġaccount Ġfor Ġchunk Ġsequences Ġwe Ġmake Ġuse Ġof Ġa Ġde Ġfacto Ġstandard Ġapproach Ġand Ġopt Ġfor Ġa Ġrecurrent Ġneural Ġmodel Ġusing Ġlong Ġshort - term Ġmemory Ġ( L ST M ) ĠB IB REF 18 . ĠIn Ġour Ġmodel , Ġthe Ġsequence Ġconsists Ġof Ġa Ġsequence Ġof Ġtweets Ġbelonging Ġto Ġone Ġchunk Ġ( Figure ĠFIG REF 11 ). ĠThe ĠL ST M Ġlearns Ġthe Ġhidden Ġstate Ġ$ h \ text sub script Ġ{ t } $ Ġby Ġcapturing Ġthe Ġprevious Ġtim est eps Ġ( past Ġtweets ). ĠThe Ġproduced Ġhidden Ġstate Ġ$ h \ text sub script Ġ{ t } $ Ġat Ġeach Ġtime Ġstep Ġis Ġpassed Ġto Ġthe Ġattention Ġlayer Ġwhich Ġcomp utes Ġa Ġ` context ' Ġvector Ġ$ c \ text sub script Ġ{ t } $ Ġas Ġthe Ġweighted Ġmean Ġof Ġthe Ġstate Ġsequence Ġ$ h $ Ġby : </s> Where Ġ$ T $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġtim est eps Ġin Ġthe Ġinput Ġsequence Ġand Ġ$\ alpha Ġ\ text sub script Ġ{ t j } $ Ġis Ġa Ġweight Ġcomputed Ġat Ġeach Ġtime Ġstep Ġ$ j $ Ġfor Ġeach Ġstate Ġh j . </s> Exper iments Ġand ĠResults </s> Data . ĠWe Ġbuild Ġa Ġdataset Ġof ĠTwitter Ġaccounts Ġbased Ġon Ġtwo Ġlists Ġannot ated Ġin Ġprevious Ġworks . ĠFor Ġthe Ġnon - fact ual Ġaccounts , Ġwe Ġrely Ġon Ġa Ġlist Ġof Ġ180 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 1 . ĠThis Ġlist Ġwas Ġcreated Ġbased Ġon Ġpublic Ġresources Ġwhere Ġsuspicious ĠTwitter Ġaccounts Ġwere Ġannot ated Ġwith Ġthe Ġmain Ġfake Ġnews Ġtypes Ġ( click b ait , Ġpropaganda , Ġsatire , Ġand Ġhoax ). ĠWe Ġdiscard Ġthe Ġsatire Ġlabeled Ġaccounts Ġsince Ġtheir Ġintention Ġis Ġnot Ġto Ġmislead Ġor Ġdeceive . ĠOn Ġthe Ġother Ġhand , Ġfor Ġthe Ġfactual Ġaccounts , Ġwe Ġuse Ġa Ġlist Ġwith Ġanother Ġ32 ĠTwitter Ġaccounts Ġfrom ĠB IB REF 19 Ġthat Ġare Ġconsidered Ġtrustworthy Ġby Ġindependent Ġthird Ġparties . ĠWe Ġdiscard Ġsome Ġaccounts Ġthat Ġpublish Ġnews Ġin Ġlanguages Ġother Ġthan ĠEnglish Ġ( e . g ., ĠRussian Ġor ĠArabic ). ĠMoreover , Ġto Ġensure Ġthe Ġquality Ġof Ġthe Ġdata , Ġwe Ġremove Ġthe Ġduplicate , Ġmedia - based , Ġand Ġlink - only Ġtweets . ĠFor Ġeach Ġaccount , Ġwe Ġcollect Ġthe Ġmaximum Ġamount Ġof Ġtweets Ġallowed Ġby ĠTwitter ĠAPI . ĠTable ĠT AB REF 13 Ġpresents Ġstatistics Ġon Ġour Ġdataset . </s> Bas elines . ĠWe Ġcompare Ġour Ġapproach Ġ( Fac Tweet ) Ġto Ġthe Ġfollowing Ġset Ġof Ġbas elines : </s> [ left margin = 4 mm ] </s> LR Ġ+ ĠBag - of - words : ĠWe Ġaggregate Ġthe Ġtweets Ġof Ġa Ġfeed Ġand Ġwe Ġuse Ġa Ġbag - of - words Ġrepresentation Ġwith Ġa Ġlog istic Ġregression Ġ( LR ) Ġclass ifier . </s> Tweet 2 vec : ĠWe Ġuse Ġthe ĠBid irection al ĠG ated Ġrecurrent Ġneural Ġnetwork Ġmodel Ġproposed Ġin ĠB IB REF 20 . ĠWe Ġkeep Ġthe Ġdefault Ġparameters Ġthat Ġwere Ġprovided Ġwith Ġthe Ġimplementation . ĠTo Ġrepresent Ġthe Ġtweets , Ġwe Ġuse Ġthe Ġdec oded Ġembed ding Ġproduced Ġby Ġthe Ġmodel . ĠWith Ġthis Ġbaseline Ġwe Ġaim Ġat Ġassessing Ġif Ġthe Ġtweets ' Ġhasht ags Ġmay Ġhelp Ġdetecting Ġthe Ġnon - fact ual Ġaccounts . </s> LR Ġ+ ĠAll ĠFeatures Ġ( t weet - level ): ĠWe Ġextract Ġall Ġour Ġfeatures Ġfrom Ġeach Ġtweet Ġand Ġfeed Ġthem Ġinto Ġa ĠLR Ġclass ifier . ĠHere , Ġwe Ġdo Ġnot Ġaggregate Ġover Ġtweets Ġand Ġthus Ġview Ġeach Ġtweet Ġindependently . </s> LR Ġ+ ĠAll ĠFeatures Ġ( ch unk - level ): ĠWe Ġconc aten ate Ġthe Ġfeatures ' Ġvectors Ġof Ġthe Ġtweets Ġin Ġa Ġchunk Ġand Ġfeed Ġthem Ġinto Ġa ĠLR Ġclass ifier . </s> Fac Tweet Ġ( t weet - level ): ĠSimilar Ġto Ġthe ĠFac Tweet Ġapproach , Ġbut Ġat Ġtweet - level ; Ġthe Ġsequential Ġflow Ġof Ġthe Ġtweets Ġis Ġnot Ġutilized . ĠWe Ġaim Ġat Ġinvestigating Ġthe Ġimportance Ġof Ġthe Ġsequential Ġflow Ġof Ġtweets . </s> Top -$ k $ Ġreplies , Ġlikes , Ġor Ġre - t we ets : ĠSome Ġapproaches Ġin Ġrumors Ġdetection Ġuse Ġthe Ġnumber Ġof Ġreplies , Ġlikes , Ġand Ġre - t we ets Ġto Ġdetect Ġrumors ĠB IB REF 21 . ĠThus , Ġwe Ġextract Ġtop Ġ$ k $ Ġreplied , Ġliked Ġor Ġre - t weet ed Ġtweets Ġfrom Ġeach Ġaccount Ġto Ġassess Ġthe Ġaccounts Ġfact uality . ĠWe Ġtested Ġdifferent Ġ$ k $ Ġvalues Ġbetween Ġ10 Ġtweets Ġto Ġthe Ġmax Ġnumber Ġof Ġtweets Ġfrom Ġeach Ġaccount . ĠFigure ĠFIG REF 24 Ġshows Ġthe Ġmacro - F 1 Ġvalues Ġfor Ġdifferent Ġ$ k $ Ġvalues . ĠIt Ġseems Ġthat Ġ$ k = 500 $ Ġfor Ġthe Ġtop Ġreplied Ġtweets Ġachieves Ġthe Ġhighest Ġresult . ĠTherefore , Ġwe Ġconsider Ġthis Ġas Ġa Ġbaseline . </s> Exper imental ĠSetup . ĠWe Ġapply Ġa Ġ5 Ġcross - valid ation Ġon Ġthe Ġaccount 's Ġlevel . ĠFor Ġthe ĠFac Tweet Ġmodel , Ġwe Ġexperiment Ġwith Ġ25 % Ġof Ġthe Ġaccounts Ġfor Ġvalidation Ġand Ġparameters Ġselection . ĠWe Ġuse Ġhyper opt Ġlibrary Ġto Ġselect Ġthe Ġhyper - param eters Ġon Ġthe Ġfollowing Ġvalues : ĠL ST M Ġlayer Ġsize Ġ( 16 , Ġ32 , Ġ64 ), Ġdrop out Ġ($ 0 . 0 - 0 . 9 $ ), Ġactivation Ġfunction Ġ($ re lu $, Ġ$ sel u $, Ġ$ tan h $ ), Ġoptim izer Ġ($ s gd $, Ġ$ ad am $, Ġ$ r ms prop $ ) Ġwith Ġvarying Ġthe Ġvalue Ġof Ġthe Ġlearning Ġrate Ġ( 1 e - 1 , .. ,- 5 ), Ġand Ġbatch Ġsize Ġ( 4 , Ġ8 , Ġ16 ). ĠThe Ġvalidation Ġsplit Ġis Ġextracted Ġon Ġthe Ġclass Ġlevel Ġusing Ġstrat ified Ġsampling : Ġwe Ġtook Ġa Ġrandom Ġ25 % Ġof Ġthe Ġaccounts Ġfrom Ġeach Ġclass Ġsince Ġthe Ġdataset Ġis Ġun balanced . ĠDisc arding Ġthe Ġclasses ' Ġsize Ġin Ġthe Ġsplitting Ġprocess Ġmay Ġaffect Ġthe Ġminority Ġclasses Ġ( e . g . Ġhoax ). ĠFor Ġthe Ġbas elines ' Ġclass ifier , Ġwe Ġtested Ġmany Ġclass ifiers Ġand Ġthe ĠLR Ġshowed Ġthe Ġbest Ġoverall Ġperformance . </s> Results . ĠTable ĠT AB REF 25 Ġpresents Ġthe Ġresults . ĠWe Ġpresent Ġthe Ġresults Ġusing Ġa Ġchunk Ġsize Ġof Ġ20 , Ġwhich Ġwas Ġfound Ġto Ġbe Ġthe Ġbest Ġsize Ġon Ġthe Ġheld - out Ġdata . ĠFigure ĠFIG REF 24 Ġshows Ġthe Ġresults Ġof Ġdifferent Ġchunks Ġsizes . ĠFac Tweet Ġperforms Ġbetter Ġthan Ġthe Ġproposed Ġbas elines Ġand Ġob tains Ġthe Ġhighest Ġmacro - F 1 Ġvalue Ġof Ġ$ 0 . 565 $. ĠOur Ġresults Ġindicate Ġthe Ġimportance Ġof Ġtaking Ġinto Ġaccount Ġthe Ġsequence Ġof Ġthe Ġtweets Ġin Ġthe Ġaccounts ' Ġtimelines . ĠThe Ġsequence Ġof Ġthese Ġtweets Ġis Ġbetter Ġcaptured Ġby Ġour Ġproposed Ġmodel Ġsequence - agn ostic Ġor Ġnon - ne ural Ġclass ifiers . ĠMoreover , Ġthe Ġresults Ġdemonstrate Ġthat Ġthe Ġfeatures Ġat Ġtweet - level Ġdo Ġnot Ġperform Ġwell Ġto Ġdetect Ġthe ĠTwitter Ġaccounts Ġfact uality , Ġsince Ġthey Ġobtain Ġa Ġresult Ġnear Ġto Ġthe Ġmajority Ġclass Ġ($ 0 . 18 $ ). ĠAnother Ġfinding Ġfrom Ġour Ġexperiments Ġshows Ġthat Ġthe Ġperformance Ġof Ġthe ĠTweet 2 vec Ġis Ġweak . ĠThis Ġdemonstrates Ġthat Ġtweets ' Ġhasht ags Ġare Ġnot Ġinformative Ġto Ġdetect Ġnon - fact ual Ġaccounts . ĠIn ĠTable ĠT AB REF 25 , Ġwe Ġpresent Ġab lation Ġtests Ġso Ġas Ġto Ġquantify Ġthe Ġcontribution Ġof Ġsubset Ġof Ġfeatures . ĠThe Ġresults Ġindicate Ġthat Ġmost Ġperformance Ġgains Ġcome Ġfrom Ġwords Ġembed d ings , Ġstyle , Ġand Ġmorality Ġfeatures . ĠOther Ġfeatures Ġ( em otion Ġand Ġsentiment ) Ġshow Ġlower Ġimportance : Ġnevertheless , Ġthey Ġstill Ġimprove Ġthe Ġoverall Ġsystem Ġperformance Ġ( on Ġaverage Ġ0 . 35 % ĠMacro - F $ _ 1 $ Ġimprovement ). ĠThese Ġperformance Ġfigures Ġsuggest Ġthat Ġnon - fact ual Ġaccounts Ġuse Ġsemantic Ġand Ġstyl istic Ġhidden Ġsignatures Ġmostly Ġwhile Ġtweeting Ġnews , Ġso Ġas Ġto Ġbe Ġable Ġto Ġmislead Ġthe Ġreaders Ġand Ġbehave Ġas Ġreputable Ġ( i . e ., Ġfactual ) Ġsources . ĠWe Ġleave Ġa Ġmore Ġfine - gr ained , Ġdi ach ronic Ġanalysis Ġof Ġsemantic Ġand Ġstyl istic Ġfeatures ĠâĢĵ Ġhow Ġsemantic Ġand Ġstyl istic Ġsignature Ġevolve Ġacross Ġtime Ġand Ġchange Ġacross Ġthe Ġaccounts ' Ġtimelines ĠâĢĵ Ġfor Ġfuture Ġwork . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġmodel Ġthat Ġutilizes Ġchunk ed Ġtimelines Ġof Ġtweets Ġand Ġa Ġrecurrent Ġneural Ġmodel Ġin Ġorder Ġto Ġinfer Ġthe Ġfact uality Ġof Ġa ĠTwitter Ġnews Ġaccount . ĠOur Ġexperimental Ġresults Ġindicate Ġthe Ġimportance Ġof Ġanalyzing Ġtweet Ġstream Ġinto Ġchunks , Ġas Ġwell Ġas Ġthe Ġbenefits Ġof Ġheter ogeneous Ġknowledge Ġsource Ġ( i . e ., Ġlex ica Ġas Ġwell Ġas Ġtext ) Ġin Ġorder Ġto Ġcapture Ġfact uality . ĠIn Ġfuture Ġwork , Ġwe Ġwould Ġlike Ġto Ġextend Ġthis Ġline Ġof Ġresearch Ġwith Ġfurther Ġin - depth Ġanalysis Ġto Ġunderstand Ġthe Ġflow Ġchange Ġof Ġthe Ġused Ġfeatures Ġin Ġthe Ġaccounts ' Ġstreams . ĠMoreover , Ġwe Ġwould Ġlike Ġto Ġtake Ġour Ġapproach Ġone Ġstep Ġfurther Ġincorporating Ġexplicit Ġtemporal Ġinformation , Ġe . g ., Ġusing Ġtim est amps . ĠCru cially , Ġwe Ġare Ġalso Ġinterested Ġin Ġdeveloping Ġa Ġmult ilingual Ġversion Ġof Ġour Ġapproach , Ġfor Ġinstance Ġby Ġleveraging Ġthe Ġnow Ġubiquitous Ġcross - ling ual Ġembed d ings ĠB IB REF 22 , ĠB IB REF 23 .