an intentional and potentially multicast communication, “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends"
EXTRACTIVE
First, propaganda is a social phenomenon and takes place as an act of communication BIBREF19, and so it is more than a simple information-theoretic message of zeros and ones—it also incorporates an addresser and addressee(s), each in phatic contact (typically via broadcast media), ideally with a shared denotational code and contextual surround(s) BIBREF20.
EXTRACTIVE
The term `propaganda' derives from propagare in post-classical Latin, as in “propagation of the faith" BIBREF1, and thus has from the beginning been associated with an intentional and potentially multicast communication; only later did it become a pejorative term. It was pragmatically defined in the World War II era as “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends" BIBREF2.
The term `propaganda' derives from propagare in post-classical Latin, as in “propagation of the faith" BIBREF1, and thus has from the beginning been associated with an intentional and potentially multicast communication; only later did it become a pejorative term. It was pragmatically defined in the World War II era as “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends" BIBREF2. For the philosopher and sociologist Jacques Ellul, however, in a society with mass communication, propaganda is inevitable and thus it is necessary to become more aware of it BIBREF3; but whether or not to classify a given strip of text as propaganda depends not just on its content but on its use on the part of both addressers and addressees BIBREF1, and this fact makes the automated detection of propaganda intrinsically challenging. It is worth reflecting on the nature of the shared task dataset (PTC corpus) and its structural correspondence (or lack thereof) to some of the definitions of propaganda mentioned in the introduction. First, propaganda is a social phenomenon and takes place as an act of communication BIBREF19, and so it is more than a simple information-theoretic message of zeros and ones—it also incorporates an addresser and addressee(s), each in phatic contact (typically via broadcast media), ideally with a shared denotational code and contextual surround(s) BIBREF20. FLOAT SELECTED: Figure 1: Excerpt of an example (truncated) news document with three separate field-level classification (FLC) tags, for LOADED LANGUAGE, FLAG-WAVING, AND WHATABOUTISM. As such, a dataset of decontextualised documents with labelled sentences, devoid of authorial or publisher metadata, has taken us at some remove from even a simple everyday definition of propaganda. Our models for this shared task cannot easily incorporate information about the addresser or addressee; are left to assume a shared denotational code between author and reader (one perhaps simulated with the use of pre-trained word embeddings); and they are unaware of when or where the act(s) of propagandistic communication took place. This slipperiness is illustrated in our example document (Fig. FIGREF13): note that while Sentences 3 and 7, labelled as propaganda, reflect a propagandistic attitude on the part of the journalist and/or publisher, Sentence 4—also labelled as propaganda in the training data—instead reflects a “flag-waving" propagandistic attitude on the part of U.S. congressman Jeff Flake, via the conventions of reported speech BIBREF21. While reported speech often is signaled by specific morphosyntactic patterns (e.g. the use of double-quotes and “Flake said") BIBREF22, we argue that human readers routinely distinguish propagandistic reportage from the propagandastic speech acts of its subjects, and to conflate these categories in a propaganda detection corpus may contribute to the occurrence of false positives/negatives.
2003.11563
Introduction </s> The Ġchallenges Ġof Ġim balanced Ġclassification âĢĶ in Ġwhich Ġthe Ġproportion Ġof Ġelements Ġin Ġeach Ġclass Ġfor Ġa Ġclassification Ġtask Ġsignificantly Ġdiffer âĢĶ and Ġof Ġthe Ġability Ġto Ġgeneral ise Ġon Ġdiss imilar Ġdata Ġhave Ġremained Ġimportant Ġproblems Ġin ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġand ĠMachine ĠLearning Ġin Ġgeneral . ĠPopular ĠN LP Ġtasks Ġincluding Ġsentiment Ġanalysis , Ġpropaganda Ġdetection , Ġand Ġevent Ġextraction Ġfrom Ġsocial Ġmedia Ġare Ġall Ġexamples Ġof Ġim balanced Ġclassification Ġproblems . ĠIn Ġeach Ġcase Ġthe Ġnumber Ġof Ġelements Ġin Ġone Ġof Ġthe Ġclasses Ġ( e . g . Ġnegative Ġsentiment , Ġpropag and istic Ġcontent , Ġor Ġspecific Ġevents Ġdiscussed Ġon Ġsocial Ġmedia , Ġrespectively ) Ġis Ġsignificantly Ġlower Ġthan Ġthe Ġnumber Ġof Ġelements Ġin Ġthe Ġother Ġclasses . </s> The Ġrecently Ġintroduced ĠB ERT Ġlanguage Ġmodel Ġfor Ġtransfer Ġlearning ĠB IB REF 0 Ġuses Ġa Ġdeep Ġbid irection al Ġtransformer Ġarchitecture Ġto Ġproduce Ġpre - trained Ġcontext - dependent Ġembed d ings . ĠIt Ġhas Ġproven Ġto Ġbe Ġpowerful Ġin Ġsolving Ġmany ĠN LP Ġtasks Ġand , Ġas Ġwe Ġfind , Ġalso Ġappears Ġto Ġhandle Ġim balanced Ġclassification Ġwell , Ġthus Ġremoving Ġthe Ġneed Ġto Ġuse Ġstandard Ġmethods Ġof Ġdata Ġaug mentation Ġto Ġmitigate Ġthis Ġproblem Ġ( see ĠSection ĠSEC REF 11 Ġfor Ġrelated Ġwork Ġand ĠSection ĠSEC REF 16 Ġfor Ġanalysis ). </s> BER T Ġis Ġcredited Ġwith Ġthe Ġability Ġto Ġadapt Ġto Ġmany Ġtasks Ġand Ġdata Ġwith Ġvery Ġlittle Ġtraining ĠB IB REF 0 . ĠHowever , Ġwe Ġshow Ġthat ĠB ERT Ġfails Ġto Ġperform Ġwell Ġwhen Ġthe Ġtraining Ġand Ġtest Ġdata Ġare Ġsignificantly Ġdiss imilar , Ġas Ġis Ġthe Ġcase Ġwith Ġseveral Ġtasks Ġthat Ġdeal Ġwith Ġsocial Ġand Ġnews Ġdata . ĠIn Ġthese Ġcases , Ġthe Ġtraining Ġdata Ġis Ġnecessarily Ġa Ġsubset Ġof Ġpast Ġdata , Ġwhile Ġthe Ġmodel Ġis Ġlikely Ġto Ġbe Ġused Ġon Ġfuture Ġdata Ġwhich Ġdeals Ġwith Ġdifferent Ġtopics . ĠThis Ġwork Ġaddresses Ġthis Ġproblem Ġby Ġincorporating Ġcost - s ensitivity Ġ( Section ĠSEC REF 19 ) Ġinto ĠB ERT . </s> We Ġtest Ġthese Ġmethods Ġby Ġparticipating Ġin Ġthe ĠShared ĠTask Ġon ĠFine - Gr ained ĠProp aganda ĠDetection Ġfor Ġthe Ġ2 nd ĠWorkshop Ġon ĠN LP Ġfor ĠInternet ĠFreedom , Ġfor Ġwhich Ġwe Ġachieve Ġthe Ġsecond Ġrank Ġon Ġsentence - level Ġclassification Ġof Ġpropaganda , Ġconfirming Ġthe Ġimportance Ġof Ġcost - s ensitivity Ġwhen Ġthe Ġtraining Ġand Ġtest Ġsets Ġare Ġdiss imilar . </s> Introduction Ġ: :: ĠDetect ing ĠProp aganda </s> The Ġterm Ġ` prop aganda ' Ġderives Ġfrom Ġpropag are Ġin Ġpost - class ical ĠLatin , Ġas Ġin ĠâĢ ľ prop ag ation Ġof Ġthe Ġfaith " ĠB IB REF 1 , Ġand Ġthus Ġhas Ġfrom Ġthe Ġbeginning Ġbeen Ġassociated Ġwith Ġan Ġintentional Ġand Ġpotentially Ġmultic ast Ġcommunication ; Ġonly Ġlater Ġdid Ġit Ġbecome Ġa Ġpe j orative Ġterm . ĠIt Ġwas Ġprag matically Ġdefined Ġin Ġthe ĠWorld ĠWar ĠII Ġera Ġas ĠâĢ ľ the Ġexpression Ġof Ġan Ġopinion Ġor Ġan Ġaction Ġby Ġindividuals Ġor Ġgroups Ġdeliberately Ġdesigned Ġto Ġinfluence Ġthe Ġopinions Ġor Ġthe Ġactions Ġof Ġother Ġindividuals Ġor Ġgroups Ġwith Ġreference Ġto Ġpredetermined Ġends " ĠB IB REF 2 . </s> For Ġthe Ġphilosopher Ġand Ġsoc iologist ĠJacques ĠEll ul , Ġhowever , Ġin Ġa Ġsociety Ġwith Ġmass Ġcommunication , Ġpropaganda Ġis Ġinevitable Ġand Ġthus Ġit Ġis Ġnecessary Ġto Ġbecome Ġmore Ġaware Ġof Ġit ĠB IB REF 3 ; Ġbut Ġwhether Ġor Ġnot Ġto Ġclassify Ġa Ġgiven Ġstrip Ġof Ġtext Ġas Ġpropaganda Ġdepends Ġnot Ġjust Ġon Ġits Ġcontent Ġbut Ġon Ġits Ġuse Ġon Ġthe Ġpart Ġof Ġboth Ġaddress ers Ġand Ġadd res se es ĠB IB REF 1 , Ġand Ġthis Ġfact Ġmakes Ġthe Ġautomated Ġdetection Ġof Ġpropaganda Ġintrinsically Ġchallenging . </s> Despite Ġthis Ġdifficulty , Ġinterest Ġin Ġautomatically Ġdetecting Ġmisinformation Ġand / or Ġpropaganda Ġhas Ġgained Ġsignificance Ġdue Ġto Ġthe Ġexponential Ġgrowth Ġin Ġonline Ġsources Ġof Ġinformation Ġcombined Ġwith Ġthe Ġspeed Ġwith Ġwhich Ġinformation Ġis Ġshared Ġtoday . ĠThe Ġsheer Ġvolume Ġof Ġsocial Ġinteractions Ġmakes Ġit Ġimpossible Ġto Ġmanually Ġcheck Ġthe Ġver acity Ġof Ġall Ġinformation Ġbeing Ġshared . ĠAutom ation Ġthus Ġremains Ġa Ġpotentially Ġviable Ġmethod Ġof Ġensuring Ġthat Ġwe Ġcontinue Ġto Ġenjoy Ġthe Ġbenefits Ġof Ġa Ġconnected Ġworld Ġwithout Ġthe Ġspread Ġof Ġmisinformation Ġthrough Ġeither Ġignorance Ġor Ġmalicious Ġintent . </s> In Ġthe Ġtask Ġintroduced Ġby ĠB IB REF 4 , Ġwe Ġare Ġprovided Ġwith Ġarticles Ġtagged Ġas Ġpropaganda Ġat Ġthe Ġsentence Ġand Ġfragment Ġ( or Ġspan ) Ġlevel Ġand Ġare Ġtasked Ġwith Ġmaking Ġpredictions Ġon Ġa Ġdevelopment Ġset Ġfollowed Ġby Ġa Ġfinal Ġheld - out Ġtest Ġset . ĠWe Ġnote Ġthis Ġgives Ġus Ġaccess Ġto Ġthe Ġarticles Ġin Ġthe Ġdevelopment Ġand Ġtest Ġsets Ġbut Ġnot Ġtheir Ġlabels . </s> We Ġparticipated Ġin Ġthis Ġtask Ġunder Ġthe Ġteam Ġname ĠProper G ander Ġand Ġwere Ġplaced Ġ2 nd Ġon Ġthe Ġsentence Ġlevel Ġclassification Ġtask Ġwhere Ġwe Ġmake Ġuse Ġof Ġour Ġmethods Ġof Ġincorporating Ġcost - s ensitivity Ġinto ĠB ERT . ĠWe Ġalso Ġparticipated Ġin Ġthe Ġfragment Ġlevel Ġtask Ġand Ġwere Ġplaced Ġ7 th . ĠThe Ġsignificant Ġcontributions Ġof Ġthis Ġwork Ġare : </s> We Ġshow Ġthat Ġcommon Ġ( ` easy ') Ġmethods Ġof Ġdata Ġaug mentation Ġfor Ġdealing Ġwith Ġclass Ġimbalance Ġdo Ġnot Ġimprove Ġbase ĠB ERT Ġperformance . </s> We Ġprovide Ġa Ġstatistical Ġmethod Ġof Ġestablishing Ġthe Ġsimilarity Ġof Ġdatasets . </s> We Ġincorporate Ġcost - s ensitivity Ġinto ĠB ERT Ġto Ġenable Ġmodels Ġto Ġadapt Ġto Ġdiss imilar Ġdatasets . </s> We Ġrelease Ġall Ġour Ġprogram Ġcode Ġon ĠGitHub Ġand ĠGoogle ĠCol abor atory , Ġso Ġthat Ġother Ġresearchers Ġcan Ġbenefit Ġfrom Ġthis Ġwork . </s> Related Ġwork Ġ: :: ĠProp aganda Ġdetection </s> Most Ġof Ġthe Ġexisting Ġworks Ġon Ġpropaganda Ġdetection Ġfocus Ġon Ġidentifying Ġpropaganda Ġat Ġthe Ġnews Ġarticle Ġlevel , Ġor Ġeven Ġat Ġthe Ġnews Ġoutlet Ġlevel Ġwith Ġthe Ġassumption Ġthat Ġeach Ġof Ġthe Ġarticles Ġof Ġthe Ġsuspected Ġpropag and istic Ġoutlet Ġare Ġpropaganda ĠB IB REF 5 , ĠB IB REF 6 . </s> Here Ġwe Ġstudy Ġtwo Ġtasks Ġthat Ġare Ġmore Ġfine - gr ained , Ġspecifically Ġpropaganda Ġdetection Ġat Ġthe Ġsentence Ġand Ġphrase Ġ( fr ag ment ) Ġlevels ĠB IB REF 4 . ĠThis Ġfine - gr ained Ġsetup Ġaims Ġto Ġtrain Ġmodels Ġthat Ġidentify Ġlinguistic Ġpropaganda Ġtechniques Ġrather Ġthan Ġdistinguishing Ġbetween Ġthe Ġarticle Ġsource Ġstyles . </s> BI B REF 4 ĠEM N LP 19 Da San Mart ino Ġwere Ġthe Ġfirst Ġto Ġpropose Ġthis Ġproblem Ġsetup Ġand Ġrelease Ġit Ġas Ġa Ġshared Ġtask . ĠAlong Ġwith Ġthe Ġreleased Ġdataset , ĠB IB REF 4 Ġproposed Ġa Ġmulti - gran ularity Ġneural Ġnetwork , Ġwhich Ġuses Ġthe Ġdeep Ġbid irection al Ġtransformer Ġarchitecture Ġknown Ġas ĠB ERT , Ġwhich Ġfeatures Ġpre - trained Ġcontext - dependent Ġembed d ings ĠB IB REF 0 . ĠTheir Ġsystem Ġtakes Ġa Ġjoint Ġlearning Ġapproach Ġto Ġthe Ġsentence - Ġand Ġphrase - level Ġtasks , Ġconc aten ating Ġthe Ġoutput Ġrepresentation Ġof Ġthe Ġless Ġgran ular Ġ( sent ence - level ) Ġtask Ġwith Ġthe Ġmore Ġfine - gr ained Ġtask Ġusing Ġlearned Ġweights . </s> In Ġthis Ġwork Ġwe Ġalso Ġtake Ġthe ĠB ERT Ġmodel Ġas Ġthe Ġbasis Ġof Ġour Ġapproach Ġand Ġfocus Ġon Ġthe Ġclass Ġimbalance Ġas Ġwell Ġas Ġthe Ġlack Ġof Ġsimilarity Ġbetween Ġtraining Ġand Ġtest Ġdata Ġinherent Ġto Ġthe Ġtask . </s> Related Ġwork Ġ: :: ĠClass Ġimbalance </s> A Ġcommon Ġissue Ġfor Ġmany ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġclassification Ġtasks Ġis Ġclass Ġimbalance , Ġthe Ġsituation Ġwhere Ġone Ġof Ġthe Ġclass Ġcategories Ġcomprises Ġa Ġsignificantly Ġlarger Ġproportion Ġof Ġthe Ġdataset Ġthan Ġthe Ġother Ġclasses . ĠIt Ġis Ġespecially Ġprominent Ġin Ġreal - world Ġdatasets Ġand Ġcompl icates Ġclassification Ġwhen Ġthe Ġidentification Ġof Ġthe Ġminority Ġclass Ġis Ġof Ġspecific Ġimportance . </s> Mod els Ġtrained Ġon Ġthe Ġbasis Ġof Ġminim ising Ġerrors Ġfor Ġim balanced Ġdatasets Ġtend Ġto Ġmore Ġfrequently Ġpredict Ġthe Ġmajority Ġclass ; Ġachieving Ġhigh Ġaccuracy Ġin Ġsuch Ġcases Ġcan Ġbe Ġmisleading . ĠBecause Ġof Ġthis , Ġthe Ġmacro - aver aged ĠF - score , Ġchosen Ġfor Ġthis Ġcompetition , Ġis Ġa Ġmore Ġsuitable Ġmetric Ġas Ġit Ġweights Ġthe Ġperformance Ġon Ġeach Ġclass Ġequally . </s> As Ġclass Ġimbalance Ġis Ġa Ġwidespread Ġissue , Ġmultiple Ġtechniques Ġhave Ġbeen Ġdeveloped Ġthat Ġhelp Ġalleviate Ġit ĠB IB REF 7 , ĠB IB REF 8 , Ġby Ġeither Ġadjusting Ġthe Ġmodel Ġ( e . g . Ġchanging Ġthe Ġperformance Ġmetric ) Ġor Ġchanging Ġthe Ġdata Ġ( e . g . Ġovers am pling Ġthe Ġminority Ġclass Ġor Ġunders am pling Ġthe Ġmajority Ġclass ). </s> Related Ġwork Ġ: :: ĠClass Ġimbalance Ġ: :: ĠCost - sensitive Ġlearning </s> Cost - sensitive Ġclassification Ġcan Ġbe Ġused Ġwhen Ġthe ĠâĢ ľ cost âĢ Ŀ Ġof Ġmis lab elling Ġone Ġclass Ġis Ġhigher Ġthan Ġthat Ġof Ġmis lab elling Ġother Ġclasses ĠB IB REF 9 , ĠB IB REF 10 . ĠFor Ġexample , Ġthe Ġreal Ġcost Ġto Ġa Ġbank Ġof Ġmisc ategor ising Ġa Ġlarge Ġfraudulent Ġtransaction Ġas Ġauthentic Ġis Ġpotentially Ġhigher Ġthan Ġmisc ategor ising Ġ( perhaps Ġonly Ġtemporarily ) Ġa Ġvalid Ġtransaction Ġas Ġfraudulent . ĠCost - sensitive Ġlearning Ġtackles Ġthe Ġissue Ġof Ġclass Ġimbalance Ġby Ġchanging Ġthe Ġcost Ġfunction Ġof Ġthe Ġmodel Ġsuch Ġthat Ġmis class ification Ġof Ġtraining Ġexamples Ġfrom Ġthe Ġminority Ġclass Ġcarries Ġmore Ġweight Ġand Ġis Ġthus Ġmore Ġ` expensive '. ĠThis Ġis Ġachieved Ġby Ġsimply Ġmultiplying Ġthe Ġloss Ġof Ġeach Ġexample Ġby Ġa Ġcertain Ġfactor . ĠThis Ġcost - sensitive Ġlearning Ġtechnique Ġtakes Ġmis class ification Ġcosts Ġinto Ġaccount Ġduring Ġmodel Ġtraining , Ġand Ġdoes Ġnot Ġmodify Ġthe Ġim balanced Ġdata Ġdistribution Ġdirectly . </s> Related Ġwork Ġ: :: ĠClass Ġimbalance Ġ: :: ĠData Ġaug mentation </s> Common Ġmethods Ġthat Ġtackle Ġthe Ġproblem Ġof Ġclass Ġimbalance Ġby Ġmodifying Ġthe Ġdata Ġto Ġcreate Ġbalanced Ġdatasets Ġare Ġunders am pling Ġand Ġovers am pling . ĠUnd ers am pling Ġrandomly Ġremoves Ġinstances Ġfrom Ġthe Ġmajority Ġclass Ġand Ġis Ġonly Ġsuitable Ġfor Ġproblems Ġwith Ġan Ġabundance Ġof Ġdata . ĠOvers am pling Ġmeans Ġcreating Ġmore Ġminority Ġclass Ġinstances Ġto Ġmatch Ġthe Ġsize Ġof Ġthe Ġmajority Ġclass . ĠOvers am pling Ġmethods Ġrange Ġfrom Ġsimple Ġrandom Ġovers am pling , Ġi . e . Ġrepeating Ġthe Ġtraining Ġprocedure Ġon Ġinstances Ġfrom Ġthe Ġminority Ġclass , Ġchosen Ġat Ġrandom , Ġto Ġthe Ġmore Ġcomplex , Ġwhich Ġinvolves Ġconstructing Ġsynthetic Ġminority - class Ġsamples . ĠRandom Ġovers am pling Ġis Ġsimilar Ġto Ġcost - sensitive Ġlearning Ġas Ġrepeating Ġthe Ġsample Ġseveral Ġtimes Ġmakes Ġthe Ġcost Ġof Ġits Ġmis - class ification Ġgrow Ġproportion ally . ĠKol om iy ets Ġet Ġal . Ġk ol om iy ets 2011 model , ĠZhang Ġet Ġal . Ġz hang 2015 character , Ġand ĠWang Ġand ĠYang Ġw ang 2015 s Ġperform Ġdata Ġaug mentation Ġusing Ġsyn onym Ġreplacement , Ġi . e . Ġreplacing Ġrandom Ġwords Ġin Ġsentences Ġwith Ġtheir Ġsyn onyms Ġor Ġnearest - ne igh bor Ġembed d ings , Ġand Ġshow Ġits Ġeffectiveness Ġon Ġmultiple Ġtasks Ġand Ġdatasets . ĠWei Ġet Ġal . Ġwe i 2019 eda Ġprovide Ġa Ġgreat Ġoverview Ġof Ġ` easy ' Ġdata Ġaug mentation Ġ( ED A ) Ġtechniques Ġfor ĠN LP , Ġincluding Ġsyn onym Ġreplacement Ġas Ġdescribed Ġabove , Ġand Ġrandom Ġdeletion , Ġi . e . Ġremoving Ġwords Ġin Ġthe Ġsentence Ġat Ġrandom Ġwith Ġpre - defined Ġprobability . ĠThey Ġshow Ġthe Ġeffectiveness Ġof ĠE DA Ġacross Ġfive Ġtext Ġclassification Ġtasks . ĠHowever , Ġthey Ġmention Ġthat ĠE DA Ġmay Ġnot Ġlead Ġto Ġsubstantial Ġimprovements Ġwhen Ġusing Ġpre - trained Ġmodels . ĠIn Ġthis Ġwork Ġwe Ġtest Ġthis Ġclaim Ġby Ġcomparing Ġperformance Ġgains Ġof Ġusing Ġcost - sensitive Ġlearning Ġversus Ġtwo Ġdata Ġaug mentation Ġmethods , Ġsyn onym Ġreplacement Ġand Ġrandom Ġdeletion , Ġwith Ġa Ġpre - trained ĠB ERT Ġmodel . </s> More Ġcomplex Ġaug mentation Ġmethods Ġinclude Ġback - translation ĠB IB REF 11 , Ġtransl ational Ġdata Ġaug mentation ĠB IB REF 12 , Ġand Ġno ising ĠB IB REF 13 , Ġbut Ġthese Ġare Ġout Ġof Ġthe Ġscope Ġof Ġthis Ġstudy . </s> Dat as et </s> The ĠProp aganda ĠTechniques ĠCorpus Ġ( P TC ) Ġdataset Ġfor Ġthe Ġ2019 ĠShared ĠTask Ġon ĠFine - Gr ained ĠProp aganda Ġconsists Ġof Ġa Ġtraining Ġset Ġof Ġ350 Ġnews Ġarticles , Ġconsisting Ġof Ġjust Ġover Ġ16 , 9 65 Ġtotal Ġsentences , Ġin Ġwhich Ġspecifically Ġpropag and istic Ġfragments Ġhave Ġbeen Ġmanually Ġspotted Ġand Ġlabelled Ġby Ġexperts . ĠThis Ġis Ġaccompanied Ġby Ġa Ġdevelopment Ġset Ġ( or Ġdev Ġset ) Ġof Ġ61 Ġarticles Ġwith Ġ2 , 235 Ġtotal Ġsentences , Ġwhose Ġlabels Ġare Ġmaintained Ġby Ġthe Ġshared Ġtask Ġorganisers ; Ġand Ġtwo Ġmonths Ġafter Ġthe Ġrelease Ġof Ġthis Ġdata , Ġthe Ġorganisers Ġreleased Ġa Ġtest Ġset Ġof Ġ86 Ġarticles Ġand Ġ3 , 526 Ġtotal Ġsentences . ĠIn Ġthe Ġtraining Ġset , Ġ4 , 720 Ġ($ \ sim Ġ28 \ % $ ) Ġof Ġthe Ġsentences Ġhave Ġbeen Ġassessed Ġas Ġcontaining Ġpropaganda , Ġwith Ġ12 , 245 Ġsentences Ġ($ \ sim Ġ72 Ġ\ % $ ) Ġas Ġnon - prop aganda , Ġdemonstrating Ġa Ġclear Ġclass Ġimbalance . </s> In Ġthe Ġbinary Ġsentence - level Ġclassification Ġ( S LC ) Ġtask , Ġa Ġmodel Ġis Ġtrained Ġto Ġdetect Ġwhether Ġeach Ġand Ġevery Ġsentence Ġis Ġeither Ġ' prop aganda ' Ġor Ġ' non - prop aganda '; Ġin Ġthe Ġmore Ġchallenging Ġfield - level Ġclassification Ġ( FL C ) Ġtask , Ġa Ġmodel Ġis Ġtrained Ġto Ġdetect Ġone Ġof Ġ18 Ġpossible Ġpropaganda Ġtechnique Ġtypes Ġin Ġspans Ġof Ġcharacters Ġwithin Ġsentences . ĠThese Ġpropaganda Ġtypes Ġare Ġlisted Ġin ĠB IB REF 4 Ġand Ġrange Ġfrom Ġthose Ġwhich Ġmight Ġbe Ġrecogn isable Ġat Ġthe Ġlex ical Ġlevel Ġ( e . g . ĠName _ Calling , ĠRep etition ), Ġand Ġthose Ġwhich Ġwould Ġlikely Ġneed Ġto Ġincorporate Ġsemantic Ġunderstanding Ġ( Red _ Her ring , ĠStraw _ Man ). </s> For Ġseveral Ġexample Ġsentences Ġfrom Ġa Ġsample Ġdocument Ġannot ated Ġwith Ġfragment - level Ġclass ifications Ġ( FL C ) Ġ( Figure ĠFIG REF 13 ). ĠThe Ġcorresponding Ġsentence - level Ġclassification Ġ( S LC ) Ġlabels Ġwould Ġindicate Ġthat Ġsentences Ġ3 , Ġ4 , Ġand Ġ7 Ġare Ġ' prop aganda ' Ġwhile Ġthe Ġthe Ġother Ġsentences Ġare Ġ` non - prop aganda '. </s> Dat as et Ġ: :: ĠData ĠDistribution </s> One Ġof Ġthe Ġmost Ġinteresting Ġaspects Ġof Ġthe Ġdata Ġprovided Ġfor Ġthis Ġtask Ġis Ġthe Ġnotable Ġdifference Ġbetween Ġthe Ġtraining Ġand Ġthe Ġdevelopment / test Ġsets . ĠWe Ġemphas ise Ġthat Ġthis Ġdifference Ġis Ġrealistic Ġand Ġreflective Ġof Ġreal Ġworld Ġnews Ġdata , Ġin Ġwhich Ġmajor Ġstories Ġare Ġoften Ġaccompanied Ġby Ġthe Ġintroduction Ġof Ġnew Ġterms , Ġnames , Ġand Ġeven Ġphrases . ĠThis Ġis Ġbecause Ġthe Ġtraining Ġdata Ġis Ġa Ġsubset Ġof Ġpast Ġdata Ġwhile Ġthe Ġmodel Ġis Ġto Ġbe Ġused Ġon Ġfuture Ġdata Ġwhich Ġdeals Ġwith Ġdifferent Ġnews worthy Ġtopics . </s> We Ġdemonstrate Ġthis Ġdifference Ġstatistically Ġby Ġusing Ġa Ġmethod Ġfor Ġfinding Ġthe Ġsimilarity Ġof Ġcorpor a Ġsuggested Ġby ĠB IB REF 14 . ĠWe Ġuse Ġthe ĠWil co xon Ġsigned - rank Ġtest ĠB IB REF 15 Ġwhich Ġcompares Ġthe Ġfrequency Ġcounts Ġof Ġrandomly Ġsampled Ġelements Ġfrom Ġdifferent Ġdatasets Ġto Ġdetermine Ġif Ġthose Ġdatasets Ġhave Ġa Ġstatistically Ġsimilar Ġdistribution Ġof Ġelements . </s> We Ġimplement Ġthis Ġas Ġfollows . ĠFor Ġeach Ġof Ġthe Ġtraining , Ġdevelopment Ġand Ġtest Ġsets , Ġwe Ġextract Ġall Ġwords Ġ( ret aining Ġthe Ġrepeats ) Ġwhile Ġignoring Ġa Ġset Ġof Ġstop words Ġ( identified Ġthrough Ġthe ĠPython ĠNatural ĠLanguage ĠTool kit ). ĠWe Ġthen Ġextract Ġ10 , 000 Ġsamples Ġ( with Ġreplacements ) Ġfor Ġvarious Ġpairs Ġof Ġthese Ġdatasets Ġ( training , Ġdevelopment , Ġand Ġtest Ġsets Ġalong Ġwith Ġsplits Ġof Ġeach Ġof Ġthese Ġdatasets ). ĠFinally , Ġwe Ġuse Ġcomparative Ġword Ġfrequencies Ġfrom Ġthe Ġtwo Ġsets Ġto Ġcalculate Ġthe Ġp - value Ġusing Ġthe ĠWil co xon Ġsigned - rank Ġtest . ĠTable ĠT AB REF 15 Ġprovides Ġthe Ġminimum Ġand Ġmaximum Ġp - values Ġand Ġtheir Ġinterpretations Ġfor Ġten Ġsuch Ġruns Ġof Ġeach Ġpair Ġreported . </s> With Ġp - value Ġless Ġthan Ġ0 . 05 , Ġwe Ġshow Ġthat Ġthe Ġtrain , Ġdevelopment Ġand Ġtest Ġsets Ġare Ġself - similar Ġand Ġalso Ġsignificantly Ġdifferent Ġfrom Ġeach Ġother . ĠIn Ġmeasuring Ġself - similar ity , Ġwe Ġsplit Ġeach Ġdataset Ġafter Ġshuff ling Ġall Ġsentences . ĠWhile Ġthis Ġcomparison Ġis Ġmade Ġat Ġthe Ġsentence Ġlevel Ġ( as Ġopposed Ġto Ġthe Ġarticle Ġlevel ), Ġit Ġis Ġconsistent Ġwith Ġthe Ġgran ularity Ġused Ġfor Ġpropaganda Ġdetection , Ġwhich Ġis Ġalso Ġat Ġthe Ġsentence Ġlevel . ĠWe Ġalso Ġperform Ġmeasurements Ġof Ġself Ġsimilarity Ġafter Ġsplitting Ġthe Ġdata Ġat Ġthe Ġarticle Ġlevel Ġand Ġfind Ġthat Ġthe Ġconclusions Ġof Ġsimilarity Ġbetween Ġthe Ġsets Ġhold Ġwith Ġa Ġp - value Ġthreshold Ġof Ġ0 . 001 , Ġwhere Ġp - values Ġfor Ġsimilarity Ġbetween Ġthe Ġtraining Ġand Ġdev / test Ġsets Ġare Ġorders Ġof Ġmagnitude Ġlower Ġcompared Ġto Ġself - similar ity . ĠSince Ġwe Ġuse Ġrandom Ġsampling Ġwe Ġrun Ġthis Ġtest Ġ10 Ġtimes Ġand Ġpresent Ġthe Ġboth Ġthe Ġmaximum Ġand Ġminimum Ġp - values . ĠWe Ġinclude Ġthe Ġsimilarity Ġbetween Ġ25 % Ġof Ġa Ġdataset Ġand Ġthe Ġremaining Ġ75 % Ġof Ġthat Ġset Ġbecause Ġthat Ġis Ġthe Ġtrain / test Ġratio Ġwe Ġuse Ġin Ġour Ġexperiments , Ġfurther Ġdescribed Ġin Ġour Ġmethodology Ġ( Section ĠSEC REF 4 ). </s> This Ġanalysis Ġshows Ġthat Ġwhile Ġall Ġsplits Ġof Ġeach Ġof Ġthe Ġdatasets Ġare Ġstatistically Ġsimilar , Ġthe Ġtraining Ġset Ġ( and Ġthe Ġsplit Ġof Ġthe Ġtraining Ġset Ġthat Ġwe Ġuse Ġfor Ġexperimentation ) Ġare Ġsignificantly Ġdifferent Ġfrom Ġthe Ġdevelopment Ġand Ġtest Ġsets . ĠWhile Ġour Ġanalysis Ġdoes Ġshow Ġthat Ġthe Ġdevelopment Ġand Ġthe Ġtest Ġsets Ġare Ġdiss imilar , Ġwe Ġnote Ġ( based Ġon Ġthe Ġp - values ) Ġthat Ġthey Ġare Ġsignificantly Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġthey Ġare Ġto Ġthe Ġtraining Ġset . </s> Method ology </s> We Ġwere Ġprovided Ġwith Ġtwo Ġtasks : Ġ( 1 ) Ġpropaganda Ġfragment - level Ġidentification Ġ( FL C ) Ġand Ġ( 2 ) Ġpropag and istic Ġsentence - level Ġidentification Ġ( S LC ). ĠWhile Ġwe Ġdevelop Ġsystems Ġfor Ġboth Ġtasks , Ġour Ġmain Ġfocus Ġis Ġtoward Ġthe Ġlatter . ĠGiven Ġthe Ġdifferences Ġbetween Ġthe Ġtraining , Ġdevelopment , Ġand Ġtest Ġsets , Ġwe Ġfocus Ġon Ġmethods Ġfor Ġgeneral ising Ġour Ġmodels . ĠWe Ġnote Ġthat Ġpropaganda Ġidentification Ġis , Ġin Ġgeneral , Ġan Ġim balanced Ġbinary Ġclassification Ġproblem Ġas Ġmost Ġsentences Ġare Ġnot Ġpropag and istic . </s> Due Ġto Ġthe Ġnon - d eter ministic Ġnature Ġof Ġfast ĠGPU Ġcomput ations , Ġwe Ġrun Ġeach Ġof Ġour Ġmodels Ġthree Ġtimes Ġand Ġreport Ġthe Ġaverage Ġof Ġthese Ġthree Ġruns Ġthrough Ġthe Ġrest Ġof Ġthis Ġsection . ĠWhen Ġpicking Ġthe Ġmodel Ġto Ġuse Ġfor Ġour Ġfinal Ġsubmission , Ġwe Ġpick Ġthe Ġmodel Ġthat Ġperforms Ġbest Ġon Ġthe Ġdevelopment Ġset . </s> When Ġtesting Ġour Ġmodels , Ġwe Ġsplit Ġthe Ġlabelled Ġtraining Ġdata Ġinto Ġtwo Ġnon - over l apping Ġparts : Ġthe Ġfirst Ġone , Ġconsisting Ġof Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġis Ġused Ġto Ġtrain Ġmodels , Ġwhereas Ġthe Ġother Ġis Ġused Ġto Ġtest Ġthe Ġeffectiveness Ġof Ġthe Ġmodels . ĠAll Ġmodels Ġare Ġtrained Ġand Ġtested Ġon Ġthe Ġsame Ġsplit Ġto Ġensure Ġcompar ability . ĠSimilarly , Ġto Ġensure Ġthat Ġour Ġmodels Ġremain Ġcomparable , Ġwe Ġcontinue Ġto Ġtrain Ġon Ġthe Ġsame Ġ75 % Ġof Ġthe Ġtraining Ġset Ġeven Ġwhen Ġtesting Ġon Ġthe Ġdevelopment Ġset . </s> Once Ġthe Ġbest Ġmodel Ġis Ġfound Ġusing Ġthese Ġmethods , Ġwe Ġtrain Ġthat Ġmodel Ġon Ġall Ġof Ġthe Ġtraining Ġdata Ġavailable Ġbefore Ġthen Ġsubmitting Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġto Ġthe Ġleader board . ĠThese Ġresults Ġare Ġdetailed Ġin Ġthe Ġsection Ġdescribing Ġour Ġresults Ġ( Section ĠSEC REF 5 ). </s> Method ology Ġ: :: ĠClass ĠIm balance Ġin ĠSent ence ĠLevel ĠClassification </s> The Ġsentence Ġlevel Ġclassification Ġtask Ġis Ġan Ġim balanced Ġbinary Ġclassification Ġproblem Ġthat Ġwe Ġaddress Ġusing ĠB ERT ĠB IB REF 0 . ĠWe Ġuse ĠB ERT B ASE , Ġunc ased , Ġwhich Ġconsists Ġof Ġ12 Ġself - att ention Ġlayers , Ġand Ġreturns Ġa Ġ768 - dimension Ġvector Ġthat Ġrepresentation Ġa Ġsentence . ĠSo Ġas Ġto Ġmake Ġuse Ġof ĠB ERT Ġfor Ġsentence Ġclassification , Ġwe Ġinclude Ġa Ġfully Ġconnected Ġlayer Ġon Ġtop Ġof Ġthe ĠB ERT Ġself - att ention Ġlayers , Ġwhich Ġclass ifies Ġthe Ġsentence Ġembed ding Ġprovided Ġby ĠB ERT Ġinto Ġthe Ġtwo Ġclasses Ġof Ġinterest Ġ( prop aganda Ġor Ġnon - prop aganda ). </s> We Ġattempt Ġto Ġexploit Ġvarious Ġdata Ġaug mentation Ġtechniques Ġto Ġaddress Ġthe Ġproblem Ġof Ġclass Ġimbalance . ĠTable ĠT AB REF 17 Ġshows Ġthe Ġresults Ġof Ġour Ġexperiments Ġfor Ġdifferent Ġdata Ġaug mentation Ġtechniques Ġwhen , Ġafter Ġshuff ling Ġthe Ġtraining Ġdata , Ġwe Ġtrain Ġthe Ġmodel Ġon Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġand Ġtest Ġit Ġon Ġthe Ġremaining Ġ25 % Ġof Ġthe Ġtraining Ġdata Ġand Ġthe Ġdevelopment Ġdata . </s> We Ġobserve Ġthat ĠB ERT Ġwithout Ġaug mentation Ġconsistently Ġoutper forms ĠB ERT Ġwith Ġaug mentation Ġin Ġthe Ġexperiments Ġwhen Ġthe Ġmodel Ġis Ġtrained Ġon Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġand Ġevaluated Ġon Ġthe Ġrest , Ġi . e Ġtrained Ġand Ġevaluated Ġon Ġsimilar Ġdata , Ġcoming Ġfrom Ġthe Ġsame Ġdistribution . ĠThis Ġis Ġconsistent Ġwith Ġobservations Ġby ĠWei Ġet Ġal . Ġwe i 2019 eda Ġthat Ġcontextual Ġword Ġembed d ings Ġdo Ġnot Ġgain Ġfrom Ġdata Ġaug mentation . ĠThe Ġfact Ġthat Ġwe Ġshuffle Ġthe Ġtraining Ġdata Ġprior Ġto Ġsplitting Ġit Ġinto Ġtraining Ġand Ġtesting Ġsubs ets Ġcould Ġimply Ġthat Ġthe Ġmodel Ġis Ġlearning Ġto Ġassociate Ġtopic Ġwords , Ġsuch Ġas Ġ` M ueller ', Ġas Ġpropaganda . ĠHowever , Ġwhen Ġwe Ġperform Ġmodel Ġevaluation Ġusing Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġdiss imilar Ġto Ġthe Ġtraining , Ġwe Ġobserve Ġthat Ġsyn onym Ġinsertion Ġand Ġword Ġdropping Ġtechniques Ġalso Ġdo Ġnot Ġbring Ġperformance Ġgains , Ġwhile Ġrandom Ġovers am pling Ġincreases Ġperformance Ġover Ġbase ĠB ERT Ġby Ġ4 %. ĠSyn onym Ġinsertion Ġprovides Ġresults Ġvery Ġsimilar Ġto Ġbase ĠB ERT , Ġwhile Ġrandom Ġdeletion Ġharms Ġmodel Ġperformance Ġproducing Ġlower Ġscores . ĠWe Ġbelieve Ġthat Ġthis Ġcould Ġbe Ġattributed Ġto Ġthe Ġfact Ġthat Ġsyn onym Ġinsertion Ġand Ġrandom Ġword Ġdropping Ġinvolve Ġthe Ġintroduction Ġof Ġnoise Ġto Ġthe Ġdata , Ġwhile Ġovers am pling Ġdoes Ġnot . ĠAs Ġwe Ġare Ġworking Ġwith Ġnatural Ġlanguage Ġdata , Ġthis Ġtype Ġof Ġnoise Ġcan Ġin Ġfact Ġchange Ġthe Ġmeaning Ġof Ġthe Ġsentence . ĠOvers am pling Ġon Ġthe Ġother Ġhand Ġpurely Ġincreases Ġthe Ġimportance Ġof Ġthe Ġminority Ġclass Ġby Ġrepeating Ġtraining Ġon Ġthe Ġunchanged Ġinstances . </s> So Ġas Ġto Ġbetter Ġunderstand Ġthe Ġaspects Ġof Ġovers am pling Ġthat Ġcontribute Ġto Ġthese Ġgains , Ġwe Ġperform Ġa Ġclass - wise Ġperformance Ġanalysis Ġof ĠB ERT Ġwith / without Ġovers am pling . ĠThe Ġresults Ġof Ġthese Ġexperiments Ġ( Table ĠT AB REF 18 ) Ġshow Ġthat Ġovers am pling Ġincreases Ġthe Ġoverall Ġrecall Ġwhile Ġmaintaining Ġprecision . ĠThis Ġis Ġachieved Ġby Ġsignificantly Ġimproving Ġthe Ġrecall Ġof Ġthe Ġminority Ġclass Ġ( prop aganda ) Ġat Ġthe Ġcost Ġof Ġthe Ġrecall Ġof Ġthe Ġmajority Ġclass . </s> So Ġfar Ġwe Ġhave Ġbeen Ġable Ġto Ġestablish Ġthat Ġa ) Ġthe Ġtraining Ġand Ġtest Ġsets Ġare Ġdiss imilar , Ġthus Ġrequiring Ġus Ġto Ġgeneral ise Ġour Ġmodel , Ġb ) Ġovers am pling Ġprovides Ġa Ġmethod Ġof Ġgeneral isation , Ġand Ġc ) Ġovers am pling Ġdoes Ġthis Ġwhile Ġmaintaining Ġrecall Ġon Ġthe Ġminority Ġ( and Ġthus Ġmore Ġinteresting ) Ġclass . </s> Given Ġthis Ġwe Ġexplore Ġalternative Ġmethods Ġof Ġincreasing Ġminority Ġclass Ġrecall Ġwithout Ġa Ġsignificant Ġdrop Ġin Ġprecision . ĠOne Ġsuch Ġmethod Ġis Ġcost - sensitive Ġclassification , Ġwhich Ġdiffers Ġfrom Ġrandom Ġovers am pling Ġin Ġthat Ġit Ġprovides Ġa Ġmore Ġcontinuous - valued Ġand Ġconsistent Ġmethod Ġof Ġweight ing Ġsamples Ġof Ġim balanced Ġtraining Ġdata ; Ġfor Ġexample , Ġrandom Ġovers am pling Ġwill Ġinevitably Ġemphas ise Ġsome Ġtraining Ġinstances Ġat Ġthe Ġexpense Ġof Ġothers . ĠWe Ġdetail Ġour Ġmethods Ġof Ġusing Ġcost - sensitive Ġclassification Ġin Ġthe Ġnext Ġsection . ĠFurther Ġexperiments Ġwith Ġovers am pling Ġmight Ġhave Ġprovided Ġinsights Ġinto Ġthe Ġrelationships Ġbetween Ġthese Ġmethods , Ġwhich Ġwe Ġleave Ġfor Ġfuture Ġexploration . </s> Method ology Ġ: :: ĠCost - sensitive ĠClassification </s> As Ġdiscussed Ġin ĠSection ĠSEC REF 10 , Ġcost - sensitive Ġclassification Ġcan Ġbe Ġperformed Ġby Ġweight ing Ġthe Ġcost Ġfunction . ĠWe Ġincrease Ġthe Ġweight Ġof Ġincorrectly Ġlab elling Ġa Ġpropag and istic Ġsentence Ġby Ġaltering Ġthe Ġcost Ġfunction Ġof Ġthe Ġtraining Ġof Ġthe Ġfinal Ġfully Ġconnected Ġlayer Ġof Ġour Ġmodel Ġpreviously Ġdescribed Ġin ĠSection ĠSEC REF 16 . ĠWe Ġmake Ġthese Ġchanges Ġthrough Ġthe Ġuse Ġof ĠPy Tor ch ĠB IB REF 16 Ġwhich Ġcalculates Ġthe Ġcross - ent ropy Ġloss Ġfor Ġa Ġsingle Ġprediction Ġ$ x $, Ġan Ġarray Ġwhere Ġthe Ġ$ j ^{ th } $ Ġelement Ġrepresents Ġthe Ġmodels Ġprediction Ġfor Ġclass Ġ$ j $, Ġlabelled Ġwith Ġthe Ġclass Ġ$ class $ Ġas Ġgiven Ġby ĠEqu ation ĠDIS PLAY _ FORM 20 . </s> The Ġcross - ent ropy Ġloss Ġgiven Ġin ĠEqu ation ĠDIS PLAY _ FORM 20 Ġis Ġmodified Ġto Ġaccommodate Ġan Ġarray Ġ$ weight $, Ġthe Ġ$ i ^{ th } $ Ġelement Ġof Ġwhich Ġrepresents Ġthe Ġweight Ġof Ġthe Ġ$ i ^{ th } $ Ġclass , Ġas Ġdescribed Ġin ĠEqu ation ĠDIS PLAY _ FORM 21 . </s> Int uitive ly , Ġwe Ġincrease Ġthe Ġcost Ġof Ġgetting Ġthe Ġclassification Ġof Ġan ĠâĢ ľ important âĢ Ŀ Ġclass Ġwrong Ġand Ġcorresponding Ġdecre es Ġthe Ġcost Ġof Ġgetting Ġa Ġless Ġimportant Ġclass Ġwrong . ĠIn Ġour Ġcase , Ġwe Ġincrease Ġthe Ġcost Ġof Ġmis lab elling Ġthe Ġminority Ġclass Ġwhich Ġis ĠâĢ ľ prop aganda âĢ Ŀ Ġ( as Ġopposed Ġto ĠâĢ ľ non - prop aganda âĢ Ŀ ). </s> We Ġexpect Ġthe Ġeffect Ġof Ġthis Ġto Ġbe Ġsimilar Ġto Ġthat Ġof Ġovers am pling , Ġin Ġthat Ġit Ġis Ġlikely Ġto Ġenable Ġus Ġto Ġincrease Ġthe Ġrecall Ġof Ġthe Ġminority Ġclass Ġthus Ġresulting Ġin Ġthe Ġdecrease Ġin Ġrecall Ġof Ġthe Ġoverall Ġmodel Ġwhile Ġmaintaining Ġhigh Ġprecision . ĠWe Ġreiterate Ġthat Ġthis Ġspecific Ġchange Ġto Ġa Ġmodel Ġresults Ġin Ġincreasing Ġthe Ġmodel 's Ġability Ġto Ġbetter Ġidentify Ġelements Ġbelonging Ġto Ġthe Ġminority Ġclass Ġin Ġdiss imilar Ġdatasets Ġwhen Ġusing ĠB ERT . </s> We Ġexplore Ġthe Ġvalidity Ġof Ġthis Ġby Ġperforming Ġseveral Ġexperiments Ġwith Ġdifferent Ġweights Ġassigned Ġto Ġthe Ġminority Ġclass . ĠWe Ġnote Ġthat Ġin Ġour Ġexperiments Ġuse Ġsignificantly Ġhigher Ġweights Ġthan Ġthe Ġweights Ġproportional Ġto Ġclass Ġfrequencies Ġin Ġthe Ġtraining Ġdata , Ġthat Ġare Ġcommon Ġin Ġliterature ĠB IB REF 17 . ĠRather Ġthan Ġdirectly Ġusing Ġthe Ġclass Ġproportions Ġof Ġthe Ġtraining Ġset , Ġwe Ġshow Ġthat Ġtuning Ġweights Ġbased Ġon Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġis Ġmore Ġbeneficial . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġresults Ġof Ġthese Ġexperiments Ġwherein Ġwe Ġare Ġable Ġto Ġmaintain Ġthe Ġprecision Ġon Ġthe Ġsubset Ġof Ġthe Ġtraining Ġset Ġused Ġfor Ġtesting Ġwhile Ġreducing Ġits Ġrecall Ġand Ġthus Ġgeneral ising Ġthe Ġmodel . ĠThe Ġfact Ġthat Ġthe Ġmodel Ġis Ġgeneral ising Ġon Ġa Ġdiss imilar Ġdataset Ġis Ġconfirmed Ġby Ġthe Ġincrease Ġin Ġthe Ġdevelopment Ġset ĠF 1 Ġscore . ĠWe Ġnote Ġthat Ġthe Ġgains Ġare Ġnot Ġinfinite Ġand Ġthat Ġa Ġbalance Ġmust Ġbe Ġstruck Ġbased Ġon Ġthe Ġamount Ġof Ġgeneral isation Ġand Ġthe Ġcorresponding Ġloss Ġin Ġaccuracy . ĠThe Ġexact Ġweight Ġto Ġuse Ġfor Ġthe Ġbest Ġtransfer Ġof Ġclassification Ġaccuracy Ġis Ġrelated Ġto Ġthe Ġdiss imilar ity Ġof Ġthat Ġother Ġdataset Ġand Ġhence Ġis Ġto Ġbe Ġobtained Ġexperiment ally Ġthrough Ġhyper param eter Ġsearch . ĠOur Ġexperiments Ġshowed Ġthat Ġa Ġvalue Ġof Ġ4 Ġis Ġbest Ġsuited Ġfor Ġthis Ġtask . </s> We Ġdo Ġnot Ġinclude Ġthe Ġcomplete Ġresults Ġof Ġour Ġexperiments Ġhere Ġdue Ġto Ġspace Ġconstraints Ġbut Ġinclude Ġthem Ġalong Ġwith Ġcharts Ġand Ġprogram Ġcode Ġon Ġour Ġproject Ġwebsite . ĠBased Ġon Ġthis Ġexploration Ġwe Ġfind Ġthat Ġthe Ġbest Ġweights Ġfor Ġthis Ġparticular Ġdataset Ġare Ġ1 Ġfor Ġnon - prop aganda Ġand Ġ4 Ġfor Ġpropaganda Ġand Ġwe Ġuse Ġthis Ġto Ġtrain Ġthe Ġfinal Ġmodel Ġused Ġto Ġsubmit Ġresults Ġto Ġthe Ġleader board . ĠWe Ġalso Ġfound Ġthat Ġadding ĠPart Ġof ĠSpeech Ġtags Ġand ĠNamed ĠEntity Ġinformation Ġto ĠB ERT Ġembed d ings Ġby Ġconc aten ating Ġthese Ġone - hot Ġvectors Ġto Ġthe ĠB ERT Ġembed d ings Ġdoes Ġnot Ġimprove Ġmodel Ġperformance . ĠWe Ġdescribe Ġthese Ġresults Ġin ĠSection ĠSEC REF 5 . </s> Method ology Ġ: :: ĠFrag ment - level Ġclassification Ġ( FL C ) </s> In Ġaddition Ġto Ġparticipating Ġin Ġthe ĠSent ence ĠLevel ĠClassification Ġtask Ġwe Ġalso Ġparticipate Ġin Ġthe ĠFrag ment ĠLevel ĠClassification Ġtask . ĠWe Ġnote Ġthat Ġextracting Ġfragments Ġthat Ġare Ġpropag and istic Ġis Ġsimilar Ġto Ġthe Ġtask Ġof ĠNamed ĠEntity ĠRecogn ition , Ġin Ġthat Ġthey Ġare Ġboth Ġspan Ġextraction Ġtasks , Ġand Ġso Ġuse Ġa ĠB ERT Ġbased Ġmodel Ġdesigned Ġfor Ġthis Ġtask Ġ- ĠWe Ġbuild Ġon Ġthe Ġwork Ġby ĠB IB REF 18 Ġwhich Ġmakes Ġuse Ġof ĠContinuous ĠRandom ĠField Ġstacked Ġon Ġtop Ġof Ġan ĠL ST M Ġto Ġpredict Ġspans . ĠThis Ġarchitecture Ġis Ġstandard Ġamongst Ġstate Ġof Ġthe Ġart Ġmodels Ġthat Ġperform Ġspan Ġidentification . </s> While Ġthe Ġsame Ġspan Ġof Ġtext Ġcannot Ġhave Ġmultiple Ġnamed Ġentity Ġlabels , Ġit Ġcan Ġhave Ġdifferent Ġpropaganda Ġlabels . ĠWe Ġget Ġaround Ġthis Ġproblem Ġby Ġpicking Ġone Ġof Ġthe Ġlabels Ġat Ġrandom . ĠAdditionally , Ġso Ġas Ġto Ġspeed Ġup Ġtraining , Ġwe Ġonly Ġtrain Ġour Ġmodel Ġon Ġthose Ġsentences Ġthat Ġcontain Ġsome Ġpropag and istic Ġfragment . ĠIn Ġhindsight , Ġwe Ġnote Ġthat Ġboth Ġthese Ġdecisions Ġwere Ġnot Ġideal Ġand Ġdiscuss Ġwhat Ġwe Ġmight Ġhave Ġotherwise Ġdone Ġin ĠSection ĠSEC REF 7 . </s> Results </s> In Ġthis Ġsection , Ġwe Ġshow Ġour Ġrankings Ġon Ġthe Ġleader board Ġon Ġthe Ġtest Ġset . ĠUnlike Ġthe Ġprevious Ġexplor atory Ġsections , Ġin Ġwhich Ġwe Ġtrained Ġour Ġmodel Ġon Ġpart Ġof Ġthe Ġtraining Ġset , Ġwe Ġtrain Ġmodels Ġdescribed Ġin Ġthis Ġsection Ġon Ġthe Ġcomplete Ġtraining Ġset . </s> Results Ġ: :: ĠResults Ġon Ġthe ĠS LC Ġtask </s> Our Ġbest Ġperforming Ġmodel , Ġselected Ġon Ġthe Ġbasis Ġof Ġa Ġsystematic Ġanalysis Ġof Ġthe Ġrelationship Ġbetween Ġcost Ġweights Ġand Ġrecall , Ġplaces Ġus Ġsecond Ġamongst Ġthe Ġ25 Ġteams Ġthat Ġsubmitted Ġtheir Ġresults Ġon Ġthis Ġtask . ĠWe Ġpresent Ġour Ġscore Ġon Ġthe Ġtest Ġset Ġalongside Ġthose Ġof Ġcomparable Ġteams Ġin ĠTable ĠT AB REF 25 . ĠWe Ġnote Ġthat Ġthe Ġtask Ġdescription Ġpaper ĠB IB REF 4 Ġdescribes Ġa Ġmethod Ġof Ġachieving Ġan ĠF 1 Ġscore Ġof Ġ60 . 98 % Ġon Ġa Ġsimilar Ġtask Ġalthough Ġthis Ġreported Ġscore Ġis Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġresults Ġon Ġthis Ġtask Ġbecause Ġof Ġthe Ġdifferences Ġin Ġtesting Ġsets . </s> Results Ġ: :: ĠResults Ġon Ġthe ĠF LC Ġtask </s> We Ġtrain Ġthe Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 23 Ġon Ġthe Ġcomplete Ġtraining Ġset Ġbefore Ġsubmitting Ġto Ġthe Ġleader board . ĠOur Ġbest Ġperforming Ġmodel Ġwas Ġplaced Ġ7 th Ġamongst Ġthe Ġ13 Ġteams Ġthat Ġsubmitted Ġresults Ġfor Ġthis Ġtask . ĠWe Ġpresent Ġour Ġscore Ġon Ġthe Ġtest Ġset Ġalongside Ġthose Ġof Ġcomparable Ġteams Ġin ĠTable ĠT AB REF 27 . ĠWe Ġnote Ġthat Ġthe Ġtask Ġdescription Ġpaper ĠB IB REF 4 Ġdescribes Ġa Ġmethod Ġof Ġachieving Ġan ĠF 1 Ġscore Ġof Ġ22 . 58 % Ġon Ġa Ġsimilar Ġtask Ġalthough , Ġthis Ġreported Ġscore Ġis Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġresults Ġon Ġthis Ġtask . </s> One Ġof Ġthe Ġmajor Ġsetbacks Ġto Ġour Ġmethod Ġfor Ġidentifying Ġsentence Ġfragments Ġwas Ġthe Ġloss Ġof Ġtraining Ġdata Ġas Ġa Ġresult Ġof Ġrandomly Ġpicking Ġone Ġlabel Ġwhen Ġthe Ġsame Ġfragment Ġhad Ġmultiple Ġlabels . ĠThis Ġcould Ġhave Ġbeen Ġavoided Ġby Ġtraining Ġdifferent Ġmodels Ġfor Ġeach Ġlabel Ġand Ġsimply Ġconc aten ating Ġthe Ġresults . ĠAdditionally , Ġtraining Ġon Ġall Ġsentences , Ġincluding Ġthose Ġthat Ġdid Ġnot Ġcontain Ġany Ġfragments Ġlabelled Ġas Ġpropag and istic Ġwould Ġhave Ġlikely Ġimproved Ġour Ġmodel Ġperformance . ĠWe Ġintend Ġto Ġperform Ġthese Ġexperiments Ġas Ġpart Ġof Ġour Ġongoing Ġresearch . </s> Iss ues Ġof ĠDec on text ual ization Ġin ĠAutom ated ĠProp aganda ĠDetection </s> It Ġis Ġworth Ġreflecting Ġon Ġthe Ġnature Ġof Ġthe Ġshared Ġtask Ġdataset Ġ( P TC Ġcorpus ) Ġand Ġits Ġstructural Ġcorrespondence Ġ( or Ġlack Ġthereof ) Ġto Ġsome Ġof Ġthe Ġdefinitions Ġof Ġpropaganda Ġmentioned Ġin Ġthe Ġintroduction . ĠFirst , Ġpropaganda Ġis Ġa Ġsocial Ġphenomenon Ġand Ġtakes Ġplace Ġas Ġan Ġact Ġof Ġcommunication ĠB IB REF 19 , Ġand Ġso Ġit Ġis Ġmore Ġthan Ġa Ġsimple Ġinformation - the oret ic Ġmessage Ġof Ġz eros Ġand Ġones âĢĶ it Ġalso Ġincorporates Ġan Ġaddress er Ġand Ġadd res see ( s ), Ġeach Ġin Ġph atic Ġcontact Ġ( typically Ġvia Ġbroadcast Ġmedia ), Ġideally Ġwith Ġa Ġshared Ġden ot ational Ġcode Ġand Ġcontextual Ġsurround ( s ) ĠB IB REF 20 . </s> As Ġsuch , Ġa Ġdataset Ġof Ġdecon text ual ised Ġdocuments Ġwith Ġlabelled Ġsentences , Ġdevoid Ġof Ġauthor ial Ġor Ġpublisher Ġmetadata , Ġhas Ġtaken Ġus Ġat Ġsome Ġremove Ġfrom Ġeven Ġa Ġsimple Ġeveryday Ġdefinition Ġof Ġpropaganda . ĠOur Ġmodels Ġfor Ġthis Ġshared Ġtask Ġcannot Ġeasily Ġincorporate Ġinformation Ġabout Ġthe Ġaddress er Ġor Ġadd res see ; Ġare Ġleft Ġto Ġassume Ġa Ġshared Ġden ot ational Ġcode Ġbetween Ġauthor Ġand Ġreader Ġ( one Ġperhaps Ġsimulated Ġwith Ġthe Ġuse Ġof Ġpre - trained Ġword Ġembed d ings ); Ġand Ġthey Ġare Ġunaware Ġof Ġwhen Ġor Ġwhere Ġthe Ġact ( s ) Ġof Ġpropag and istic Ġcommunication Ġtook Ġplace . ĠThis Ġsl ipper iness Ġis Ġillustrated Ġin Ġour Ġexample Ġdocument Ġ( Fig . ĠFIG REF 13 ): Ġnote Ġthat Ġwhile ĠSent ences Ġ3 Ġand Ġ7 , Ġlabelled Ġas Ġpropaganda , Ġreflect Ġa Ġpropag and istic Ġattitude Ġon Ġthe Ġpart Ġof Ġthe Ġjournalist Ġand / or Ġpublisher , ĠSent ence Ġ4 âĢĶ also Ġlabelled Ġas Ġpropaganda Ġin Ġthe Ġtraining Ġdata âĢĶ instead Ġreflects Ġa ĠâĢ ľ flag - w aving " Ġpropag and istic Ġattitude Ġon Ġthe Ġpart Ġof ĠU . S . Ġcongressman ĠJeff ĠFlake , Ġvia Ġthe Ġconventions Ġof Ġreported Ġspeech ĠB IB REF 21 . ĠWhile Ġreported Ġspeech Ġoften Ġis Ġsignaled Ġby Ġspecific Ġmorph os ynt actic Ġpatterns Ġ( e . g . Ġthe Ġuse Ġof Ġdouble - qu otes Ġand ĠâĢ ľ Fl ake Ġsaid ") ĠB IB REF 22 , Ġwe Ġargue Ġthat Ġhuman Ġreaders Ġroutinely Ġdistinguish Ġpropag and istic Ġreport age Ġfrom Ġthe Ġpropag and astic Ġspeech Ġacts Ġof Ġits Ġsubjects , Ġand Ġto Ġconf late Ġthese Ġcategories Ġin Ġa Ġpropaganda Ġdetection Ġcorpus Ġmay Ġcontribute Ġto Ġthe Ġoccurrence Ġof Ġfalse Ġpositives / neg atives . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork Ġwe Ġhave Ġpresented Ġa Ġmethod Ġof Ġincorporating Ġcost - s ensitivity Ġinto ĠB ERT Ġto Ġallow Ġfor Ġbetter Ġgeneral isation Ġand Ġadditionally , Ġwe Ġprovide Ġa Ġsimple Ġmeasure Ġof Ġcorpus Ġsimilarity Ġto Ġdetermine Ġwhen Ġthis Ġmethod Ġis Ġlikely Ġto Ġbe Ġuseful . ĠWe Ġintend Ġto Ġextend Ġour Ġanalysis Ġof Ġthe Ġability Ġto Ġgeneral ise Ġmodels Ġto Ġless Ġsimilar Ġdata Ġby Ġexperimenting Ġon Ġother Ġdatasets Ġand Ġmodels . ĠWe Ġhope Ġthat Ġthe Ġrelease Ġof Ġprogram Ġcode Ġand Ġdocumentation Ġwill Ġallow Ġthe Ġresearch Ġcommunity Ġto Ġhelp Ġin Ġthis Ġexperimentation Ġwhile Ġexploiting Ġthese Ġmethods . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank ĠDr ĠLe andro ĠM ink u Ġfrom Ġthe ĠUniversity Ġof ĠBirmingham Ġfor Ġhis Ġinsights Ġinto Ġand Ġhelp Ġwith Ġthe Ġstatistical Ġanalysis Ġpresented Ġin Ġthis Ġpaper . </s> This Ġwork Ġwas Ġalso Ġpartially Ġsupported Ġby ĠThe ĠAlan ĠTuring ĠInstitute Ġunder Ġthe ĠEPS RC Ġgrant ĠEP / N 51 01 29 / 1 . ĠWork Ġby ĠElena ĠKoch k ina Ġwas Ġpartially Ġsupported Ġby Ġthe ĠLever h ul me ĠTrust Ġthrough Ġthe ĠBridges ĠProgramme Ġand ĠWarwick ĠCD T Ġfor ĠUrban ĠScience Ġ& ĠProgress Ġunder Ġthe ĠEPS RC ĠGrant ĠNumber ĠEP / L 01 64 00 / 1 .
Introduction The challenges of imbalanced classification—in which the proportion of elements in each class for a classification task significantly differ—and of the ability to generalise on dissimilar data have remained important problems in Natural Language Processing (NLP) and Machine Learning in general. Popular NLP tasks including sentiment analysis, propaganda detection, and event extraction from social media are all examples of imbalanced classification problems. In each case the number of elements in one of the classes (e.g. negative sentiment, propagandistic content, or specific events discussed on social media, respectively) is significantly lower than the number of elements in the other classes. The recently introduced BERT language model for transfer learning BIBREF0 uses a deep bidirectional transformer architecture to produce pre-trained context-dependent embeddings. It has proven to be powerful in solving many NLP tasks and, as we find, also appears to handle imbalanced classification well, thus removing the need to use standard methods of data augmentation to mitigate this problem (see Section SECREF11 for related work and Section SECREF16 for analysis). BERT is credited with the ability to adapt to many tasks and data with very little training BIBREF0. However, we show that BERT fails to perform well when the training and test data are significantly dissimilar, as is the case with several tasks that deal with social and news data. In these cases, the training data is necessarily a subset of past data, while the model is likely to be used on future data which deals with different topics. This work addresses this problem by incorporating cost-sensitivity (Section SECREF19) into BERT. We test these methods by participating in the Shared Task on Fine-Grained Propaganda Detection for the 2nd Workshop on NLP for Internet Freedom, for which we achieve the second rank on sentence-level classification of propaganda, confirming the importance of cost-sensitivity when the training and test sets are dissimilar. Introduction ::: Detecting Propaganda The term `propaganda' derives from propagare in post-classical Latin, as in “propagation of the faith" BIBREF1, and thus has from the beginning been associated with an intentional and potentially multicast communication; only later did it become a pejorative term. It was pragmatically defined in the World War II era as “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends" BIBREF2. For the philosopher and sociologist Jacques Ellul, however, in a society with mass communication, propaganda is inevitable and thus it is necessary to become more aware of it BIBREF3; but whether or not to classify a given strip of text as propaganda depends not just on its content but on its use on the part of both addressers and addressees BIBREF1, and this fact makes the automated detection of propaganda intrinsically challenging. Despite this difficulty, interest in automatically detecting misinformation and/or propaganda has gained significance due to the exponential growth in online sources of information combined with the speed with which information is shared today. The sheer volume of social interactions makes it impossible to manually check the veracity of all information being shared. Automation thus remains a potentially viable method of ensuring that we continue to enjoy the benefits of a connected world without the spread of misinformation through either ignorance or malicious intent. In the task introduced by BIBREF4, we are provided with articles tagged as propaganda at the sentence and fragment (or span) level and are tasked with making predictions on a development set followed by a final held-out test set. We note this gives us access to the articles in the development and test sets but not their labels. We participated in this task under the team name ProperGander and were placed 2nd on the sentence level classification task where we make use of our methods of incorporating cost-sensitivity into BERT. We also participated in the fragment level task and were placed 7th. The significant contributions of this work are: We show that common (`easy') methods of data augmentation for dealing with class imbalance do not improve base BERT performance. We provide a statistical method of establishing the similarity of datasets. We incorporate cost-sensitivity into BERT to enable models to adapt to dissimilar datasets. We release all our program code on GitHub and Google Colaboratory, so that other researchers can benefit from this work. Related work ::: Propaganda detection Most of the existing works on propaganda detection focus on identifying propaganda at the news article level, or even at the news outlet level with the assumption that each of the articles of the suspected propagandistic outlet are propaganda BIBREF5, BIBREF6. Here we study two tasks that are more fine-grained, specifically propaganda detection at the sentence and phrase (fragment) levels BIBREF4. This fine-grained setup aims to train models that identify linguistic propaganda techniques rather than distinguishing between the article source styles. BIBREF4 EMNLP19DaSanMartino were the first to propose this problem setup and release it as a shared task. Along with the released dataset, BIBREF4 proposed a multi-granularity neural network, which uses the deep bidirectional transformer architecture known as BERT, which features pre-trained context-dependent embeddings BIBREF0. Their system takes a joint learning approach to the sentence- and phrase-level tasks, concatenating the output representation of the less granular (sentence-level) task with the more fine-grained task using learned weights. In this work we also take the BERT model as the basis of our approach and focus on the class imbalance as well as the lack of similarity between training and test data inherent to the task. Related work ::: Class imbalance A common issue for many Natural Language Processing (NLP) classification tasks is class imbalance, the situation where one of the class categories comprises a significantly larger proportion of the dataset than the other classes. It is especially prominent in real-world datasets and complicates classification when the identification of the minority class is of specific importance. Models trained on the basis of minimising errors for imbalanced datasets tend to more frequently predict the majority class; achieving high accuracy in such cases can be misleading. Because of this, the macro-averaged F-score, chosen for this competition, is a more suitable metric as it weights the performance on each class equally. As class imbalance is a widespread issue, multiple techniques have been developed that help alleviate it BIBREF7, BIBREF8, by either adjusting the model (e.g. changing the performance metric) or changing the data (e.g. oversampling the minority class or undersampling the majority class). Related work ::: Class imbalance ::: Cost-sensitive learning Cost-sensitive classification can be used when the “cost” of mislabelling one class is higher than that of mislabelling other classes BIBREF9, BIBREF10. For example, the real cost to a bank of miscategorising a large fraudulent transaction as authentic is potentially higher than miscategorising (perhaps only temporarily) a valid transaction as fraudulent. Cost-sensitive learning tackles the issue of class imbalance by changing the cost function of the model such that misclassification of training examples from the minority class carries more weight and is thus more `expensive'. This is achieved by simply multiplying the loss of each example by a certain factor. This cost-sensitive learning technique takes misclassification costs into account during model training, and does not modify the imbalanced data distribution directly. Related work ::: Class imbalance ::: Data augmentation Common methods that tackle the problem of class imbalance by modifying the data to create balanced datasets are undersampling and oversampling. Undersampling randomly removes instances from the majority class and is only suitable for problems with an abundance of data. Oversampling means creating more minority class instances to match the size of the majority class. Oversampling methods range from simple random oversampling, i.e. repeating the training procedure on instances from the minority class, chosen at random, to the more complex, which involves constructing synthetic minority-class samples. Random oversampling is similar to cost-sensitive learning as repeating the sample several times makes the cost of its mis-classification grow proportionally. Kolomiyets et al. kolomiyets2011model, Zhang et al. zhang2015character, and Wang and Yang wang2015s perform data augmentation using synonym replacement, i.e. replacing random words in sentences with their synonyms or nearest-neighbor embeddings, and show its effectiveness on multiple tasks and datasets. Wei et al. wei2019eda provide a great overview of `easy' data augmentation (EDA) techniques for NLP, including synonym replacement as described above, and random deletion, i.e. removing words in the sentence at random with pre-defined probability. They show the effectiveness of EDA across five text classification tasks. However, they mention that EDA may not lead to substantial improvements when using pre-trained models. In this work we test this claim by comparing performance gains of using cost-sensitive learning versus two data augmentation methods, synonym replacement and random deletion, with a pre-trained BERT model. More complex augmentation methods include back-translation BIBREF11, translational data augmentation BIBREF12, and noising BIBREF13, but these are out of the scope of this study. Dataset The Propaganda Techniques Corpus (PTC) dataset for the 2019 Shared Task on Fine-Grained Propaganda consists of a training set of 350 news articles, consisting of just over 16,965 total sentences, in which specifically propagandistic fragments have been manually spotted and labelled by experts. This is accompanied by a development set (or dev set) of 61 articles with 2,235 total sentences, whose labels are maintained by the shared task organisers; and two months after the release of this data, the organisers released a test set of 86 articles and 3,526 total sentences. In the training set, 4,720 ($\sim 28\%$) of the sentences have been assessed as containing propaganda, with 12,245 sentences ($\sim 72 \%$) as non-propaganda, demonstrating a clear class imbalance. In the binary sentence-level classification (SLC) task, a model is trained to detect whether each and every sentence is either 'propaganda' or 'non-propaganda'; in the more challenging field-level classification (FLC) task, a model is trained to detect one of 18 possible propaganda technique types in spans of characters within sentences. These propaganda types are listed in BIBREF4 and range from those which might be recognisable at the lexical level (e.g. Name_Calling, Repetition), and those which would likely need to incorporate semantic understanding (Red_Herring, Straw_Man). For several example sentences from a sample document annotated with fragment-level classifications (FLC) (Figure FIGREF13). The corresponding sentence-level classification (SLC) labels would indicate that sentences 3, 4, and 7 are 'propaganda' while the the other sentences are `non-propaganda'. Dataset ::: Data Distribution One of the most interesting aspects of the data provided for this task is the notable difference between the training and the development/test sets. We emphasise that this difference is realistic and reflective of real world news data, in which major stories are often accompanied by the introduction of new terms, names, and even phrases. This is because the training data is a subset of past data while the model is to be used on future data which deals with different newsworthy topics. We demonstrate this difference statistically by using a method for finding the similarity of corpora suggested by BIBREF14. We use the Wilcoxon signed-rank test BIBREF15 which compares the frequency counts of randomly sampled elements from different datasets to determine if those datasets have a statistically similar distribution of elements. We implement this as follows. For each of the training, development and test sets, we extract all words (retaining the repeats) while ignoring a set of stopwords (identified through the Python Natural Language Toolkit). We then extract 10,000 samples (with replacements) for various pairs of these datasets (training, development, and test sets along with splits of each of these datasets). Finally, we use comparative word frequencies from the two sets to calculate the p-value using the Wilcoxon signed-rank test. Table TABREF15 provides the minimum and maximum p-values and their interpretations for ten such runs of each pair reported. With p-value less than 0.05, we show that the train, development and test sets are self-similar and also significantly different from each other. In measuring self-similarity, we split each dataset after shuffling all sentences. While this comparison is made at the sentence level (as opposed to the article level), it is consistent with the granularity used for propaganda detection, which is also at the sentence level. We also perform measurements of self similarity after splitting the data at the article level and find that the conclusions of similarity between the sets hold with a p-value threshold of 0.001, where p-values for similarity between the training and dev/test sets are orders of magnitude lower compared to self-similarity. Since we use random sampling we run this test 10 times and present the both the maximum and minimum p-values. We include the similarity between 25% of a dataset and the remaining 75% of that set because that is the train/test ratio we use in our experiments, further described in our methodology (Section SECREF4). This analysis shows that while all splits of each of the datasets are statistically similar, the training set (and the split of the training set that we use for experimentation) are significantly different from the development and test sets. While our analysis does show that the development and the test sets are dissimilar, we note (based on the p-values) that they are significantly more similar to each other than they are to the training set. Methodology We were provided with two tasks: (1) propaganda fragment-level identification (FLC) and (2) propagandistic sentence-level identification (SLC). While we develop systems for both tasks, our main focus is toward the latter. Given the differences between the training, development, and test sets, we focus on methods for generalising our models. We note that propaganda identification is, in general, an imbalanced binary classification problem as most sentences are not propagandistic. Due to the non-deterministic nature of fast GPU computations, we run each of our models three times and report the average of these three runs through the rest of this section. When picking the model to use for our final submission, we pick the model that performs best on the development set. When testing our models, we split the labelled training data into two non-overlapping parts: the first one, consisting of 75% of the training data is used to train models, whereas the other is used to test the effectiveness of the models. All models are trained and tested on the same split to ensure comparability. Similarly, to ensure that our models remain comparable, we continue to train on the same 75% of the training set even when testing on the development set. Once the best model is found using these methods, we train that model on all of the training data available before then submitting the results on the development set to the leaderboard. These results are detailed in the section describing our results (Section SECREF5). Methodology ::: Class Imbalance in Sentence Level Classification The sentence level classification task is an imbalanced binary classification problem that we address using BERT BIBREF0. We use BERTBASE, uncased, which consists of 12 self-attention layers, and returns a 768-dimension vector that representation a sentence. So as to make use of BERT for sentence classification, we include a fully connected layer on top of the BERT self-attention layers, which classifies the sentence embedding provided by BERT into the two classes of interest (propaganda or non-propaganda). We attempt to exploit various data augmentation techniques to address the problem of class imbalance. Table TABREF17 shows the results of our experiments for different data augmentation techniques when, after shuffling the training data, we train the model on 75% of the training data and test it on the remaining 25% of the training data and the development data. We observe that BERT without augmentation consistently outperforms BERT with augmentation in the experiments when the model is trained on 75% of the training data and evaluated on the rest, i.e trained and evaluated on similar data, coming from the same distribution. This is consistent with observations by Wei et al. wei2019eda that contextual word embeddings do not gain from data augmentation. The fact that we shuffle the training data prior to splitting it into training and testing subsets could imply that the model is learning to associate topic words, such as `Mueller', as propaganda. However, when we perform model evaluation using the development set, which is dissimilar to the training, we observe that synonym insertion and word dropping techniques also do not bring performance gains, while random oversampling increases performance over base BERT by 4%. Synonym insertion provides results very similar to base BERT, while random deletion harms model performance producing lower scores. We believe that this could be attributed to the fact that synonym insertion and random word dropping involve the introduction of noise to the data, while oversampling does not. As we are working with natural language data, this type of noise can in fact change the meaning of the sentence. Oversampling on the other hand purely increases the importance of the minority class by repeating training on the unchanged instances. So as to better understand the aspects of oversampling that contribute to these gains, we perform a class-wise performance analysis of BERT with/without oversampling. The results of these experiments (Table TABREF18) show that oversampling increases the overall recall while maintaining precision. This is achieved by significantly improving the recall of the minority class (propaganda) at the cost of the recall of the majority class. So far we have been able to establish that a) the training and test sets are dissimilar, thus requiring us to generalise our model, b) oversampling provides a method of generalisation, and c) oversampling does this while maintaining recall on the minority (and thus more interesting) class. Given this we explore alternative methods of increasing minority class recall without a significant drop in precision. One such method is cost-sensitive classification, which differs from random oversampling in that it provides a more continuous-valued and consistent method of weighting samples of imbalanced training data; for example, random oversampling will inevitably emphasise some training instances at the expense of others. We detail our methods of using cost-sensitive classification in the next section. Further experiments with oversampling might have provided insights into the relationships between these methods, which we leave for future exploration. Methodology ::: Cost-sensitive Classification As discussed in Section SECREF10, cost-sensitive classification can be performed by weighting the cost function. We increase the weight of incorrectly labelling a propagandistic sentence by altering the cost function of the training of the final fully connected layer of our model previously described in Section SECREF16. We make these changes through the use of PyTorch BIBREF16 which calculates the cross-entropy loss for a single prediction $x$, an array where the $j^{th}$ element represents the models prediction for class $j$, labelled with the class $class$ as given by Equation DISPLAY_FORM20. The cross-entropy loss given in Equation DISPLAY_FORM20 is modified to accommodate an array $weight$, the $i^{th}$ element of which represents the weight of the $i^{th}$ class, as described in Equation DISPLAY_FORM21. Intuitively, we increase the cost of getting the classification of an “important” class wrong and corresponding decrees the cost of getting a less important class wrong. In our case, we increase the cost of mislabelling the minority class which is “propaganda” (as opposed to “non-propaganda”). We expect the effect of this to be similar to that of oversampling, in that it is likely to enable us to increase the recall of the minority class thus resulting in the decrease in recall of the overall model while maintaining high precision. We reiterate that this specific change to a model results in increasing the model's ability to better identify elements belonging to the minority class in dissimilar datasets when using BERT. We explore the validity of this by performing several experiments with different weights assigned to the minority class. We note that in our experiments use significantly higher weights than the weights proportional to class frequencies in the training data, that are common in literature BIBREF17. Rather than directly using the class proportions of the training set, we show that tuning weights based on performance on the development set is more beneficial. Figure FIGREF22 shows the results of these experiments wherein we are able to maintain the precision on the subset of the training set used for testing while reducing its recall and thus generalising the model. The fact that the model is generalising on a dissimilar dataset is confirmed by the increase in the development set F1 score. We note that the gains are not infinite and that a balance must be struck based on the amount of generalisation and the corresponding loss in accuracy. The exact weight to use for the best transfer of classification accuracy is related to the dissimilarity of that other dataset and hence is to be obtained experimentally through hyperparameter search. Our experiments showed that a value of 4 is best suited for this task. We do not include the complete results of our experiments here due to space constraints but include them along with charts and program code on our project website. Based on this exploration we find that the best weights for this particular dataset are 1 for non-propaganda and 4 for propaganda and we use this to train the final model used to submit results to the leaderboard. We also found that adding Part of Speech tags and Named Entity information to BERT embeddings by concatenating these one-hot vectors to the BERT embeddings does not improve model performance. We describe these results in Section SECREF5. Methodology ::: Fragment-level classification (FLC) In addition to participating in the Sentence Level Classification task we also participate in the Fragment Level Classification task. We note that extracting fragments that are propagandistic is similar to the task of Named Entity Recognition, in that they are both span extraction tasks, and so use a BERT based model designed for this task - We build on the work by BIBREF18 which makes use of Continuous Random Field stacked on top of an LSTM to predict spans. This architecture is standard amongst state of the art models that perform span identification. While the same span of text cannot have multiple named entity labels, it can have different propaganda labels. We get around this problem by picking one of the labels at random. Additionally, so as to speed up training, we only train our model on those sentences that contain some propagandistic fragment. In hindsight, we note that both these decisions were not ideal and discuss what we might have otherwise done in Section SECREF7. Results In this section, we show our rankings on the leaderboard on the test set. Unlike the previous exploratory sections, in which we trained our model on part of the training set, we train models described in this section on the complete training set. Results ::: Results on the SLC task Our best performing model, selected on the basis of a systematic analysis of the relationship between cost weights and recall, places us second amongst the 25 teams that submitted their results on this task. We present our score on the test set alongside those of comparable teams in Table TABREF25. We note that the task description paper BIBREF4 describes a method of achieving an F1 score of 60.98% on a similar task although this reported score is not directly comparable to the results on this task because of the differences in testing sets. Results ::: Results on the FLC task We train the model described in Section SECREF23 on the complete training set before submitting to the leaderboard. Our best performing model was placed 7th amongst the 13 teams that submitted results for this task. We present our score on the test set alongside those of comparable teams in Table TABREF27. We note that the task description paper BIBREF4 describes a method of achieving an F1 score of 22.58% on a similar task although, this reported score is not directly comparable to the results on this task. One of the major setbacks to our method for identifying sentence fragments was the loss of training data as a result of randomly picking one label when the same fragment had multiple labels. This could have been avoided by training different models for each label and simply concatenating the results. Additionally, training on all sentences, including those that did not contain any fragments labelled as propagandistic would have likely improved our model performance. We intend to perform these experiments as part of our ongoing research. Issues of Decontextualization in Automated Propaganda Detection It is worth reflecting on the nature of the shared task dataset (PTC corpus) and its structural correspondence (or lack thereof) to some of the definitions of propaganda mentioned in the introduction. First, propaganda is a social phenomenon and takes place as an act of communication BIBREF19, and so it is more than a simple information-theoretic message of zeros and ones—it also incorporates an addresser and addressee(s), each in phatic contact (typically via broadcast media), ideally with a shared denotational code and contextual surround(s) BIBREF20. As such, a dataset of decontextualised documents with labelled sentences, devoid of authorial or publisher metadata, has taken us at some remove from even a simple everyday definition of propaganda. Our models for this shared task cannot easily incorporate information about the addresser or addressee; are left to assume a shared denotational code between author and reader (one perhaps simulated with the use of pre-trained word embeddings); and they are unaware of when or where the act(s) of propagandistic communication took place. This slipperiness is illustrated in our example document (Fig. FIGREF13): note that while Sentences 3 and 7, labelled as propaganda, reflect a propagandistic attitude on the part of the journalist and/or publisher, Sentence 4—also labelled as propaganda in the training data—instead reflects a “flag-waving" propagandistic attitude on the part of U.S. congressman Jeff Flake, via the conventions of reported speech BIBREF21. While reported speech often is signaled by specific morphosyntactic patterns (e.g. the use of double-quotes and “Flake said") BIBREF22, we argue that human readers routinely distinguish propagandistic reportage from the propagandastic speech acts of its subjects, and to conflate these categories in a propaganda detection corpus may contribute to the occurrence of false positives/negatives. Conclusions and Future Work In this work we have presented a method of incorporating cost-sensitivity into BERT to allow for better generalisation and additionally, we provide a simple measure of corpus similarity to determine when this method is likely to be useful. We intend to extend our analysis of the ability to generalise models to less similar data by experimenting on other datasets and models. We hope that the release of program code and documentation will allow the research community to help in this experimentation while exploiting these methods. Acknowledgements We would like to thank Dr Leandro Minku from the University of Birmingham for his insights into and help with the statistical analysis presented in this paper. This work was also partially supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1. Work by Elena Kochkina was partially supported by the Leverhulme Trust through the Bridges Programme and Warwick CDT for Urban Science & Progress under the EPSRC Grant Number EP/L016400/1.
How is "propaganda" defined for the purposes of this study?
64af7f5c109ed10eda4fb1b70ecda21e6d5b96c8
How Ġis Ġ" prop aganda " Ġdefined Ġfor Ġthe Ġpurposes Ġof Ġthis Ġstudy ?
an Ġintentional Ġand Ġpotentially Ġmultic ast Ġcommunication , ĠâĢ ľ the Ġexpression Ġof Ġan Ġopinion Ġor Ġan Ġaction Ġby Ġindividuals Ġor Ġgroups Ġdeliberately Ġdesigned Ġto Ġinfluence Ġthe Ġopinions Ġor Ġthe Ġactions Ġof Ġother Ġindividuals Ġor Ġgroups Ġwith Ġreference Ġto Ġpredetermined Ġends "
<s> How Ġis Ġ" prop aganda " Ġdefined Ġfor Ġthe Ġpurposes Ġof Ġthis Ġstudy ? </s> Introduction </s> The Ġchallenges Ġof Ġim balanced Ġclassification âĢĶ in Ġwhich Ġthe Ġproportion Ġof Ġelements Ġin Ġeach Ġclass Ġfor Ġa Ġclassification Ġtask Ġsignificantly Ġdiffer âĢĶ and Ġof Ġthe Ġability Ġto Ġgeneral ise Ġon Ġdiss imilar Ġdata Ġhave Ġremained Ġimportant Ġproblems Ġin ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġand ĠMachine ĠLearning Ġin Ġgeneral . ĠPopular ĠN LP Ġtasks Ġincluding Ġsentiment Ġanalysis , Ġpropaganda Ġdetection , Ġand Ġevent Ġextraction Ġfrom Ġsocial Ġmedia Ġare Ġall Ġexamples Ġof Ġim balanced Ġclassification Ġproblems . ĠIn Ġeach Ġcase Ġthe Ġnumber Ġof Ġelements Ġin Ġone Ġof Ġthe Ġclasses Ġ( e . g . Ġnegative Ġsentiment , Ġpropag and istic Ġcontent , Ġor Ġspecific Ġevents Ġdiscussed Ġon Ġsocial Ġmedia , Ġrespectively ) Ġis Ġsignificantly Ġlower Ġthan Ġthe Ġnumber Ġof Ġelements Ġin Ġthe Ġother Ġclasses . </s> The Ġrecently Ġintroduced ĠB ERT Ġlanguage Ġmodel Ġfor Ġtransfer Ġlearning ĠB IB REF 0 Ġuses Ġa Ġdeep Ġbid irection al Ġtransformer Ġarchitecture Ġto Ġproduce Ġpre - trained Ġcontext - dependent Ġembed d ings . ĠIt Ġhas Ġproven Ġto Ġbe Ġpowerful Ġin Ġsolving Ġmany ĠN LP Ġtasks Ġand , Ġas Ġwe Ġfind , Ġalso Ġappears Ġto Ġhandle Ġim balanced Ġclassification Ġwell , Ġthus Ġremoving Ġthe Ġneed Ġto Ġuse Ġstandard Ġmethods Ġof Ġdata Ġaug mentation Ġto Ġmitigate Ġthis Ġproblem Ġ( see ĠSection ĠSEC REF 11 Ġfor Ġrelated Ġwork Ġand ĠSection ĠSEC REF 16 Ġfor Ġanalysis ). </s> BER T Ġis Ġcredited Ġwith Ġthe Ġability Ġto Ġadapt Ġto Ġmany Ġtasks Ġand Ġdata Ġwith Ġvery Ġlittle Ġtraining ĠB IB REF 0 . ĠHowever , Ġwe Ġshow Ġthat ĠB ERT Ġfails Ġto Ġperform Ġwell Ġwhen Ġthe Ġtraining Ġand Ġtest Ġdata Ġare Ġsignificantly Ġdiss imilar , Ġas Ġis Ġthe Ġcase Ġwith Ġseveral Ġtasks Ġthat Ġdeal Ġwith Ġsocial Ġand Ġnews Ġdata . ĠIn Ġthese Ġcases , Ġthe Ġtraining Ġdata Ġis Ġnecessarily Ġa Ġsubset Ġof Ġpast Ġdata , Ġwhile Ġthe Ġmodel Ġis Ġlikely Ġto Ġbe Ġused Ġon Ġfuture Ġdata Ġwhich Ġdeals Ġwith Ġdifferent Ġtopics . ĠThis Ġwork Ġaddresses Ġthis Ġproblem Ġby Ġincorporating Ġcost - s ensitivity Ġ( Section ĠSEC REF 19 ) Ġinto ĠB ERT . </s> We Ġtest Ġthese Ġmethods Ġby Ġparticipating Ġin Ġthe ĠShared ĠTask Ġon ĠFine - Gr ained ĠProp aganda ĠDetection Ġfor Ġthe Ġ2 nd ĠWorkshop Ġon ĠN LP Ġfor ĠInternet ĠFreedom , Ġfor Ġwhich Ġwe Ġachieve Ġthe Ġsecond Ġrank Ġon Ġsentence - level Ġclassification Ġof Ġpropaganda , Ġconfirming Ġthe Ġimportance Ġof Ġcost - s ensitivity Ġwhen Ġthe Ġtraining Ġand Ġtest Ġsets Ġare Ġdiss imilar . </s> Introduction Ġ: :: ĠDetect ing ĠProp aganda </s> The Ġterm Ġ` prop aganda ' Ġderives Ġfrom Ġpropag are Ġin Ġpost - class ical ĠLatin , Ġas Ġin ĠâĢ ľ prop ag ation Ġof Ġthe Ġfaith " ĠB IB REF 1 , Ġand Ġthus Ġhas Ġfrom Ġthe Ġbeginning Ġbeen Ġassociated Ġwith Ġan Ġintentional Ġand Ġpotentially Ġmultic ast Ġcommunication ; Ġonly Ġlater Ġdid Ġit Ġbecome Ġa Ġpe j orative Ġterm . ĠIt Ġwas Ġprag matically Ġdefined Ġin Ġthe ĠWorld ĠWar ĠII Ġera Ġas ĠâĢ ľ the Ġexpression Ġof Ġan Ġopinion Ġor Ġan Ġaction Ġby Ġindividuals Ġor Ġgroups Ġdeliberately Ġdesigned Ġto Ġinfluence Ġthe Ġopinions Ġor Ġthe Ġactions Ġof Ġother Ġindividuals Ġor Ġgroups Ġwith Ġreference Ġto Ġpredetermined Ġends " ĠB IB REF 2 . </s> For Ġthe Ġphilosopher Ġand Ġsoc iologist ĠJacques ĠEll ul , Ġhowever , Ġin Ġa Ġsociety Ġwith Ġmass Ġcommunication , Ġpropaganda Ġis Ġinevitable Ġand Ġthus Ġit Ġis Ġnecessary Ġto Ġbecome Ġmore Ġaware Ġof Ġit ĠB IB REF 3 ; Ġbut Ġwhether Ġor Ġnot Ġto Ġclassify Ġa Ġgiven Ġstrip Ġof Ġtext Ġas Ġpropaganda Ġdepends Ġnot Ġjust Ġon Ġits Ġcontent Ġbut Ġon Ġits Ġuse Ġon Ġthe Ġpart Ġof Ġboth Ġaddress ers Ġand Ġadd res se es ĠB IB REF 1 , Ġand Ġthis Ġfact Ġmakes Ġthe Ġautomated Ġdetection Ġof Ġpropaganda Ġintrinsically Ġchallenging . </s> Despite Ġthis Ġdifficulty , Ġinterest Ġin Ġautomatically Ġdetecting Ġmisinformation Ġand / or Ġpropaganda Ġhas Ġgained Ġsignificance Ġdue Ġto Ġthe Ġexponential Ġgrowth Ġin Ġonline Ġsources Ġof Ġinformation Ġcombined Ġwith Ġthe Ġspeed Ġwith Ġwhich Ġinformation Ġis Ġshared Ġtoday . ĠThe Ġsheer Ġvolume Ġof Ġsocial Ġinteractions Ġmakes Ġit Ġimpossible Ġto Ġmanually Ġcheck Ġthe Ġver acity Ġof Ġall Ġinformation Ġbeing Ġshared . ĠAutom ation Ġthus Ġremains Ġa Ġpotentially Ġviable Ġmethod Ġof Ġensuring Ġthat Ġwe Ġcontinue Ġto Ġenjoy Ġthe Ġbenefits Ġof Ġa Ġconnected Ġworld Ġwithout Ġthe Ġspread Ġof Ġmisinformation Ġthrough Ġeither Ġignorance Ġor Ġmalicious Ġintent . </s> In Ġthe Ġtask Ġintroduced Ġby ĠB IB REF 4 , Ġwe Ġare Ġprovided Ġwith Ġarticles Ġtagged Ġas Ġpropaganda Ġat Ġthe Ġsentence Ġand Ġfragment Ġ( or Ġspan ) Ġlevel Ġand Ġare Ġtasked Ġwith Ġmaking Ġpredictions Ġon Ġa Ġdevelopment Ġset Ġfollowed Ġby Ġa Ġfinal Ġheld - out Ġtest Ġset . ĠWe Ġnote Ġthis Ġgives Ġus Ġaccess Ġto Ġthe Ġarticles Ġin Ġthe Ġdevelopment Ġand Ġtest Ġsets Ġbut Ġnot Ġtheir Ġlabels . </s> We Ġparticipated Ġin Ġthis Ġtask Ġunder Ġthe Ġteam Ġname ĠProper G ander Ġand Ġwere Ġplaced Ġ2 nd Ġon Ġthe Ġsentence Ġlevel Ġclassification Ġtask Ġwhere Ġwe Ġmake Ġuse Ġof Ġour Ġmethods Ġof Ġincorporating Ġcost - s ensitivity Ġinto ĠB ERT . ĠWe Ġalso Ġparticipated Ġin Ġthe Ġfragment Ġlevel Ġtask Ġand Ġwere Ġplaced Ġ7 th . ĠThe Ġsignificant Ġcontributions Ġof Ġthis Ġwork Ġare : </s> We Ġshow Ġthat Ġcommon Ġ( ` easy ') Ġmethods Ġof Ġdata Ġaug mentation Ġfor Ġdealing Ġwith Ġclass Ġimbalance Ġdo Ġnot Ġimprove Ġbase ĠB ERT Ġperformance . </s> We Ġprovide Ġa Ġstatistical Ġmethod Ġof Ġestablishing Ġthe Ġsimilarity Ġof Ġdatasets . </s> We Ġincorporate Ġcost - s ensitivity Ġinto ĠB ERT Ġto Ġenable Ġmodels Ġto Ġadapt Ġto Ġdiss imilar Ġdatasets . </s> We Ġrelease Ġall Ġour Ġprogram Ġcode Ġon ĠGitHub Ġand ĠGoogle ĠCol abor atory , Ġso Ġthat Ġother Ġresearchers Ġcan Ġbenefit Ġfrom Ġthis Ġwork . </s> Related Ġwork Ġ: :: ĠProp aganda Ġdetection </s> Most Ġof Ġthe Ġexisting Ġworks Ġon Ġpropaganda Ġdetection Ġfocus Ġon Ġidentifying Ġpropaganda Ġat Ġthe Ġnews Ġarticle Ġlevel , Ġor Ġeven Ġat Ġthe Ġnews Ġoutlet Ġlevel Ġwith Ġthe Ġassumption Ġthat Ġeach Ġof Ġthe Ġarticles Ġof Ġthe Ġsuspected Ġpropag and istic Ġoutlet Ġare Ġpropaganda ĠB IB REF 5 , ĠB IB REF 6 . </s> Here Ġwe Ġstudy Ġtwo Ġtasks Ġthat Ġare Ġmore Ġfine - gr ained , Ġspecifically Ġpropaganda Ġdetection Ġat Ġthe Ġsentence Ġand Ġphrase Ġ( fr ag ment ) Ġlevels ĠB IB REF 4 . ĠThis Ġfine - gr ained Ġsetup Ġaims Ġto Ġtrain Ġmodels Ġthat Ġidentify Ġlinguistic Ġpropaganda Ġtechniques Ġrather Ġthan Ġdistinguishing Ġbetween Ġthe Ġarticle Ġsource Ġstyles . </s> BI B REF 4 ĠEM N LP 19 Da San Mart ino Ġwere Ġthe Ġfirst Ġto Ġpropose Ġthis Ġproblem Ġsetup Ġand Ġrelease Ġit Ġas Ġa Ġshared Ġtask . ĠAlong Ġwith Ġthe Ġreleased Ġdataset , ĠB IB REF 4 Ġproposed Ġa Ġmulti - gran ularity Ġneural Ġnetwork , Ġwhich Ġuses Ġthe Ġdeep Ġbid irection al Ġtransformer Ġarchitecture Ġknown Ġas ĠB ERT , Ġwhich Ġfeatures Ġpre - trained Ġcontext - dependent Ġembed d ings ĠB IB REF 0 . ĠTheir Ġsystem Ġtakes Ġa Ġjoint Ġlearning Ġapproach Ġto Ġthe Ġsentence - Ġand Ġphrase - level Ġtasks , Ġconc aten ating Ġthe Ġoutput Ġrepresentation Ġof Ġthe Ġless Ġgran ular Ġ( sent ence - level ) Ġtask Ġwith Ġthe Ġmore Ġfine - gr ained Ġtask Ġusing Ġlearned Ġweights . </s> In Ġthis Ġwork Ġwe Ġalso Ġtake Ġthe ĠB ERT Ġmodel Ġas Ġthe Ġbasis Ġof Ġour Ġapproach Ġand Ġfocus Ġon Ġthe Ġclass Ġimbalance Ġas Ġwell Ġas Ġthe Ġlack Ġof Ġsimilarity Ġbetween Ġtraining Ġand Ġtest Ġdata Ġinherent Ġto Ġthe Ġtask . </s> Related Ġwork Ġ: :: ĠClass Ġimbalance </s> A Ġcommon Ġissue Ġfor Ġmany ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġclassification Ġtasks Ġis Ġclass Ġimbalance , Ġthe Ġsituation Ġwhere Ġone Ġof Ġthe Ġclass Ġcategories Ġcomprises Ġa Ġsignificantly Ġlarger Ġproportion Ġof Ġthe Ġdataset Ġthan Ġthe Ġother Ġclasses . ĠIt Ġis Ġespecially Ġprominent Ġin Ġreal - world Ġdatasets Ġand Ġcompl icates Ġclassification Ġwhen Ġthe Ġidentification Ġof Ġthe Ġminority Ġclass Ġis Ġof Ġspecific Ġimportance . </s> Mod els Ġtrained Ġon Ġthe Ġbasis Ġof Ġminim ising Ġerrors Ġfor Ġim balanced Ġdatasets Ġtend Ġto Ġmore Ġfrequently Ġpredict Ġthe Ġmajority Ġclass ; Ġachieving Ġhigh Ġaccuracy Ġin Ġsuch Ġcases Ġcan Ġbe Ġmisleading . ĠBecause Ġof Ġthis , Ġthe Ġmacro - aver aged ĠF - score , Ġchosen Ġfor Ġthis Ġcompetition , Ġis Ġa Ġmore Ġsuitable Ġmetric Ġas Ġit Ġweights Ġthe Ġperformance Ġon Ġeach Ġclass Ġequally . </s> As Ġclass Ġimbalance Ġis Ġa Ġwidespread Ġissue , Ġmultiple Ġtechniques Ġhave Ġbeen Ġdeveloped Ġthat Ġhelp Ġalleviate Ġit ĠB IB REF 7 , ĠB IB REF 8 , Ġby Ġeither Ġadjusting Ġthe Ġmodel Ġ( e . g . Ġchanging Ġthe Ġperformance Ġmetric ) Ġor Ġchanging Ġthe Ġdata Ġ( e . g . Ġovers am pling Ġthe Ġminority Ġclass Ġor Ġunders am pling Ġthe Ġmajority Ġclass ). </s> Related Ġwork Ġ: :: ĠClass Ġimbalance Ġ: :: ĠCost - sensitive Ġlearning </s> Cost - sensitive Ġclassification Ġcan Ġbe Ġused Ġwhen Ġthe ĠâĢ ľ cost âĢ Ŀ Ġof Ġmis lab elling Ġone Ġclass Ġis Ġhigher Ġthan Ġthat Ġof Ġmis lab elling Ġother Ġclasses ĠB IB REF 9 , ĠB IB REF 10 . ĠFor Ġexample , Ġthe Ġreal Ġcost Ġto Ġa Ġbank Ġof Ġmisc ategor ising Ġa Ġlarge Ġfraudulent Ġtransaction Ġas Ġauthentic Ġis Ġpotentially Ġhigher Ġthan Ġmisc ategor ising Ġ( perhaps Ġonly Ġtemporarily ) Ġa Ġvalid Ġtransaction Ġas Ġfraudulent . ĠCost - sensitive Ġlearning Ġtackles Ġthe Ġissue Ġof Ġclass Ġimbalance Ġby Ġchanging Ġthe Ġcost Ġfunction Ġof Ġthe Ġmodel Ġsuch Ġthat Ġmis class ification Ġof Ġtraining Ġexamples Ġfrom Ġthe Ġminority Ġclass Ġcarries Ġmore Ġweight Ġand Ġis Ġthus Ġmore Ġ` expensive '. ĠThis Ġis Ġachieved Ġby Ġsimply Ġmultiplying Ġthe Ġloss Ġof Ġeach Ġexample Ġby Ġa Ġcertain Ġfactor . ĠThis Ġcost - sensitive Ġlearning Ġtechnique Ġtakes Ġmis class ification Ġcosts Ġinto Ġaccount Ġduring Ġmodel Ġtraining , Ġand Ġdoes Ġnot Ġmodify Ġthe Ġim balanced Ġdata Ġdistribution Ġdirectly . </s> Related Ġwork Ġ: :: ĠClass Ġimbalance Ġ: :: ĠData Ġaug mentation </s> Common Ġmethods Ġthat Ġtackle Ġthe Ġproblem Ġof Ġclass Ġimbalance Ġby Ġmodifying Ġthe Ġdata Ġto Ġcreate Ġbalanced Ġdatasets Ġare Ġunders am pling Ġand Ġovers am pling . ĠUnd ers am pling Ġrandomly Ġremoves Ġinstances Ġfrom Ġthe Ġmajority Ġclass Ġand Ġis Ġonly Ġsuitable Ġfor Ġproblems Ġwith Ġan Ġabundance Ġof Ġdata . ĠOvers am pling Ġmeans Ġcreating Ġmore Ġminority Ġclass Ġinstances Ġto Ġmatch Ġthe Ġsize Ġof Ġthe Ġmajority Ġclass . ĠOvers am pling Ġmethods Ġrange Ġfrom Ġsimple Ġrandom Ġovers am pling , Ġi . e . Ġrepeating Ġthe Ġtraining Ġprocedure Ġon Ġinstances Ġfrom Ġthe Ġminority Ġclass , Ġchosen Ġat Ġrandom , Ġto Ġthe Ġmore Ġcomplex , Ġwhich Ġinvolves Ġconstructing Ġsynthetic Ġminority - class Ġsamples . ĠRandom Ġovers am pling Ġis Ġsimilar Ġto Ġcost - sensitive Ġlearning Ġas Ġrepeating Ġthe Ġsample Ġseveral Ġtimes Ġmakes Ġthe Ġcost Ġof Ġits Ġmis - class ification Ġgrow Ġproportion ally . ĠKol om iy ets Ġet Ġal . Ġk ol om iy ets 2011 model , ĠZhang Ġet Ġal . Ġz hang 2015 character , Ġand ĠWang Ġand ĠYang Ġw ang 2015 s Ġperform Ġdata Ġaug mentation Ġusing Ġsyn onym Ġreplacement , Ġi . e . Ġreplacing Ġrandom Ġwords Ġin Ġsentences Ġwith Ġtheir Ġsyn onyms Ġor Ġnearest - ne igh bor Ġembed d ings , Ġand Ġshow Ġits Ġeffectiveness Ġon Ġmultiple Ġtasks Ġand Ġdatasets . ĠWei Ġet Ġal . Ġwe i 2019 eda Ġprovide Ġa Ġgreat Ġoverview Ġof Ġ` easy ' Ġdata Ġaug mentation Ġ( ED A ) Ġtechniques Ġfor ĠN LP , Ġincluding Ġsyn onym Ġreplacement Ġas Ġdescribed Ġabove , Ġand Ġrandom Ġdeletion , Ġi . e . Ġremoving Ġwords Ġin Ġthe Ġsentence Ġat Ġrandom Ġwith Ġpre - defined Ġprobability . ĠThey Ġshow Ġthe Ġeffectiveness Ġof ĠE DA Ġacross Ġfive Ġtext Ġclassification Ġtasks . ĠHowever , Ġthey Ġmention Ġthat ĠE DA Ġmay Ġnot Ġlead Ġto Ġsubstantial Ġimprovements Ġwhen Ġusing Ġpre - trained Ġmodels . ĠIn Ġthis Ġwork Ġwe Ġtest Ġthis Ġclaim Ġby Ġcomparing Ġperformance Ġgains Ġof Ġusing Ġcost - sensitive Ġlearning Ġversus Ġtwo Ġdata Ġaug mentation Ġmethods , Ġsyn onym Ġreplacement Ġand Ġrandom Ġdeletion , Ġwith Ġa Ġpre - trained ĠB ERT Ġmodel . </s> More Ġcomplex Ġaug mentation Ġmethods Ġinclude Ġback - translation ĠB IB REF 11 , Ġtransl ational Ġdata Ġaug mentation ĠB IB REF 12 , Ġand Ġno ising ĠB IB REF 13 , Ġbut Ġthese Ġare Ġout Ġof Ġthe Ġscope Ġof Ġthis Ġstudy . </s> Dat as et </s> The ĠProp aganda ĠTechniques ĠCorpus Ġ( P TC ) Ġdataset Ġfor Ġthe Ġ2019 ĠShared ĠTask Ġon ĠFine - Gr ained ĠProp aganda Ġconsists Ġof Ġa Ġtraining Ġset Ġof Ġ350 Ġnews Ġarticles , Ġconsisting Ġof Ġjust Ġover Ġ16 , 9 65 Ġtotal Ġsentences , Ġin Ġwhich Ġspecifically Ġpropag and istic Ġfragments Ġhave Ġbeen Ġmanually Ġspotted Ġand Ġlabelled Ġby Ġexperts . ĠThis Ġis Ġaccompanied Ġby Ġa Ġdevelopment Ġset Ġ( or Ġdev Ġset ) Ġof Ġ61 Ġarticles Ġwith Ġ2 , 235 Ġtotal Ġsentences , Ġwhose Ġlabels Ġare Ġmaintained Ġby Ġthe Ġshared Ġtask Ġorganisers ; Ġand Ġtwo Ġmonths Ġafter Ġthe Ġrelease Ġof Ġthis Ġdata , Ġthe Ġorganisers Ġreleased Ġa Ġtest Ġset Ġof Ġ86 Ġarticles Ġand Ġ3 , 526 Ġtotal Ġsentences . ĠIn Ġthe Ġtraining Ġset , Ġ4 , 720 Ġ($ \ sim Ġ28 \ % $ ) Ġof Ġthe Ġsentences Ġhave Ġbeen Ġassessed Ġas Ġcontaining Ġpropaganda , Ġwith Ġ12 , 245 Ġsentences Ġ($ \ sim Ġ72 Ġ\ % $ ) Ġas Ġnon - prop aganda , Ġdemonstrating Ġa Ġclear Ġclass Ġimbalance . </s> In Ġthe Ġbinary Ġsentence - level Ġclassification Ġ( S LC ) Ġtask , Ġa Ġmodel Ġis Ġtrained Ġto Ġdetect Ġwhether Ġeach Ġand Ġevery Ġsentence Ġis Ġeither Ġ' prop aganda ' Ġor Ġ' non - prop aganda '; Ġin Ġthe Ġmore Ġchallenging Ġfield - level Ġclassification Ġ( FL C ) Ġtask , Ġa Ġmodel Ġis Ġtrained Ġto Ġdetect Ġone Ġof Ġ18 Ġpossible Ġpropaganda Ġtechnique Ġtypes Ġin Ġspans Ġof Ġcharacters Ġwithin Ġsentences . ĠThese Ġpropaganda Ġtypes Ġare Ġlisted Ġin ĠB IB REF 4 Ġand Ġrange Ġfrom Ġthose Ġwhich Ġmight Ġbe Ġrecogn isable Ġat Ġthe Ġlex ical Ġlevel Ġ( e . g . ĠName _ Calling , ĠRep etition ), Ġand Ġthose Ġwhich Ġwould Ġlikely Ġneed Ġto Ġincorporate Ġsemantic Ġunderstanding Ġ( Red _ Her ring , ĠStraw _ Man ). </s> For Ġseveral Ġexample Ġsentences Ġfrom Ġa Ġsample Ġdocument Ġannot ated Ġwith Ġfragment - level Ġclass ifications Ġ( FL C ) Ġ( Figure ĠFIG REF 13 ). ĠThe Ġcorresponding Ġsentence - level Ġclassification Ġ( S LC ) Ġlabels Ġwould Ġindicate Ġthat Ġsentences Ġ3 , Ġ4 , Ġand Ġ7 Ġare Ġ' prop aganda ' Ġwhile Ġthe Ġthe Ġother Ġsentences Ġare Ġ` non - prop aganda '. </s> Dat as et Ġ: :: ĠData ĠDistribution </s> One Ġof Ġthe Ġmost Ġinteresting Ġaspects Ġof Ġthe Ġdata Ġprovided Ġfor Ġthis Ġtask Ġis Ġthe Ġnotable Ġdifference Ġbetween Ġthe Ġtraining Ġand Ġthe Ġdevelopment / test Ġsets . ĠWe Ġemphas ise Ġthat Ġthis Ġdifference Ġis Ġrealistic Ġand Ġreflective Ġof Ġreal Ġworld Ġnews Ġdata , Ġin Ġwhich Ġmajor Ġstories Ġare Ġoften Ġaccompanied Ġby Ġthe Ġintroduction Ġof Ġnew Ġterms , Ġnames , Ġand Ġeven Ġphrases . ĠThis Ġis Ġbecause Ġthe Ġtraining Ġdata Ġis Ġa Ġsubset Ġof Ġpast Ġdata Ġwhile Ġthe Ġmodel Ġis Ġto Ġbe Ġused Ġon Ġfuture Ġdata Ġwhich Ġdeals Ġwith Ġdifferent Ġnews worthy Ġtopics . </s> We Ġdemonstrate Ġthis Ġdifference Ġstatistically Ġby Ġusing Ġa Ġmethod Ġfor Ġfinding Ġthe Ġsimilarity Ġof Ġcorpor a Ġsuggested Ġby ĠB IB REF 14 . ĠWe Ġuse Ġthe ĠWil co xon Ġsigned - rank Ġtest ĠB IB REF 15 Ġwhich Ġcompares Ġthe Ġfrequency Ġcounts Ġof Ġrandomly Ġsampled Ġelements Ġfrom Ġdifferent Ġdatasets Ġto Ġdetermine Ġif Ġthose Ġdatasets Ġhave Ġa Ġstatistically Ġsimilar Ġdistribution Ġof Ġelements . </s> We Ġimplement Ġthis Ġas Ġfollows . ĠFor Ġeach Ġof Ġthe Ġtraining , Ġdevelopment Ġand Ġtest Ġsets , Ġwe Ġextract Ġall Ġwords Ġ( ret aining Ġthe Ġrepeats ) Ġwhile Ġignoring Ġa Ġset Ġof Ġstop words Ġ( identified Ġthrough Ġthe ĠPython ĠNatural ĠLanguage ĠTool kit ). ĠWe Ġthen Ġextract Ġ10 , 000 Ġsamples Ġ( with Ġreplacements ) Ġfor Ġvarious Ġpairs Ġof Ġthese Ġdatasets Ġ( training , Ġdevelopment , Ġand Ġtest Ġsets Ġalong Ġwith Ġsplits Ġof Ġeach Ġof Ġthese Ġdatasets ). ĠFinally , Ġwe Ġuse Ġcomparative Ġword Ġfrequencies Ġfrom Ġthe Ġtwo Ġsets Ġto Ġcalculate Ġthe Ġp - value Ġusing Ġthe ĠWil co xon Ġsigned - rank Ġtest . ĠTable ĠT AB REF 15 Ġprovides Ġthe Ġminimum Ġand Ġmaximum Ġp - values Ġand Ġtheir Ġinterpretations Ġfor Ġten Ġsuch Ġruns Ġof Ġeach Ġpair Ġreported . </s> With Ġp - value Ġless Ġthan Ġ0 . 05 , Ġwe Ġshow Ġthat Ġthe Ġtrain , Ġdevelopment Ġand Ġtest Ġsets Ġare Ġself - similar Ġand Ġalso Ġsignificantly Ġdifferent Ġfrom Ġeach Ġother . ĠIn Ġmeasuring Ġself - similar ity , Ġwe Ġsplit Ġeach Ġdataset Ġafter Ġshuff ling Ġall Ġsentences . ĠWhile Ġthis Ġcomparison Ġis Ġmade Ġat Ġthe Ġsentence Ġlevel Ġ( as Ġopposed Ġto Ġthe Ġarticle Ġlevel ), Ġit Ġis Ġconsistent Ġwith Ġthe Ġgran ularity Ġused Ġfor Ġpropaganda Ġdetection , Ġwhich Ġis Ġalso Ġat Ġthe Ġsentence Ġlevel . ĠWe Ġalso Ġperform Ġmeasurements Ġof Ġself Ġsimilarity Ġafter Ġsplitting Ġthe Ġdata Ġat Ġthe Ġarticle Ġlevel Ġand Ġfind Ġthat Ġthe Ġconclusions Ġof Ġsimilarity Ġbetween Ġthe Ġsets Ġhold Ġwith Ġa Ġp - value Ġthreshold Ġof Ġ0 . 001 , Ġwhere Ġp - values Ġfor Ġsimilarity Ġbetween Ġthe Ġtraining Ġand Ġdev / test Ġsets Ġare Ġorders Ġof Ġmagnitude Ġlower Ġcompared Ġto Ġself - similar ity . ĠSince Ġwe Ġuse Ġrandom Ġsampling Ġwe Ġrun Ġthis Ġtest Ġ10 Ġtimes Ġand Ġpresent Ġthe Ġboth Ġthe Ġmaximum Ġand Ġminimum Ġp - values . ĠWe Ġinclude Ġthe Ġsimilarity Ġbetween Ġ25 % Ġof Ġa Ġdataset Ġand Ġthe Ġremaining Ġ75 % Ġof Ġthat Ġset Ġbecause Ġthat Ġis Ġthe Ġtrain / test Ġratio Ġwe Ġuse Ġin Ġour Ġexperiments , Ġfurther Ġdescribed Ġin Ġour Ġmethodology Ġ( Section ĠSEC REF 4 ). </s> This Ġanalysis Ġshows Ġthat Ġwhile Ġall Ġsplits Ġof Ġeach Ġof Ġthe Ġdatasets Ġare Ġstatistically Ġsimilar , Ġthe Ġtraining Ġset Ġ( and Ġthe Ġsplit Ġof Ġthe Ġtraining Ġset Ġthat Ġwe Ġuse Ġfor Ġexperimentation ) Ġare Ġsignificantly Ġdifferent Ġfrom Ġthe Ġdevelopment Ġand Ġtest Ġsets . ĠWhile Ġour Ġanalysis Ġdoes Ġshow Ġthat Ġthe Ġdevelopment Ġand Ġthe Ġtest Ġsets Ġare Ġdiss imilar , Ġwe Ġnote Ġ( based Ġon Ġthe Ġp - values ) Ġthat Ġthey Ġare Ġsignificantly Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġthey Ġare Ġto Ġthe Ġtraining Ġset . </s> Method ology </s> We Ġwere Ġprovided Ġwith Ġtwo Ġtasks : Ġ( 1 ) Ġpropaganda Ġfragment - level Ġidentification Ġ( FL C ) Ġand Ġ( 2 ) Ġpropag and istic Ġsentence - level Ġidentification Ġ( S LC ). ĠWhile Ġwe Ġdevelop Ġsystems Ġfor Ġboth Ġtasks , Ġour Ġmain Ġfocus Ġis Ġtoward Ġthe Ġlatter . ĠGiven Ġthe Ġdifferences Ġbetween Ġthe Ġtraining , Ġdevelopment , Ġand Ġtest Ġsets , Ġwe Ġfocus Ġon Ġmethods Ġfor Ġgeneral ising Ġour Ġmodels . ĠWe Ġnote Ġthat Ġpropaganda Ġidentification Ġis , Ġin Ġgeneral , Ġan Ġim balanced Ġbinary Ġclassification Ġproblem Ġas Ġmost Ġsentences Ġare Ġnot Ġpropag and istic . </s> Due Ġto Ġthe Ġnon - d eter ministic Ġnature Ġof Ġfast ĠGPU Ġcomput ations , Ġwe Ġrun Ġeach Ġof Ġour Ġmodels Ġthree Ġtimes Ġand Ġreport Ġthe Ġaverage Ġof Ġthese Ġthree Ġruns Ġthrough Ġthe Ġrest Ġof Ġthis Ġsection . ĠWhen Ġpicking Ġthe Ġmodel Ġto Ġuse Ġfor Ġour Ġfinal Ġsubmission , Ġwe Ġpick Ġthe Ġmodel Ġthat Ġperforms Ġbest Ġon Ġthe Ġdevelopment Ġset . </s> When Ġtesting Ġour Ġmodels , Ġwe Ġsplit Ġthe Ġlabelled Ġtraining Ġdata Ġinto Ġtwo Ġnon - over l apping Ġparts : Ġthe Ġfirst Ġone , Ġconsisting Ġof Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġis Ġused Ġto Ġtrain Ġmodels , Ġwhereas Ġthe Ġother Ġis Ġused Ġto Ġtest Ġthe Ġeffectiveness Ġof Ġthe Ġmodels . ĠAll Ġmodels Ġare Ġtrained Ġand Ġtested Ġon Ġthe Ġsame Ġsplit Ġto Ġensure Ġcompar ability . ĠSimilarly , Ġto Ġensure Ġthat Ġour Ġmodels Ġremain Ġcomparable , Ġwe Ġcontinue Ġto Ġtrain Ġon Ġthe Ġsame Ġ75 % Ġof Ġthe Ġtraining Ġset Ġeven Ġwhen Ġtesting Ġon Ġthe Ġdevelopment Ġset . </s> Once Ġthe Ġbest Ġmodel Ġis Ġfound Ġusing Ġthese Ġmethods , Ġwe Ġtrain Ġthat Ġmodel Ġon Ġall Ġof Ġthe Ġtraining Ġdata Ġavailable Ġbefore Ġthen Ġsubmitting Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġto Ġthe Ġleader board . ĠThese Ġresults Ġare Ġdetailed Ġin Ġthe Ġsection Ġdescribing Ġour Ġresults Ġ( Section ĠSEC REF 5 ). </s> Method ology Ġ: :: ĠClass ĠIm balance Ġin ĠSent ence ĠLevel ĠClassification </s> The Ġsentence Ġlevel Ġclassification Ġtask Ġis Ġan Ġim balanced Ġbinary Ġclassification Ġproblem Ġthat Ġwe Ġaddress Ġusing ĠB ERT ĠB IB REF 0 . ĠWe Ġuse ĠB ERT B ASE , Ġunc ased , Ġwhich Ġconsists Ġof Ġ12 Ġself - att ention Ġlayers , Ġand Ġreturns Ġa Ġ768 - dimension Ġvector Ġthat Ġrepresentation Ġa Ġsentence . ĠSo Ġas Ġto Ġmake Ġuse Ġof ĠB ERT Ġfor Ġsentence Ġclassification , Ġwe Ġinclude Ġa Ġfully Ġconnected Ġlayer Ġon Ġtop Ġof Ġthe ĠB ERT Ġself - att ention Ġlayers , Ġwhich Ġclass ifies Ġthe Ġsentence Ġembed ding Ġprovided Ġby ĠB ERT Ġinto Ġthe Ġtwo Ġclasses Ġof Ġinterest Ġ( prop aganda Ġor Ġnon - prop aganda ). </s> We Ġattempt Ġto Ġexploit Ġvarious Ġdata Ġaug mentation Ġtechniques Ġto Ġaddress Ġthe Ġproblem Ġof Ġclass Ġimbalance . ĠTable ĠT AB REF 17 Ġshows Ġthe Ġresults Ġof Ġour Ġexperiments Ġfor Ġdifferent Ġdata Ġaug mentation Ġtechniques Ġwhen , Ġafter Ġshuff ling Ġthe Ġtraining Ġdata , Ġwe Ġtrain Ġthe Ġmodel Ġon Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġand Ġtest Ġit Ġon Ġthe Ġremaining Ġ25 % Ġof Ġthe Ġtraining Ġdata Ġand Ġthe Ġdevelopment Ġdata . </s> We Ġobserve Ġthat ĠB ERT Ġwithout Ġaug mentation Ġconsistently Ġoutper forms ĠB ERT Ġwith Ġaug mentation Ġin Ġthe Ġexperiments Ġwhen Ġthe Ġmodel Ġis Ġtrained Ġon Ġ75 % Ġof Ġthe Ġtraining Ġdata Ġand Ġevaluated Ġon Ġthe Ġrest , Ġi . e Ġtrained Ġand Ġevaluated Ġon Ġsimilar Ġdata , Ġcoming Ġfrom Ġthe Ġsame Ġdistribution . ĠThis Ġis Ġconsistent Ġwith Ġobservations Ġby ĠWei Ġet Ġal . Ġwe i 2019 eda Ġthat Ġcontextual Ġword Ġembed d ings Ġdo Ġnot Ġgain Ġfrom Ġdata Ġaug mentation . ĠThe Ġfact Ġthat Ġwe Ġshuffle Ġthe Ġtraining Ġdata Ġprior Ġto Ġsplitting Ġit Ġinto Ġtraining Ġand Ġtesting Ġsubs ets Ġcould Ġimply Ġthat Ġthe Ġmodel Ġis Ġlearning Ġto Ġassociate Ġtopic Ġwords , Ġsuch Ġas Ġ` M ueller ', Ġas Ġpropaganda . ĠHowever , Ġwhen Ġwe Ġperform Ġmodel Ġevaluation Ġusing Ġthe Ġdevelopment Ġset , Ġwhich Ġis Ġdiss imilar Ġto Ġthe Ġtraining , Ġwe Ġobserve Ġthat Ġsyn onym Ġinsertion Ġand Ġword Ġdropping Ġtechniques Ġalso Ġdo Ġnot Ġbring Ġperformance Ġgains , Ġwhile Ġrandom Ġovers am pling Ġincreases Ġperformance Ġover Ġbase ĠB ERT Ġby Ġ4 %. ĠSyn onym Ġinsertion Ġprovides Ġresults Ġvery Ġsimilar Ġto Ġbase ĠB ERT , Ġwhile Ġrandom Ġdeletion Ġharms Ġmodel Ġperformance Ġproducing Ġlower Ġscores . ĠWe Ġbelieve Ġthat Ġthis Ġcould Ġbe Ġattributed Ġto Ġthe Ġfact Ġthat Ġsyn onym Ġinsertion Ġand Ġrandom Ġword Ġdropping Ġinvolve Ġthe Ġintroduction Ġof Ġnoise Ġto Ġthe Ġdata , Ġwhile Ġovers am pling Ġdoes Ġnot . ĠAs Ġwe Ġare Ġworking Ġwith Ġnatural Ġlanguage Ġdata , Ġthis Ġtype Ġof Ġnoise Ġcan Ġin Ġfact Ġchange Ġthe Ġmeaning Ġof Ġthe Ġsentence . ĠOvers am pling Ġon Ġthe Ġother Ġhand Ġpurely Ġincreases Ġthe Ġimportance Ġof Ġthe Ġminority Ġclass Ġby Ġrepeating Ġtraining Ġon Ġthe Ġunchanged Ġinstances . </s> So Ġas Ġto Ġbetter Ġunderstand Ġthe Ġaspects Ġof Ġovers am pling Ġthat Ġcontribute Ġto Ġthese Ġgains , Ġwe Ġperform Ġa Ġclass - wise Ġperformance Ġanalysis Ġof ĠB ERT Ġwith / without Ġovers am pling . ĠThe Ġresults Ġof Ġthese Ġexperiments Ġ( Table ĠT AB REF 18 ) Ġshow Ġthat Ġovers am pling Ġincreases Ġthe Ġoverall Ġrecall Ġwhile Ġmaintaining Ġprecision . ĠThis Ġis Ġachieved Ġby Ġsignificantly Ġimproving Ġthe Ġrecall Ġof Ġthe Ġminority Ġclass Ġ( prop aganda ) Ġat Ġthe Ġcost Ġof Ġthe Ġrecall Ġof Ġthe Ġmajority Ġclass . </s> So Ġfar Ġwe Ġhave Ġbeen Ġable Ġto Ġestablish Ġthat Ġa ) Ġthe Ġtraining Ġand Ġtest Ġsets Ġare Ġdiss imilar , Ġthus Ġrequiring Ġus Ġto Ġgeneral ise Ġour Ġmodel , Ġb ) Ġovers am pling Ġprovides Ġa Ġmethod Ġof Ġgeneral isation , Ġand Ġc ) Ġovers am pling Ġdoes Ġthis Ġwhile Ġmaintaining Ġrecall Ġon Ġthe Ġminority Ġ( and Ġthus Ġmore Ġinteresting ) Ġclass . </s> Given Ġthis Ġwe Ġexplore Ġalternative Ġmethods Ġof Ġincreasing Ġminority Ġclass Ġrecall Ġwithout Ġa Ġsignificant Ġdrop Ġin Ġprecision . ĠOne Ġsuch Ġmethod Ġis Ġcost - sensitive Ġclassification , Ġwhich Ġdiffers Ġfrom Ġrandom Ġovers am pling Ġin Ġthat Ġit Ġprovides Ġa Ġmore Ġcontinuous - valued Ġand Ġconsistent Ġmethod Ġof Ġweight ing Ġsamples Ġof Ġim balanced Ġtraining Ġdata ; Ġfor Ġexample , Ġrandom Ġovers am pling Ġwill Ġinevitably Ġemphas ise Ġsome Ġtraining Ġinstances Ġat Ġthe Ġexpense Ġof Ġothers . ĠWe Ġdetail Ġour Ġmethods Ġof Ġusing Ġcost - sensitive Ġclassification Ġin Ġthe Ġnext Ġsection . ĠFurther Ġexperiments Ġwith Ġovers am pling Ġmight Ġhave Ġprovided Ġinsights Ġinto Ġthe Ġrelationships Ġbetween Ġthese Ġmethods , Ġwhich Ġwe Ġleave Ġfor Ġfuture Ġexploration . </s> Method ology Ġ: :: ĠCost - sensitive ĠClassification </s> As Ġdiscussed Ġin ĠSection ĠSEC REF 10 , Ġcost - sensitive Ġclassification Ġcan Ġbe Ġperformed Ġby Ġweight ing Ġthe Ġcost Ġfunction . ĠWe Ġincrease Ġthe Ġweight Ġof Ġincorrectly Ġlab elling Ġa Ġpropag and istic Ġsentence Ġby Ġaltering Ġthe Ġcost Ġfunction Ġof Ġthe Ġtraining Ġof Ġthe Ġfinal Ġfully Ġconnected Ġlayer Ġof Ġour Ġmodel Ġpreviously Ġdescribed Ġin ĠSection ĠSEC REF 16 . ĠWe Ġmake Ġthese Ġchanges Ġthrough Ġthe Ġuse Ġof ĠPy Tor ch ĠB IB REF 16 Ġwhich Ġcalculates Ġthe Ġcross - ent ropy Ġloss Ġfor Ġa Ġsingle Ġprediction Ġ$ x $, Ġan Ġarray Ġwhere Ġthe Ġ$ j ^{ th } $ Ġelement Ġrepresents Ġthe Ġmodels Ġprediction Ġfor Ġclass Ġ$ j $, Ġlabelled Ġwith Ġthe Ġclass Ġ$ class $ Ġas Ġgiven Ġby ĠEqu ation ĠDIS PLAY _ FORM 20 . </s> The Ġcross - ent ropy Ġloss Ġgiven Ġin ĠEqu ation ĠDIS PLAY _ FORM 20 Ġis Ġmodified Ġto Ġaccommodate Ġan Ġarray Ġ$ weight $, Ġthe Ġ$ i ^{ th } $ Ġelement Ġof Ġwhich Ġrepresents Ġthe Ġweight Ġof Ġthe Ġ$ i ^{ th } $ Ġclass , Ġas Ġdescribed Ġin ĠEqu ation ĠDIS PLAY _ FORM 21 . </s> Int uitive ly , Ġwe Ġincrease Ġthe Ġcost Ġof Ġgetting Ġthe Ġclassification Ġof Ġan ĠâĢ ľ important âĢ Ŀ Ġclass Ġwrong Ġand Ġcorresponding Ġdecre es Ġthe Ġcost Ġof Ġgetting Ġa Ġless Ġimportant Ġclass Ġwrong . ĠIn Ġour Ġcase , Ġwe Ġincrease Ġthe Ġcost Ġof Ġmis lab elling Ġthe Ġminority Ġclass Ġwhich Ġis ĠâĢ ľ prop aganda âĢ Ŀ Ġ( as Ġopposed Ġto ĠâĢ ľ non - prop aganda âĢ Ŀ ). </s> We Ġexpect Ġthe Ġeffect Ġof Ġthis Ġto Ġbe Ġsimilar Ġto Ġthat Ġof Ġovers am pling , Ġin Ġthat Ġit Ġis Ġlikely Ġto Ġenable Ġus Ġto Ġincrease Ġthe Ġrecall Ġof Ġthe Ġminority Ġclass Ġthus Ġresulting Ġin Ġthe Ġdecrease Ġin Ġrecall Ġof Ġthe Ġoverall Ġmodel Ġwhile Ġmaintaining Ġhigh Ġprecision . ĠWe Ġreiterate Ġthat Ġthis Ġspecific Ġchange Ġto Ġa Ġmodel Ġresults Ġin Ġincreasing Ġthe Ġmodel 's Ġability Ġto Ġbetter Ġidentify Ġelements Ġbelonging Ġto Ġthe Ġminority Ġclass Ġin Ġdiss imilar Ġdatasets Ġwhen Ġusing ĠB ERT . </s> We Ġexplore Ġthe Ġvalidity Ġof Ġthis Ġby Ġperforming Ġseveral Ġexperiments Ġwith Ġdifferent Ġweights Ġassigned Ġto Ġthe Ġminority Ġclass . ĠWe Ġnote Ġthat Ġin Ġour Ġexperiments Ġuse Ġsignificantly Ġhigher Ġweights Ġthan Ġthe Ġweights Ġproportional Ġto Ġclass Ġfrequencies Ġin Ġthe Ġtraining Ġdata , Ġthat Ġare Ġcommon Ġin Ġliterature ĠB IB REF 17 . ĠRather Ġthan Ġdirectly Ġusing Ġthe Ġclass Ġproportions Ġof Ġthe Ġtraining Ġset , Ġwe Ġshow Ġthat Ġtuning Ġweights Ġbased Ġon Ġperformance Ġon Ġthe Ġdevelopment Ġset Ġis Ġmore Ġbeneficial . ĠFigure ĠFIG REF 22 Ġshows Ġthe Ġresults Ġof Ġthese Ġexperiments Ġwherein Ġwe Ġare Ġable Ġto Ġmaintain Ġthe Ġprecision Ġon Ġthe Ġsubset Ġof Ġthe Ġtraining Ġset Ġused Ġfor Ġtesting Ġwhile Ġreducing Ġits Ġrecall Ġand Ġthus Ġgeneral ising Ġthe Ġmodel . ĠThe Ġfact Ġthat Ġthe Ġmodel Ġis Ġgeneral ising Ġon Ġa Ġdiss imilar Ġdataset Ġis Ġconfirmed Ġby Ġthe Ġincrease Ġin Ġthe Ġdevelopment Ġset ĠF 1 Ġscore . ĠWe Ġnote Ġthat Ġthe Ġgains Ġare Ġnot Ġinfinite Ġand Ġthat Ġa Ġbalance Ġmust Ġbe Ġstruck Ġbased Ġon Ġthe Ġamount Ġof Ġgeneral isation Ġand Ġthe Ġcorresponding Ġloss Ġin Ġaccuracy . ĠThe Ġexact Ġweight Ġto Ġuse Ġfor Ġthe Ġbest Ġtransfer Ġof Ġclassification Ġaccuracy Ġis Ġrelated Ġto Ġthe Ġdiss imilar ity Ġof Ġthat Ġother Ġdataset Ġand Ġhence Ġis Ġto Ġbe Ġobtained Ġexperiment ally Ġthrough Ġhyper param eter Ġsearch . ĠOur Ġexperiments Ġshowed Ġthat Ġa Ġvalue Ġof Ġ4 Ġis Ġbest Ġsuited Ġfor Ġthis Ġtask . </s> We Ġdo Ġnot Ġinclude Ġthe Ġcomplete Ġresults Ġof Ġour Ġexperiments Ġhere Ġdue Ġto Ġspace Ġconstraints Ġbut Ġinclude Ġthem Ġalong Ġwith Ġcharts Ġand Ġprogram Ġcode Ġon Ġour Ġproject Ġwebsite . ĠBased Ġon Ġthis Ġexploration Ġwe Ġfind Ġthat Ġthe Ġbest Ġweights Ġfor Ġthis Ġparticular Ġdataset Ġare Ġ1 Ġfor Ġnon - prop aganda Ġand Ġ4 Ġfor Ġpropaganda Ġand Ġwe Ġuse Ġthis Ġto Ġtrain Ġthe Ġfinal Ġmodel Ġused Ġto Ġsubmit Ġresults Ġto Ġthe Ġleader board . ĠWe Ġalso Ġfound Ġthat Ġadding ĠPart Ġof ĠSpeech Ġtags Ġand ĠNamed ĠEntity Ġinformation Ġto ĠB ERT Ġembed d ings Ġby Ġconc aten ating Ġthese Ġone - hot Ġvectors Ġto Ġthe ĠB ERT Ġembed d ings Ġdoes Ġnot Ġimprove Ġmodel Ġperformance . ĠWe Ġdescribe Ġthese Ġresults Ġin ĠSection ĠSEC REF 5 . </s> Method ology Ġ: :: ĠFrag ment - level Ġclassification Ġ( FL C ) </s> In Ġaddition Ġto Ġparticipating Ġin Ġthe ĠSent ence ĠLevel ĠClassification Ġtask Ġwe Ġalso Ġparticipate Ġin Ġthe ĠFrag ment ĠLevel ĠClassification Ġtask . ĠWe Ġnote Ġthat Ġextracting Ġfragments Ġthat Ġare Ġpropag and istic Ġis Ġsimilar Ġto Ġthe Ġtask Ġof ĠNamed ĠEntity ĠRecogn ition , Ġin Ġthat Ġthey Ġare Ġboth Ġspan Ġextraction Ġtasks , Ġand Ġso Ġuse Ġa ĠB ERT Ġbased Ġmodel Ġdesigned Ġfor Ġthis Ġtask Ġ- ĠWe Ġbuild Ġon Ġthe Ġwork Ġby ĠB IB REF 18 Ġwhich Ġmakes Ġuse Ġof ĠContinuous ĠRandom ĠField Ġstacked Ġon Ġtop Ġof Ġan ĠL ST M Ġto Ġpredict Ġspans . ĠThis Ġarchitecture Ġis Ġstandard Ġamongst Ġstate Ġof Ġthe Ġart Ġmodels Ġthat Ġperform Ġspan Ġidentification . </s> While Ġthe Ġsame Ġspan Ġof Ġtext Ġcannot Ġhave Ġmultiple Ġnamed Ġentity Ġlabels , Ġit Ġcan Ġhave Ġdifferent Ġpropaganda Ġlabels . ĠWe Ġget Ġaround Ġthis Ġproblem Ġby Ġpicking Ġone Ġof Ġthe Ġlabels Ġat Ġrandom . ĠAdditionally , Ġso Ġas Ġto Ġspeed Ġup Ġtraining , Ġwe Ġonly Ġtrain Ġour Ġmodel Ġon Ġthose Ġsentences Ġthat Ġcontain Ġsome Ġpropag and istic Ġfragment . ĠIn Ġhindsight , Ġwe Ġnote Ġthat Ġboth Ġthese Ġdecisions Ġwere Ġnot Ġideal Ġand Ġdiscuss Ġwhat Ġwe Ġmight Ġhave Ġotherwise Ġdone Ġin ĠSection ĠSEC REF 7 . </s> Results </s> In Ġthis Ġsection , Ġwe Ġshow Ġour Ġrankings Ġon Ġthe Ġleader board Ġon Ġthe Ġtest Ġset . ĠUnlike Ġthe Ġprevious Ġexplor atory Ġsections , Ġin Ġwhich Ġwe Ġtrained Ġour Ġmodel Ġon Ġpart Ġof Ġthe Ġtraining Ġset , Ġwe Ġtrain Ġmodels Ġdescribed Ġin Ġthis Ġsection Ġon Ġthe Ġcomplete Ġtraining Ġset . </s> Results Ġ: :: ĠResults Ġon Ġthe ĠS LC Ġtask </s> Our Ġbest Ġperforming Ġmodel , Ġselected Ġon Ġthe Ġbasis Ġof Ġa Ġsystematic Ġanalysis Ġof Ġthe Ġrelationship Ġbetween Ġcost Ġweights Ġand Ġrecall , Ġplaces Ġus Ġsecond Ġamongst Ġthe Ġ25 Ġteams Ġthat Ġsubmitted Ġtheir Ġresults Ġon Ġthis Ġtask . ĠWe Ġpresent Ġour Ġscore Ġon Ġthe Ġtest Ġset Ġalongside Ġthose Ġof Ġcomparable Ġteams Ġin ĠTable ĠT AB REF 25 . ĠWe Ġnote Ġthat Ġthe Ġtask Ġdescription Ġpaper ĠB IB REF 4 Ġdescribes Ġa Ġmethod Ġof Ġachieving Ġan ĠF 1 Ġscore Ġof Ġ60 . 98 % Ġon Ġa Ġsimilar Ġtask Ġalthough Ġthis Ġreported Ġscore Ġis Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġresults Ġon Ġthis Ġtask Ġbecause Ġof Ġthe Ġdifferences Ġin Ġtesting Ġsets . </s> Results Ġ: :: ĠResults Ġon Ġthe ĠF LC Ġtask </s> We Ġtrain Ġthe Ġmodel Ġdescribed Ġin ĠSection ĠSEC REF 23 Ġon Ġthe Ġcomplete Ġtraining Ġset Ġbefore Ġsubmitting Ġto Ġthe Ġleader board . ĠOur Ġbest Ġperforming Ġmodel Ġwas Ġplaced Ġ7 th Ġamongst Ġthe Ġ13 Ġteams Ġthat Ġsubmitted Ġresults Ġfor Ġthis Ġtask . ĠWe Ġpresent Ġour Ġscore Ġon Ġthe Ġtest Ġset Ġalongside Ġthose Ġof Ġcomparable Ġteams Ġin ĠTable ĠT AB REF 27 . ĠWe Ġnote Ġthat Ġthe Ġtask Ġdescription Ġpaper ĠB IB REF 4 Ġdescribes Ġa Ġmethod Ġof Ġachieving Ġan ĠF 1 Ġscore Ġof Ġ22 . 58 % Ġon Ġa Ġsimilar Ġtask Ġalthough , Ġthis Ġreported Ġscore Ġis Ġnot Ġdirectly Ġcomparable Ġto Ġthe Ġresults Ġon Ġthis Ġtask . </s> One Ġof Ġthe Ġmajor Ġsetbacks Ġto Ġour Ġmethod Ġfor Ġidentifying Ġsentence Ġfragments Ġwas Ġthe Ġloss Ġof Ġtraining Ġdata Ġas Ġa Ġresult Ġof Ġrandomly Ġpicking Ġone Ġlabel Ġwhen Ġthe Ġsame Ġfragment Ġhad Ġmultiple Ġlabels . ĠThis Ġcould Ġhave Ġbeen Ġavoided Ġby Ġtraining Ġdifferent Ġmodels Ġfor Ġeach Ġlabel Ġand Ġsimply Ġconc aten ating Ġthe Ġresults . ĠAdditionally , Ġtraining Ġon Ġall Ġsentences , Ġincluding Ġthose Ġthat Ġdid Ġnot Ġcontain Ġany Ġfragments Ġlabelled Ġas Ġpropag and istic Ġwould Ġhave Ġlikely Ġimproved Ġour Ġmodel Ġperformance . ĠWe Ġintend Ġto Ġperform Ġthese Ġexperiments Ġas Ġpart Ġof Ġour Ġongoing Ġresearch . </s> Iss ues Ġof ĠDec on text ual ization Ġin ĠAutom ated ĠProp aganda ĠDetection </s> It Ġis Ġworth Ġreflecting Ġon Ġthe Ġnature Ġof Ġthe Ġshared Ġtask Ġdataset Ġ( P TC Ġcorpus ) Ġand Ġits Ġstructural Ġcorrespondence Ġ( or Ġlack Ġthereof ) Ġto Ġsome Ġof Ġthe Ġdefinitions Ġof Ġpropaganda Ġmentioned Ġin Ġthe Ġintroduction . ĠFirst , Ġpropaganda Ġis Ġa Ġsocial Ġphenomenon Ġand Ġtakes Ġplace Ġas Ġan Ġact Ġof Ġcommunication ĠB IB REF 19 , Ġand Ġso Ġit Ġis Ġmore Ġthan Ġa Ġsimple Ġinformation - the oret ic Ġmessage Ġof Ġz eros Ġand Ġones âĢĶ it Ġalso Ġincorporates Ġan Ġaddress er Ġand Ġadd res see ( s ), Ġeach Ġin Ġph atic Ġcontact Ġ( typically Ġvia Ġbroadcast Ġmedia ), Ġideally Ġwith Ġa Ġshared Ġden ot ational Ġcode Ġand Ġcontextual Ġsurround ( s ) ĠB IB REF 20 . </s> As Ġsuch , Ġa Ġdataset Ġof Ġdecon text ual ised Ġdocuments Ġwith Ġlabelled Ġsentences , Ġdevoid Ġof Ġauthor ial Ġor Ġpublisher Ġmetadata , Ġhas Ġtaken Ġus Ġat Ġsome Ġremove Ġfrom Ġeven Ġa Ġsimple Ġeveryday Ġdefinition Ġof Ġpropaganda . ĠOur Ġmodels Ġfor Ġthis Ġshared Ġtask Ġcannot Ġeasily Ġincorporate Ġinformation Ġabout Ġthe Ġaddress er Ġor Ġadd res see ; Ġare Ġleft Ġto Ġassume Ġa Ġshared Ġden ot ational Ġcode Ġbetween Ġauthor Ġand Ġreader Ġ( one Ġperhaps Ġsimulated Ġwith Ġthe Ġuse Ġof Ġpre - trained Ġword Ġembed d ings ); Ġand Ġthey Ġare Ġunaware Ġof Ġwhen Ġor Ġwhere Ġthe Ġact ( s ) Ġof Ġpropag and istic Ġcommunication Ġtook Ġplace . ĠThis Ġsl ipper iness Ġis Ġillustrated Ġin Ġour Ġexample Ġdocument Ġ( Fig . ĠFIG REF 13 ): Ġnote Ġthat Ġwhile ĠSent ences Ġ3 Ġand Ġ7 , Ġlabelled Ġas Ġpropaganda , Ġreflect Ġa Ġpropag and istic Ġattitude Ġon Ġthe Ġpart Ġof Ġthe Ġjournalist Ġand / or Ġpublisher , ĠSent ence Ġ4 âĢĶ also Ġlabelled Ġas Ġpropaganda Ġin Ġthe Ġtraining Ġdata âĢĶ instead Ġreflects Ġa ĠâĢ ľ flag - w aving " Ġpropag and istic Ġattitude Ġon Ġthe Ġpart Ġof ĠU . S . Ġcongressman ĠJeff ĠFlake , Ġvia Ġthe Ġconventions Ġof Ġreported Ġspeech ĠB IB REF 21 . ĠWhile Ġreported Ġspeech Ġoften Ġis Ġsignaled Ġby Ġspecific Ġmorph os ynt actic Ġpatterns Ġ( e . g . Ġthe Ġuse Ġof Ġdouble - qu otes Ġand ĠâĢ ľ Fl ake Ġsaid ") ĠB IB REF 22 , Ġwe Ġargue Ġthat Ġhuman Ġreaders Ġroutinely Ġdistinguish Ġpropag and istic Ġreport age Ġfrom Ġthe Ġpropag and astic Ġspeech Ġacts Ġof Ġits Ġsubjects , Ġand Ġto Ġconf late Ġthese Ġcategories Ġin Ġa Ġpropaganda Ġdetection Ġcorpus Ġmay Ġcontribute Ġto Ġthe Ġoccurrence Ġof Ġfalse Ġpositives / neg atives . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġwork Ġwe Ġhave Ġpresented Ġa Ġmethod Ġof Ġincorporating Ġcost - s ensitivity Ġinto ĠB ERT Ġto Ġallow Ġfor Ġbetter Ġgeneral isation Ġand Ġadditionally , Ġwe Ġprovide Ġa Ġsimple Ġmeasure Ġof Ġcorpus Ġsimilarity Ġto Ġdetermine Ġwhen Ġthis Ġmethod Ġis Ġlikely Ġto Ġbe Ġuseful . ĠWe Ġintend Ġto Ġextend Ġour Ġanalysis Ġof Ġthe Ġability Ġto Ġgeneral ise Ġmodels Ġto Ġless Ġsimilar Ġdata Ġby Ġexperimenting Ġon Ġother Ġdatasets Ġand Ġmodels . ĠWe Ġhope Ġthat Ġthe Ġrelease Ġof Ġprogram Ġcode Ġand Ġdocumentation Ġwill Ġallow Ġthe Ġresearch Ġcommunity Ġto Ġhelp Ġin Ġthis Ġexperimentation Ġwhile Ġexploiting Ġthese Ġmethods . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank ĠDr ĠLe andro ĠM ink u Ġfrom Ġthe ĠUniversity Ġof ĠBirmingham Ġfor Ġhis Ġinsights Ġinto Ġand Ġhelp Ġwith Ġthe Ġstatistical Ġanalysis Ġpresented Ġin Ġthis Ġpaper . </s> This Ġwork Ġwas Ġalso Ġpartially Ġsupported Ġby ĠThe ĠAlan ĠTuring ĠInstitute Ġunder Ġthe ĠEPS RC Ġgrant ĠEP / N 51 01 29 / 1 . ĠWork Ġby ĠElena ĠKoch k ina Ġwas Ġpartially Ġsupported Ġby Ġthe ĠLever h ul me ĠTrust Ġthrough Ġthe ĠBridges ĠProgramme Ġand ĠWarwick ĠCD T Ġfor ĠUrban ĠScience Ġ& ĠProgress Ġunder Ġthe ĠEPS RC ĠGrant ĠNumber ĠEP / L 01 64 00 / 1 .