 news articles in free-text format
EXTRACTIVE
collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators
EXTRACTIVE
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators. More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles. We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators.
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators. More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles. We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators. The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively. Figure FIGREF15 shows an annotated example, which contains several propaganda techniques. For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling. Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language.
1910.09982
Introduction </s> Prop aganda Ġaims Ġat Ġinfluencing Ġpeople 's Ġmindset Ġwith Ġthe Ġpurpose Ġof Ġadvancing Ġa Ġspecific Ġagenda . ĠIn Ġthe ĠInternet Ġera , Ġthanks Ġto Ġthe Ġmechanism Ġof Ġsharing Ġin Ġsocial Ġnetworks , Ġpropaganda Ġcampaigns Ġhave Ġthe Ġpotential Ġof Ġreaching Ġvery Ġlarge Ġaudiences ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . </s> Prop ag and ist Ġnews Ġarticles Ġuse Ġspecific Ġtechniques Ġto Ġconvey Ġtheir Ġmessage , Ġsuch Ġas Ġwhat about ism , Ġred ĠHer ring , Ġand Ġname Ġcalling , Ġamong Ġmany Ġothers Ġ( cf . ĠSection ĠSEC REF 3 ). ĠWhereas Ġproving Ġintent Ġis Ġnot Ġeasy , Ġwe Ġcan Ġanalyse Ġthe Ġlanguage Ġof Ġa Ġclaim / article Ġand Ġlook Ġfor Ġthe Ġuse Ġof Ġspecific Ġpropaganda Ġtechniques . ĠGoing Ġat Ġthis Ġfine - gr ained Ġlevel Ġcan Ġyield Ġmore Ġreliable Ġsystems Ġand Ġit Ġalso Ġmakes Ġit Ġpossible Ġto Ġexplain Ġto Ġthe Ġuser Ġwhy Ġan Ġarticle Ġwas Ġjudged Ġas Ġpropag and ist Ġby Ġan Ġautomatic Ġsystem . </s> With Ġthis Ġin Ġmind , Ġwe Ġorganised Ġthe Ġshared Ġtask Ġon Ġfine - gr ained Ġpropaganda Ġdetection Ġat Ġthe ĠN LP 4 IF @ EM N LP - IJ CN LP Ġ2019 Ġworkshop . ĠThe Ġtask Ġis Ġbased Ġon Ġa Ġcorpus Ġof Ġnews Ġarticles Ġannot ated Ġwith Ġan Ġinventory Ġof Ġ18 Ġpropag and ist Ġtechniques Ġat Ġthe Ġfragment Ġlevel . ĠWe Ġhope Ġthat Ġthe Ġcorpus Ġwould Ġraise Ġinterest Ġoutside Ġof Ġthe Ġcommunity Ġof Ġresearchers Ġstudying Ġpropaganda . ĠFor Ġexample , Ġthe Ġtechniques Ġrelated Ġto Ġfall acies Ġand Ġthe Ġones Ġrelying Ġon Ġemotions Ġmight Ġprovide Ġa Ġnovel Ġsetting Ġfor Ġresearchers Ġinterested Ġin ĠArgument ation Ġand ĠSent iment ĠAnalysis . </s> Related ĠWork </s> Prop aganda Ġhas Ġbeen Ġtackled Ġmostly Ġat Ġthe Ġarticle Ġlevel . ĠB IB REF 3 Ġcreated Ġa Ġcorpus Ġof Ġnews Ġarticles Ġlabelled Ġas Ġpropaganda , Ġtrusted , Ġhoax , Ġor Ġsatire . ĠB IB REF 4 Ġexperimented Ġwith Ġa Ġbin ar ized Ġversion Ġof Ġthat Ġcorpus : Ġpropaganda Ġvs . Ġthe Ġother Ġthree Ġcategories . ĠB IB REF 5 Ġannot ated Ġa Ġlarge Ġbinary Ġcorpus Ġof Ġpropag and ist Ġvs . Ġnon - prop ag and ist Ġarticles Ġand Ġproposed Ġa Ġfeature - based Ġsystem Ġfor Ġdiscriminating Ġbetween Ġthem . ĠIn Ġall Ġthese Ġcases , Ġthe Ġlabels Ġwere Ġobtained Ġusing Ġdistant Ġsupervision , Ġassuming Ġthat Ġall Ġarticles Ġfrom Ġa Ġgiven Ġnews Ġoutlet Ġshare Ġthe Ġlabel Ġof Ġthat Ġoutlet , Ġwhich Ġinevitably Ġintroduces Ġnoise ĠB IB REF 6 . </s> A Ġrelated Ġfield Ġis Ġthat Ġof Ġcomputational Ġargument ation Ġwhich , Ġamong Ġothers , Ġdeals Ġwith Ġsome Ġlogical Ġfall acies Ġrelated Ġto Ġpropaganda . ĠB IB REF 7 Ġpresented Ġa Ġcorpus Ġof ĠWeb Ġforum Ġdiscussions Ġwith Ġinstances Ġof Ġad Ġhom inem Ġfallacy . ĠB IB REF 8 , ĠB IB REF 9 Ġintroduced ĠArg ot ario , Ġa Ġgame Ġto Ġeducate Ġpeople Ġto Ġrecognize Ġand Ġcreate Ġfall acies , Ġa Ġby - product Ġof Ġwhich Ġis Ġa Ġcorpus Ġwith Ġ$ 1 . 3 k $ Ġarguments Ġannot ated Ġwith Ġfive Ġfall acies Ġsuch Ġas Ġad Ġhom inem , Ġred Ġher ring Ġand Ġirrelevant Ġauthority , Ġwhich Ġdirectly Ġrelate Ġto Ġpropaganda . </s> Unlike ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 7 , Ġour Ġcorpus Ġuses Ġ18 Ġtechniques Ġannot ated Ġon Ġthe Ġsame Ġset Ġof Ġnews Ġarticles . ĠMoreover , Ġour Ġannotations Ġaim Ġat Ġidentifying Ġthe Ġminimal Ġfragments Ġrelated Ġto Ġa Ġtechnique Ġinstead Ġof Ġflag ging Ġentire Ġarguments . </s> The Ġmost Ġrelevant Ġrelated Ġwork Ġis Ġour Ġown , Ġwhich Ġis Ġpublished Ġin Ġparallel Ġto Ġthis Ġpaper Ġat ĠEM N LP - IJ CN LP Ġ2019 ĠB IB REF 10 Ġand Ġdescribes Ġa Ġcorpus Ġthat Ġis Ġa Ġsubset Ġof Ġthe Ġone Ġused Ġfor Ġthis Ġshared Ġtask . </s> Prop aganda ĠTechniques </s> Prop aganda Ġuses Ġpsychological Ġand Ġrhetorical Ġtechniques Ġto Ġachieve Ġits Ġobjective . ĠSuch Ġtechniques Ġinclude Ġthe Ġuse Ġof Ġlogical Ġfall acies Ġand Ġappeal Ġto Ġemotions . ĠFor Ġthe Ġshared Ġtask , Ġwe Ġuse Ġ18 Ġtechniques Ġthat Ġcan Ġbe Ġfound Ġin Ġnews Ġarticles Ġand Ġcan Ġbe Ġjudged Ġintrinsically , Ġwithout Ġthe Ġneed Ġto Ġretrieve Ġsupporting Ġinformation Ġfrom Ġexternal Ġresources . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 10 Ġfor Ġmore Ġdetails Ġon Ġthe Ġpropaganda Ġtechniques ; Ġbelow Ġwe Ġreport Ġthe Ġlist Ġof Ġtechniques : </s> Prop aganda ĠTechniques Ġ: :: Ġ1 . ĠLoaded Ġlanguage . </s> Using Ġwords / ph r ases Ġwith Ġstrong Ġemotional Ġimplications Ġ( positive Ġor Ġnegative ) Ġto Ġinfluence Ġan Ġaudience ĠB IB REF 11 . </s> Prop aganda ĠTechniques Ġ: :: Ġ2 . ĠName Ġcalling Ġor Ġlabeling . </s> Label ing Ġthe Ġobject Ġof Ġthe Ġpropaganda Ġas Ġsomething Ġthe Ġtarget Ġaudience Ġfears , Ġhates , Ġfinds Ġundesirable Ġor Ġotherwise Ġloves Ġor Ġpraises ĠB IB REF 12 . </s> Prop aganda ĠTechniques Ġ: :: Ġ3 . ĠRep etition . </s> Repe ating Ġthe Ġsame Ġmessage Ġover Ġand Ġover Ġagain , Ġso Ġthat Ġthe Ġaudience Ġwill Ġeventually Ġaccept Ġit ĠB IB REF 13 , ĠB IB REF 12 . </s> Prop aganda ĠTechniques Ġ: :: Ġ4 . ĠEx agger ation Ġor Ġminim ization . </s> Either Ġrepresenting Ġsomething Ġin Ġan Ġexcessive Ġmanner : Ġmaking Ġthings Ġlarger , Ġbetter , Ġworse , Ġor Ġmaking Ġsomething Ġseem Ġless Ġimportant Ġor Ġsmaller Ġthan Ġit Ġactually Ġis ĠB IB REF 14 , Ġe . g ., Ġsaying Ġthat Ġan Ġinsult Ġwas Ġjust Ġa Ġjoke . </s> Prop aganda ĠTechniques Ġ: :: Ġ5 . ĠDou bt . </s> Question ing Ġthe Ġcredibility Ġof Ġsomeone Ġor Ġsomething . </s> Prop aganda ĠTechniques Ġ: :: Ġ6 . ĠAppeal Ġto Ġfear / pre jud ice . </s> Se eking Ġto Ġbuild Ġsupport Ġfor Ġan Ġidea Ġby Ġinst illing Ġanxiety Ġand / or Ġpanic Ġin Ġthe Ġpopulation Ġtowards Ġan Ġalternative , Ġpossibly Ġbased Ġon Ġprecon ceived Ġjudgments . </s> Prop aganda ĠTechniques Ġ: :: Ġ7 . ĠFlag - w aving . </s> Playing Ġon Ġstrong Ġnational Ġfeeling Ġ( or Ġwith Ġrespect Ġto Ġa Ġgroup , Ġe . g ., Ġrace , Ġgender , Ġpolitical Ġpreference ) Ġto Ġjustify Ġor Ġpromote Ġan Ġaction Ġor Ġidea ĠB IB REF 15 . </s> Prop aganda ĠTechniques Ġ: :: Ġ8 . ĠCa usal Ġovers impl ification . </s> Assuming Ġone Ġcause Ġwhen Ġthere Ġare Ġmultiple Ġcauses Ġbehind Ġan Ġissue . ĠWe Ġinclude Ġscapego ating Ġas Ġwell : Ġthe Ġtransfer Ġof Ġthe Ġblame Ġto Ġone Ġperson Ġor Ġgroup Ġof Ġpeople Ġwithout Ġinvestigating Ġthe Ġcomplexities Ġof Ġan Ġissue . </s> Prop aganda ĠTechniques Ġ: :: Ġ9 . ĠSl og ans . </s> A Ġbrief Ġand Ġstriking Ġphrase Ġthat Ġmay Ġinclude Ġlabeling Ġand Ġstereotyp ing . ĠSl og ans Ġtend Ġto Ġact Ġas Ġemotional Ġappeals ĠB IB REF 16 . </s> Prop aganda ĠTechniques Ġ: :: Ġ10 . ĠAppeal Ġto Ġauthority . </s> St ating Ġthat Ġa Ġclaim Ġis Ġtrue Ġsimply Ġbecause Ġa Ġvalid Ġauthority / ex pert Ġon Ġthe Ġissue Ġsupports Ġit , Ġwithout Ġany Ġother Ġsupporting Ġevidence ĠB IB REF 17 . ĠWe Ġinclude Ġthe Ġspecial Ġcase Ġwhere Ġthe Ġreference Ġis Ġnot Ġan Ġauthority / ex pert , Ġalthough Ġit Ġis Ġreferred Ġto Ġas Ġtestim onial Ġin Ġthe Ġliterature ĠB IB REF 14 . </s> Prop aganda ĠTechniques Ġ: :: Ġ11 . ĠBlack - and - white Ġfallacy , Ġdictatorship . </s> Present ing Ġtwo Ġalternative Ġoptions Ġas Ġthe Ġonly Ġpossibilities , Ġwhen Ġin Ġfact Ġmore Ġpossibilities Ġexist ĠB IB REF 13 . ĠAs Ġan Ġextreme Ġcase , Ġtelling Ġthe Ġaudience Ġexactly Ġwhat Ġactions Ġto Ġtake , Ġeliminating Ġany Ġother Ġpossible Ġchoice Ġ( dict ators hip ). </s> Prop aganda ĠTechniques Ġ: :: Ġ12 . ĠThought - termin ating ĠclichÃ© . </s> Words Ġor Ġphrases Ġthat Ġdiscourage Ġcritical Ġthought Ġand Ġmeaningful Ġdiscussion Ġabout Ġa Ġgiven Ġtopic . ĠThey Ġare Ġtypically Ġshort Ġand Ġgeneric Ġsentences Ġthat Ġoffer Ġseemingly Ġsimple Ġanswers Ġto Ġcomplex Ġquestions Ġor Ġthat Ġdistract Ġattention Ġaway Ġfrom Ġother Ġlines Ġof Ġthought ĠB IB REF 18 . </s> Prop aganda ĠTechniques Ġ: :: Ġ13 . ĠWhat about ism . </s> Disc redit Ġan Ġopponent 's Ġposition Ġby Ġcharging Ġthem Ġwith Ġhypocrisy Ġwithout Ġdirectly Ġdispro ving Ġtheir Ġargument ĠB IB REF 19 . </s> Prop aganda ĠTechniques Ġ: :: Ġ14 . ĠRed uct io Ġad ĠHitler um . </s> Pers u ading Ġan Ġaudience Ġto Ġdisapprove Ġan Ġaction Ġor Ġidea Ġby Ġsuggesting Ġthat Ġthe Ġidea Ġis Ġpopular Ġwith Ġgroups Ġhated Ġin Ġcontempt Ġby Ġthe Ġtarget Ġaudience . ĠIt Ġcan Ġrefer Ġto Ġany Ġperson Ġor Ġconcept Ġwith Ġa Ġnegative Ġcon notation ĠB IB REF 20 . </s> Prop aganda ĠTechniques Ġ: :: Ġ15 . ĠRed Ġher ring . </s> Introdu cing Ġirrelevant Ġmaterial Ġto Ġthe Ġissue Ġbeing Ġdiscussed , Ġso Ġthat Ġeveryone 's Ġattention Ġis Ġdiverted Ġaway Ġfrom Ġthe Ġpoints Ġmade ĠB IB REF 11 . ĠThose Ġsubjected Ġto Ġa Ġred Ġher ring Ġargument Ġare Ġled Ġaway Ġfrom Ġthe Ġissue Ġthat Ġhad Ġbeen Ġthe Ġfocus Ġof Ġthe Ġdiscussion Ġand Ġurged Ġto Ġfollow Ġan Ġobservation Ġor Ġclaim Ġthat Ġmay Ġbe Ġassociated Ġwith Ġthe Ġoriginal Ġclaim , Ġbut Ġis Ġnot Ġhighly Ġrelevant Ġto Ġthe Ġissue Ġin Ġdispute ĠB IB REF 20 . </s> Prop aganda ĠTechniques Ġ: :: Ġ16 . ĠBand wagon . </s> Attempt ing Ġto Ġpersuade Ġthe Ġtarget Ġaudience Ġto Ġjoin Ġin Ġand Ġtake Ġthe Ġcourse Ġof Ġaction Ġbecause ĠâĢ ľ everyone Ġelse Ġis Ġtaking Ġthe Ġsame Ġaction âĢ Ŀ ĠB IB REF 15 . </s> Prop aganda ĠTechniques Ġ: :: Ġ17 . ĠOb fusc ation , Ġintentional Ġv agu eness , Ġconfusion . </s> Using Ġdeliberately Ġunclear Ġwords , Ġto Ġlet Ġthe Ġaudience Ġhave Ġits Ġown Ġinterpretation ĠB IB REF 21 , ĠB IB REF 11 . ĠFor Ġinstance , Ġwhen Ġan Ġunclear Ġphrase Ġwith Ġmultiple Ġpossible Ġmeanings Ġis Ġused Ġwithin Ġthe Ġargument Ġand , Ġtherefore , Ġit Ġdoes Ġnot Ġreally Ġsupport Ġthe Ġconclusion . </s> Prop aganda ĠTechniques Ġ: :: Ġ18 . ĠStraw Ġman . </s> When Ġan Ġopponent 's Ġproposition Ġis Ġsubstituted Ġwith Ġa Ġsimilar Ġone Ġwhich Ġis Ġthen Ġrefuted Ġin Ġplace Ġof Ġthe Ġoriginal ĠB IB REF 22 . </s> T asks </s> The Ġshared Ġtask Ġfeatures Ġtwo Ġsubt asks : </s> T asks Ġ: :: ĠFrag ment - Level ĠClassification Ġtask Ġ( FL C ). </s> Given Ġa Ġnews Ġarticle , Ġdetect Ġall Ġspans Ġof Ġthe Ġtext Ġin Ġwhich Ġa Ġpropaganda Ġtechnique Ġis Ġused . ĠIn Ġaddition , Ġfor Ġeach Ġspan Ġthe Ġpropaganda Ġtechnique Ġapplied Ġmust Ġbe Ġidentified . </s> T asks Ġ: :: ĠSent ence - Level ĠClassification Ġtask Ġ( S LC ). </s> A Ġsentence Ġis Ġconsidered Ġpropag and ist Ġif Ġit Ġcontains Ġat Ġleast Ġone Ġpropag and ist Ġfragment . ĠWe Ġthen Ġdefine Ġa Ġbinary Ġclassification Ġtask Ġin Ġwhich , Ġgiven Ġa Ġsentence , Ġthe Ġcorrect Ġlabel , Ġeither Ġpropaganda Ġor Ġnon - prop aganda , Ġis Ġto Ġbe Ġpredicted . </s> Data </s> The Ġinput Ġfor Ġboth Ġtasks Ġconsists Ġof Ġnews Ġarticles Ġin Ġfree - text Ġformat , Ġcollected Ġfrom Ġ36 Ġpropag and ist Ġand Ġ12 Ġnon - prop ag and ist Ġnews Ġoutlets Ġand Ġthen Ġannot ated Ġby Ġprofessional Ġannot ators . ĠMore Ġdetails Ġabout Ġthe Ġdata Ġcollection Ġand Ġthe Ġannotation , Ġas Ġwell Ġas Ġstatistics Ġabout Ġthe Ġcorpus Ġcan Ġbe Ġfound Ġin ĠB IB REF 10 , Ġwhere Ġan Ġearlier Ġversion Ġof Ġthe Ġcorpus Ġis Ġdescribed , Ġwhich Ġincludes Ġ450 Ġnews Ġarticles . ĠWe Ġfurther Ġannot ated Ġ47 Ġadditional Ġarticles Ġfor Ġthe Ġpurpose Ġof Ġthe Ġshared Ġtask Ġusing Ġthe Ġsame Ġprotocol Ġand Ġthe Ġsame Ġannot ators . </s> The Ġtraining , Ġthe Ġdevelopment , Ġand Ġthe Ġtest Ġpartitions Ġof Ġthe Ġcorpus Ġused Ġfor Ġthe Ġshared Ġtask Ġconsist Ġof Ġ350 , Ġ61 , Ġand Ġ86 Ġarticles Ġand Ġof Ġ16 , 9 65 , Ġ2 , 235 , Ġand Ġ3 , 526 Ġsentences , Ġrespectively . ĠFigure ĠFIG REF 15 Ġshows Ġan Ġannot ated Ġexample , Ġwhich Ġcontains Ġseveral Ġpropaganda Ġtechniques . ĠFor Ġexample , Ġthe Ġfragment Ġbabies Ġon Ġline Ġ1 Ġis Ġan Ġinstance Ġof Ġboth ĠName _ Calling Ġand ĠLab eling . ĠNote Ġthat Ġthe Ġfragment Ġnot Ġlooking Ġas Ġthough ĠTrump Ġkilled Ġhis Ġgrandma Ġon Ġline Ġ4 Ġis Ġan Ġinstance Ġof ĠEx agger ation _ or _ Min im isation Ġand Ġit Ġoverl aps Ġwith Ġthe Ġfragment Ġkilled Ġhis Ġgrandma , Ġwhich Ġis Ġan Ġinstance Ġof ĠLoaded _ Language . </s> Table ĠT AB REF 23 Ġreports Ġthe Ġtotal Ġnumber Ġof Ġinstances Ġper Ġtechnique Ġand Ġthe Ġpercentage Ġwith Ġrespect Ġto Ġthe Ġtotal Ġnumber Ġof Ġannotations , Ġfor Ġthe Ġtraining Ġand Ġfor Ġthe Ġdevelopment Ġsets . </s> Setup </s> The Ġshared Ġtask Ġhad Ġtwo Ġphases : ĠIn Ġthe Ġdevelopment Ġphase , Ġthe Ġparticipants Ġwere Ġprovided Ġlabeled Ġtraining Ġand Ġdevelopment Ġdatasets ; Ġin Ġthe Ġtesting Ġphase , Ġtesting Ġinput Ġwas Ġfurther Ġprovided . </s> The Ġparticipants Ġtried Ġto Ġachieve Ġthe Ġbest Ġperformance Ġon Ġthe Ġdevelopment Ġset . ĠA Ġlive Ġleader board Ġkept Ġtrack Ġof Ġthe Ġsubmissions . </s> The Ġtest Ġset Ġwas Ġreleased Ġand Ġthe Ġparticipants Ġhad Ġfew Ġdays Ġto Ġmake Ġfinal Ġpredictions . </s> In Ġphase Ġ2 , Ġno Ġimmediate Ġfeedback Ġon Ġthe Ġsubmissions Ġwas Ġprovided . ĠThe Ġwinner Ġwas Ġdetermined Ġbased Ġon Ġthe Ġperformance Ġon Ġthe Ġtest Ġset . </s> E val uation Ġ: :: ĠF LC Ġtask . </s> FL C Ġis Ġa Ġcomposition Ġof Ġtwo Ġsubt asks : Ġthe Ġidentification Ġof Ġthe Ġpropag and ist Ġtext Ġfragments Ġand Ġthe Ġidentification Ġof Ġthe Ġtechniques Ġused Ġ( 18 - way Ġclassification Ġtask ). ĠWhile ĠF $ _ 1 $ Ġmeasure Ġis Ġappropriate Ġfor Ġa Ġmulti - class Ġclassification Ġtask , Ġwe Ġmodified Ġit Ġto Ġaccount Ġfor Ġpartial Ġmatching Ġbetween Ġthe Ġspans ; Ġsee ĠB IB REF 10 Ġfor Ġmore Ġdetails . ĠWe Ġfurther Ġcomputed Ġan ĠF $ _ 1 $ Ġvalue Ġfor Ġeach Ġpropaganda Ġtechnique Ġ( not Ġshown Ġbelow Ġfor Ġthe Ġsake Ġof Ġsaving Ġspace , Ġbut Ġavailable Ġon Ġthe Ġleader board ). </s> E val uation Ġ: :: ĠS LC Ġtask . </s> S LC Ġis Ġa Ġbinary Ġclassification Ġtask Ġwith Ġim balanced Ġdata . ĠTherefore , Ġthe Ġofficial Ġevaluation Ġmeasure Ġfor Ġthe Ġtask Ġis Ġthe Ġstandard ĠF $ _ 1 $ Ġmeasure . ĠWe Ġfurther Ġreport ĠPrecision Ġand ĠRecall . </s> Bas elines </s> The Ġbaseline Ġsystem Ġfor Ġthe ĠS LC Ġtask Ġis Ġa Ġvery Ġsimple Ġlog istic Ġregression Ġclass ifier Ġwith Ġdefault Ġparameters , Ġwhere Ġwe Ġrepresent Ġthe Ġinput Ġinstances Ġwith Ġa Ġsingle Ġfeature : Ġthe Ġlength Ġof Ġthe Ġsentence . ĠThe Ġperformance Ġof Ġthis Ġbaseline Ġon Ġthe ĠS LC Ġtask Ġis Ġshown Ġin ĠTables ĠT AB REF 33 Ġand ĠT AB REF 34 . </s> The Ġbaseline Ġfor Ġthe ĠF LC Ġtask Ġgenerates Ġspans Ġand Ġselects Ġone Ġof Ġthe Ġ18 Ġtechniques Ġrandomly . ĠThe Ġin effic acy Ġof Ġsuch Ġa Ġsimple Ġrandom Ġbaseline Ġis Ġillustrated Ġin ĠTables ĠT AB REF 36 Ġand ĠT AB REF 41 . </s> Particip ants Ġand ĠAppro aches </s> A Ġtotal Ġof Ġ90 Ġteams Ġregistered Ġfor Ġthe Ġshared Ġtask , Ġand Ġ39 Ġof Ġthem Ġsubmitted Ġpredictions Ġfor Ġa Ġtotal Ġof Ġ3 , 0 65 Ġsubmissions . ĠFor Ġthe ĠF LC Ġtask , Ġ21 Ġteams Ġmade Ġa Ġtotal Ġof Ġ5 27 Ġsubmissions , Ġand Ġfor Ġthe ĠS LC Ġtask , Ġ35 Ġteams Ġmade Ġa Ġtotal Ġof Ġ2 , 538 Ġsubmissions . </s> Below , Ġwe Ġgive Ġan Ġoverview Ġof Ġthe Ġapproaches Ġas Ġdescribed Ġin Ġthe Ġparticipants ' Ġpapers . ĠTables ĠT AB REF 28 Ġand ĠT AB REF 29 Ġoffer Ġa Ġhigh - level Ġsummary . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin Ġthe ĠFrag ment - Level ĠClassification ĠOnly </s> Team Ġnews peak ĠB IB REF 23 Ġachieved Ġthe Ġbest Ġresults Ġon Ġthe Ġtest Ġset Ġfor Ġthe ĠF LC Ġtask Ġusing Ġ20 - way Ġword - level Ġclassification Ġbased Ġon ĠB ERT ĠB IB REF 24 : Ġa Ġword Ġcould Ġbelong Ġto Ġone Ġof Ġthe Ġ18 Ġpropaganda Ġtechniques , Ġto Ġnone Ġof Ġthem , Ġor Ġto Ġan Ġauxiliary Ġ( token - derived ) Ġclass . ĠThe Ġteam Ġfed Ġone Ġsentence Ġat Ġa Ġtime Ġin Ġorder Ġto Ġreduce Ġthe Ġworkload . ĠIn Ġaddition Ġto Ġexperimenting Ġwith Ġan Ġout - of - the - box ĠB ERT , Ġthey Ġalso Ġtried Ġun super vised Ġfine - tun ing Ġboth Ġon Ġthe Ġ1 M Ġnews Ġdataset Ġand Ġon ĠWikipedia . ĠTheir Ġbest Ġmodel Ġwas Ġbased Ġon Ġthe Ġunc ased Ġbase Ġmodel Ġof ĠB ERT , Ġwith Ġ12 ĠTrans former Ġlayers ĠB IB REF 25 , Ġand Ġ110 Ġmillion Ġparameters . ĠMoreover , Ġovers am pling Ġof Ġthe Ġleast Ġrepresented Ġclasses Ġproved Ġto Ġbe Ġcrucial Ġfor Ġthe Ġfinal Ġperformance . ĠFinally , Ġcareful Ġanalysis Ġhas Ġshown Ġthat Ġthe Ġmodel Ġpays Ġspecial Ġattention Ġto Ġadject ives Ġand Ġad verbs . </s> Team ĠStalin ĠB IB REF 26 Ġfocused Ġon Ġdata Ġaug mentation Ġto Ġaddress Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġthe Ġdata Ġfor Ġfine - tun ing Ġcontextual Ġembed ding Ġrepresentations Ġbased Ġon ĠEL Mo ĠB IB REF 27 , ĠB ERT , Ġand ĠGro ver ĠB IB REF 28 . ĠThe Ġbalancing Ġof Ġthe Ġembed ding Ġspace Ġwas Ġcarried Ġout Ġby Ġmeans Ġof Ġsynthetic Ġminority Ġclass Ġover - sam pling . ĠThen , Ġthe Ġlearned Ġrepresentations Ġwere Ġfed Ġinto Ġan ĠL ST M . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin Ġthe ĠSent ence - Level ĠClassification ĠOnly </s> Team ĠCA Un LP ĠB IB REF 29 Ġused Ġtwo Ġcontext - aware Ġrepresentations Ġbased Ġon ĠB ERT . ĠIn Ġthe Ġfirst Ġrepresentation , Ġthe Ġtarget Ġsentence Ġis Ġfollowed Ġby Ġthe Ġtitle Ġof Ġthe Ġarticle . ĠIn Ġthe Ġsecond Ġrepresentation , Ġthe Ġprevious Ġsentence Ġis Ġalso Ġadded . ĠThey Ġperformed Ġsubs am pling Ġin Ġorder Ġto Ġdeal Ġwith Ġclass Ġimbalance , Ġand Ġexperimented Ġwith ĠB ERT $ _{ B ASE } $ Ġand ĠB ERT $ _{ L AR GE } $ </s> Team ĠLI ACC ĠB IB REF 30 Ġused Ġhand - crafted Ġfeatures Ġand Ġpre - trained ĠEL Mo Ġembed d ings . ĠThey Ġalso Ġobserved Ġa Ġboost Ġin Ġperformance Ġwhen Ġbalancing Ġthe Ġdataset Ġby Ġdropping Ġsome Ġnegative Ġexamples . </s> Team ĠJUST Deep ĠB IB REF 31 Ġused Ġa Ġcombination Ġof Ġmodels Ġand Ġfeatures , Ġincluding Ġword Ġembed d ings Ġbased Ġon ĠGl o Ve ĠB IB REF 32 Ġconc aten ated Ġwith Ġvectors Ġrepresenting Ġaffection Ġand Ġlex ical Ġfeatures . ĠThese Ġwere Ġcombined Ġin Ġan Ġensemble Ġof Ġsupervised Ġmodels : Ġbi - L ST M , ĠX GB o ost , Ġand Ġvariations Ġof ĠB ERT . </s> Team ĠY M JA ĠB IB REF 33 Ġalso Ġbased Ġtheir Ġapproach Ġon Ġfine - tun ed ĠB ERT . ĠInspired Ġby Ġk agg le Ġcompetitions Ġon Ġsentiment Ġanalysis , Ġthey Ġcreated Ġan Ġensemble Ġof Ġmodels Ġvia Ġcross - valid ation . </s> Team Ġj inf en ĠB IB REF 34 Ġused Ġa Ġlog istic Ġregression Ġmodel Ġfed Ġwith Ġa Ġmanifold Ġof Ġrepresentations , Ġincluding ĠTF . ID F Ġand ĠB ERT Ġvectors , Ġas Ġwell Ġas Ġvoc ab ular ies Ġand Ġread ability Ġmeasures . </s> Team ĠTh a 3 ar oon ĠB IB REF 35 Ġimplemented Ġan Ġensemble Ġof Ġthree Ġclass ifiers : Ġtwo Ġbased Ġon ĠB ERT Ġand Ġone Ġbased Ġon Ġa Ġuniversal Ġsentence Ġenc oder ĠB IB REF 36 . </s> Team ĠNS IT ĠB IB REF 37 Ġexplored Ġthree Ġof Ġthe Ġmost Ġpopular Ġtransfer Ġlearning Ġmodels : Ġvarious Ġversions Ġof ĠEL Mo , ĠB ERT , Ġand ĠRo BER Ta ĠB IB REF 38 . </s> Team ĠMind cod ers ĠB IB REF 39 Ġcombined ĠB ERT , ĠBi - L ST M Ġand ĠCaps ule Ġnetworks ĠB IB REF 40 Ġinto Ġa Ġsingle Ġdeep Ġneural Ġnetwork Ġand Ġpre - trained Ġthe Ġresulting Ġnetwork Ġon Ġcorpor a Ġused Ġfor Ġrelated Ġtasks , Ġe . g ., Ġemotion Ġclassification . </s> Finally , Ġteam Ġl tu orp ĠB IB REF 41 Ġused Ġan Ġattention Ġtransformer Ġusing ĠB ERT Ġtrained Ġon ĠWikipedia Ġand ĠBook Corp us . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin ĠBoth ĠT asks </s> Team ĠMIC - C IS ĠB IB REF 42 Ġparticipated Ġin Ġboth Ġtasks . ĠFor Ġthe Ġsentence - level Ġclassification , Ġthey Ġused Ġa Ġvoting Ġensemble Ġincluding Ġlog istic Ġregression , Ġconv olution al Ġneural Ġnetworks , Ġand ĠB ERT , Ġin Ġall Ġcases Ġusing ĠFast Text Ġembed d ings ĠB IB REF 43 Ġand Ġpre - trained ĠB ERT Ġmodels . ĠBes ide Ġthese Ġrepresentations , Ġmultiple Ġfeatures Ġof Ġread ability , Ġsentiment Ġand Ġemotions Ġwere Ġconsidered . ĠFor Ġthe Ġfragment - level Ġtask , Ġthey Ġused Ġa Ġmulti - task Ġneural Ġsequence Ġtag ger , Ġbased Ġon ĠL ST M - CR F ĠB IB REF 44 , Ġin Ġconjunction Ġwith Ġlinguistic Ġfeatures . ĠFinally , Ġthey Ġapplied Ġsentence - Ġand Ġfragment - level Ġmodels Ġjointly . </s> Team ĠC UN LP ĠB IB REF 45 Ġconsidered Ġtwo Ġapproaches Ġfor Ġthe Ġsentence - level Ġtask . ĠThe Ġfirst Ġapproach Ġwas Ġbased Ġon Ġfine - tun ing ĠB ERT . ĠThe Ġsecond Ġapproach Ġcomple mented Ġthe Ġfine - tun ed ĠB ERT Ġapproach Ġby Ġfeeding Ġits Ġdecision Ġinto Ġa Ġlog istic Ġreg ressor , Ġtogether Ġwith Ġfeatures Ġfrom Ġthe ĠL ingu istic ĠInquiry Ġand ĠWord ĠCount Ġ( LI WC ) Ġlex icon Ġand Ġpunct uation - derived Ġfeatures . ĠSimilarly Ġto ĠB IB REF 42 , Ġfor Ġthe Ġfragment - level Ġproblem Ġthey Ġused Ġa ĠBi - L ST M - CR F Ġarchitecture , Ġcombining Ġboth Ġcharacter - Ġand Ġword - level Ġembed d ings . </s> Team ĠProper G ander ĠB IB REF 46 Ġalso Ġused ĠB ERT , Ġbut Ġthey Ġpaid Ġspecial Ġattention Ġto Ġthe Ġimbalance Ġof Ġthe Ġdata , Ġas Ġwell Ġas Ġto Ġthe Ġdifferences Ġbetween Ġtraining Ġand Ġtesting . ĠThey Ġshowed Ġthat Ġaugment ing Ġthe Ġtraining Ġdata Ġby Ġovers am pling Ġyielded Ġimprovements Ġwhen Ġtesting Ġon Ġdata Ġthat Ġis Ġtempor ally Ġfar Ġfrom Ġthe Ġtraining Ġ( by Ġincreasing Ġrecall ). ĠIn Ġorder Ġto Ġdeal Ġwith Ġthe Ġimbalance , Ġthey Ġperformed Ġcost - sensitive Ġclassification , Ġi . e ., Ġthe Ġerrors Ġon Ġthe Ġsmaller Ġpositive Ġclass Ġwere Ġmore Ġcostly . ĠFor Ġthe Ġfragment - level Ġclassification , Ġinspired Ġby Ġnamed Ġentity Ġrecognition , Ġthey Ġused Ġa Ġmodel Ġbased Ġon ĠB ERT Ġusing ĠContinuous ĠRandom ĠField Ġstacked Ġon Ġtop Ġof Ġan ĠL ST M . </s> E val uation ĠResults </s> The Ġresults Ġon Ġthe Ġtest Ġset Ġfor Ġthe ĠS LC Ġtask Ġare Ġshown Ġin ĠTable ĠT AB REF 33 , Ġwhile ĠTable ĠT AB REF 34 Ġpresents Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġat Ġthe Ġend Ġof Ġphase Ġ1 Ġ( cf . ĠSection ĠSEC REF 6 ). ĠThe Ġgeneral Ġdecrease Ġof Ġthe ĠF $ _ 1 $ Ġvalues Ġbetween Ġthe Ġdevelopment Ġand Ġthe Ġtest Ġset Ġcould Ġindicate Ġthat Ġsystems Ġtend Ġto Ġover fit Ġon Ġthe Ġdevelopment Ġset . ĠIndeed , Ġthe Ġwinning Ġteam Ġl tu orp Ġchose Ġthe Ġparameters Ġof Ġtheir Ġsystem Ġboth Ġon Ġthe Ġdevelopment Ġset Ġand Ġon Ġa Ġsubset Ġof Ġthe Ġtraining Ġset Ġin Ġorder Ġto Ġimprove Ġthe Ġrobust ness Ġof Ġtheir Ġsystem . </s> T ables ĠT AB REF 36 Ġand ĠT AB REF 41 Ġreport Ġthe Ġresults Ġon Ġthe Ġtest Ġand Ġon Ġthe Ġdevelopment Ġsets Ġfor Ġthe ĠF LC Ġtask . ĠFor Ġthis Ġtask , Ġthe Ġresults Ġtend Ġto Ġbe Ġmore Ġstable Ġacross Ġthe Ġtwo Ġsets . ĠIndeed , Ġteam Ġnews peak Ġmanaged Ġto Ġalmost Ġkeep Ġthe Ġsame Ġdifference Ġin Ġperformance Ġwith Ġrespect Ġto Ġteam ĠAnt ig anda . ĠNote Ġthat Ġteam ĠMIC - C IS Ġmanaged Ġto Ġreach Ġthe Ġthird Ġposition Ġdespite Ġnever Ġhaving Ġsubmitted Ġa Ġrun Ġon Ġthe Ġdevelopment Ġset . </s> Conclusion Ġand ĠFurther ĠWork </s> We Ġhave Ġdescribed Ġthe ĠN LP 4 IF @ EM N LP - IJ CN LP Ġ2019 Ġshared Ġtask Ġon Ġfine - gr ained Ġpropaganda Ġidentification . ĠWe Ġreceived Ġ25 Ġand Ġ12 Ġsubmissions Ġon Ġthe Ġtest Ġset Ġfor Ġthe Ġsentence - level Ġclassification Ġand Ġthe Ġfragment - level Ġclassification Ġtasks , Ġrespectively . ĠOverall , Ġthe Ġsentence - level Ġtask Ġwas Ġeasier Ġand Ġmost Ġsubmitted Ġsystems Ġmanaged Ġto Ġoutper form Ġthe Ġbaseline . ĠThe Ġfragment - level Ġtask Ġproved Ġto Ġbe Ġmuch Ġmore Ġchallenging , Ġwith Ġlower Ġabsolute Ġscores , Ġbut Ġmost Ġteams Ġstill Ġmanaged Ġto Ġoutper form Ġthe Ġbaseline . </s> We Ġplan Ġto Ġmake Ġthe Ġschema Ġand Ġthe Ġdataset Ġpublicly Ġavailable Ġto Ġbe Ġused Ġbeyond ĠN LP 4 IF . ĠWe Ġhope Ġthat Ġthe Ġcorpus Ġwould Ġraise Ġinterest Ġoutside Ġof Ġthe Ġcommunity Ġof Ġresearchers Ġstudying Ġpropaganda : Ġthe Ġtechniques Ġrelated Ġto Ġfall acies Ġand Ġthe Ġones Ġrelying Ġon Ġemotions Ġmight Ġprovide Ġa Ġnovel Ġsetting Ġfor Ġresearchers Ġinterested Ġin ĠArgument ation Ġand ĠSent iment ĠAnalysis . </s> As Ġa Ġkind Ġof Ġadvertisement , ĠTask Ġ11 Ġat ĠSem E val Ġ2020 Ġis Ġa Ġfollow Ġup Ġof Ġthis Ġshared Ġtask . ĠIt Ġfeatures Ġtwo Ġcomplimentary Ġtasks : </s> Given Ġa Ġfree - text Ġarticle , Ġidentify Ġthe Ġpropag and ist Ġtext Ġspans . </s> Given Ġa Ġtext Ġspan Ġalready Ġflagged Ġas Ġpropag and ist Ġand Ġits Ġcontext , Ġidentify Ġthe Ġspecific Ġpropaganda Ġtechnique Ġit Ġcontains . </s> This Ġsetting Ġwould Ġallow Ġparticipants Ġto Ġfocus Ġtheir Ġefforts Ġon Ġbinary Ġsequence Ġlabeling Ġfor ĠTask Ġ1 Ġand Ġon Ġmulti - class Ġclassification Ġfor ĠTask Ġ2 . </s> Acknowled gments </s> This Ġresearch Ġis Ġpart Ġof Ġthe ĠProp aganda ĠAnalysis ĠProject , Ġwhich Ġis Ġframed Ġwithin Ġthe ĠTan b ih Ġproject . ĠThe ĠTan b ih Ġproject Ġaims Ġto Ġlimit Ġthe Ġeffect Ġof ĠâĢ ľ fake Ġnews âĢ Ŀ , Ġpropaganda , Ġand Ġmedia Ġbias Ġby Ġmaking Ġusers Ġaware Ġof Ġwhat Ġthey Ġare Ġreading , Ġthus Ġpromoting Ġmedia Ġliteracy Ġand Ġcritical Ġthinking , Ġwhich Ġis Ġarguably Ġthe Ġbest Ġway Ġto Ġaddress Ġdisinformation Ġand ĠâĢ ľ fake Ġnews . âĢ Ŀ ĠThe Ġproject Ġis Ġdeveloped Ġin Ġcollaboration Ġbetween Ġthe ĠQatar ĠComputing ĠResearch ĠInstitute Ġ( Q C RI ), ĠHB K U Ġand Ġthe ĠMIT ĠComputer ĠScience Ġand ĠArtificial ĠIntelligence ĠLaboratory Ġ( C SA IL ). </s> The Ġcorpus Ġfor Ġthe Ġtask Ġwas Ġannot ated Ġby ĠA ĠData ĠPro , Ġa Ġcompany Ġthat Ġperforms Ġhigh - quality Ġmanual Ġannotations .
Introduction Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda. In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2. Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf. Section SECREF3). Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques. Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system. With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop. The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level. We hope that the corpus would raise interest outside of the community of researchers studying propaganda. For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis. Related Work Propaganda has been tackled mostly at the article level. BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire. BIBREF4 experimented with a binarized version of that corpus: propaganda vs. the other three categories. BIBREF5 annotated a large binary corpus of propagandist vs. non-propagandist articles and proposed a feature-based system for discriminating between them. In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6. A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda. BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy. BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1.3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda. Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles. Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments. The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task. Propaganda Techniques Propaganda uses psychological and rhetorical techniques to achieve its objective. Such techniques include the use of logical fallacies and appeal to emotions. For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources. We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques: Propaganda Techniques ::: 1. Loaded language. Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11. Propaganda Techniques ::: 2. Name calling or labeling. Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12. Propaganda Techniques ::: 3. Repetition. Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12. Propaganda Techniques ::: 4. Exaggeration or minimization. Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e.g., saying that an insult was just a joke. Propaganda Techniques ::: 5. Doubt. Questioning the credibility of someone or something. Propaganda Techniques ::: 6. Appeal to fear/prejudice. Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments. Propaganda Techniques ::: 7. Flag-waving. Playing on strong national feeling (or with respect to a group, e.g., race, gender, political preference) to justify or promote an action or idea BIBREF15. Propaganda Techniques ::: 8. Causal oversimplification. Assuming one cause when there are multiple causes behind an issue. We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue. Propaganda Techniques ::: 9. Slogans. A brief and striking phrase that may include labeling and stereotyping. Slogans tend to act as emotional appeals BIBREF16. Propaganda Techniques ::: 10. Appeal to authority. Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17. We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14. Propaganda Techniques ::: 11. Black-and-white fallacy, dictatorship. Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13. As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship). Propaganda Techniques ::: 12. Thought-terminating cliché. Words or phrases that discourage critical thought and meaningful discussion about a given topic. They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18. Propaganda Techniques ::: 13. Whataboutism. Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19. Propaganda Techniques ::: 14. Reductio ad Hitlerum. Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience. It can refer to any person or concept with a negative connotation BIBREF20. Propaganda Techniques ::: 15. Red herring. Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11. Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20. Propaganda Techniques ::: 16. Bandwagon. Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15. Propaganda Techniques ::: 17. Obfuscation, intentional vagueness, confusion. Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11. For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion. Propaganda Techniques ::: 18. Straw man. When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22. Tasks The shared task features two subtasks: Tasks ::: Fragment-Level Classification task (FLC). Given a news article, detect all spans of the text in which a propaganda technique is used. In addition, for each span the propaganda technique applied must be identified. Tasks ::: Sentence-Level Classification task (SLC). A sentence is considered propagandist if it contains at least one propagandist fragment. We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted. Data The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators. More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles. We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators. The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively. Figure FIGREF15 shows an annotated example, which contains several propaganda techniques. For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling. Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language. Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets. Setup The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided. The participants tried to achieve the best performance on the development set. A live leaderboard kept track of the submissions. The test set was released and the participants had few days to make final predictions. In phase 2, no immediate feedback on the submissions was provided. The winner was determined based on the performance on the test set. Evaluation ::: FLC task. FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task). While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details. We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard). Evaluation ::: SLC task. SLC is a binary classification task with imbalanced data. Therefore, the official evaluation measure for the task is the standard F$_1$ measure. We further report Precision and Recall. Baselines The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence. The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly. The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41. Participants and Approaches A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions. For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions. Below, we give an overview of the approaches as described in the participants' papers. Tables TABREF28 and TABREF29 offer a high-level summary. Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class. The team fed one sentence at a time in order to reduce the workload. In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia. Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters. Moreover, oversampling of the least represented classes proved to be crucial for the final performance. Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs. Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28. The balancing of the embedding space was carried out by means of synthetic minority class over-sampling. Then, the learned representations were fed into an LSTM. Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only Team CAUnLP BIBREF29 used two context-aware representations based on BERT. In the first representation, the target sentence is followed by the title of the article. In the second representation, the previous sentence is also added. They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$ Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings. They also observed a boost in performance when balancing the dataset by dropping some negative examples. Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features. These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT. Team YMJA BIBREF33 also based their approach on fine-tuned BERT. Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation. Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF.IDF and BERT vectors, as well as vocabularies and readability measures. Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36. Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38. Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e.g., emotion classification. Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus. Participants and Approaches ::: Teams Participating in Both Tasks Team MIC-CIS BIBREF42 participated in both tasks. For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models. Beside these representations, multiple features of readability, sentiment and emotions were considered. For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features. Finally, they applied sentence- and fragment-level models jointly. Team CUNLP BIBREF45 considered two approaches for the sentence-level task. The first approach was based on fine-tuning BERT. The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features. Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings. Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing. They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall). In order to deal with the imbalance, they performed cost-sensitive classification, i.e., the errors on the smaller positive class were more costly. For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM. Evaluation Results The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf. Section SECREF6). The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set. Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system. Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task. For this task, the results tend to be more stable across the two sets. Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda. Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set. Conclusion and Further Work We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification. We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively. Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline. The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline. We plan to make the schema and the dataset publicly available to be used beyond NLP4IF. We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis. As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task. It features two complimentary tasks: Given a free-text article, identify the propagandist text spans. Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains. This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2. Acknowledgments This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project. The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news.” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations.
What dataset was used?
df5a4505edccc0ee11349ed6e7958cf6b84c9ed4
What Ġdataset Ġwas Ġused ?
Ġnews Ġarticles Ġin Ġfree - text Ġformat
<s> What Ġdataset Ġwas Ġused ? </s> Introduction </s> Prop aganda Ġaims Ġat Ġinfluencing Ġpeople 's Ġmindset Ġwith Ġthe Ġpurpose Ġof Ġadvancing Ġa Ġspecific Ġagenda . ĠIn Ġthe ĠInternet Ġera , Ġthanks Ġto Ġthe Ġmechanism Ġof Ġsharing Ġin Ġsocial Ġnetworks , Ġpropaganda Ġcampaigns Ġhave Ġthe Ġpotential Ġof Ġreaching Ġvery Ġlarge Ġaudiences ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . </s> Prop ag and ist Ġnews Ġarticles Ġuse Ġspecific Ġtechniques Ġto Ġconvey Ġtheir Ġmessage , Ġsuch Ġas Ġwhat about ism , Ġred ĠHer ring , Ġand Ġname Ġcalling , Ġamong Ġmany Ġothers Ġ( cf . ĠSection ĠSEC REF 3 ). ĠWhereas Ġproving Ġintent Ġis Ġnot Ġeasy , Ġwe Ġcan Ġanalyse Ġthe Ġlanguage Ġof Ġa Ġclaim / article Ġand Ġlook Ġfor Ġthe Ġuse Ġof Ġspecific Ġpropaganda Ġtechniques . ĠGoing Ġat Ġthis Ġfine - gr ained Ġlevel Ġcan Ġyield Ġmore Ġreliable Ġsystems Ġand Ġit Ġalso Ġmakes Ġit Ġpossible Ġto Ġexplain Ġto Ġthe Ġuser Ġwhy Ġan Ġarticle Ġwas Ġjudged Ġas Ġpropag and ist Ġby Ġan Ġautomatic Ġsystem . </s> With Ġthis Ġin Ġmind , Ġwe Ġorganised Ġthe Ġshared Ġtask Ġon Ġfine - gr ained Ġpropaganda Ġdetection Ġat Ġthe ĠN LP 4 IF @ EM N LP - IJ CN LP Ġ2019 Ġworkshop . ĠThe Ġtask Ġis Ġbased Ġon Ġa Ġcorpus Ġof Ġnews Ġarticles Ġannot ated Ġwith Ġan Ġinventory Ġof Ġ18 Ġpropag and ist Ġtechniques Ġat Ġthe Ġfragment Ġlevel . ĠWe Ġhope Ġthat Ġthe Ġcorpus Ġwould Ġraise Ġinterest Ġoutside Ġof Ġthe Ġcommunity Ġof Ġresearchers Ġstudying Ġpropaganda . ĠFor Ġexample , Ġthe Ġtechniques Ġrelated Ġto Ġfall acies Ġand Ġthe Ġones Ġrelying Ġon Ġemotions Ġmight Ġprovide Ġa Ġnovel Ġsetting Ġfor Ġresearchers Ġinterested Ġin ĠArgument ation Ġand ĠSent iment ĠAnalysis . </s> Related ĠWork </s> Prop aganda Ġhas Ġbeen Ġtackled Ġmostly Ġat Ġthe Ġarticle Ġlevel . ĠB IB REF 3 Ġcreated Ġa Ġcorpus Ġof Ġnews Ġarticles Ġlabelled Ġas Ġpropaganda , Ġtrusted , Ġhoax , Ġor Ġsatire . ĠB IB REF 4 Ġexperimented Ġwith Ġa Ġbin ar ized Ġversion Ġof Ġthat Ġcorpus : Ġpropaganda Ġvs . Ġthe Ġother Ġthree Ġcategories . ĠB IB REF 5 Ġannot ated Ġa Ġlarge Ġbinary Ġcorpus Ġof Ġpropag and ist Ġvs . Ġnon - prop ag and ist Ġarticles Ġand Ġproposed Ġa Ġfeature - based Ġsystem Ġfor Ġdiscriminating Ġbetween Ġthem . ĠIn Ġall Ġthese Ġcases , Ġthe Ġlabels Ġwere Ġobtained Ġusing Ġdistant Ġsupervision , Ġassuming Ġthat Ġall Ġarticles Ġfrom Ġa Ġgiven Ġnews Ġoutlet Ġshare Ġthe Ġlabel Ġof Ġthat Ġoutlet , Ġwhich Ġinevitably Ġintroduces Ġnoise ĠB IB REF 6 . </s> A Ġrelated Ġfield Ġis Ġthat Ġof Ġcomputational Ġargument ation Ġwhich , Ġamong Ġothers , Ġdeals Ġwith Ġsome Ġlogical Ġfall acies Ġrelated Ġto Ġpropaganda . ĠB IB REF 7 Ġpresented Ġa Ġcorpus Ġof ĠWeb Ġforum Ġdiscussions Ġwith Ġinstances Ġof Ġad Ġhom inem Ġfallacy . ĠB IB REF 8 , ĠB IB REF 9 Ġintroduced ĠArg ot ario , Ġa Ġgame Ġto Ġeducate Ġpeople Ġto Ġrecognize Ġand Ġcreate Ġfall acies , Ġa Ġby - product Ġof Ġwhich Ġis Ġa Ġcorpus Ġwith Ġ$ 1 . 3 k $ Ġarguments Ġannot ated Ġwith Ġfive Ġfall acies Ġsuch Ġas Ġad Ġhom inem , Ġred Ġher ring Ġand Ġirrelevant Ġauthority , Ġwhich Ġdirectly Ġrelate Ġto Ġpropaganda . </s> Unlike ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 7 , Ġour Ġcorpus Ġuses Ġ18 Ġtechniques Ġannot ated Ġon Ġthe Ġsame Ġset Ġof Ġnews Ġarticles . ĠMoreover , Ġour Ġannotations Ġaim Ġat Ġidentifying Ġthe Ġminimal Ġfragments Ġrelated Ġto Ġa Ġtechnique Ġinstead Ġof Ġflag ging Ġentire Ġarguments . </s> The Ġmost Ġrelevant Ġrelated Ġwork Ġis Ġour Ġown , Ġwhich Ġis Ġpublished Ġin Ġparallel Ġto Ġthis Ġpaper Ġat ĠEM N LP - IJ CN LP Ġ2019 ĠB IB REF 10 Ġand Ġdescribes Ġa Ġcorpus Ġthat Ġis Ġa Ġsubset Ġof Ġthe Ġone Ġused Ġfor Ġthis Ġshared Ġtask . </s> Prop aganda ĠTechniques </s> Prop aganda Ġuses Ġpsychological Ġand Ġrhetorical Ġtechniques Ġto Ġachieve Ġits Ġobjective . ĠSuch Ġtechniques Ġinclude Ġthe Ġuse Ġof Ġlogical Ġfall acies Ġand Ġappeal Ġto Ġemotions . ĠFor Ġthe Ġshared Ġtask , Ġwe Ġuse Ġ18 Ġtechniques Ġthat Ġcan Ġbe Ġfound Ġin Ġnews Ġarticles Ġand Ġcan Ġbe Ġjudged Ġintrinsically , Ġwithout Ġthe Ġneed Ġto Ġretrieve Ġsupporting Ġinformation Ġfrom Ġexternal Ġresources . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 10 Ġfor Ġmore Ġdetails Ġon Ġthe Ġpropaganda Ġtechniques ; Ġbelow Ġwe Ġreport Ġthe Ġlist Ġof Ġtechniques : </s> Prop aganda ĠTechniques Ġ: :: Ġ1 . ĠLoaded Ġlanguage . </s> Using Ġwords / ph r ases Ġwith Ġstrong Ġemotional Ġimplications Ġ( positive Ġor Ġnegative ) Ġto Ġinfluence Ġan Ġaudience ĠB IB REF 11 . </s> Prop aganda ĠTechniques Ġ: :: Ġ2 . ĠName Ġcalling Ġor Ġlabeling . </s> Label ing Ġthe Ġobject Ġof Ġthe Ġpropaganda Ġas Ġsomething Ġthe Ġtarget Ġaudience Ġfears , Ġhates , Ġfinds Ġundesirable Ġor Ġotherwise Ġloves Ġor Ġpraises ĠB IB REF 12 . </s> Prop aganda ĠTechniques Ġ: :: Ġ3 . ĠRep etition . </s> Repe ating Ġthe Ġsame Ġmessage Ġover Ġand Ġover Ġagain , Ġso Ġthat Ġthe Ġaudience Ġwill Ġeventually Ġaccept Ġit ĠB IB REF 13 , ĠB IB REF 12 . </s> Prop aganda ĠTechniques Ġ: :: Ġ4 . ĠEx agger ation Ġor Ġminim ization . </s> Either Ġrepresenting Ġsomething Ġin Ġan Ġexcessive Ġmanner : Ġmaking Ġthings Ġlarger , Ġbetter , Ġworse , Ġor Ġmaking Ġsomething Ġseem Ġless Ġimportant Ġor Ġsmaller Ġthan Ġit Ġactually Ġis ĠB IB REF 14 , Ġe . g ., Ġsaying Ġthat Ġan Ġinsult Ġwas Ġjust Ġa Ġjoke . </s> Prop aganda ĠTechniques Ġ: :: Ġ5 . ĠDou bt . </s> Question ing Ġthe Ġcredibility Ġof Ġsomeone Ġor Ġsomething . </s> Prop aganda ĠTechniques Ġ: :: Ġ6 . ĠAppeal Ġto Ġfear / pre jud ice . </s> Se eking Ġto Ġbuild Ġsupport Ġfor Ġan Ġidea Ġby Ġinst illing Ġanxiety Ġand / or Ġpanic Ġin Ġthe Ġpopulation Ġtowards Ġan Ġalternative , Ġpossibly Ġbased Ġon Ġprecon ceived Ġjudgments . </s> Prop aganda ĠTechniques Ġ: :: Ġ7 . ĠFlag - w aving . </s> Playing Ġon Ġstrong Ġnational Ġfeeling Ġ( or Ġwith Ġrespect Ġto Ġa Ġgroup , Ġe . g ., Ġrace , Ġgender , Ġpolitical Ġpreference ) Ġto Ġjustify Ġor Ġpromote Ġan Ġaction Ġor Ġidea ĠB IB REF 15 . </s> Prop aganda ĠTechniques Ġ: :: Ġ8 . ĠCa usal Ġovers impl ification . </s> Assuming Ġone Ġcause Ġwhen Ġthere Ġare Ġmultiple Ġcauses Ġbehind Ġan Ġissue . ĠWe Ġinclude Ġscapego ating Ġas Ġwell : Ġthe Ġtransfer Ġof Ġthe Ġblame Ġto Ġone Ġperson Ġor Ġgroup Ġof Ġpeople Ġwithout Ġinvestigating Ġthe Ġcomplexities Ġof Ġan Ġissue . </s> Prop aganda ĠTechniques Ġ: :: Ġ9 . ĠSl og ans . </s> A Ġbrief Ġand Ġstriking Ġphrase Ġthat Ġmay Ġinclude Ġlabeling Ġand Ġstereotyp ing . ĠSl og ans Ġtend Ġto Ġact Ġas Ġemotional Ġappeals ĠB IB REF 16 . </s> Prop aganda ĠTechniques Ġ: :: Ġ10 . ĠAppeal Ġto Ġauthority . </s> St ating Ġthat Ġa Ġclaim Ġis Ġtrue Ġsimply Ġbecause Ġa Ġvalid Ġauthority / ex pert Ġon Ġthe Ġissue Ġsupports Ġit , Ġwithout Ġany Ġother Ġsupporting Ġevidence ĠB IB REF 17 . ĠWe Ġinclude Ġthe Ġspecial Ġcase Ġwhere Ġthe Ġreference Ġis Ġnot Ġan Ġauthority / ex pert , Ġalthough Ġit Ġis Ġreferred Ġto Ġas Ġtestim onial Ġin Ġthe Ġliterature ĠB IB REF 14 . </s> Prop aganda ĠTechniques Ġ: :: Ġ11 . ĠBlack - and - white Ġfallacy , Ġdictatorship . </s> Present ing Ġtwo Ġalternative Ġoptions Ġas Ġthe Ġonly Ġpossibilities , Ġwhen Ġin Ġfact Ġmore Ġpossibilities Ġexist ĠB IB REF 13 . ĠAs Ġan Ġextreme Ġcase , Ġtelling Ġthe Ġaudience Ġexactly Ġwhat Ġactions Ġto Ġtake , Ġeliminating Ġany Ġother Ġpossible Ġchoice Ġ( dict ators hip ). </s> Prop aganda ĠTechniques Ġ: :: Ġ12 . ĠThought - termin ating ĠclichÃ© . </s> Words Ġor Ġphrases Ġthat Ġdiscourage Ġcritical Ġthought Ġand Ġmeaningful Ġdiscussion Ġabout Ġa Ġgiven Ġtopic . ĠThey Ġare Ġtypically Ġshort Ġand Ġgeneric Ġsentences Ġthat Ġoffer Ġseemingly Ġsimple Ġanswers Ġto Ġcomplex Ġquestions Ġor Ġthat Ġdistract Ġattention Ġaway Ġfrom Ġother Ġlines Ġof Ġthought ĠB IB REF 18 . </s> Prop aganda ĠTechniques Ġ: :: Ġ13 . ĠWhat about ism . </s> Disc redit Ġan Ġopponent 's Ġposition Ġby Ġcharging Ġthem Ġwith Ġhypocrisy Ġwithout Ġdirectly Ġdispro ving Ġtheir Ġargument ĠB IB REF 19 . </s> Prop aganda ĠTechniques Ġ: :: Ġ14 . ĠRed uct io Ġad ĠHitler um . </s> Pers u ading Ġan Ġaudience Ġto Ġdisapprove Ġan Ġaction Ġor Ġidea Ġby Ġsuggesting Ġthat Ġthe Ġidea Ġis Ġpopular Ġwith Ġgroups Ġhated Ġin Ġcontempt Ġby Ġthe Ġtarget Ġaudience . ĠIt Ġcan Ġrefer Ġto Ġany Ġperson Ġor Ġconcept Ġwith Ġa Ġnegative Ġcon notation ĠB IB REF 20 . </s> Prop aganda ĠTechniques Ġ: :: Ġ15 . ĠRed Ġher ring . </s> Introdu cing Ġirrelevant Ġmaterial Ġto Ġthe Ġissue Ġbeing Ġdiscussed , Ġso Ġthat Ġeveryone 's Ġattention Ġis Ġdiverted Ġaway Ġfrom Ġthe Ġpoints Ġmade ĠB IB REF 11 . ĠThose Ġsubjected Ġto Ġa Ġred Ġher ring Ġargument Ġare Ġled Ġaway Ġfrom Ġthe Ġissue Ġthat Ġhad Ġbeen Ġthe Ġfocus Ġof Ġthe Ġdiscussion Ġand Ġurged Ġto Ġfollow Ġan Ġobservation Ġor Ġclaim Ġthat Ġmay Ġbe Ġassociated Ġwith Ġthe Ġoriginal Ġclaim , Ġbut Ġis Ġnot Ġhighly Ġrelevant Ġto Ġthe Ġissue Ġin Ġdispute ĠB IB REF 20 . </s> Prop aganda ĠTechniques Ġ: :: Ġ16 . ĠBand wagon . </s> Attempt ing Ġto Ġpersuade Ġthe Ġtarget Ġaudience Ġto Ġjoin Ġin Ġand Ġtake Ġthe Ġcourse Ġof Ġaction Ġbecause ĠâĢ ľ everyone Ġelse Ġis Ġtaking Ġthe Ġsame Ġaction âĢ Ŀ ĠB IB REF 15 . </s> Prop aganda ĠTechniques Ġ: :: Ġ17 . ĠOb fusc ation , Ġintentional Ġv agu eness , Ġconfusion . </s> Using Ġdeliberately Ġunclear Ġwords , Ġto Ġlet Ġthe Ġaudience Ġhave Ġits Ġown Ġinterpretation ĠB IB REF 21 , ĠB IB REF 11 . ĠFor Ġinstance , Ġwhen Ġan Ġunclear Ġphrase Ġwith Ġmultiple Ġpossible Ġmeanings Ġis Ġused Ġwithin Ġthe Ġargument Ġand , Ġtherefore , Ġit Ġdoes Ġnot Ġreally Ġsupport Ġthe Ġconclusion . </s> Prop aganda ĠTechniques Ġ: :: Ġ18 . ĠStraw Ġman . </s> When Ġan Ġopponent 's Ġproposition Ġis Ġsubstituted Ġwith Ġa Ġsimilar Ġone Ġwhich Ġis Ġthen Ġrefuted Ġin Ġplace Ġof Ġthe Ġoriginal ĠB IB REF 22 . </s> T asks </s> The Ġshared Ġtask Ġfeatures Ġtwo Ġsubt asks : </s> T asks Ġ: :: ĠFrag ment - Level ĠClassification Ġtask Ġ( FL C ). </s> Given Ġa Ġnews Ġarticle , Ġdetect Ġall Ġspans Ġof Ġthe Ġtext Ġin Ġwhich Ġa Ġpropaganda Ġtechnique Ġis Ġused . ĠIn Ġaddition , Ġfor Ġeach Ġspan Ġthe Ġpropaganda Ġtechnique Ġapplied Ġmust Ġbe Ġidentified . </s> T asks Ġ: :: ĠSent ence - Level ĠClassification Ġtask Ġ( S LC ). </s> A Ġsentence Ġis Ġconsidered Ġpropag and ist Ġif Ġit Ġcontains Ġat Ġleast Ġone Ġpropag and ist Ġfragment . ĠWe Ġthen Ġdefine Ġa Ġbinary Ġclassification Ġtask Ġin Ġwhich , Ġgiven Ġa Ġsentence , Ġthe Ġcorrect Ġlabel , Ġeither Ġpropaganda Ġor Ġnon - prop aganda , Ġis Ġto Ġbe Ġpredicted . </s> Data </s> The Ġinput Ġfor Ġboth Ġtasks Ġconsists Ġof Ġnews Ġarticles Ġin Ġfree - text Ġformat , Ġcollected Ġfrom Ġ36 Ġpropag and ist Ġand Ġ12 Ġnon - prop ag and ist Ġnews Ġoutlets Ġand Ġthen Ġannot ated Ġby Ġprofessional Ġannot ators . ĠMore Ġdetails Ġabout Ġthe Ġdata Ġcollection Ġand Ġthe Ġannotation , Ġas Ġwell Ġas Ġstatistics Ġabout Ġthe Ġcorpus Ġcan Ġbe Ġfound Ġin ĠB IB REF 10 , Ġwhere Ġan Ġearlier Ġversion Ġof Ġthe Ġcorpus Ġis Ġdescribed , Ġwhich Ġincludes Ġ450 Ġnews Ġarticles . ĠWe Ġfurther Ġannot ated Ġ47 Ġadditional Ġarticles Ġfor Ġthe Ġpurpose Ġof Ġthe Ġshared Ġtask Ġusing Ġthe Ġsame Ġprotocol Ġand Ġthe Ġsame Ġannot ators . </s> The Ġtraining , Ġthe Ġdevelopment , Ġand Ġthe Ġtest Ġpartitions Ġof Ġthe Ġcorpus Ġused Ġfor Ġthe Ġshared Ġtask Ġconsist Ġof Ġ350 , Ġ61 , Ġand Ġ86 Ġarticles Ġand Ġof Ġ16 , 9 65 , Ġ2 , 235 , Ġand Ġ3 , 526 Ġsentences , Ġrespectively . ĠFigure ĠFIG REF 15 Ġshows Ġan Ġannot ated Ġexample , Ġwhich Ġcontains Ġseveral Ġpropaganda Ġtechniques . ĠFor Ġexample , Ġthe Ġfragment Ġbabies Ġon Ġline Ġ1 Ġis Ġan Ġinstance Ġof Ġboth ĠName _ Calling Ġand ĠLab eling . ĠNote Ġthat Ġthe Ġfragment Ġnot Ġlooking Ġas Ġthough ĠTrump Ġkilled Ġhis Ġgrandma Ġon Ġline Ġ4 Ġis Ġan Ġinstance Ġof ĠEx agger ation _ or _ Min im isation Ġand Ġit Ġoverl aps Ġwith Ġthe Ġfragment Ġkilled Ġhis Ġgrandma , Ġwhich Ġis Ġan Ġinstance Ġof ĠLoaded _ Language . </s> Table ĠT AB REF 23 Ġreports Ġthe Ġtotal Ġnumber Ġof Ġinstances Ġper Ġtechnique Ġand Ġthe Ġpercentage Ġwith Ġrespect Ġto Ġthe Ġtotal Ġnumber Ġof Ġannotations , Ġfor Ġthe Ġtraining Ġand Ġfor Ġthe Ġdevelopment Ġsets . </s> Setup </s> The Ġshared Ġtask Ġhad Ġtwo Ġphases : ĠIn Ġthe Ġdevelopment Ġphase , Ġthe Ġparticipants Ġwere Ġprovided Ġlabeled Ġtraining Ġand Ġdevelopment Ġdatasets ; Ġin Ġthe Ġtesting Ġphase , Ġtesting Ġinput Ġwas Ġfurther Ġprovided . </s> The Ġparticipants Ġtried Ġto Ġachieve Ġthe Ġbest Ġperformance Ġon Ġthe Ġdevelopment Ġset . ĠA Ġlive Ġleader board Ġkept Ġtrack Ġof Ġthe Ġsubmissions . </s> The Ġtest Ġset Ġwas Ġreleased Ġand Ġthe Ġparticipants Ġhad Ġfew Ġdays Ġto Ġmake Ġfinal Ġpredictions . </s> In Ġphase Ġ2 , Ġno Ġimmediate Ġfeedback Ġon Ġthe Ġsubmissions Ġwas Ġprovided . ĠThe Ġwinner Ġwas Ġdetermined Ġbased Ġon Ġthe Ġperformance Ġon Ġthe Ġtest Ġset . </s> E val uation Ġ: :: ĠF LC Ġtask . </s> FL C Ġis Ġa Ġcomposition Ġof Ġtwo Ġsubt asks : Ġthe Ġidentification Ġof Ġthe Ġpropag and ist Ġtext Ġfragments Ġand Ġthe Ġidentification Ġof Ġthe Ġtechniques Ġused Ġ( 18 - way Ġclassification Ġtask ). ĠWhile ĠF $ _ 1 $ Ġmeasure Ġis Ġappropriate Ġfor Ġa Ġmulti - class Ġclassification Ġtask , Ġwe Ġmodified Ġit Ġto Ġaccount Ġfor Ġpartial Ġmatching Ġbetween Ġthe Ġspans ; Ġsee ĠB IB REF 10 Ġfor Ġmore Ġdetails . ĠWe Ġfurther Ġcomputed Ġan ĠF $ _ 1 $ Ġvalue Ġfor Ġeach Ġpropaganda Ġtechnique Ġ( not Ġshown Ġbelow Ġfor Ġthe Ġsake Ġof Ġsaving Ġspace , Ġbut Ġavailable Ġon Ġthe Ġleader board ). </s> E val uation Ġ: :: ĠS LC Ġtask . </s> S LC Ġis Ġa Ġbinary Ġclassification Ġtask Ġwith Ġim balanced Ġdata . ĠTherefore , Ġthe Ġofficial Ġevaluation Ġmeasure Ġfor Ġthe Ġtask Ġis Ġthe Ġstandard ĠF $ _ 1 $ Ġmeasure . ĠWe Ġfurther Ġreport ĠPrecision Ġand ĠRecall . </s> Bas elines </s> The Ġbaseline Ġsystem Ġfor Ġthe ĠS LC Ġtask Ġis Ġa Ġvery Ġsimple Ġlog istic Ġregression Ġclass ifier Ġwith Ġdefault Ġparameters , Ġwhere Ġwe Ġrepresent Ġthe Ġinput Ġinstances Ġwith Ġa Ġsingle Ġfeature : Ġthe Ġlength Ġof Ġthe Ġsentence . ĠThe Ġperformance Ġof Ġthis Ġbaseline Ġon Ġthe ĠS LC Ġtask Ġis Ġshown Ġin ĠTables ĠT AB REF 33 Ġand ĠT AB REF 34 . </s> The Ġbaseline Ġfor Ġthe ĠF LC Ġtask Ġgenerates Ġspans Ġand Ġselects Ġone Ġof Ġthe Ġ18 Ġtechniques Ġrandomly . ĠThe Ġin effic acy Ġof Ġsuch Ġa Ġsimple Ġrandom Ġbaseline Ġis Ġillustrated Ġin ĠTables ĠT AB REF 36 Ġand ĠT AB REF 41 . </s> Particip ants Ġand ĠAppro aches </s> A Ġtotal Ġof Ġ90 Ġteams Ġregistered Ġfor Ġthe Ġshared Ġtask , Ġand Ġ39 Ġof Ġthem Ġsubmitted Ġpredictions Ġfor Ġa Ġtotal Ġof Ġ3 , 0 65 Ġsubmissions . ĠFor Ġthe ĠF LC Ġtask , Ġ21 Ġteams Ġmade Ġa Ġtotal Ġof Ġ5 27 Ġsubmissions , Ġand Ġfor Ġthe ĠS LC Ġtask , Ġ35 Ġteams Ġmade Ġa Ġtotal Ġof Ġ2 , 538 Ġsubmissions . </s> Below , Ġwe Ġgive Ġan Ġoverview Ġof Ġthe Ġapproaches Ġas Ġdescribed Ġin Ġthe Ġparticipants ' Ġpapers . ĠTables ĠT AB REF 28 Ġand ĠT AB REF 29 Ġoffer Ġa Ġhigh - level Ġsummary . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin Ġthe ĠFrag ment - Level ĠClassification ĠOnly </s> Team Ġnews peak ĠB IB REF 23 Ġachieved Ġthe Ġbest Ġresults Ġon Ġthe Ġtest Ġset Ġfor Ġthe ĠF LC Ġtask Ġusing Ġ20 - way Ġword - level Ġclassification Ġbased Ġon ĠB ERT ĠB IB REF 24 : Ġa Ġword Ġcould Ġbelong Ġto Ġone Ġof Ġthe Ġ18 Ġpropaganda Ġtechniques , Ġto Ġnone Ġof Ġthem , Ġor Ġto Ġan Ġauxiliary Ġ( token - derived ) Ġclass . ĠThe Ġteam Ġfed Ġone Ġsentence Ġat Ġa Ġtime Ġin Ġorder Ġto Ġreduce Ġthe Ġworkload . ĠIn Ġaddition Ġto Ġexperimenting Ġwith Ġan Ġout - of - the - box ĠB ERT , Ġthey Ġalso Ġtried Ġun super vised Ġfine - tun ing Ġboth Ġon Ġthe Ġ1 M Ġnews Ġdataset Ġand Ġon ĠWikipedia . ĠTheir Ġbest Ġmodel Ġwas Ġbased Ġon Ġthe Ġunc ased Ġbase Ġmodel Ġof ĠB ERT , Ġwith Ġ12 ĠTrans former Ġlayers ĠB IB REF 25 , Ġand Ġ110 Ġmillion Ġparameters . ĠMoreover , Ġovers am pling Ġof Ġthe Ġleast Ġrepresented Ġclasses Ġproved Ġto Ġbe Ġcrucial Ġfor Ġthe Ġfinal Ġperformance . ĠFinally , Ġcareful Ġanalysis Ġhas Ġshown Ġthat Ġthe Ġmodel Ġpays Ġspecial Ġattention Ġto Ġadject ives Ġand Ġad verbs . </s> Team ĠStalin ĠB IB REF 26 Ġfocused Ġon Ġdata Ġaug mentation Ġto Ġaddress Ġthe Ġrelatively Ġsmall Ġsize Ġof Ġthe Ġdata Ġfor Ġfine - tun ing Ġcontextual Ġembed ding Ġrepresentations Ġbased Ġon ĠEL Mo ĠB IB REF 27 , ĠB ERT , Ġand ĠGro ver ĠB IB REF 28 . ĠThe Ġbalancing Ġof Ġthe Ġembed ding Ġspace Ġwas Ġcarried Ġout Ġby Ġmeans Ġof Ġsynthetic Ġminority Ġclass Ġover - sam pling . ĠThen , Ġthe Ġlearned Ġrepresentations Ġwere Ġfed Ġinto Ġan ĠL ST M . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin Ġthe ĠSent ence - Level ĠClassification ĠOnly </s> Team ĠCA Un LP ĠB IB REF 29 Ġused Ġtwo Ġcontext - aware Ġrepresentations Ġbased Ġon ĠB ERT . ĠIn Ġthe Ġfirst Ġrepresentation , Ġthe Ġtarget Ġsentence Ġis Ġfollowed Ġby Ġthe Ġtitle Ġof Ġthe Ġarticle . ĠIn Ġthe Ġsecond Ġrepresentation , Ġthe Ġprevious Ġsentence Ġis Ġalso Ġadded . ĠThey Ġperformed Ġsubs am pling Ġin Ġorder Ġto Ġdeal Ġwith Ġclass Ġimbalance , Ġand Ġexperimented Ġwith ĠB ERT $ _{ B ASE } $ Ġand ĠB ERT $ _{ L AR GE } $ </s> Team ĠLI ACC ĠB IB REF 30 Ġused Ġhand - crafted Ġfeatures Ġand Ġpre - trained ĠEL Mo Ġembed d ings . ĠThey Ġalso Ġobserved Ġa Ġboost Ġin Ġperformance Ġwhen Ġbalancing Ġthe Ġdataset Ġby Ġdropping Ġsome Ġnegative Ġexamples . </s> Team ĠJUST Deep ĠB IB REF 31 Ġused Ġa Ġcombination Ġof Ġmodels Ġand Ġfeatures , Ġincluding Ġword Ġembed d ings Ġbased Ġon ĠGl o Ve ĠB IB REF 32 Ġconc aten ated Ġwith Ġvectors Ġrepresenting Ġaffection Ġand Ġlex ical Ġfeatures . ĠThese Ġwere Ġcombined Ġin Ġan Ġensemble Ġof Ġsupervised Ġmodels : Ġbi - L ST M , ĠX GB o ost , Ġand Ġvariations Ġof ĠB ERT . </s> Team ĠY M JA ĠB IB REF 33 Ġalso Ġbased Ġtheir Ġapproach Ġon Ġfine - tun ed ĠB ERT . ĠInspired Ġby Ġk agg le Ġcompetitions Ġon Ġsentiment Ġanalysis , Ġthey Ġcreated Ġan Ġensemble Ġof Ġmodels Ġvia Ġcross - valid ation . </s> Team Ġj inf en ĠB IB REF 34 Ġused Ġa Ġlog istic Ġregression Ġmodel Ġfed Ġwith Ġa Ġmanifold Ġof Ġrepresentations , Ġincluding ĠTF . ID F Ġand ĠB ERT Ġvectors , Ġas Ġwell Ġas Ġvoc ab ular ies Ġand Ġread ability Ġmeasures . </s> Team ĠTh a 3 ar oon ĠB IB REF 35 Ġimplemented Ġan Ġensemble Ġof Ġthree Ġclass ifiers : Ġtwo Ġbased Ġon ĠB ERT Ġand Ġone Ġbased Ġon Ġa Ġuniversal Ġsentence Ġenc oder ĠB IB REF 36 . </s> Team ĠNS IT ĠB IB REF 37 Ġexplored Ġthree Ġof Ġthe Ġmost Ġpopular Ġtransfer Ġlearning Ġmodels : Ġvarious Ġversions Ġof ĠEL Mo , ĠB ERT , Ġand ĠRo BER Ta ĠB IB REF 38 . </s> Team ĠMind cod ers ĠB IB REF 39 Ġcombined ĠB ERT , ĠBi - L ST M Ġand ĠCaps ule Ġnetworks ĠB IB REF 40 Ġinto Ġa Ġsingle Ġdeep Ġneural Ġnetwork Ġand Ġpre - trained Ġthe Ġresulting Ġnetwork Ġon Ġcorpor a Ġused Ġfor Ġrelated Ġtasks , Ġe . g ., Ġemotion Ġclassification . </s> Finally , Ġteam Ġl tu orp ĠB IB REF 41 Ġused Ġan Ġattention Ġtransformer Ġusing ĠB ERT Ġtrained Ġon ĠWikipedia Ġand ĠBook Corp us . </s> Particip ants Ġand ĠAppro aches Ġ: :: ĠTeams ĠParticip ating Ġin ĠBoth ĠT asks </s> Team ĠMIC - C IS ĠB IB REF 42 Ġparticipated Ġin Ġboth Ġtasks . ĠFor Ġthe Ġsentence - level Ġclassification , Ġthey Ġused Ġa Ġvoting Ġensemble Ġincluding Ġlog istic Ġregression , Ġconv olution al Ġneural Ġnetworks , Ġand ĠB ERT , Ġin Ġall Ġcases Ġusing ĠFast Text Ġembed d ings ĠB IB REF 43 Ġand Ġpre - trained ĠB ERT Ġmodels . ĠBes ide Ġthese Ġrepresentations , Ġmultiple Ġfeatures Ġof Ġread ability , Ġsentiment Ġand Ġemotions Ġwere Ġconsidered . ĠFor Ġthe Ġfragment - level Ġtask , Ġthey Ġused Ġa Ġmulti - task Ġneural Ġsequence Ġtag ger , Ġbased Ġon ĠL ST M - CR F ĠB IB REF 44 , Ġin Ġconjunction Ġwith Ġlinguistic Ġfeatures . ĠFinally , Ġthey Ġapplied Ġsentence - Ġand Ġfragment - level Ġmodels Ġjointly . </s> Team ĠC UN LP ĠB IB REF 45 Ġconsidered Ġtwo Ġapproaches Ġfor Ġthe Ġsentence - level Ġtask . ĠThe Ġfirst Ġapproach Ġwas Ġbased Ġon Ġfine - tun ing ĠB ERT . ĠThe Ġsecond Ġapproach Ġcomple mented Ġthe Ġfine - tun ed ĠB ERT Ġapproach Ġby Ġfeeding Ġits Ġdecision Ġinto Ġa Ġlog istic Ġreg ressor , Ġtogether Ġwith Ġfeatures Ġfrom Ġthe ĠL ingu istic ĠInquiry Ġand ĠWord ĠCount Ġ( LI WC ) Ġlex icon Ġand Ġpunct uation - derived Ġfeatures . ĠSimilarly Ġto ĠB IB REF 42 , Ġfor Ġthe Ġfragment - level Ġproblem Ġthey Ġused Ġa ĠBi - L ST M - CR F Ġarchitecture , Ġcombining Ġboth Ġcharacter - Ġand Ġword - level Ġembed d ings . </s> Team ĠProper G ander ĠB IB REF 46 Ġalso Ġused ĠB ERT , Ġbut Ġthey Ġpaid Ġspecial Ġattention Ġto Ġthe Ġimbalance Ġof Ġthe Ġdata , Ġas Ġwell Ġas Ġto Ġthe Ġdifferences Ġbetween Ġtraining Ġand Ġtesting . ĠThey Ġshowed Ġthat Ġaugment ing Ġthe Ġtraining Ġdata Ġby Ġovers am pling Ġyielded Ġimprovements Ġwhen Ġtesting Ġon Ġdata Ġthat Ġis Ġtempor ally Ġfar Ġfrom Ġthe Ġtraining Ġ( by Ġincreasing Ġrecall ). ĠIn Ġorder Ġto Ġdeal Ġwith Ġthe Ġimbalance , Ġthey Ġperformed Ġcost - sensitive Ġclassification , Ġi . e ., Ġthe Ġerrors Ġon Ġthe Ġsmaller Ġpositive Ġclass Ġwere Ġmore Ġcostly . ĠFor Ġthe Ġfragment - level Ġclassification , Ġinspired Ġby Ġnamed Ġentity Ġrecognition , Ġthey Ġused Ġa Ġmodel Ġbased Ġon ĠB ERT Ġusing ĠContinuous ĠRandom ĠField Ġstacked Ġon Ġtop Ġof Ġan ĠL ST M . </s> E val uation ĠResults </s> The Ġresults Ġon Ġthe Ġtest Ġset Ġfor Ġthe ĠS LC Ġtask Ġare Ġshown Ġin ĠTable ĠT AB REF 33 , Ġwhile ĠTable ĠT AB REF 34 Ġpresents Ġthe Ġresults Ġon Ġthe Ġdevelopment Ġset Ġat Ġthe Ġend Ġof Ġphase Ġ1 Ġ( cf . ĠSection ĠSEC REF 6 ). ĠThe Ġgeneral Ġdecrease Ġof Ġthe ĠF $ _ 1 $ Ġvalues Ġbetween Ġthe Ġdevelopment Ġand Ġthe Ġtest Ġset Ġcould Ġindicate Ġthat Ġsystems Ġtend Ġto Ġover fit Ġon Ġthe Ġdevelopment Ġset . ĠIndeed , Ġthe Ġwinning Ġteam Ġl tu orp Ġchose Ġthe Ġparameters Ġof Ġtheir Ġsystem Ġboth Ġon Ġthe Ġdevelopment Ġset Ġand Ġon Ġa Ġsubset Ġof Ġthe Ġtraining Ġset Ġin Ġorder Ġto Ġimprove Ġthe Ġrobust ness Ġof Ġtheir Ġsystem . </s> T ables ĠT AB REF 36 Ġand ĠT AB REF 41 Ġreport Ġthe Ġresults Ġon Ġthe Ġtest Ġand Ġon Ġthe Ġdevelopment Ġsets Ġfor Ġthe ĠF LC Ġtask . ĠFor Ġthis Ġtask , Ġthe Ġresults Ġtend Ġto Ġbe Ġmore Ġstable Ġacross Ġthe Ġtwo Ġsets . ĠIndeed , Ġteam Ġnews peak Ġmanaged Ġto Ġalmost Ġkeep Ġthe Ġsame Ġdifference Ġin Ġperformance Ġwith Ġrespect Ġto Ġteam ĠAnt ig anda . ĠNote Ġthat Ġteam ĠMIC - C IS Ġmanaged Ġto Ġreach Ġthe Ġthird Ġposition Ġdespite Ġnever Ġhaving Ġsubmitted Ġa Ġrun Ġon Ġthe Ġdevelopment Ġset . </s> Conclusion Ġand ĠFurther ĠWork </s> We Ġhave Ġdescribed Ġthe ĠN LP 4 IF @ EM N LP - IJ CN LP Ġ2019 Ġshared Ġtask Ġon Ġfine - gr ained Ġpropaganda Ġidentification . ĠWe Ġreceived Ġ25 Ġand Ġ12 Ġsubmissions Ġon Ġthe Ġtest Ġset Ġfor Ġthe Ġsentence - level Ġclassification Ġand Ġthe Ġfragment - level Ġclassification Ġtasks , Ġrespectively . ĠOverall , Ġthe Ġsentence - level Ġtask Ġwas Ġeasier Ġand Ġmost Ġsubmitted Ġsystems Ġmanaged Ġto Ġoutper form Ġthe Ġbaseline . ĠThe Ġfragment - level Ġtask Ġproved Ġto Ġbe Ġmuch Ġmore Ġchallenging , Ġwith Ġlower Ġabsolute Ġscores , Ġbut Ġmost Ġteams Ġstill Ġmanaged Ġto Ġoutper form Ġthe Ġbaseline . </s> We Ġplan Ġto Ġmake Ġthe Ġschema Ġand Ġthe Ġdataset Ġpublicly Ġavailable Ġto Ġbe Ġused Ġbeyond ĠN LP 4 IF . ĠWe Ġhope Ġthat Ġthe Ġcorpus Ġwould Ġraise Ġinterest Ġoutside Ġof Ġthe Ġcommunity Ġof Ġresearchers Ġstudying Ġpropaganda : Ġthe Ġtechniques Ġrelated Ġto Ġfall acies Ġand Ġthe Ġones Ġrelying Ġon Ġemotions Ġmight Ġprovide Ġa Ġnovel Ġsetting Ġfor Ġresearchers Ġinterested Ġin ĠArgument ation Ġand ĠSent iment ĠAnalysis . </s> As Ġa Ġkind Ġof Ġadvertisement , ĠTask Ġ11 Ġat ĠSem E val Ġ2020 Ġis Ġa Ġfollow Ġup Ġof Ġthis Ġshared Ġtask . ĠIt Ġfeatures Ġtwo Ġcomplimentary Ġtasks : </s> Given Ġa Ġfree - text Ġarticle , Ġidentify Ġthe Ġpropag and ist Ġtext Ġspans . </s> Given Ġa Ġtext Ġspan Ġalready Ġflagged Ġas Ġpropag and ist Ġand Ġits Ġcontext , Ġidentify Ġthe Ġspecific Ġpropaganda Ġtechnique Ġit Ġcontains . </s> This Ġsetting Ġwould Ġallow Ġparticipants Ġto Ġfocus Ġtheir Ġefforts Ġon Ġbinary Ġsequence Ġlabeling Ġfor ĠTask Ġ1 Ġand Ġon Ġmulti - class Ġclassification Ġfor ĠTask Ġ2 . </s> Acknowled gments </s> This Ġresearch Ġis Ġpart Ġof Ġthe ĠProp aganda ĠAnalysis ĠProject , Ġwhich Ġis Ġframed Ġwithin Ġthe ĠTan b ih Ġproject . ĠThe ĠTan b ih Ġproject Ġaims Ġto Ġlimit Ġthe Ġeffect Ġof ĠâĢ ľ fake Ġnews âĢ Ŀ , Ġpropaganda , Ġand Ġmedia Ġbias Ġby Ġmaking Ġusers Ġaware Ġof Ġwhat Ġthey Ġare Ġreading , Ġthus Ġpromoting Ġmedia Ġliteracy Ġand Ġcritical Ġthinking , Ġwhich Ġis Ġarguably Ġthe Ġbest Ġway Ġto Ġaddress Ġdisinformation Ġand ĠâĢ ľ fake Ġnews . âĢ Ŀ ĠThe Ġproject Ġis Ġdeveloped Ġin Ġcollaboration Ġbetween Ġthe ĠQatar ĠComputing ĠResearch ĠInstitute Ġ( Q C RI ), ĠHB K U Ġand Ġthe ĠMIT ĠComputer ĠScience Ġand ĠArtificial ĠIntelligence ĠLaboratory Ġ( C SA IL ). </s> The Ġcorpus Ġfor Ġthe Ġtask Ġwas Ġannot ated Ġby ĠA ĠData ĠPro , Ġa Ġcompany Ġthat Ġperforms Ġhigh - quality Ġmanual Ġannotations .