The best ensemble topped the best single model by 0.029 in F1 score on dev (external).
ABSTRACTIVE
They increased F1 Score by 0.029 in Sentence Level Classification, and by 0.044 in Fragment-Level classification
ABSTRACTIVE
FLOAT SELECTED: Table 3: SLC: Scores on Dev (internal) of Fold1 and Dev (external) using different classifiers and features.
Table TABREF10 shows the scores on dev (internal and external) for SLC task. Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation. In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb. Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal). Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets. Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0.35$ is found optimal. We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i.e., 15 models) to obtain predictions for dev (external). We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i.e., 0.673). In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i.e., BERTSentEmb) in the document. If the cosine-similarity is greater than $\lambda \in \lbrace .99, .95\rbrace $, then the current sentence is labeled as propaganda due to repetition. Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess. Finally, we use the configuration of r19 on the test set. The ensemble+ of (r4, r7 r12) was analyzed after test submission. Table TABREF9 (SLC) shows that our submission is ranked at 4th position. FLOAT SELECTED: Table 3: SLC: Scores on Dev (internal) of Fold1 and Dev (external) using different classifiers and features. Table TABREF11 shows the scores on dev (internal and external) for FLC task. Observe that the features (i.e., polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1. We run multi-grained LSTM-CRF without BERTSentEmb (i.e., row III) and with it (i.e., row IV), where the latter improves scores on dev (internal), however not on dev (external). Finally, we perform multi-tasking with another auxiliary task of PFD. Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration. Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external). Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i.e., $|{\mathcal {M}}|=6$ models to obtain predictions on test. Table TABREF9 (FLC) shows that our submission is ranked at 3rd position.
1909.06162
Introduction </s> In Ġthe Ġage Ġof Ġinformation Ġdissemination Ġwithout Ġquality Ġcontrol , Ġit Ġhas Ġenabled Ġmalicious Ġusers Ġto Ġspread Ġmisinformation Ġvia Ġsocial Ġmedia Ġand Ġaim Ġindividual Ġusers Ġwith Ġpropaganda Ġcampaigns Ġto Ġachieve Ġpolitical Ġand Ġfinancial Ġgains Ġas Ġwell Ġas Ġadvance Ġa Ġspecific Ġagenda . ĠOften Ġdisinformation Ġis Ġcomplied Ġin Ġthe Ġtwo Ġmajor Ġforms : Ġfake Ġnews Ġand Ġpropaganda , Ġwhere Ġthey Ġdiffer Ġin Ġthe Ġsense Ġthat Ġthe Ġpropaganda Ġis Ġpossibly Ġbuilt Ġupon Ġtrue Ġinformation Ġ( e . g ., Ġbiased , Ġloaded Ġlanguage , Ġrepetition , Ġetc .). </s> Prior Ġworks ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 Ġin Ġdetecting Ġpropaganda Ġhave Ġfocused Ġprimarily Ġat Ġdocument Ġlevel , Ġtypically Ġlabeling Ġall Ġarticles Ġfrom Ġa Ġpropag and istic Ġnews Ġoutlet Ġas Ġpropaganda Ġand Ġthus , Ġoften Ġnon - prop ag and istic Ġarticles Ġfrom Ġthe Ġoutlet Ġare Ġmis label ed . ĠTo Ġthis Ġend , ĠEM N LP 19 Da San Mart ino Ġfocuses Ġon Ġanalyzing Ġthe Ġuse Ġof Ġpropaganda Ġand Ġdetecting Ġspecific Ġpropag and istic Ġtechniques Ġin Ġnews Ġarticles Ġat Ġsentence Ġand Ġfragment Ġlevel , Ġrespectively Ġand Ġthus , Ġpromotes Ġexplain able ĠAI . ĠFor Ġinstance , Ġthe Ġfollowing Ġtext Ġis Ġa Ġpropaganda Ġof Ġtype Ġ` s log an '. </s> Trump Ġtweeted : Ġ$\ under brace {\ text { ` } ` {\ text tt Ġ{ BU ILD ĠTHE ĠW ALL ! }" }} _{ \ text { s log an }} $ </s> Sh ared ĠTask : ĠThis Ġwork Ġaddresses Ġthe Ġtwo Ġtasks Ġin Ġpropaganda Ġdetection ĠB IB REF 3 Ġof Ġdifferent Ġgran ular ities : Ġ( 1 ) ĠSent ence - level ĠClassification Ġ( S LC ), Ġa Ġbinary Ġclassification Ġthat Ġpredicts Ġwhether Ġa Ġsentence Ġcontains Ġat Ġleast Ġone Ġpropaganda Ġtechnique , Ġand Ġ( 2 ) ĠFrag ment - level ĠClassification Ġ( FL C ), Ġa Ġtoken - level Ġ( multi - label ) Ġclassification Ġthat Ġidentifies Ġboth Ġthe Ġspans Ġand Ġthe Ġtype Ġof Ġpropaganda Ġtechnique ( s ). </s> Contribut ions : Ġ( 1 ) ĠTo Ġaddress ĠS LC , Ġwe Ġdesign Ġan Ġensemble Ġof Ġdifferent Ġclass ifiers Ġbased Ġon ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT , Ġand Ġleverage Ġtransfer Ġlearning Ġbenefits Ġusing Ġthe Ġpre - trained Ġembed d ings / models Ġfrom ĠFast Text Ġand ĠB ERT . ĠWe Ġalso Ġemployed Ġdifferent Ġfeatures Ġsuch Ġas Ġlinguistic Ġ( sent iment , Ġread ability , Ġemotion , Ġpart - of - speech Ġand Ġnamed Ġentity Ġtags , Ġetc .), Ġlayout , Ġtopics , Ġetc . Ġ( 2 ) ĠTo Ġaddress ĠF LC , Ġwe Ġdesign Ġa Ġmulti - task Ġneural Ġsequence Ġtag ger Ġbased Ġon ĠL ST M - CR F Ġand Ġlinguistic Ġfeatures Ġto Ġjointly Ġdetect Ġpropag and istic Ġfragments Ġand Ġits Ġtype . ĠMoreover , Ġwe Ġinvestigate Ġperforming ĠF LC Ġand ĠS LC Ġjointly Ġin Ġa Ġmulti - gran ularity Ġnetwork Ġbased Ġon ĠL ST M - CR F Ġand ĠB ERT . Ġ( 3 ) ĠOur Ġsystem Ġ( MIC - C IS ) Ġis Ġranked Ġ3 rd Ġ( out Ġof Ġ12 Ġparticipants ) Ġand Ġ4 th Ġ( out Ġof Ġ25 Ġparticipants ) Ġin ĠF LC Ġand ĠS LC Ġtasks , Ġrespectively . </s> System ĠDescription Ġ: :: ĠL ingu istic , ĠLayout Ġand ĠTop ical ĠFeatures </s> Some Ġof Ġthe Ġpropaganda Ġtechniques ĠB IB REF 3 Ġinvolve Ġword Ġand Ġphrases Ġthat Ġexpress Ġstrong Ġemotional Ġimplications , Ġexaggeration , Ġminim ization , Ġdoubt , Ġnational Ġfeeling , Ġlabeling Ġ, Ġstereotyp ing , Ġetc . ĠThis Ġinspires Ġus Ġin Ġextracting Ġdifferent Ġfeatures Ġ( Table ĠT AB REF 1 ) Ġincluding Ġthe Ġcomplexity Ġof Ġtext , Ġsentiment , Ġemotion , Ġlex ical Ġ( POS , ĠN ER , Ġetc .), Ġlayout , Ġetc . ĠTo Ġfurther Ġinvestigate , Ġwe Ġuse Ġtopical Ġfeatures Ġ( e . g ., Ġdocument - topic Ġproportion ) ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġat Ġsentence Ġand Ġdocument Ġlevels Ġin Ġorder Ġto Ġdetermine Ġirrelevant Ġthemes , Ġif Ġintroduced Ġto Ġthe Ġissue Ġbeing Ġdiscussed Ġ( e . g ., ĠRed ĠHer ring ). </s> For Ġword Ġand Ġsentence Ġrepresentations , Ġwe Ġuse Ġpre - trained Ġvectors Ġfrom ĠFast Text ĠB IB REF 7 Ġand ĠB ERT ĠB IB REF 8 . </s> System ĠDescription Ġ: :: ĠSent ence - level ĠProp aganda ĠDetection </s> Figure ĠFIG REF 2 Ġ( left ) Ġdescribes Ġthe Ġthree Ġcomponents Ġof Ġour Ġsystem Ġfor ĠS LC Ġtask : Ġfeatures , Ġclass ifiers Ġand Ġensemble . ĠThe Ġarrows Ġfrom Ġfeatures - to - class ifier Ġindicate Ġthat Ġwe Ġinvestigate Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġin Ġthe Ġtwo Ġbinary Ġclass ifiers : ĠLog istic Reg ression Ġand ĠCNN . ĠFor ĠCNN , Ġwe Ġfollow Ġthe Ġarchitecture Ġof ĠD BL P : conf / em nl p / Kim 14 Ġfor Ġsentence - level Ġclassification , Ġinitial izing Ġthe Ġword Ġvectors Ġby ĠFast Text Ġor ĠB ERT . ĠWe Ġconc aten ate Ġfeatures Ġin Ġthe Ġlast Ġhidden Ġlayer Ġbefore Ġclassification . </s> One Ġof Ġour Ġstrong Ġclass ifiers Ġincludes ĠB ERT Ġthat Ġhas Ġachieved Ġstate - of - the - art Ġperformance Ġon Ġmultiple ĠN LP Ġbenchmarks . ĠFollowing ĠD BL P : conf / na acl / Dev lin CL T 19 , Ġwe Ġfine - t une ĠB ERT Ġfor Ġbinary Ġclassification , Ġinitial izing Ġwith Ġa Ġpre - trained Ġmodel Ġ( i . e ., ĠB ERT - base , ĠC ased ). ĠAdditionally , Ġwe Ġapply Ġa Ġdecision Ġfunction Ġsuch Ġthat Ġa Ġsentence Ġis Ġtagged Ġas Ġpropaganda Ġif Ġprediction Ġprobability Ġof Ġthe Ġclass ifier Ġis Ġgreater Ġthan Ġa Ġthreshold Ġ($ \ t au Ġ$ ). ĠWe Ġrelax Ġthe Ġbinary Ġdecision Ġboundary Ġto Ġboost Ġrecall , Ġsimilar Ġto Ġp ank aj gu pta : Cross RE 2019 . </s> En semble Ġof ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT : ĠIn Ġthe Ġfinal Ġcomponent , Ġwe Ġcollect Ġpredictions Ġ( i . e ., Ġpropaganda Ġlabel ) Ġfor Ġeach Ġsentence Ġfrom Ġthe Ġthree Ġ($ \ math cal Ġ{ M } = 3 $ ) Ġclass ifiers Ġand Ġthus , Ġobtain Ġ$\ math cal Ġ{ M } $ Ġnumber Ġof Ġpredictions Ġfor Ġeach Ġsentence . ĠWe Ġexplore Ġtwo Ġensemble Ġstrategies Ġ( Table ĠT AB REF 1 ): Ġmajority - v oting Ġand Ġrelax - v oting Ġto Ġboost Ġprecision Ġand Ġrecall , Ġrespectively . </s> System ĠDescription Ġ: :: ĠFrag ment - level ĠProp aganda ĠDetection </s> Figure ĠFIG REF 2 Ġ( right ) Ġdescribes Ġour Ġsystem Ġfor ĠF LC Ġtask , Ġwhere Ġwe Ġdesign Ġsequence Ġtag gers ĠB IB REF 9 , ĠB IB REF 10 Ġin Ġthree Ġmodes : Ġ( 1 ) ĠL ST M - CR F ĠB IB REF 11 Ġwith Ġword Ġembed d ings Ġ($ w \ _ e $ ) Ġand Ġcharacter Ġembed d ings Ġ$ c \ _ e $, Ġtoken - level Ġfeatures Ġ($ t \ _ f $ ) Ġsuch Ġas Ġpol arity , ĠPOS , ĠN ER , Ġetc . Ġ( 2 ) ĠL ST M - CR F + Multi - grain Ġthat Ġjointly Ġperforms ĠF LC Ġand ĠS LC Ġwith ĠFast Text Word Emb Ġand ĠB ER TS ent Emb , Ġrespectively . ĠHere , Ġwe Ġadd Ġbinary Ġsentence Ġclassification Ġloss Ġto Ġsequence Ġtagging Ġweighted Ġby Ġa Ġfactor Ġof Ġ$\ alpha Ġ$ . Ġ( 3 ) ĠL ST M - CR F + Multi - task Ġthat Ġperforms Ġpropag and istic Ġspan / fr ag ment Ġdetection Ġ( P FD ) Ġand ĠF LC Ġ( fr ag ment Ġdetection Ġ+ Ġ19 - way Ġclassification ). </s> En semble Ġof ĠMulti - grain , ĠMulti - task ĠL ST M - CR F Ġwith ĠB ERT : ĠHere , Ġwe Ġbuild Ġan Ġensemble Ġby Ġconsidering Ġpropag and istic Ġfragments Ġ( and Ġits Ġtype ) Ġfrom Ġeach Ġof Ġthe Ġsequence Ġtag gers . ĠIn Ġdoing Ġso , Ġwe Ġfirst Ġperform Ġmajority Ġvoting Ġat Ġthe Ġfragment Ġlevel Ġfor Ġthe Ġfragment Ġwhere Ġtheir Ġspans Ġexactly Ġoverlap . ĠIn Ġcase Ġof Ġnon - over l apping Ġfragments , Ġwe Ġconsider Ġall . ĠHowever , Ġwhen Ġthe Ġspans Ġoverlap Ġ( though Ġwith Ġthe Ġsame Ġlabel ), Ġwe Ġconsider Ġthe Ġfragment Ġwith Ġthe Ġlargest Ġspan . </s> Exper iments Ġand ĠEvaluation </s> Data : ĠWhile Ġthe ĠS LC Ġtask Ġis Ġbinary , Ġthe ĠF LC Ġconsists Ġof Ġ18 Ġpropaganda Ġtechniques ĠB IB REF 3 . ĠWe Ġsplit Ġ( 80 - 20 %) Ġthe Ġannot ated Ġcorpus Ġinto Ġ5 - f olds Ġand Ġ3 - f olds Ġfor ĠS LC Ġand ĠF LC Ġtasks , Ġrespectively . ĠThe Ġdevelopment Ġset Ġof Ġeach Ġthe Ġfolds Ġis Ġrepresented Ġby Ġdev Ġ( internal ); Ġhowever , Ġthe Ġun - annot ated Ġcorpus Ġused Ġin Ġleader board Ġcomparisons Ġby Ġdev Ġ( external ). ĠWe Ġremove Ġempty Ġand Ġsingle Ġtoken Ġsentences Ġafter Ġtoken ization . ĠExperimental ĠSetup : ĠWe Ġuse ĠPy Tor ch Ġframework Ġfor Ġthe Ġpre - trained ĠB ERT Ġmodel Ġ( B ert - base - c ased ), Ġfine - tun ed Ġfor ĠS LC Ġtask . ĠIn Ġthe Ġmulti - gran ularity Ġloss , Ġwe Ġset Ġ$\ alpha Ġ= Ġ0 . 1 $ Ġfor Ġsentence Ġclassification Ġbased Ġon Ġdev Ġ( internal , Ġfold 1 ) Ġscores . ĠWe Ġuse ĠB IO Ġtagging Ġscheme Ġof ĠN ER Ġin ĠF LC Ġtask . ĠFor ĠCNN , Ġwe Ġfollow ĠD BL P : conf / em nl p / Kim 14 Ġwith Ġfilter - s izes Ġof Ġ[ 2 , Ġ3 , Ġ4 , Ġ5 , Ġ6 ], Ġ128 Ġfilters Ġand Ġ16 Ġbatch - size . ĠWe Ġcompute Ġbinary - F 1 and Ġmacro - F 1 ĠB IB REF 12 Ġin ĠS LC Ġand ĠF LC , Ġrespectively Ġon Ġdev Ġ( internal ). </s> Exper iments Ġand ĠEvaluation Ġ: :: ĠResults : ĠSent ence - Level ĠProp aganda </s> Table ĠT AB REF 10 Ġshows Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġfor ĠS LC Ġtask . ĠObs erve Ġthat Ġthe Ġpre - trained Ġembed d ings Ġ( Fast Text Ġor ĠB ERT ) Ġoutper form ĠTF - ID F Ġvector Ġrepresentation . ĠIn Ġrow Ġr 2 , Ġwe Ġapply Ġlog istic Ġregression Ġclass ifier Ġwith ĠB ER TS ent Emb Ġthat Ġleads Ġto Ġimproved Ġscores Ġover ĠFast Text Sent Emb . ĠSub sequently , Ġwe Ġaugment Ġthe Ġsentence Ġvector Ġwith Ġadditional Ġfeatures Ġthat Ġimproves ĠF 1 Ġon Ġdev Ġ( external ), Ġhowever Ġnot Ġdev Ġ( internal ). ĠNext , Ġwe Ġinitialize ĠCNN Ġby ĠFast Text Word Emb Ġor ĠB ERT Word Emb Ġand Ġaugment Ġthe Ġlast Ġhidden Ġlayer Ġ( before Ġclassification ) Ġwith ĠB ER TS ent Emb Ġand Ġfeature Ġvectors , Ġleading Ġto Ġgains Ġin ĠF 1 Ġfor Ġboth Ġthe Ġdev Ġsets . ĠFurther , Ġwe Ġfine - t une ĠB ERT Ġand Ġapply Ġdifferent Ġthresholds Ġin Ġrelaxing Ġthe Ġdecision Ġboundary , Ġwhere Ġ$\ t au Ġ\ ge Ġ0 . 35 $ Ġis Ġfound Ġoptimal . </s> We Ġchoose Ġthe Ġthree Ġdifferent Ġmodels Ġin Ġthe Ġensemble : ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT Ġon Ġfold 1 Ġand Ġsubsequently Ġan Ġensemble + Ġof Ġr 3 , Ġr 6 Ġand Ġr 12 Ġfrom Ġeach Ġfold 1 - 5 Ġ( i . e ., Ġ15 Ġmodels ) Ġto Ġobtain Ġpredictions Ġfor Ġdev Ġ( external ). ĠWe Ġinvestigate Ġdifferent Ġensemble Ġschemes Ġ( r 17 - r 19 ), Ġwhere Ġwe Ġobserve Ġthat Ġthe Ġrelax - v oting Ġimproves Ġrecall Ġand Ġtherefore , Ġthe Ġhigher ĠF 1 Ġ( i . e ., Ġ0 . 673 ). ĠIn Ġpost process Ġstep , Ġwe Ġcheck Ġfor Ġrepetition Ġpropaganda Ġtechnique Ġby Ġcomputing Ġcos ine Ġsimilarity Ġbetween Ġthe Ġcurrent Ġsentence Ġand Ġits Ġpreceding Ġ$ w = 10 $ Ġsentence Ġvectors Ġ( i . e ., ĠB ER TS ent Emb ) Ġin Ġthe Ġdocument . ĠIf Ġthe Ġcos ine - similar ity Ġis Ġgreater Ġthan Ġ$\ lambda Ġ\ in Ġ\ l brace Ġ. 99 , Ġ. 95 \ r brace Ġ$ , Ġthen Ġthe Ġcurrent Ġsentence Ġis Ġlabeled Ġas Ġpropaganda Ġdue Ġto Ġrepetition . ĠComp aring Ġr 19 Ġand Ġr 21 , Ġwe Ġobserve Ġa Ġgain Ġin Ġrecall , Ġhowever Ġan Ġoverall Ġdecrease Ġin ĠF 1 Ġapplying Ġpost process . </s> Finally , Ġwe Ġuse Ġthe Ġconfiguration Ġof Ġr 19 Ġon Ġthe Ġtest Ġset . ĠThe Ġensemble + Ġof Ġ( r 4 , Ġr 7 Ġr 12 ) Ġwas Ġanalyzed Ġafter Ġtest Ġsubmission . ĠTable ĠT AB REF 9 Ġ( S LC ) Ġshows Ġthat Ġour Ġsubmission Ġis Ġranked Ġat Ġ4 th Ġposition . </s> Exper iments Ġand ĠEvaluation Ġ: :: ĠResults : ĠFrag ment - Level ĠProp aganda </s> Table ĠT AB REF 11 Ġshows Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġfor ĠF LC Ġtask . ĠObs erve Ġthat Ġthe Ġfeatures Ġ( i . e ., Ġpol arity , ĠPOS Ġand ĠN ER Ġin Ġrow ĠII ) Ġwhen Ġintroduced Ġin ĠL ST M - CR F Ġimproves ĠF 1 . ĠWe Ġrun Ġmulti - gr ained ĠL ST M - CR F Ġwithout ĠB ER TS ent Emb Ġ( i . e ., Ġrow ĠIII ) Ġand Ġwith Ġit Ġ( i . e ., Ġrow ĠIV ), Ġwhere Ġthe Ġlatter Ġimproves Ġscores Ġon Ġdev Ġ( internal ), Ġhowever Ġnot Ġon Ġdev Ġ( external ). ĠFinally , Ġwe Ġperform Ġmulti - t asking Ġwith Ġanother Ġauxiliary Ġtask Ġof ĠP FD . ĠGiven Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġusing Ġdifferent Ġconfigurations Ġ( rows ĠI - V ), Ġit Ġis Ġdifficult Ġto Ġinfer Ġthe Ġoptimal Ġconfiguration . ĠThus , Ġwe Ġchoose Ġthe Ġtwo Ġbest Ġconfigurations Ġ( II Ġand ĠIV ) Ġon Ġdev Ġ( internal ) Ġset Ġand Ġbuild Ġan Ġensemble + Ġof Ġpredictions Ġ( disc ussed Ġin Ġsection ĠSEC REF 6 ), Ġleading Ġto Ġa Ġboost Ġin Ġrecall Ġand Ġthus Ġan Ġimproved ĠF 1 Ġon Ġdev Ġ( external ). </s> Finally , Ġwe Ġuse Ġthe Ġensemble + Ġof Ġ( II Ġand ĠIV ) Ġfrom Ġeach Ġof Ġthe Ġfolds Ġ1 - 3 , Ġi . e ., Ġ$ | {\ math cal Ġ{ M }} | = 6 $ Ġmodels Ġto Ġobtain Ġpredictions Ġon Ġtest . ĠTable ĠT AB REF 9 Ġ( FL C ) Ġshows Ġthat Ġour Ġsubmission Ġis Ġranked Ġat Ġ3 rd Ġposition . </s> Conclusion Ġand ĠFuture ĠWork </s> Our Ġsystem Ġ( Team : ĠMIC - C IS ) Ġexplores Ġdifferent Ġneural Ġarchitectures Ġ( CNN , ĠB ERT Ġand ĠL ST M - CR F ) Ġwith Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġto Ġaddress Ġthe Ġtasks Ġof Ġfine - gr ained Ġpropaganda Ġdetection . ĠWe Ġhave Ġdemonstrated Ġgains Ġin Ġperformance Ġdue Ġto Ġthe Ġfeatures , Ġensemble Ġschemes , Ġmulti - t asking Ġand Ġmulti - gran ularity Ġarchitectures . ĠCompared Ġto Ġthe Ġother Ġparticipating Ġsystems , Ġour Ġsubmissions Ġare Ġranked Ġ3 rd Ġand Ġ4 th Ġin ĠF LC Ġand ĠS LC Ġtasks , Ġrespectively . </s> In Ġfuture , Ġwe Ġwould Ġlike Ġto Ġenrich ĠB ERT Ġmodels Ġwith Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġduring Ġtheir Ġfine - tun ing . ĠFurther , Ġwe Ġwould Ġalso Ġbe Ġinterested Ġin Ġunderstanding Ġand Ġanalyzing Ġthe Ġneural Ġnetwork Ġlearning , Ġi . e ., Ġextracting Ġsalient Ġfragments Ġ( or Ġkey - ph r ases ) Ġin Ġthe Ġsentence Ġthat Ġgenerate Ġpropaganda , Ġsimilar Ġto Ġp ank aj gu pta : 2018 L ISA Ġin Ġorder Ġto Ġpromote Ġexplain able ĠAI .
Introduction In the age of information dissemination without quality control, it has enabled malicious users to spread misinformation via social media and aim individual users with propaganda campaigns to achieve political and financial gains as well as advance a specific agenda. Often disinformation is complied in the two major forms: fake news and propaganda, where they differ in the sense that the propaganda is possibly built upon true information (e.g., biased, loaded language, repetition, etc.). Prior works BIBREF0, BIBREF1, BIBREF2 in detecting propaganda have focused primarily at document level, typically labeling all articles from a propagandistic news outlet as propaganda and thus, often non-propagandistic articles from the outlet are mislabeled. To this end, EMNLP19DaSanMartino focuses on analyzing the use of propaganda and detecting specific propagandistic techniques in news articles at sentence and fragment level, respectively and thus, promotes explainable AI. For instance, the following text is a propaganda of type `slogan'. Trump tweeted: $\underbrace{\text{`}`{\texttt {BUILD THE WALL!}"}}_{\text{slogan}}$ Shared Task: This work addresses the two tasks in propaganda detection BIBREF3 of different granularities: (1) Sentence-level Classification (SLC), a binary classification that predicts whether a sentence contains at least one propaganda technique, and (2) Fragment-level Classification (FLC), a token-level (multi-label) classification that identifies both the spans and the type of propaganda technique(s). Contributions: (1) To address SLC, we design an ensemble of different classifiers based on Logistic Regression, CNN and BERT, and leverage transfer learning benefits using the pre-trained embeddings/models from FastText and BERT. We also employed different features such as linguistic (sentiment, readability, emotion, part-of-speech and named entity tags, etc.), layout, topics, etc. (2) To address FLC, we design a multi-task neural sequence tagger based on LSTM-CRF and linguistic features to jointly detect propagandistic fragments and its type. Moreover, we investigate performing FLC and SLC jointly in a multi-granularity network based on LSTM-CRF and BERT. (3) Our system (MIC-CIS) is ranked 3rd (out of 12 participants) and 4th (out of 25 participants) in FLC and SLC tasks, respectively. System Description ::: Linguistic, Layout and Topical Features Some of the propaganda techniques BIBREF3 involve word and phrases that express strong emotional implications, exaggeration, minimization, doubt, national feeling, labeling , stereotyping, etc. This inspires us in extracting different features (Table TABREF1) including the complexity of text, sentiment, emotion, lexical (POS, NER, etc.), layout, etc. To further investigate, we use topical features (e.g., document-topic proportion) BIBREF4, BIBREF5, BIBREF6 at sentence and document levels in order to determine irrelevant themes, if introduced to the issue being discussed (e.g., Red Herring). For word and sentence representations, we use pre-trained vectors from FastText BIBREF7 and BERT BIBREF8. System Description ::: Sentence-level Propaganda Detection Figure FIGREF2 (left) describes the three components of our system for SLC task: features, classifiers and ensemble. The arrows from features-to-classifier indicate that we investigate linguistic, layout and topical features in the two binary classifiers: LogisticRegression and CNN. For CNN, we follow the architecture of DBLP:conf/emnlp/Kim14 for sentence-level classification, initializing the word vectors by FastText or BERT. We concatenate features in the last hidden layer before classification. One of our strong classifiers includes BERT that has achieved state-of-the-art performance on multiple NLP benchmarks. Following DBLP:conf/naacl/DevlinCLT19, we fine-tune BERT for binary classification, initializing with a pre-trained model (i.e., BERT-base, Cased). Additionally, we apply a decision function such that a sentence is tagged as propaganda if prediction probability of the classifier is greater than a threshold ($\tau $). We relax the binary decision boundary to boost recall, similar to pankajgupta:CrossRE2019. Ensemble of Logistic Regression, CNN and BERT: In the final component, we collect predictions (i.e., propaganda label) for each sentence from the three ($\mathcal {M}=3$) classifiers and thus, obtain $\mathcal {M}$ number of predictions for each sentence. We explore two ensemble strategies (Table TABREF1): majority-voting and relax-voting to boost precision and recall, respectively. System Description ::: Fragment-level Propaganda Detection Figure FIGREF2 (right) describes our system for FLC task, where we design sequence taggers BIBREF9, BIBREF10 in three modes: (1) LSTM-CRF BIBREF11 with word embeddings ($w\_e$) and character embeddings $c\_e$, token-level features ($t\_f$) such as polarity, POS, NER, etc. (2) LSTM-CRF+Multi-grain that jointly performs FLC and SLC with FastTextWordEmb and BERTSentEmb, respectively. Here, we add binary sentence classification loss to sequence tagging weighted by a factor of $\alpha $. (3) LSTM-CRF+Multi-task that performs propagandistic span/fragment detection (PFD) and FLC (fragment detection + 19-way classification). Ensemble of Multi-grain, Multi-task LSTM-CRF with BERT: Here, we build an ensemble by considering propagandistic fragments (and its type) from each of the sequence taggers. In doing so, we first perform majority voting at the fragment level for the fragment where their spans exactly overlap. In case of non-overlapping fragments, we consider all. However, when the spans overlap (though with the same label), we consider the fragment with the largest span. Experiments and Evaluation Data: While the SLC task is binary, the FLC consists of 18 propaganda techniques BIBREF3. We split (80-20%) the annotated corpus into 5-folds and 3-folds for SLC and FLC tasks, respectively. The development set of each the folds is represented by dev (internal); however, the un-annotated corpus used in leaderboard comparisons by dev (external). We remove empty and single token sentences after tokenization. Experimental Setup: We use PyTorch framework for the pre-trained BERT model (Bert-base-cased), fine-tuned for SLC task. In the multi-granularity loss, we set $\alpha = 0.1$ for sentence classification based on dev (internal, fold1) scores. We use BIO tagging scheme of NER in FLC task. For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size. We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal). Experiments and Evaluation ::: Results: Sentence-Level Propaganda Table TABREF10 shows the scores on dev (internal and external) for SLC task. Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation. In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb. Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal). Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets. Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0.35$ is found optimal. We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i.e., 15 models) to obtain predictions for dev (external). We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i.e., 0.673). In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i.e., BERTSentEmb) in the document. If the cosine-similarity is greater than $\lambda \in \lbrace .99, .95\rbrace $, then the current sentence is labeled as propaganda due to repetition. Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess. Finally, we use the configuration of r19 on the test set. The ensemble+ of (r4, r7 r12) was analyzed after test submission. Table TABREF9 (SLC) shows that our submission is ranked at 4th position. Experiments and Evaluation ::: Results: Fragment-Level Propaganda Table TABREF11 shows the scores on dev (internal and external) for FLC task. Observe that the features (i.e., polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1. We run multi-grained LSTM-CRF without BERTSentEmb (i.e., row III) and with it (i.e., row IV), where the latter improves scores on dev (internal), however not on dev (external). Finally, we perform multi-tasking with another auxiliary task of PFD. Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration. Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external). Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i.e., $|{\mathcal {M}}|=6$ models to obtain predictions on test. Table TABREF9 (FLC) shows that our submission is ranked at 3rd position. Conclusion and Future Work Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection. We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures. Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively. In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning. Further, we would also be interested in understanding and analyzing the neural network learning, i.e., extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI.
Did ensemble schemes help in boosting peformance, by how much?
2268c9044e868ba0a16e92d2063ada87f68b5d03
Did Ġensemble Ġschemes Ġhelp Ġin Ġboosting Ġpe formance , Ġby Ġhow Ġmuch ?
The Ġbest Ġensemble Ġtopped Ġthe Ġbest Ġsingle Ġmodel Ġby Ġ0 . 029 Ġin ĠF 1 Ġscore Ġon Ġdev Ġ( external ).
<s> Did Ġensemble Ġschemes Ġhelp Ġin Ġboosting Ġpe formance , Ġby Ġhow Ġmuch ? </s> Introduction </s> In Ġthe Ġage Ġof Ġinformation Ġdissemination Ġwithout Ġquality Ġcontrol , Ġit Ġhas Ġenabled Ġmalicious Ġusers Ġto Ġspread Ġmisinformation Ġvia Ġsocial Ġmedia Ġand Ġaim Ġindividual Ġusers Ġwith Ġpropaganda Ġcampaigns Ġto Ġachieve Ġpolitical Ġand Ġfinancial Ġgains Ġas Ġwell Ġas Ġadvance Ġa Ġspecific Ġagenda . ĠOften Ġdisinformation Ġis Ġcomplied Ġin Ġthe Ġtwo Ġmajor Ġforms : Ġfake Ġnews Ġand Ġpropaganda , Ġwhere Ġthey Ġdiffer Ġin Ġthe Ġsense Ġthat Ġthe Ġpropaganda Ġis Ġpossibly Ġbuilt Ġupon Ġtrue Ġinformation Ġ( e . g ., Ġbiased , Ġloaded Ġlanguage , Ġrepetition , Ġetc .). </s> Prior Ġworks ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 Ġin Ġdetecting Ġpropaganda Ġhave Ġfocused Ġprimarily Ġat Ġdocument Ġlevel , Ġtypically Ġlabeling Ġall Ġarticles Ġfrom Ġa Ġpropag and istic Ġnews Ġoutlet Ġas Ġpropaganda Ġand Ġthus , Ġoften Ġnon - prop ag and istic Ġarticles Ġfrom Ġthe Ġoutlet Ġare Ġmis label ed . ĠTo Ġthis Ġend , ĠEM N LP 19 Da San Mart ino Ġfocuses Ġon Ġanalyzing Ġthe Ġuse Ġof Ġpropaganda Ġand Ġdetecting Ġspecific Ġpropag and istic Ġtechniques Ġin Ġnews Ġarticles Ġat Ġsentence Ġand Ġfragment Ġlevel , Ġrespectively Ġand Ġthus , Ġpromotes Ġexplain able ĠAI . ĠFor Ġinstance , Ġthe Ġfollowing Ġtext Ġis Ġa Ġpropaganda Ġof Ġtype Ġ` s log an '. </s> Trump Ġtweeted : Ġ$\ under brace {\ text { ` } ` {\ text tt Ġ{ BU ILD ĠTHE ĠW ALL ! }" }} _{ \ text { s log an }} $ </s> Sh ared ĠTask : ĠThis Ġwork Ġaddresses Ġthe Ġtwo Ġtasks Ġin Ġpropaganda Ġdetection ĠB IB REF 3 Ġof Ġdifferent Ġgran ular ities : Ġ( 1 ) ĠSent ence - level ĠClassification Ġ( S LC ), Ġa Ġbinary Ġclassification Ġthat Ġpredicts Ġwhether Ġa Ġsentence Ġcontains Ġat Ġleast Ġone Ġpropaganda Ġtechnique , Ġand Ġ( 2 ) ĠFrag ment - level ĠClassification Ġ( FL C ), Ġa Ġtoken - level Ġ( multi - label ) Ġclassification Ġthat Ġidentifies Ġboth Ġthe Ġspans Ġand Ġthe Ġtype Ġof Ġpropaganda Ġtechnique ( s ). </s> Contribut ions : Ġ( 1 ) ĠTo Ġaddress ĠS LC , Ġwe Ġdesign Ġan Ġensemble Ġof Ġdifferent Ġclass ifiers Ġbased Ġon ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT , Ġand Ġleverage Ġtransfer Ġlearning Ġbenefits Ġusing Ġthe Ġpre - trained Ġembed d ings / models Ġfrom ĠFast Text Ġand ĠB ERT . ĠWe Ġalso Ġemployed Ġdifferent Ġfeatures Ġsuch Ġas Ġlinguistic Ġ( sent iment , Ġread ability , Ġemotion , Ġpart - of - speech Ġand Ġnamed Ġentity Ġtags , Ġetc .), Ġlayout , Ġtopics , Ġetc . Ġ( 2 ) ĠTo Ġaddress ĠF LC , Ġwe Ġdesign Ġa Ġmulti - task Ġneural Ġsequence Ġtag ger Ġbased Ġon ĠL ST M - CR F Ġand Ġlinguistic Ġfeatures Ġto Ġjointly Ġdetect Ġpropag and istic Ġfragments Ġand Ġits Ġtype . ĠMoreover , Ġwe Ġinvestigate Ġperforming ĠF LC Ġand ĠS LC Ġjointly Ġin Ġa Ġmulti - gran ularity Ġnetwork Ġbased Ġon ĠL ST M - CR F Ġand ĠB ERT . Ġ( 3 ) ĠOur Ġsystem Ġ( MIC - C IS ) Ġis Ġranked Ġ3 rd Ġ( out Ġof Ġ12 Ġparticipants ) Ġand Ġ4 th Ġ( out Ġof Ġ25 Ġparticipants ) Ġin ĠF LC Ġand ĠS LC Ġtasks , Ġrespectively . </s> System ĠDescription Ġ: :: ĠL ingu istic , ĠLayout Ġand ĠTop ical ĠFeatures </s> Some Ġof Ġthe Ġpropaganda Ġtechniques ĠB IB REF 3 Ġinvolve Ġword Ġand Ġphrases Ġthat Ġexpress Ġstrong Ġemotional Ġimplications , Ġexaggeration , Ġminim ization , Ġdoubt , Ġnational Ġfeeling , Ġlabeling Ġ, Ġstereotyp ing , Ġetc . ĠThis Ġinspires Ġus Ġin Ġextracting Ġdifferent Ġfeatures Ġ( Table ĠT AB REF 1 ) Ġincluding Ġthe Ġcomplexity Ġof Ġtext , Ġsentiment , Ġemotion , Ġlex ical Ġ( POS , ĠN ER , Ġetc .), Ġlayout , Ġetc . ĠTo Ġfurther Ġinvestigate , Ġwe Ġuse Ġtopical Ġfeatures Ġ( e . g ., Ġdocument - topic Ġproportion ) ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġat Ġsentence Ġand Ġdocument Ġlevels Ġin Ġorder Ġto Ġdetermine Ġirrelevant Ġthemes , Ġif Ġintroduced Ġto Ġthe Ġissue Ġbeing Ġdiscussed Ġ( e . g ., ĠRed ĠHer ring ). </s> For Ġword Ġand Ġsentence Ġrepresentations , Ġwe Ġuse Ġpre - trained Ġvectors Ġfrom ĠFast Text ĠB IB REF 7 Ġand ĠB ERT ĠB IB REF 8 . </s> System ĠDescription Ġ: :: ĠSent ence - level ĠProp aganda ĠDetection </s> Figure ĠFIG REF 2 Ġ( left ) Ġdescribes Ġthe Ġthree Ġcomponents Ġof Ġour Ġsystem Ġfor ĠS LC Ġtask : Ġfeatures , Ġclass ifiers Ġand Ġensemble . ĠThe Ġarrows Ġfrom Ġfeatures - to - class ifier Ġindicate Ġthat Ġwe Ġinvestigate Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġin Ġthe Ġtwo Ġbinary Ġclass ifiers : ĠLog istic Reg ression Ġand ĠCNN . ĠFor ĠCNN , Ġwe Ġfollow Ġthe Ġarchitecture Ġof ĠD BL P : conf / em nl p / Kim 14 Ġfor Ġsentence - level Ġclassification , Ġinitial izing Ġthe Ġword Ġvectors Ġby ĠFast Text Ġor ĠB ERT . ĠWe Ġconc aten ate Ġfeatures Ġin Ġthe Ġlast Ġhidden Ġlayer Ġbefore Ġclassification . </s> One Ġof Ġour Ġstrong Ġclass ifiers Ġincludes ĠB ERT Ġthat Ġhas Ġachieved Ġstate - of - the - art Ġperformance Ġon Ġmultiple ĠN LP Ġbenchmarks . ĠFollowing ĠD BL P : conf / na acl / Dev lin CL T 19 , Ġwe Ġfine - t une ĠB ERT Ġfor Ġbinary Ġclassification , Ġinitial izing Ġwith Ġa Ġpre - trained Ġmodel Ġ( i . e ., ĠB ERT - base , ĠC ased ). ĠAdditionally , Ġwe Ġapply Ġa Ġdecision Ġfunction Ġsuch Ġthat Ġa Ġsentence Ġis Ġtagged Ġas Ġpropaganda Ġif Ġprediction Ġprobability Ġof Ġthe Ġclass ifier Ġis Ġgreater Ġthan Ġa Ġthreshold Ġ($ \ t au Ġ$ ). ĠWe Ġrelax Ġthe Ġbinary Ġdecision Ġboundary Ġto Ġboost Ġrecall , Ġsimilar Ġto Ġp ank aj gu pta : Cross RE 2019 . </s> En semble Ġof ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT : ĠIn Ġthe Ġfinal Ġcomponent , Ġwe Ġcollect Ġpredictions Ġ( i . e ., Ġpropaganda Ġlabel ) Ġfor Ġeach Ġsentence Ġfrom Ġthe Ġthree Ġ($ \ math cal Ġ{ M } = 3 $ ) Ġclass ifiers Ġand Ġthus , Ġobtain Ġ$\ math cal Ġ{ M } $ Ġnumber Ġof Ġpredictions Ġfor Ġeach Ġsentence . ĠWe Ġexplore Ġtwo Ġensemble Ġstrategies Ġ( Table ĠT AB REF 1 ): Ġmajority - v oting Ġand Ġrelax - v oting Ġto Ġboost Ġprecision Ġand Ġrecall , Ġrespectively . </s> System ĠDescription Ġ: :: ĠFrag ment - level ĠProp aganda ĠDetection </s> Figure ĠFIG REF 2 Ġ( right ) Ġdescribes Ġour Ġsystem Ġfor ĠF LC Ġtask , Ġwhere Ġwe Ġdesign Ġsequence Ġtag gers ĠB IB REF 9 , ĠB IB REF 10 Ġin Ġthree Ġmodes : Ġ( 1 ) ĠL ST M - CR F ĠB IB REF 11 Ġwith Ġword Ġembed d ings Ġ($ w \ _ e $ ) Ġand Ġcharacter Ġembed d ings Ġ$ c \ _ e $, Ġtoken - level Ġfeatures Ġ($ t \ _ f $ ) Ġsuch Ġas Ġpol arity , ĠPOS , ĠN ER , Ġetc . Ġ( 2 ) ĠL ST M - CR F + Multi - grain Ġthat Ġjointly Ġperforms ĠF LC Ġand ĠS LC Ġwith ĠFast Text Word Emb Ġand ĠB ER TS ent Emb , Ġrespectively . ĠHere , Ġwe Ġadd Ġbinary Ġsentence Ġclassification Ġloss Ġto Ġsequence Ġtagging Ġweighted Ġby Ġa Ġfactor Ġof Ġ$\ alpha Ġ$ . Ġ( 3 ) ĠL ST M - CR F + Multi - task Ġthat Ġperforms Ġpropag and istic Ġspan / fr ag ment Ġdetection Ġ( P FD ) Ġand ĠF LC Ġ( fr ag ment Ġdetection Ġ+ Ġ19 - way Ġclassification ). </s> En semble Ġof ĠMulti - grain , ĠMulti - task ĠL ST M - CR F Ġwith ĠB ERT : ĠHere , Ġwe Ġbuild Ġan Ġensemble Ġby Ġconsidering Ġpropag and istic Ġfragments Ġ( and Ġits Ġtype ) Ġfrom Ġeach Ġof Ġthe Ġsequence Ġtag gers . ĠIn Ġdoing Ġso , Ġwe Ġfirst Ġperform Ġmajority Ġvoting Ġat Ġthe Ġfragment Ġlevel Ġfor Ġthe Ġfragment Ġwhere Ġtheir Ġspans Ġexactly Ġoverlap . ĠIn Ġcase Ġof Ġnon - over l apping Ġfragments , Ġwe Ġconsider Ġall . ĠHowever , Ġwhen Ġthe Ġspans Ġoverlap Ġ( though Ġwith Ġthe Ġsame Ġlabel ), Ġwe Ġconsider Ġthe Ġfragment Ġwith Ġthe Ġlargest Ġspan . </s> Exper iments Ġand ĠEvaluation </s> Data : ĠWhile Ġthe ĠS LC Ġtask Ġis Ġbinary , Ġthe ĠF LC Ġconsists Ġof Ġ18 Ġpropaganda Ġtechniques ĠB IB REF 3 . ĠWe Ġsplit Ġ( 80 - 20 %) Ġthe Ġannot ated Ġcorpus Ġinto Ġ5 - f olds Ġand Ġ3 - f olds Ġfor ĠS LC Ġand ĠF LC Ġtasks , Ġrespectively . ĠThe Ġdevelopment Ġset Ġof Ġeach Ġthe Ġfolds Ġis Ġrepresented Ġby Ġdev Ġ( internal ); Ġhowever , Ġthe Ġun - annot ated Ġcorpus Ġused Ġin Ġleader board Ġcomparisons Ġby Ġdev Ġ( external ). ĠWe Ġremove Ġempty Ġand Ġsingle Ġtoken Ġsentences Ġafter Ġtoken ization . ĠExperimental ĠSetup : ĠWe Ġuse ĠPy Tor ch Ġframework Ġfor Ġthe Ġpre - trained ĠB ERT Ġmodel Ġ( B ert - base - c ased ), Ġfine - tun ed Ġfor ĠS LC Ġtask . ĠIn Ġthe Ġmulti - gran ularity Ġloss , Ġwe Ġset Ġ$\ alpha Ġ= Ġ0 . 1 $ Ġfor Ġsentence Ġclassification Ġbased Ġon Ġdev Ġ( internal , Ġfold 1 ) Ġscores . ĠWe Ġuse ĠB IO Ġtagging Ġscheme Ġof ĠN ER Ġin ĠF LC Ġtask . ĠFor ĠCNN , Ġwe Ġfollow ĠD BL P : conf / em nl p / Kim 14 Ġwith Ġfilter - s izes Ġof Ġ[ 2 , Ġ3 , Ġ4 , Ġ5 , Ġ6 ], Ġ128 Ġfilters Ġand Ġ16 Ġbatch - size . ĠWe Ġcompute Ġbinary - F 1 and Ġmacro - F 1 ĠB IB REF 12 Ġin ĠS LC Ġand ĠF LC , Ġrespectively Ġon Ġdev Ġ( internal ). </s> Exper iments Ġand ĠEvaluation Ġ: :: ĠResults : ĠSent ence - Level ĠProp aganda </s> Table ĠT AB REF 10 Ġshows Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġfor ĠS LC Ġtask . ĠObs erve Ġthat Ġthe Ġpre - trained Ġembed d ings Ġ( Fast Text Ġor ĠB ERT ) Ġoutper form ĠTF - ID F Ġvector Ġrepresentation . ĠIn Ġrow Ġr 2 , Ġwe Ġapply Ġlog istic Ġregression Ġclass ifier Ġwith ĠB ER TS ent Emb Ġthat Ġleads Ġto Ġimproved Ġscores Ġover ĠFast Text Sent Emb . ĠSub sequently , Ġwe Ġaugment Ġthe Ġsentence Ġvector Ġwith Ġadditional Ġfeatures Ġthat Ġimproves ĠF 1 Ġon Ġdev Ġ( external ), Ġhowever Ġnot Ġdev Ġ( internal ). ĠNext , Ġwe Ġinitialize ĠCNN Ġby ĠFast Text Word Emb Ġor ĠB ERT Word Emb Ġand Ġaugment Ġthe Ġlast Ġhidden Ġlayer Ġ( before Ġclassification ) Ġwith ĠB ER TS ent Emb Ġand Ġfeature Ġvectors , Ġleading Ġto Ġgains Ġin ĠF 1 Ġfor Ġboth Ġthe Ġdev Ġsets . ĠFurther , Ġwe Ġfine - t une ĠB ERT Ġand Ġapply Ġdifferent Ġthresholds Ġin Ġrelaxing Ġthe Ġdecision Ġboundary , Ġwhere Ġ$\ t au Ġ\ ge Ġ0 . 35 $ Ġis Ġfound Ġoptimal . </s> We Ġchoose Ġthe Ġthree Ġdifferent Ġmodels Ġin Ġthe Ġensemble : ĠLog istic ĠReg ression , ĠCNN Ġand ĠB ERT Ġon Ġfold 1 Ġand Ġsubsequently Ġan Ġensemble + Ġof Ġr 3 , Ġr 6 Ġand Ġr 12 Ġfrom Ġeach Ġfold 1 - 5 Ġ( i . e ., Ġ15 Ġmodels ) Ġto Ġobtain Ġpredictions Ġfor Ġdev Ġ( external ). ĠWe Ġinvestigate Ġdifferent Ġensemble Ġschemes Ġ( r 17 - r 19 ), Ġwhere Ġwe Ġobserve Ġthat Ġthe Ġrelax - v oting Ġimproves Ġrecall Ġand Ġtherefore , Ġthe Ġhigher ĠF 1 Ġ( i . e ., Ġ0 . 673 ). ĠIn Ġpost process Ġstep , Ġwe Ġcheck Ġfor Ġrepetition Ġpropaganda Ġtechnique Ġby Ġcomputing Ġcos ine Ġsimilarity Ġbetween Ġthe Ġcurrent Ġsentence Ġand Ġits Ġpreceding Ġ$ w = 10 $ Ġsentence Ġvectors Ġ( i . e ., ĠB ER TS ent Emb ) Ġin Ġthe Ġdocument . ĠIf Ġthe Ġcos ine - similar ity Ġis Ġgreater Ġthan Ġ$\ lambda Ġ\ in Ġ\ l brace Ġ. 99 , Ġ. 95 \ r brace Ġ$ , Ġthen Ġthe Ġcurrent Ġsentence Ġis Ġlabeled Ġas Ġpropaganda Ġdue Ġto Ġrepetition . ĠComp aring Ġr 19 Ġand Ġr 21 , Ġwe Ġobserve Ġa Ġgain Ġin Ġrecall , Ġhowever Ġan Ġoverall Ġdecrease Ġin ĠF 1 Ġapplying Ġpost process . </s> Finally , Ġwe Ġuse Ġthe Ġconfiguration Ġof Ġr 19 Ġon Ġthe Ġtest Ġset . ĠThe Ġensemble + Ġof Ġ( r 4 , Ġr 7 Ġr 12 ) Ġwas Ġanalyzed Ġafter Ġtest Ġsubmission . ĠTable ĠT AB REF 9 Ġ( S LC ) Ġshows Ġthat Ġour Ġsubmission Ġis Ġranked Ġat Ġ4 th Ġposition . </s> Exper iments Ġand ĠEvaluation Ġ: :: ĠResults : ĠFrag ment - Level ĠProp aganda </s> Table ĠT AB REF 11 Ġshows Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġfor ĠF LC Ġtask . ĠObs erve Ġthat Ġthe Ġfeatures Ġ( i . e ., Ġpol arity , ĠPOS Ġand ĠN ER Ġin Ġrow ĠII ) Ġwhen Ġintroduced Ġin ĠL ST M - CR F Ġimproves ĠF 1 . ĠWe Ġrun Ġmulti - gr ained ĠL ST M - CR F Ġwithout ĠB ER TS ent Emb Ġ( i . e ., Ġrow ĠIII ) Ġand Ġwith Ġit Ġ( i . e ., Ġrow ĠIV ), Ġwhere Ġthe Ġlatter Ġimproves Ġscores Ġon Ġdev Ġ( internal ), Ġhowever Ġnot Ġon Ġdev Ġ( external ). ĠFinally , Ġwe Ġperform Ġmulti - t asking Ġwith Ġanother Ġauxiliary Ġtask Ġof ĠP FD . ĠGiven Ġthe Ġscores Ġon Ġdev Ġ( internal Ġand Ġexternal ) Ġusing Ġdifferent Ġconfigurations Ġ( rows ĠI - V ), Ġit Ġis Ġdifficult Ġto Ġinfer Ġthe Ġoptimal Ġconfiguration . ĠThus , Ġwe Ġchoose Ġthe Ġtwo Ġbest Ġconfigurations Ġ( II Ġand ĠIV ) Ġon Ġdev Ġ( internal ) Ġset Ġand Ġbuild Ġan Ġensemble + Ġof Ġpredictions Ġ( disc ussed Ġin Ġsection ĠSEC REF 6 ), Ġleading Ġto Ġa Ġboost Ġin Ġrecall Ġand Ġthus Ġan Ġimproved ĠF 1 Ġon Ġdev Ġ( external ). </s> Finally , Ġwe Ġuse Ġthe Ġensemble + Ġof Ġ( II Ġand ĠIV ) Ġfrom Ġeach Ġof Ġthe Ġfolds Ġ1 - 3 , Ġi . e ., Ġ$ | {\ math cal Ġ{ M }} | = 6 $ Ġmodels Ġto Ġobtain Ġpredictions Ġon Ġtest . ĠTable ĠT AB REF 9 Ġ( FL C ) Ġshows Ġthat Ġour Ġsubmission Ġis Ġranked Ġat Ġ3 rd Ġposition . </s> Conclusion Ġand ĠFuture ĠWork </s> Our Ġsystem Ġ( Team : ĠMIC - C IS ) Ġexplores Ġdifferent Ġneural Ġarchitectures Ġ( CNN , ĠB ERT Ġand ĠL ST M - CR F ) Ġwith Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġto Ġaddress Ġthe Ġtasks Ġof Ġfine - gr ained Ġpropaganda Ġdetection . ĠWe Ġhave Ġdemonstrated Ġgains Ġin Ġperformance Ġdue Ġto Ġthe Ġfeatures , Ġensemble Ġschemes , Ġmulti - t asking Ġand Ġmulti - gran ularity Ġarchitectures . ĠCompared Ġto Ġthe Ġother Ġparticipating Ġsystems , Ġour Ġsubmissions Ġare Ġranked Ġ3 rd Ġand Ġ4 th Ġin ĠF LC Ġand ĠS LC Ġtasks , Ġrespectively . </s> In Ġfuture , Ġwe Ġwould Ġlike Ġto Ġenrich ĠB ERT Ġmodels Ġwith Ġlinguistic , Ġlayout Ġand Ġtopical Ġfeatures Ġduring Ġtheir Ġfine - tun ing . ĠFurther , Ġwe Ġwould Ġalso Ġbe Ġinterested Ġin Ġunderstanding Ġand Ġanalyzing Ġthe Ġneural Ġnetwork Ġlearning , Ġi . e ., Ġextracting Ġsalient Ġfragments Ġ( or Ġkey - ph r ases ) Ġin Ġthe Ġsentence Ġthat Ġgenerate Ġpropaganda , Ġsimilar Ġto Ġp ank aj gu pta : 2018 L ISA Ġin Ġorder Ġto Ġpromote Ġexplain able ĠAI .