Yes
BOOLEAN
An alternative concept is vertical language consistency, whereby the questions are consistently presented in one language, regardless of the comment. To test whether horizontal or vertical consistency is more helpful, we train and evaluate M-Bert on a dataset variant where all questions are in their English version. We chose English as a lingua franca because it had the largest share of data during the pretraining of M-Bert.
2003.08385
Introduction </s> In Ġrecent Ġyears Ġmany Ġdatasets Ġhave Ġbeen Ġcreated Ġfor Ġthe Ġtask Ġof Ġautomated Ġstance Ġdetection , Ġadvancing Ġnatural Ġlanguage Ġunderstanding Ġsystems Ġfor Ġpolitical Ġscience , Ġopinion Ġresearch Ġand Ġother Ġapplication Ġareas . ĠTypically , Ġsuch Ġbenchmarks ĠB IB REF 0 Ġare Ġcomposed Ġof Ġshort Ġpieces Ġof Ġtext Ġcommenting Ġon Ġpoliticians Ġor Ġpublic Ġissues Ġand Ġare Ġmanually Ġannot ated Ġwith Ġtheir Ġstance Ġtowards Ġa Ġtarget Ġentity Ġ( e . g . ĠClimate ĠChange , Ġor ĠTrump ). ĠHowever , Ġthey Ġare Ġlimited Ġin Ġscope Ġon Ġmultiple Ġlevels ĠB IB REF 1 . </s> First Ġof Ġall , Ġit Ġis Ġquestionable Ġhow Ġwell Ġcurrent Ġstance Ġdetection Ġmethods Ġperform Ġin Ġa Ġcross - ling ual Ġsetting , Ġas Ġthe Ġmult ilingual Ġdatasets Ġavailable Ġtoday Ġare Ġrelatively Ġsmall , Ġand Ġspecific Ġto Ġa Ġsingle Ġtarget ĠB IB REF 2 , ĠB IB REF 3 . ĠFurthermore , Ġspecific Ġmodels Ġtend Ġto Ġbe Ġdeveloped Ġfor Ġeach Ġsingle Ġtarget Ġor Ġpair Ġof Ġtargets ĠB IB REF 4 . ĠConcern s Ġhave Ġbeen Ġraised Ġthat Ġcross - target Ġperformance Ġis Ġoften Ġconsiderably Ġlower Ġthan Ġfully Ġsupervised Ġperformance ĠB IB REF 1 . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġa Ġmuch Ġlarger Ġdataset Ġthat Ġcombines Ġmult iling uality Ġand Ġa Ġmultitude Ġof Ġtopics Ġand Ġtargets . Ġx - st ance Ġcomprises Ġmore Ġthan Ġ150 Ġquestions Ġconcerning ĠSwiss Ġpolitics Ġand Ġmore Ġthan Ġ67 k Ġanswers Ġgiven Ġin Ġthe Ġlast Ġdecade Ġby Ġcandidates Ġrunning Ġfor Ġpolitical Ġoffice Ġin ĠSwitzerland . </s> Questions Ġare Ġavailable Ġin Ġfour Ġlanguages : ĠEnglish , ĠSwiss ĠStandard ĠGerman , ĠFrench , Ġand ĠItalian . ĠThe Ġlanguage Ġof Ġa Ġcomment Ġdepends Ġon Ġthe Ġcandidate 's Ġregion Ġof Ġorigin . </s> We Ġhave Ġextracted Ġthe Ġdata Ġfrom Ġthe Ġvoting Ġadvice Ġapplication ĠSmart vote . ĠOn Ġthat Ġplatform , Ġcandidates Ġrespond Ġto Ġquestions Ġmainly Ġin Ġcateg orical Ġform Ġ( yes Ġ/ Ġrather Ġyes Ġ/ Ġrather Ġno Ġ/ Ġno ). ĠThey Ġcan Ġalso Ġsubmit Ġa Ġfree - text Ġcomment Ġin Ġorder Ġto Ġjustify , Ġexplain Ġor Ġdifferentiate Ġtheir Ġcateg orical Ġanswer . ĠAn Ġexample Ġis Ġgiven Ġin ĠFigure ĠFIG REF 1 . </s> We Ġtransform Ġthe Ġdataset Ġinto Ġa Ġstance Ġdetection Ġtask Ġby Ġinterpreting Ġthe Ġquestion Ġas Ġa Ġnatural - language Ġrepresentation Ġof Ġthe Ġtarget , Ġand Ġthe Ġcommentary Ġas Ġthe Ġinput Ġto Ġbe Ġclassified . </s> The Ġdataset Ġis Ġsplit Ġinto Ġa Ġmult ilingual Ġtraining Ġset Ġand Ġinto Ġmultiple Ġtest Ġsets Ġto Ġevaluate Ġzero - shot Ġcross - ling ual Ġand Ġcross - target Ġtransfer . ĠTo Ġprovide Ġa Ġbaseline , Ġwe Ġfine - t une Ġa Ġmult ilingual ĠBert Ġmodel ĠB IB REF 5 Ġon Ġx - st ance . ĠWe Ġshow Ġthat Ġthe Ġbaseline Ġaccuracy Ġis Ġcomparable Ġto Ġprevious Ġstance Ġdetection Ġbenchmarks Ġwhile Ġleaving Ġample Ġroom Ġfor Ġimprovement . ĠIn Ġaddition , Ġmult ilingual ĠBert Ġcan Ġgeneral ize Ġto Ġa Ġdegree Ġboth Ġcross - ling ually Ġand Ġin Ġa Ġcross - target Ġsetting . </s> We Ġhave Ġmade Ġthe Ġdataset Ġand Ġthe Ġcode Ġfor Ġreprodu cing Ġthe Ġbaseline Ġmodel Ġpublicly Ġavailable . </s> Related ĠWork Ġ: :: ĠMult ilingual ĠSt ance ĠDetection </s> In Ġthe Ġcontext Ġof Ġthe ĠI ber E val Ġshared Ġtasks , Ġtwo Ġrelated Ġmult ilingual Ġdatasets Ġhave Ġbeen Ġcreated ĠB IB REF 2 , ĠB IB REF 3 . ĠBoth Ġare Ġa Ġcollection Ġof Ġannot ated ĠSpanish Ġand ĠCatalan Ġtweets . ĠCru cially , Ġthe Ġtweets Ġin Ġboth Ġlanguages Ġfocus Ġon Ġthe Ġsame Ġissue Ġ( Catal an Ġindependence ); Ġgiven Ġthis Ġfact Ġthey Ġare Ġthe Ġfirst Ġtruly Ġmult ilingual Ġstance Ġdetection Ġdatasets Ġknown Ġto Ġus . </s> With Ġregard Ġto Ġthe Ġlanguages Ġcovered Ġby Ġx - st ance , Ġonly Ġmon oling ual Ġdatasets Ġseem Ġto Ġbe Ġavailable . ĠFor ĠFrench , Ġa Ġcollection Ġof Ġtweets Ġon ĠFrench Ġpresidential Ġcandidates Ġhas Ġbeen Ġannot ated Ġwith Ġstance ĠB IB REF 6 . ĠSimilarly , Ġtwo Ġdatasets Ġof ĠItalian Ġtweets Ġon Ġthe Ġoccasion Ġof Ġthe Ġ2016 Ġconstitutional Ġreferendum Ġhave Ġbeen Ġcreated ĠB IB REF 7 , ĠB IB REF 6 . ĠFor ĠGerman , Ġthere Ġis Ġno Ġknown Ġstance Ġdetection Ġdataset , Ġbut Ġan Ġaspect - based Ġsentiment Ġdataset Ġhas Ġbeen Ġcreated Ġfor Ġa ĠGerm E val Ġ2017 Ġshared Ġtask ĠB IB REF 8 . </s> Related ĠWork Ġ: :: ĠMulti - Target ĠSt ance ĠDetection </s> The ĠSem E val - 2016 Ġtask Ġon Ġdetecting Ġstance Ġin Ġtweets ĠB IB REF 9 Ġoffers Ġdata Ġconcerning Ġmultiple Ġtargets Ġ( A the ism , ĠClimate ĠChange , ĠFemin ism , ĠHillary ĠClinton , Ġand ĠAbortion ). ĠIn Ġthe Ġsupervised Ġsubt ask ĠA , Ġparticipants Ġtended Ġto Ġdevelop Ġa Ġtarget - specific Ġmodel Ġfor Ġeach Ġof Ġthose Ġtargets . ĠIn Ġsubt ask ĠB Ġcross - target Ġtransfer Ġto Ġthe Ġtarget ĠâĢ ľ Donald ĠTrump âĢ Ŀ Ġwas Ġtested , Ġfor Ġwhich Ġno Ġannot ated Ġtraining Ġdata Ġwere Ġprovided . ĠWhile Ġthis Ġrequired Ġthe Ġdevelopment Ġof Ġmore Ġuniversal Ġmodels , Ġtheir Ġperformance Ġwas Ġgenerally Ġmuch Ġlower . </s> BI B REF 4 Ġintroduced Ġa Ġmulti - target Ġstance Ġdataset Ġwhich Ġprovides Ġtwo Ġtargets Ġper Ġinstance . ĠFor Ġexample , Ġa Ġmodel Ġdesigned Ġin Ġthis Ġframework Ġis Ġsupposed Ġto Ġsimultaneously Ġclassify Ġa Ġtweet Ġwith Ġregard Ġto ĠClinton Ġand Ġwith Ġregard Ġto ĠTrump . ĠWhile Ġin Ġtheory Ġthe Ġframework Ġallows Ġfor Ġmore Ġthan Ġtwo Ġtargets , Ġit Ġis Ġstill Ġrestricted Ġto Ġa Ġfinite Ġand Ġclearly Ġdefined Ġset Ġof Ġtargets . ĠIt Ġfocuses Ġon Ġmodeling Ġthe Ġdependencies Ġof Ġmultiple Ġtargets Ġwithin Ġthe Ġsame Ġtext Ġsample , Ġwhile Ġour Ġapproach Ġfocuses Ġon Ġlearning Ġstance Ġdetection Ġfrom Ġmany Ġsamples Ġwith Ġmany Ġdifferent Ġtargets . </s> Related ĠWork Ġ: :: ĠRepresent ation ĠLearning Ġfor ĠSt ance ĠDetection </s> In Ġa Ġtarget - specific Ġsetting , ĠB IB REF 10 Ġperform Ġa Ġsystematic Ġevaluation Ġof Ġstance Ġdetection Ġapproaches . ĠThey Ġalso Ġevaluate ĠBert ĠB IB REF 5 Ġand Ġfind Ġthat Ġit Ġconsistently Ġoutper forms Ġprevious Ġapproaches . </s> However , Ġthey Ġonly Ġexperimented Ġwith Ġa Ġsingle - se gment Ġencoding Ġof Ġthe Ġinput , Ġpreventing Ġcross - target Ġtransfer Ġof Ġthe Ġmodel . ĠB IB REF 11 Ġpropose Ġa Ġconditional Ġencoding Ġapproach Ġto Ġencode Ġboth Ġthe Ġtarget Ġand Ġthe Ġtweet Ġas Ġsequences . ĠThey Ġuse Ġa Ġbid irection al ĠL ST M Ġto Ġcondition Ġthe Ġencoding Ġof Ġthe Ġtweets Ġon Ġthe Ġencoding Ġof Ġthe Ġtarget , Ġand Ġthen Ġapply Ġa Ġnon linear Ġprojection Ġon Ġthe Ġcondition ally Ġencoded Ġtweet . ĠThis Ġallows Ġthem Ġto Ġtrain Ġa Ġmodel Ġthat Ġcan Ġgeneral ize Ġto Ġpreviously Ġunseen Ġtargets . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠTask ĠDefinition </s> The Ġinput Ġprovided Ġby Ġx - st ance Ġis Ġtwo - fold : Ġ( A ) Ġa Ġnatural Ġlanguage Ġquestion Ġconcerning Ġa Ġpolitical Ġissue ; Ġ( B ) Ġa Ġnatural Ġlanguage Ġcommentary Ġon Ġa Ġspecific Ġstance Ġtowards Ġthe Ġquestion . </s> The Ġlabel Ġto Ġbe Ġpredicted Ġis Ġeither Ġ` f avor ' Ġor Ġ` against `. ĠThis Ġcorresponds Ġto Ġa Ġstandard Ġestablished Ġby ĠB IB REF 0 . ĠHowever , Ġx - st ance Ġdiffers Ġfrom Ġthat Ġdataset Ġin Ġthat Ġit Ġlacks Ġa Ġ` ne ither ' Ġclass ; Ġall Ġcomments Ġrefer Ġto Ġeither Ġa Ġ` f avor ' Ġor Ġan Ġ` against ` Ġposition . ĠThe Ġtask Ġposed Ġby Ġx - st ance Ġis Ġthus Ġa Ġbinary Ġclassification Ġtask . </s> As Ġan Ġevaluation Ġmetric Ġwe Ġreport Ġthe Ġmacro - average Ġof Ġthe ĠF 1 - score Ġfor Ġ` f avor ' Ġand Ġthe ĠF 1 - score Ġfor Ġ` against ', Ġsimilar Ġto ĠB IB REF 9 . ĠWe Ġuse Ġthis Ġmetric Ġmainly Ġto Ġstrengthen Ġcompar ability Ġwith Ġthe Ġprevious Ġbenchmarks . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠPro ven ance </s> We Ġdownloaded Ġquestions Ġand Ġanswers Ġvia Ġthe ĠSmart vote ĠAPI . ĠThe Ġdownloaded Ġdata Ġcover Ġ175 Ġcommunal , Ġcant onal Ġand Ġnational Ġelections Ġbetween Ġ2011 Ġand Ġ2020 . </s> All Ġcandidates Ġin Ġan Ġelection Ġwho Ġparticipate Ġin ĠSmart vote Ġare Ġasked Ġthe Ġsame Ġset Ġof Ġquestions , Ġbut Ġdepending Ġon Ġthe Ġlocale Ġthey Ġsee Ġtranslated Ġversions Ġof Ġthe Ġquestions . ĠThey Ġcan Ġanswer Ġeach Ġquestion Ġwith Ġeither Ġ` yes ', Ġ` rather Ġyes ', Ġ` rather Ġno ', Ġor Ġ` no '. ĠThey Ġcan Ġsupplement Ġeach Ġanswer Ġwith Ġa Ġcomment Ġof Ġat Ġmost Ġ500 Ġcharacters . </s> The Ġquestions Ġasked Ġon ĠSmart vote Ġhave Ġbeen Ġedited Ġby Ġa Ġteam Ġof Ġpolitical Ġscientists . ĠThey Ġare Ġintended Ġto Ġcover Ġa Ġbroad Ġrange Ġof Ġpolitical Ġissues Ġrelevant Ġat Ġthe Ġtime Ġof Ġthe Ġelection . ĠA Ġdetailed Ġdocumentation Ġof Ġthe Ġdesign Ġof ĠSmart vote Ġand Ġthe Ġediting Ġprocess Ġof Ġthe Ġquestions Ġis Ġprovided Ġby ĠB IB REF 12 . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠPre processing </s> We Ġmerged Ġthe Ġtwo Ġlabels Ġon Ġeach Ġpole Ġinto Ġa Ġsingle Ġlabel : Ġ` yes ' Ġand Ġ` rather Ġyes ' Ġwere Ġcombined Ġinto Ġ` f avor '; Ġ` rather Ġno ', Ġor Ġ` no ' Ġinto Ġ` against `. ĠThis Ġimproves Ġthe Ġconsistency Ġof Ġthe Ġdata Ġand Ġthe Ġcompar ability Ġto Ġprevious Ġstance Ġdetection Ġdatasets . </s> We Ġdid Ġnot Ġfurther Ġpre process Ġthe Ġtext Ġof Ġthe Ġcomments . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠLanguage ĠIdentification </s> As Ġthe ĠAPI Ġdoes Ġnot Ġprovide Ġthe Ġlanguage Ġof Ġcomments , Ġwe Ġemployed Ġa Ġlanguage Ġidentifier Ġto Ġautomatically Ġannot ate Ġthis Ġinformation . ĠWe Ġused Ġthe Ġlang det ect Ġlibrary ĠB IB REF 13 . ĠFor Ġeach Ġrespond er Ġwe Ġclassified Ġall Ġthe Ġcomments Ġjointly , Ġassuming Ġthat Ġresponders Ġdid Ġnot Ġswitch Ġcode Ġduring Ġthe Ġanswering Ġof Ġthe Ġquestionnaire . </s> We Ġapplied Ġthe Ġidentifier Ġin Ġa Ġtwo - step Ġapproach . ĠIn Ġthe Ġfirst Ġrun Ġwe Ġallowed Ġthe Ġidentifier Ġto Ġoutput Ġall Ġ55 Ġlanguages Ġthat Ġit Ġsupports Ġout Ġof Ġthe Ġbox , Ġplus ĠRomans h , Ġthe Ġfourth Ġofficial Ġlanguage Ġin ĠSwitzerland . ĠWe Ġfound Ġthat Ġno ĠRomans h Ġcomments Ġwere Ġdetected Ġand Ġthat Ġall Ġunexpected Ġoutputs Ġwere Ġmis class ifications Ġof ĠGerman , ĠFrench Ġor ĠItalian Ġcomments . ĠWe Ġfurther Ġconcluded Ġthat Ġlittle Ġor Ġno ĠSwiss ĠGerman Ġcomments Ġare Ġin Ġthe Ġdataset : ĠIf Ġthey Ġwere , Ġsome Ġof Ġthem Ġwould Ġhave Ġmanifested Ġthemselves Ġin Ġthe Ġform Ġof Ġmis class ifications Ġ( e . g . Ġas ĠDutch ). </s> In Ġthe Ġsecond Ġrun , Ġdrawing Ġfrom Ġthese Ġconclusions , Ġwe Ġrestricted Ġthe Ġidentifier 's Ġoutput Ġto ĠEnglish , ĠFrench , ĠGerman Ġand ĠItalian . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠFil tering </s> We Ġpre - fil tered Ġthe Ġquestions Ġand Ġanswers Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġdataset . ĠTo Ġkeep Ġthe Ġdomain Ġof Ġthe Ġdata Ġsurvey able , Ġwe Ġset Ġa Ġfocus Ġon Ġnational - level Ġquestions . ĠTherefore , Ġall Ġquestions Ġand Ġcorresponding Ġanswers Ġpertaining Ġto Ġnational Ġelections Ġwere Ġincluded . </s> In Ġthe Ġcontext Ġof Ġcommunal Ġand Ġcant onal Ġelections , Ġcandidates Ġhave Ġanswered Ġboth Ġlocal Ġquestions Ġand Ġa Ġsubset Ġof Ġthe Ġnational Ġquestions . ĠOf Ġthose Ġelections , Ġwe Ġonly Ġconsidered Ġanswers Ġto Ġthe Ġquestions Ġthat Ġalso Ġhad Ġbeen Ġasked Ġin Ġa Ġnational Ġelection . ĠFurthermore , Ġthey Ġwere Ġonly Ġused Ġto Ġaugment Ġthe Ġtraining Ġset Ġwhile Ġthe Ġvalidation Ġand Ġtest Ġsets Ġwere Ġrestricted Ġto Ġanswers Ġfrom Ġnational Ġelections . </s> We Ġdiscarded Ġthe Ġless Ġthan Ġ20 Ġcomments Ġclassified Ġas ĠEnglish . ĠFurthermore , Ġinstances Ġthat Ġmet Ġany Ġof Ġthe Ġfollowing Ġconditions Ġwere Ġfiltered Ġfrom Ġthe Ġdataset : </s> Question Ġis Ġnot Ġa Ġclosed Ġquestion Ġor Ġdoes Ġnot Ġaddress Ġa Ġclearly Ġdefined Ġpolitical Ġissue . </s> No Ġcomment Ġwas Ġsubmitted Ġby Ġthe Ġcandidate Ġor Ġthe Ġcomment Ġis Ġshorter Ġthan Ġ50 Ġcharacters . </s> Comment Ġstarts Ġwith ĠâĢ ľ but âĢ Ŀ Ġor Ġa Ġsimilar Ġindicator Ġthat Ġthe Ġcomment Ġis Ġnot Ġa Ġself - contained Ġstatement . </s> Comment Ġcontains Ġa ĠURL . </s> In Ġtotal , Ġa Ġfifth Ġof Ġthe Ġoriginal Ġcomments Ġwere Ġfiltered Ġout . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠTopics </s> The Ġquestions Ġhave Ġbeen Ġorganized Ġby Ġthe ĠSmart vote Ġeditors Ġinto Ġcategories Ġ( such Ġas ĠâĢ ľ Econom y âĢ Ŀ ). ĠWe Ġfurther Ġconsolidated Ġthe Ġpre - defined Ġcategories Ġinto Ġ12 Ġbroad Ġtopics Ġ( Table ĠT AB REF 7 ). </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠCompliance </s> The Ġdataset Ġis Ġshared Ġunder Ġa ĠCC ĠBY - NC Ġ4 . 0 Ġlicense . ĠCopyright Ġremains Ġwith Ġwww . smart vote . ch . </s> Given Ġthe Ġsensitive Ġnature Ġof Ġthe Ġdata , Ġwe Ġincrease Ġthe Ġanonymity Ġof Ġthe Ġdata Ġby Ġhashing Ġthe Ġrespondents ' ĠIDs . ĠNo Ġpersonal Ġattributes Ġof Ġthe Ġrespondents , Ġsuch Ġas Ġtheir Ġparty Ġaffiliation , Ġare Ġincluded Ġin Ġthe Ġdataset . ĠWe Ġprovide Ġa Ġdata Ġstatement ĠB IB REF 15 Ġin ĠAppendix ĠSEC REF 8 . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠSplit </s> We Ġheld Ġout Ġthe Ġtopics ĠâĢ ľ Health care âĢ Ŀ Ġand ĠâĢ ľ Political ĠSystem âĢ Ŀ Ġfrom Ġthe Ġtraining Ġdata Ġand Ġcreated Ġa Ġseparate Ġcross - topic Ġtest Ġset Ġthat Ġcontains Ġthe Ġquestions Ġand Ġanswers Ġrelated Ġto Ġthose Ġtopics . </s> Furthermore , Ġin Ġorder Ġto Ġtest Ġcross - question Ġgeneral ization Ġperformance Ġwithin Ġpreviously Ġseen Ġtopics , Ġwe Ġmanually Ġselected Ġ16 Ġheld - out Ġquestions Ġthat Ġare Ġdistributed Ġover Ġthe Ġremaining Ġ10 Ġtopics . ĠWe Ġselected Ġthe Ġheld - out Ġquestions Ġmanually Ġbecause Ġwe Ġwanted Ġto Ġmake Ġsure Ġthat Ġthey Ġare Ġtruly Ġunseen Ġand Ġthat Ġno Ġparaph r ases Ġof Ġthe Ġquestions Ġare Ġfound Ġin Ġthe Ġtraining Ġset . </s> We Ġdesignated ĠItalian Ġas Ġa Ġtest - only Ġlanguage , Ġsince Ġrelatively Ġfew Ġcomments Ġhave Ġbeen Ġwritten Ġin ĠItalian . ĠFrom Ġthe Ġremaining ĠGerman Ġand ĠFrench Ġdata Ġwe Ġrandomly Ġselected Ġa Ġpercentage Ġof Ġrespondents Ġas Ġvalidation Ġor Ġas Ġtest Ġrespondents . </s> As Ġa Ġresult Ġwe Ġreceived Ġone Ġtraining Ġset , Ġone Ġvalidation Ġset Ġand Ġfour Ġtest Ġsets . ĠThe Ġsizes Ġof Ġthe Ġsets Ġare Ġlisted Ġin ĠTable ĠT AB REF 8 . ĠWe Ġdid Ġnot Ġconsider Ġtest Ġsets Ġthat Ġare Ġcross - ling ual Ġand Ġcross - target Ġat Ġthe Ġsame Ġtime , Ġas Ġthey Ġwould Ġhave Ġbeen Ġtoo Ġsmall Ġto Ġyield Ġsignificant Ġresults . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis </s> Some Ġobservations Ġregarding Ġthe Ġcomposition Ġof Ġx - st ance Ġcan Ġbe Ġmade . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠClass ĠDistribution </s> Figure ĠFIG REF 23 Ġvisual izes Ġthe Ġproportion Ġof Ġ` f avor ' Ġand Ġ` against ` Ġstances Ġfor Ġeach Ġtarget Ġin Ġthe Ġdataset . ĠThe Ġratio Ġdiffers Ġbetween Ġquestions Ġbut Ġis Ġrelatively Ġequally Ġdistributed Ġacross Ġthe Ġtopics . ĠIn Ġparticular , Ġthe Ġquestions Ġbelonging Ġto Ġthe Ġheld - out Ġtopics Ġ( with Ġa Ġ` f avor ' Ġratio Ġof Ġ49 . 4 %) Ġhave Ġa Ġsimilar Ġclass Ġdistribution Ġas Ġthe Ġquestions Ġwithin Ġother Ġtopics Ġ( with Ġa Ġ` f avor ' Ġratio Ġof Ġ50 . 0 %). </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠL ingu istic ĠProperties </s> Not Ġevery Ġquestion Ġis Ġunique ; Ġsome Ġquestions Ġare Ġparaph r ases Ġdescribing Ġthe Ġsame Ġpolitical Ġissue . ĠFor Ġexample , Ġin Ġthe Ġ2015 Ġelection , Ġthe Ġcandidates Ġwere Ġasked : ĠâĢ ľ Should Ġthe Ġconsumption Ġof Ġcannabis Ġas Ġwell Ġas Ġits Ġpossession Ġfor Ġpersonal Ġuse Ġbe Ġlegal ised ? âĢ Ŀ ĠFour Ġyears Ġlater Ġthey Ġwere Ġasked : ĠâĢ ľ Should Ġcannabis Ġuse Ġbe Ġlegalized ? âĢ Ŀ ĠHowever , Ġwe Ġdo Ġnot Ġsee Ġany Ġneed Ġto Ġconsolidate Ġthose Ġdupl icates Ġbecause Ġthey Ġcontribute Ġto Ġthe Ġdiversity Ġof Ġthe Ġtraining Ġdata . </s> We Ġfurther Ġobserve Ġthat Ġwhile Ġsome Ġquestions Ġin Ġthe Ġdataset Ġare Ġquite Ġshort , Ġsome Ġquestions Ġare Ġrather Ġconvoluted . ĠFor Ġexample , Ġa Ġtypical Ġlong Ġquestion Ġreads : </s> Some Ġ1 % Ġof Ġdirect Ġpayments Ġto ĠSwiss Ġagriculture Ġcurrently Ġgo Ġto Ġorganic Ġfarming Ġoperations . ĠShould Ġthis Ġproportion Ġbe Ġincreased Ġat Ġthe Ġexpense Ġof Ġstandard Ġfarming Ġoperations Ġas Ġpart Ġof ĠSwitzerland 's Ġ2014 - 2017 Ġagricultural Ġpolicy ? </s> Such Ġlonger Ġquestions Ġmight Ġbe Ġmore Ġchallenging Ġto Ġprocess Ġsem antically . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠLanguages </s> The Ġx - st ance Ġdataset Ġhas Ġmore ĠGerman Ġsamples Ġthan ĠFrench Ġsamples . ĠThe Ġlanguage Ġratio Ġof Ġabout Ġ3 : 1 Ġis Ġconsistent Ġacross Ġall Ġtraining Ġand Ġtest Ġsets . ĠGiven Ġthe Ġtwo Ġlanguages Ġit Ġis Ġpossible Ġto Ġeither Ġtrain Ġtwo Ġmon oling ual Ġmodels Ġor Ġto Ġtrain Ġa Ġsingle Ġmodel Ġin Ġa Ġmulti - source Ġsetup ĠB IB REF 16 . ĠWe Ġchoose Ġa Ġmulti - source Ġbaseline Ġbecause ĠM - B ert Ġis Ġknown Ġto Ġbenefit Ġfrom Ġmult ilingual Ġtraining Ġdata Ġboth Ġin Ġa Ġsupervised Ġand Ġin Ġa Ġcross - ling ual Ġscenario ĠB IB REF 17 . </s> Bas eline ĠExper iments </s> We Ġevaluate Ġtwo Ġtypes Ġof Ġbas elines Ġto Ġobtain Ġan Ġimpression Ġof Ġthe Ġdifficulty Ġof Ġthe Ġtask . </s> Bas eline ĠExper iments Ġ: :: ĠMajority ĠClass ĠBas elines </s> The Ġfirst Ġpair Ġof Ġbas elines Ġuses Ġthe Ġmost Ġfrequent Ġclass Ġin Ġthe Ġtraining Ġset Ġfor Ġprediction . ĠSpecifically , Ġthe Ġglobal Ġmajority Ġclass Ġbaseline Ġpredicts Ġthe Ġmost Ġfrequent Ġclass Ġacross Ġall Ġtraining Ġtargets Ġwhile Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline Ġpredicts Ġthe Ġclass Ġthat Ġis Ġmost Ġfrequent Ġfor Ġa Ġgiven Ġtarget Ġquestion . ĠThe Ġlatter Ġcan Ġonly Ġbe Ġapplied Ġto Ġthe Ġsupervised Ġtest Ġset . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline </s> Secondly , Ġwe Ġfine - t une Ġmult ilingual ĠBert Ġ( M - B ert ) Ġon Ġthe Ġtask ĠB IB REF 5 Ġwhich Ġhas Ġbeen Ġpret rained Ġjointly Ġin Ġ104 Ġlanguages Ġand Ġhas Ġestablished Ġitself Ġas Ġa Ġstate Ġof Ġthe Ġart Ġfor Ġvarious Ġmult ilingual Ġtasks ĠB IB REF 18 , ĠB IB REF 19 . ĠWithin Ġthe Ġfield Ġof Ġstance Ġdetection , ĠBert Ġcan Ġoutper form Ġboth Ġfeature - based Ġand Ġother Ġneural Ġapproaches Ġin Ġa Ġmon oling ual ĠEnglish Ġsetting ĠB IB REF 10 . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠArchitecture </s> In Ġthe Ġcontext Ġof ĠBert Ġwe Ġinterpret Ġthe Ġx - st ance Ġtask Ġas Ġsequence Ġpair Ġclassification Ġinspired Ġby Ġnatural Ġlanguage Ġinference Ġtasks ĠB IB REF 20 . ĠWe Ġfollow Ġthe Ġprocedure Ġoutlined Ġby ĠB IB REF 5 Ġfor Ġsuch Ġtasks . ĠWe Ġdesignate Ġthe Ġquestion Ġas Ġsegment ĠA Ġand Ġthe Ġcomment Ġas Ġsegment ĠB . ĠThe Ġtwo Ġsegments Ġare Ġseparated Ġwith Ġthe Ġspecial Ġtoken Ġ[ SE P ], Ġand Ġthe Ġspecial Ġtoken Ġ[ CL S ] Ġis Ġprep ended Ġto Ġthe Ġsequence . ĠThe Ġfinal Ġhidden Ġstate Ġcorresponding Ġto Ġ[ CL S ] Ġis Ġthen Ġclassified Ġby Ġa Ġlinear Ġlayer . </s> We Ġfine - t une Ġthe Ġfull Ġmodel Ġwith Ġa Ġcross - ent ropy Ġloss , Ġusing Ġthe ĠAllen N LP Ġlibrary ĠB IB REF 21 Ġas Ġa Ġbasis Ġfor Ġour Ġimplementation . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠTraining </s> We Ġups am pled Ġthe Ġ` f avor ' Ġclass Ġso Ġthat Ġthe Ġtwo Ġclasses Ġare Ġbalanced Ġwhen Ġsum ming Ġover Ġall Ġquestions Ġand Ġtopics . ĠA Ġmaximum Ġsequence Ġlength Ġof Ġ512 Ġsub words Ġand Ġa Ġbatch Ġsize Ġof Ġ16 Ġwas Ġchosen Ġfor Ġall Ġtraining Ġruns . ĠWe Ġthen Ġperformed Ġa Ġgrid Ġsearch Ġwithin Ġthe Ġfollowing Ġrange Ġof Ġhyper param eters Ġbased Ġon Ġthe Ġvalidation Ġaccuracy : </s> Learning Ġrate : Ġ5 e - 5 , Ġ3 e - 5 , Ġ2 e - 5 </s> Number Ġof Ġepoch s : Ġ3 , Ġ4 </s> The Ġgrid Ġsearch Ġwas Ġrepeated Ġindependently Ġfor Ġevery Ġvariant Ġthat Ġwe Ġtested . ĠFurthermore , Ġthe Ġstandard Ġrecommendations Ġfor Ġfine - tun ing ĠBert Ġwere Ġused : ĠAdam Ġwith Ġ${ \ beta Ġ} _ 1 = 0 . 9 $ Ġand Ġ${ \ beta Ġ} _ 2 = 0 . 999 $ ; Ġan ĠL 2 Ġweight Ġdecay Ġof Ġ$ 0 . 01 $ ; Ġa Ġlearning Ġrate Ġwarm up Ġover Ġthe Ġfirst Ġ10 % Ġof Ġthe Ġsteps ; Ġand Ġa Ġlinear Ġdecay Ġof Ġthe Ġlearning Ġrate . ĠA Ġdrop out Ġprobability Ġof Ġ0 . 1 Ġwas Ġset Ġon Ġall Ġlayers . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠResults </s> Table ĠT AB REF 36 Ġshows Ġthe Ġresults Ġfor Ġthe Ġcross - ling ual Ġsetting . ĠM - B ert Ġperforms Ġconsistently Ġbetter Ġthan Ġthe Ġmajority Ġclass Ġbas elines . ĠEven Ġthe Ġzero - short Ġperformance Ġin ĠItalian , Ġwhile Ġsignificantly Ġlower Ġthan Ġthe Ġsupervised Ġscores , Ġis Ġmuch Ġbetter Ġthan Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline . </s> Results Ġfor Ġthe Ġcross - target Ġsetting Ġare Ġgiven Ġin ĠTable ĠT AB REF 37 . ĠSimilar Ġto Ġthe Ġcross - ling ual Ġsetting , ĠM - B ert Ġperforms Ġworse Ġin Ġa Ġcross - target Ġsetting Ġbut Ġeasily Ġsurpass es Ġthe Ġmajority Ġclass Ġbas elines . ĠFurthermore , Ġthe Ġcross - question Ġscore Ġof ĠM - B ert Ġis Ġslightly Ġlower Ġthan Ġthe Ġcross - topic Ġscore . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġis ĠCons istent ĠLanguage ? </s> The Ġdefault Ġsetup Ġpreserves Ġhorizontal Ġlanguage Ġconsistency Ġin Ġthat Ġthe Ġlanguage Ġof Ġthe Ġquestions Ġalways Ġmatches Ġthe Ġlanguage Ġof Ġthe Ġcomments . ĠFor Ġexample , Ġthe ĠItalian Ġtest Ġinstances Ġare Ġcombined Ġwith Ġthe ĠItalian Ġversion Ġof Ġthe Ġquestions , Ġeven Ġthough Ġduring Ġtraining Ġthe Ġmodel Ġhas Ġonly Ġever Ġseen Ġthe ĠGerman Ġand ĠFrench Ġversions Ġof Ġthe Ġquestions . </s> An Ġalternative Ġconcept Ġis Ġvertical Ġlanguage Ġconsistency , Ġwhereby Ġthe Ġquestions Ġare Ġconsistently Ġpresented Ġin Ġone Ġlanguage , Ġregardless Ġof Ġthe Ġcomment . ĠTo Ġtest Ġwhether Ġhorizontal Ġor Ġvertical Ġconsistency Ġis Ġmore Ġhelpful , Ġwe Ġtrain Ġand Ġevaluate ĠM - B ert Ġon Ġa Ġdataset Ġvariant Ġwhere Ġall Ġquestions Ġare Ġin Ġtheir ĠEnglish Ġversion . ĠWe Ġchose ĠEnglish Ġas Ġa Ġling ua Ġfr anca Ġbecause Ġit Ġhad Ġthe Ġlargest Ġshare Ġof Ġdata Ġduring Ġthe Ġpret raining Ġof ĠM - B ert . </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 39 . ĠWhile Ġthe Ġeffect Ġis Ġnegligible Ġin Ġmost Ġsettings , Ġthe Ġcross - ling ual Ġperformance Ġclearly Ġincreases Ġwhen Ġall Ġquestions Ġare Ġgiven Ġin ĠEnglish . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġare Ġthe ĠSe gments ? </s> In Ġorder Ġto Ġrule Ġout Ġthat Ġonly Ġthe Ġquestions Ġor Ġonly Ġthe Ġcomments Ġare Ġnecessary Ġto Ġoptim ally Ġsolve Ġthe Ġtask , Ġwe Ġconduct Ġsome Ġadditional Ġexperiments : </s> Only Ġuse Ġa Ġsingle Ġsegment Ġcontaining Ġthe Ġcomment , Ġremoving Ġthe Ġquestions Ġfrom Ġthe Ġtraining Ġand Ġtest Ġdata Ġ( missing Ġquestions ). </s> Only Ġuse Ġthe Ġquestion Ġand Ġremove Ġthe Ġcomment Ġ( missing Ġcomments ). </s> In Ġboth Ġcases Ġthe Ġperformance Ġdecreases Ġacross Ġall Ġevaluation Ġsettings Ġ( Table ĠT AB REF 39 ). ĠThe Ġloss Ġin Ġperformance Ġis Ġmuch Ġhigher Ġwhen Ġcomments Ġare Ġmissing , Ġindicating Ġthat Ġthe Ġcomments Ġcontain Ġthe Ġmost Ġimportant Ġinformation Ġabout Ġstance . ĠAs Ġcan Ġbe Ġexpected , Ġthe Ġscore Ġachieved Ġwithout Ġcomments Ġis Ġonly Ġslightly Ġdifferent Ġfrom Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline . </s> But Ġthere Ġis Ġalso Ġa Ġloss Ġin Ġperformance Ġwhen Ġthe Ġquestions Ġare Ġmissing , Ġwhich Ġunder lines Ġthe Ġimportance Ġof Ġpairing Ġboth Ġpieces Ġof Ġtext . ĠThe Ġeffect Ġof Ġmissing Ġquestions Ġis Ġespecially Ġstrong Ġin Ġthe Ġsupervised Ġand Ġcross - ling ual Ġsettings . ĠTo Ġillustrate Ġthis , Ġwe Ġprovide Ġin ĠTable ĠT AB REF 48 Ġsome Ġexamples Ġof Ġcomments Ġthat Ġoccur Ġwith Ġmultiple Ġdifferent Ġtargets Ġin Ġthe Ġtraining Ġset . ĠThose Ġexamples Ġcan Ġexplain Ġwhy Ġthe Ġtarget Ġcan Ġbe Ġessential Ġfor Ġdis amb ig uating Ġa Ġstance Ġdetection Ġproblem . ĠOn Ġthe Ġother Ġhand , Ġthe Ġeffect Ġof Ġo mitting Ġthe Ġquestions Ġis Ġless Ġpronounced Ġin Ġthe Ġcross - target Ġsettings . </s> The Ġabove Ġsingle - se gment Ġexperiments Ġtell Ġus Ġthat Ġboth Ġthe Ġcomment Ġand Ġthe Ġquestion Ġprovide Ġcrucial Ġinformation . ĠBut Ġit Ġis Ġpossible Ġthat Ġthe ĠM - B ert Ġmodel , Ġeven Ġthough Ġtrained Ġon Ġboth Ġsegments , Ġmainly Ġlooks Ġat Ġa Ġsingle Ġsegment Ġat Ġtest Ġtime . ĠTo Ġrule Ġthis Ġout , Ġwe Ġprobe Ġthe Ġmodel Ġwith Ġrandomized Ġdata Ġat Ġtest Ġtime : </s> Test Ġthe Ġmodel Ġon Ġversions Ġof Ġthe Ġtest Ġsets Ġwhere Ġthe Ġcomments Ġremain Ġin Ġplace Ġbut Ġthe Ġquestions Ġare Ġshuff led Ġrandomly Ġ( random Ġquestions ). ĠWe Ġmake Ġsure Ġthat Ġthe Ġrandom Ġquestions Ġcome Ġfrom Ġthe Ġsame Ġtest Ġset Ġand Ġlanguage Ġas Ġthe Ġoriginal Ġquestions . </s> Keep Ġthe Ġquestions Ġin Ġplace Ġand Ġrandom ize Ġthe Ġcomments Ġ( random Ġcomments ). ĠAgain Ġwe Ġshuffle Ġthe Ġcomments Ġonly Ġwithin Ġtest Ġset Ġboundaries . </s> The Ġresults Ġin ĠTable ĠT AB REF 39 Ġshow Ġthat Ġthe Ġperformance Ġof Ġthe Ġmodel Ġdecreases Ġin Ġboth Ġcases , Ġconfirming Ġthat Ġit Ġlearns Ġto Ġtake Ġinto Ġaccount Ġboth Ġsegments . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġare ĠSpe lled - Out ĠTarg ets ? </s> Finally Ġwe Ġtest Ġwhether Ġthe Ġtarget Ġreally Ġneeds Ġto Ġbe Ġrepresented Ġby Ġnatural Ġlanguage Ġ( e . g . ĠâĢ ľ Do Ġyou Ġsupport ĠX ? âĢ Ŀ ). ĠName ly , Ġan Ġalternative Ġis Ġto Ġrepresent Ġthe Ġtarget Ġwith Ġa Ġtrain able Ġembed ding Ġinstead Ġof Ġa Ġquestion . </s> In Ġorder Ġto Ġfit Ġtarget Ġembed d ings Ġsmoothly Ġinto Ġour Ġarchitecture , Ġwe Ġrepresent Ġeach Ġtarget Ġtype Ġwith Ġa Ġdifferent Ġreserved Ġsymbol Ġfrom Ġthe ĠM - B ert Ġvocabulary . ĠSe gment ĠA Ġis Ġthen Ġset Ġto Ġthis Ġsymbol Ġinstead Ġof Ġa Ġnatural Ġlanguage Ġquestion . </s> The Ġresults Ġfor Ġthis Ġexperiment Ġare Ġlisted Ġin Ġthe Ġbottom Ġrow Ġof ĠTable ĠT AB REF 39 . ĠAn ĠM - B ert Ġmodel Ġthat Ġlearns Ġtarget Ġembed d ings Ġinstead Ġof Ġencoding Ġa Ġquestion Ġperforms Ġclearly Ġworse Ġin Ġthe Ġsupervised Ġand Ġcross - ling ual Ġsettings . ĠFrom Ġthis Ġwe Ġconclude Ġthat Ġspelled - out Ġnatural Ġlanguage Ġquestions Ġprovide Ġimportant Ġlinguistic Ġdetail Ġthat Ġcan Ġhelp Ġin Ġstance Ġdetection . </s> Discussion </s> The Ġbaseline Ġexperiments Ġconfirm Ġthat ĠM - B ert Ġcan Ġachieve Ġa Ġreasonable Ġaccuracy Ġon Ġx - st ance . </s> To Ġput Ġthe Ġsupervised Ġscore Ġinto Ġcontext Ġwe Ġlist Ġscores Ġthat Ġvariants Ġof ĠBert Ġhave Ġachieved Ġon Ġother Ġstance Ġdetection Ġdatasets Ġin ĠTable ĠT AB REF 46 . ĠIt Ġseems Ġthat Ġthe Ġsupervised Ġpart Ġof Ġx - st ance Ġhas Ġa Ġsimilar Ġdifficulty Ġas Ġthe ĠSem E val - 2016 ĠB IB REF 0 Ġor ĠMP CH I ĠB IB REF 22 Ġdatasets Ġon Ġwhich ĠBert Ġhas Ġpreviously Ġbeen Ġevaluated . </s> On Ġthe Ġother Ġhand , Ġin Ġthe Ġcross - ling ual Ġand Ġcross - target Ġsettings , Ġthe Ġmean Ġscore Ġdrops Ġby Ġ6 âĢĵ 8 Ġpercentage Ġpoints Ġcompared Ġto Ġthe Ġsupervised Ġsetting ; Ġwhile Ġzero - shot Ġtransfer Ġis Ġpossible Ġto Ġa Ġdegree , Ġit Ġcan Ġstill Ġbe Ġimproved . </s> The Ġadditional Ġexperiments Ġ( Table ĠT AB REF 39 ) Ġvalidate Ġthe Ġresults Ġand Ġshow Ġthat Ġthe Ġsequence - pair Ġclassification Ġapproach Ġto Ġstance Ġdetection Ġis Ġjustified . </s> It Ġis Ġinteresting Ġto Ġsee Ġwhat Ġerrors Ġthe ĠM - B ert Ġmodel Ġmakes . ĠTable ĠT AB REF 47 Ġpresents Ġinstances Ġwhere Ġit Ġpredicts Ġthe Ġwrong Ġlabel Ġwith Ġa Ġhigh Ġconfidence . ĠThese Ġexamples Ġindicate Ġthat Ġmany Ġcomments Ġexpress Ġtheir Ġstance Ġonly Ġon Ġa Ġvery Ġimplicit Ġlevel , Ġand Ġthus Ġhint Ġat Ġa Ġpotential Ġweakness Ġof Ġthe Ġdataset . ĠBecause Ġon Ġthe Ġvoting Ġadvice Ġplatform Ġthe Ġlabel Ġis Ġexplicitly Ġshown Ġto Ġreaders Ġin Ġaddition Ġto Ġthe Ġcomments , Ġthe Ġcomments Ġdo Ġnot Ġneed Ġto Ġexpress Ġthe Ġstance Ġexplicitly . ĠManual Ġannotation Ġcould Ġeliminate Ġvery Ġimplicit Ġsamples Ġin Ġa Ġfuture Ġversion Ġof Ġthe Ġdataset . ĠHowever , Ġthe Ġsheer Ġsize Ġand Ġbreadth Ġof Ġthe Ġdataset Ġcould Ġnot Ġrealistically Ġbe Ġachieved Ġwith Ġmanual Ġannotation , Ġand , Ġin Ġour Ġview , Ġlargely Ġcompens ates Ġfor Ġthe Ġimplicit ness Ġof Ġthe Ġtexts . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġnew Ġdataset Ġfor Ġpolitical Ġstance Ġdetection Ġcalled Ġx - st ance . ĠThe Ġdataset Ġextends Ġover Ġa Ġbroad Ġrange Ġof Ġtopics Ġand Ġissues Ġregarding Ġnational ĠSwiss Ġpolitics . ĠThis Ġdiversity Ġof Ġtopics Ġopens Ġup Ġan Ġopportunity Ġto Ġfurther Ġstudy Ġmulti - target Ġlearning . ĠMoreover , Ġbeing Ġpartly ĠSwiss ĠStandard ĠGerman , Ġpartly ĠFrench Ġand ĠItalian , Ġthe Ġdataset Ġpromotes Ġa Ġmult ilingual Ġapproach Ġto Ġstance Ġdetection . </s> By Ġcompiling Ġformal Ġcommentary Ġthat Ġpoliticians Ġhave Ġwritten Ġon Ġpolitical Ġquestions , Ġwe Ġadd Ġa Ġnew Ġtext Ġgenre Ġto Ġthe Ġfield Ġof Ġstance Ġdetection . ĠWe Ġalso Ġpropose Ġa Ġquestion âĢĵ answer Ġformat Ġthat Ġallows Ġus Ġto Ġcondition Ġstance Ġdetection Ġmodels Ġon Ġa Ġtarget Ġnaturally . </s> Our Ġbaseline Ġresults Ġwith Ġmult ilingual ĠBert Ġshow Ġthat Ġthe Ġmodel Ġhas Ġsome Ġcapability Ġto Ġperform Ġzero - shot Ġtransfer Ġto Ġunseen Ġlanguages Ġand Ġto Ġunseen Ġtargets Ġ( both Ġwithin Ġa Ġtopic Ġand Ġto Ġunseen Ġtopics ). ĠHowever , Ġthere Ġis Ġsome Ġgap Ġin Ġperformance Ġthat Ġfuture Ġwork Ġcould Ġaddress . ĠWe Ġexpect Ġthat Ġthe Ġx - st ance Ġdataset Ġcould Ġfurthermore Ġbe Ġa Ġvaluable Ġresource Ġfor Ġfields Ġsuch Ġas Ġargument Ġmining , Ġargument Ġsearch Ġor Ġtopic Ġclassification . </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( project ĠM UT AM UR ; Ġno . Ġ17 67 27 ). ĠWe Ġwould Ġlike Ġto Ġthank ĠIsabel le ĠAug enstein Ġfor Ġhelpful Ġfeedback . </s> Data ĠStatement Ġ: :: ĠC uration Ġrationale </s> In Ġorder Ġto Ġstudy Ġthe Ġautomatic Ġdetection Ġof Ġstances Ġon Ġpolitical Ġissues , Ġquestions Ġand Ġcandidate Ġresponses Ġon Ġthe Ġvoting Ġadvice Ġapplication Ġsmart vote . ch Ġwere Ġdownloaded . ĠMain ly Ġdata Ġpertaining Ġto Ġnational - level Ġissues Ġwere Ġincluded Ġto Ġreduce Ġvariability . </s> Data ĠStatement Ġ: :: ĠLanguage Ġvariety </s> The Ġtraining Ġset Ġconsists Ġof Ġquestions Ġand Ġanswers Ġin ĠSwiss ĠStandard ĠGerman Ġand ĠSwiss ĠFrench Ġ( 74 . 1 % Ġde - CH ; Ġ25 . 9 % Ġfr - CH ). ĠThe Ġtest Ġsets Ġalso Ġcontain Ġquestions Ġand Ġanswers Ġin ĠSwiss ĠItalian Ġ( 67 . 1 % Ġde - CH ; Ġ24 . 7 % Ġfr - CH ; Ġ8 . 2 % Ġit - CH ). ĠThe Ġquestions Ġhave Ġalso Ġbeen Ġtranslated Ġinto ĠEnglish . </s> Data ĠStatement Ġ: :: ĠSpeaker Ġdemographic Ġ( ans w ers ) </s> Cand idates Ġfor Ġcommunal , Ġcant onal Ġor Ġnational Ġelections Ġin ĠSwitzerland Ġwho Ġhave Ġfilled Ġout Ġan Ġonline Ġquestionnaire . </s> Age : Ġ18 Ġor Ġolder ĠâĢĵ Ġmixed . </s> Gender : ĠUnknown ĠâĢĵ Ġmixed . </s> Race / ethnic ity : ĠUnknown ĠâĢĵ Ġmixed . </s> Native Ġlanguage : ĠUnknown ĠâĢĵ Ġmixed . </s> Soc io economic Ġstatus : ĠUnknown ĠâĢĵ Ġmixed . </s> Different Ġspeakers Ġrepresented : Ġ7 581 . </s> Pres ence Ġof Ġdis ordered Ġspeech : ĠUnknown . </s> Data ĠStatement Ġ: :: ĠSpeech Ġsituation </s> The Ġquestions Ġwere Ġedited Ġand Ġtranslated Ġby Ġpolitical Ġscientists Ġfor Ġa Ġpublic Ġvoting Ġadvice Ġwebsite . </s> The Ġanswers Ġwere Ġwritten Ġbetween Ġ2011 Ġand Ġ2020 Ġby Ġthe Ġusers Ġof Ġthe Ġwebsite . </s> Data ĠStatement Ġ: :: ĠText Ġcharacteristics </s> Questions , Ġanswers , Ġarguments Ġand Ġcomments Ġregarding Ġpolitical Ġissues .
Introduction In recent years many datasets have been created for the task of automated stance detection, advancing natural language understanding systems for political science, opinion research and other application areas. Typically, such benchmarks BIBREF0 are composed of short pieces of text commenting on politicians or public issues and are manually annotated with their stance towards a target entity (e.g. Climate Change, or Trump). However, they are limited in scope on multiple levels BIBREF1. First of all, it is questionable how well current stance detection methods perform in a cross-lingual setting, as the multilingual datasets available today are relatively small, and specific to a single target BIBREF2, BIBREF3. Furthermore, specific models tend to be developed for each single target or pair of targets BIBREF4. Concerns have been raised that cross-target performance is often considerably lower than fully supervised performance BIBREF1. In this paper we propose a much larger dataset that combines multilinguality and a multitude of topics and targets. x-stance comprises more than 150 questions concerning Swiss politics and more than 67k answers given in the last decade by candidates running for political office in Switzerland. Questions are available in four languages: English, Swiss Standard German, French, and Italian. The language of a comment depends on the candidate's region of origin. We have extracted the data from the voting advice application Smartvote. On that platform, candidates respond to questions mainly in categorical form (yes / rather yes / rather no / no). They can also submit a free-text comment in order to justify, explain or differentiate their categorical answer. An example is given in Figure FIGREF1. We transform the dataset into a stance detection task by interpreting the question as a natural-language representation of the target, and the commentary as the input to be classified. The dataset is split into a multilingual training set and into multiple test sets to evaluate zero-shot cross-lingual and cross-target transfer. To provide a baseline, we fine-tune a multilingual Bert model BIBREF5 on x-stance. We show that the baseline accuracy is comparable to previous stance detection benchmarks while leaving ample room for improvement. In addition, multilingual Bert can generalize to a degree both cross-lingually and in a cross-target setting. We have made the dataset and the code for reproducing the baseline model publicly available. Related Work ::: Multilingual Stance Detection In the context of the IberEval shared tasks, two related multilingual datasets have been created BIBREF2, BIBREF3. Both are a collection of annotated Spanish and Catalan tweets. Crucially, the tweets in both languages focus on the same issue (Catalan independence); given this fact they are the first truly multilingual stance detection datasets known to us. With regard to the languages covered by x-stance, only monolingual datasets seem to be available. For French, a collection of tweets on French presidential candidates has been annotated with stance BIBREF6. Similarly, two datasets of Italian tweets on the occasion of the 2016 constitutional referendum have been created BIBREF7, BIBREF6. For German, there is no known stance detection dataset, but an aspect-based sentiment dataset has been created for a GermEval 2017 shared task BIBREF8. Related Work ::: Multi-Target Stance Detection The SemEval-2016 task on detecting stance in tweets BIBREF9 offers data concerning multiple targets (Atheism, Climate Change, Feminism, Hillary Clinton, and Abortion). In the supervised subtask A, participants tended to develop a target-specific model for each of those targets. In subtask B cross-target transfer to the target “Donald Trump” was tested, for which no annotated training data were provided. While this required the development of more universal models, their performance was generally much lower. BIBREF4 introduced a multi-target stance dataset which provides two targets per instance. For example, a model designed in this framework is supposed to simultaneously classify a tweet with regard to Clinton and with regard to Trump. While in theory the framework allows for more than two targets, it is still restricted to a finite and clearly defined set of targets. It focuses on modeling the dependencies of multiple targets within the same text sample, while our approach focuses on learning stance detection from many samples with many different targets. Related Work ::: Representation Learning for Stance Detection In a target-specific setting, BIBREF10 perform a systematic evaluation of stance detection approaches. They also evaluate Bert BIBREF5 and find that it consistently outperforms previous approaches. However, they only experimented with a single-segment encoding of the input, preventing cross-target transfer of the model. BIBREF11 propose a conditional encoding approach to encode both the target and the tweet as sequences. They use a bidirectional LSTM to condition the encoding of the tweets on the encoding of the target, and then apply a nonlinear projection on the conditionally encoded tweet. This allows them to train a model that can generalize to previously unseen targets. The x-stance Dataset ::: Task Definition The input provided by x-stance is two-fold: (A) a natural language question concerning a political issue; (B) a natural language commentary on a specific stance towards the question. The label to be predicted is either `favor' or `against`. This corresponds to a standard established by BIBREF0. However, x-stance differs from that dataset in that it lacks a `neither' class; all comments refer to either a `favor' or an `against` position. The task posed by x-stance is thus a binary classification task. As an evaluation metric we report the macro-average of the F1-score for `favor' and the F1-score for `against', similar to BIBREF9. We use this metric mainly to strengthen comparability with the previous benchmarks. The x-stance Dataset ::: Data Collection ::: Provenance We downloaded questions and answers via the Smartvote API. The downloaded data cover 175 communal, cantonal and national elections between 2011 and 2020. All candidates in an election who participate in Smartvote are asked the same set of questions, but depending on the locale they see translated versions of the questions. They can answer each question with either `yes', `rather yes', `rather no', or `no'. They can supplement each answer with a comment of at most 500 characters. The questions asked on Smartvote have been edited by a team of political scientists. They are intended to cover a broad range of political issues relevant at the time of the election. A detailed documentation of the design of Smartvote and the editing process of the questions is provided by BIBREF12. The x-stance Dataset ::: Data Collection ::: Preprocessing We merged the two labels on each pole into a single label: `yes' and `rather yes' were combined into `favor'; `rather no', or `no' into `against`. This improves the consistency of the data and the comparability to previous stance detection datasets. We did not further preprocess the text of the comments. The x-stance Dataset ::: Data Collection ::: Language Identification As the API does not provide the language of comments, we employed a language identifier to automatically annotate this information. We used the langdetect library BIBREF13. For each responder we classified all the comments jointly, assuming that responders did not switch code during the answering of the questionnaire. We applied the identifier in a two-step approach. In the first run we allowed the identifier to output all 55 languages that it supports out of the box, plus Romansh, the fourth official language in Switzerland. We found that no Romansh comments were detected and that all unexpected outputs were misclassifications of German, French or Italian comments. We further concluded that little or no Swiss German comments are in the dataset: If they were, some of them would have manifested themselves in the form of misclassifications (e.g. as Dutch). In the second run, drawing from these conclusions, we restricted the identifier's output to English, French, German and Italian. The x-stance Dataset ::: Data Collection ::: Filtering We pre-filtered the questions and answers to improve the quality of the dataset. To keep the domain of the data surveyable, we set a focus on national-level questions. Therefore, all questions and corresponding answers pertaining to national elections were included. In the context of communal and cantonal elections, candidates have answered both local questions and a subset of the national questions. Of those elections, we only considered answers to the questions that also had been asked in a national election. Furthermore, they were only used to augment the training set while the validation and test sets were restricted to answers from national elections. We discarded the less than 20 comments classified as English. Furthermore, instances that met any of the following conditions were filtered from the dataset: Question is not a closed question or does not address a clearly defined political issue. No comment was submitted by the candidate or the comment is shorter than 50 characters. Comment starts with “but” or a similar indicator that the comment is not a self-contained statement. Comment contains a URL. In total, a fifth of the original comments were filtered out. The x-stance Dataset ::: Data Collection ::: Topics The questions have been organized by the Smartvote editors into categories (such as “Economy”). We further consolidated the pre-defined categories into 12 broad topics (Table TABREF7). The x-stance Dataset ::: Data Collection ::: Compliance The dataset is shared under a CC BY-NC 4.0 license. Copyright remains with www.smartvote.ch. Given the sensitive nature of the data, we increase the anonymity of the data by hashing the respondents' IDs. No personal attributes of the respondents, such as their party affiliation, are included in the dataset. We provide a data statement BIBREF15 in Appendix SECREF8. The x-stance Dataset ::: Data Split We held out the topics “Healthcare” and “Political System” from the training data and created a separate cross-topic test set that contains the questions and answers related to those topics. Furthermore, in order to test cross-question generalization performance within previously seen topics, we manually selected 16 held-out questions that are distributed over the remaining 10 topics. We selected the held-out questions manually because we wanted to make sure that they are truly unseen and that no paraphrases of the questions are found in the training set. We designated Italian as a test-only language, since relatively few comments have been written in Italian. From the remaining German and French data we randomly selected a percentage of respondents as validation or as test respondents. As a result we received one training set, one validation set and four test sets. The sizes of the sets are listed in Table TABREF8. We did not consider test sets that are cross-lingual and cross-target at the same time, as they would have been too small to yield significant results. The x-stance Dataset ::: Analysis Some observations regarding the composition of x-stance can be made. The x-stance Dataset ::: Analysis ::: Class Distribution Figure FIGREF23 visualizes the proportion of `favor' and `against` stances for each target in the dataset. The ratio differs between questions but is relatively equally distributed across the topics. In particular, the questions belonging to the held-out topics (with a `favor' ratio of 49.4%) have a similar class distribution as the questions within other topics (with a `favor' ratio of 50.0%). The x-stance Dataset ::: Analysis ::: Linguistic Properties Not every question is unique; some questions are paraphrases describing the same political issue. For example, in the 2015 election, the candidates were asked: “Should the consumption of cannabis as well as its possession for personal use be legalised?” Four years later they were asked: “Should cannabis use be legalized?” However, we do not see any need to consolidate those duplicates because they contribute to the diversity of the training data. We further observe that while some questions in the dataset are quite short, some questions are rather convoluted. For example, a typical long question reads: Some 1% of direct payments to Swiss agriculture currently go to organic farming operations. Should this proportion be increased at the expense of standard farming operations as part of Switzerland's 2014-2017 agricultural policy? Such longer questions might be more challenging to process semantically. The x-stance Dataset ::: Analysis ::: Languages The x-stance dataset has more German samples than French samples. The language ratio of about 3:1 is consistent across all training and test sets. Given the two languages it is possible to either train two monolingual models or to train a single model in a multi-source setup BIBREF16. We choose a multi-source baseline because M-Bert is known to benefit from multilingual training data both in a supervised and in a cross-lingual scenario BIBREF17. Baseline Experiments We evaluate two types of baselines to obtain an impression of the difficulty of the task. Baseline Experiments ::: Majority Class Baselines The first pair of baselines uses the most frequent class in the training set for prediction. Specifically, the global majority class baseline predicts the most frequent class across all training targets while the target-wise majority class baseline predicts the class that is most frequent for a given target question. The latter can only be applied to the supervised test set. Baseline Experiments ::: Multilingual BERT Baseline Secondly, we fine-tune multilingual Bert (M-Bert) on the task BIBREF5 which has been pretrained jointly in 104 languages and has established itself as a state of the art for various multilingual tasks BIBREF18, BIBREF19. Within the field of stance detection, Bert can outperform both feature-based and other neural approaches in a monolingual English setting BIBREF10. Baseline Experiments ::: Multilingual BERT Baseline ::: Architecture In the context of Bert we interpret the x-stance task as sequence pair classification inspired by natural language inference tasks BIBREF20. We follow the procedure outlined by BIBREF5 for such tasks. We designate the question as segment A and the comment as segment B. The two segments are separated with the special token [SEP], and the special token [CLS] is prepended to the sequence. The final hidden state corresponding to [CLS] is then classified by a linear layer. We fine-tune the full model with a cross-entropy loss, using the AllenNLP library BIBREF21 as a basis for our implementation. Baseline Experiments ::: Multilingual BERT Baseline ::: Training We upsampled the `favor' class so that the two classes are balanced when summing over all questions and topics. A maximum sequence length of 512 subwords and a batch size of 16 was chosen for all training runs. We then performed a grid search within the following range of hyperparameters based on the validation accuracy: Learning rate: 5e-5, 3e-5, 2e-5 Number of epochs: 3, 4 The grid search was repeated independently for every variant that we tested. Furthermore, the standard recommendations for fine-tuning Bert were used: Adam with ${\beta }_1=0.9$ and ${\beta }_2=0.999$; an L2 weight decay of $0.01$; a learning rate warmup over the first 10% of the steps; and a linear decay of the learning rate. A dropout probability of 0.1 was set on all layers. Baseline Experiments ::: Multilingual BERT Baseline ::: Results Table TABREF36 shows the results for the cross-lingual setting. M-Bert performs consistently better than the majority class baselines. Even the zero-short performance in Italian, while significantly lower than the supervised scores, is much better than the target-wise majority class baseline. Results for the cross-target setting are given in Table TABREF37. Similar to the cross-lingual setting, M-Bert performs worse in a cross-target setting but easily surpasses the majority class baselines. Furthermore, the cross-question score of M-Bert is slightly lower than the cross-topic score. Baseline Experiments ::: How Important is Consistent Language? The default setup preserves horizontal language consistency in that the language of the questions always matches the language of the comments. For example, the Italian test instances are combined with the Italian version of the questions, even though during training the model has only ever seen the German and French versions of the questions. An alternative concept is vertical language consistency, whereby the questions are consistently presented in one language, regardless of the comment. To test whether horizontal or vertical consistency is more helpful, we train and evaluate M-Bert on a dataset variant where all questions are in their English version. We chose English as a lingua franca because it had the largest share of data during the pretraining of M-Bert. The results are shown in Table TABREF39. While the effect is negligible in most settings, the cross-lingual performance clearly increases when all questions are given in English. Baseline Experiments ::: How Important are the Segments? In order to rule out that only the questions or only the comments are necessary to optimally solve the task, we conduct some additional experiments: Only use a single segment containing the comment, removing the questions from the training and test data (missing questions). Only use the question and remove the comment (missing comments). In both cases the performance decreases across all evaluation settings (Table TABREF39). The loss in performance is much higher when comments are missing, indicating that the comments contain the most important information about stance. As can be expected, the score achieved without comments is only slightly different from the target-wise majority class baseline. But there is also a loss in performance when the questions are missing, which underlines the importance of pairing both pieces of text. The effect of missing questions is especially strong in the supervised and cross-lingual settings. To illustrate this, we provide in Table TABREF48 some examples of comments that occur with multiple different targets in the training set. Those examples can explain why the target can be essential for disambiguating a stance detection problem. On the other hand, the effect of omitting the questions is less pronounced in the cross-target settings. The above single-segment experiments tell us that both the comment and the question provide crucial information. But it is possible that the M-Bert model, even though trained on both segments, mainly looks at a single segment at test time. To rule this out, we probe the model with randomized data at test time: Test the model on versions of the test sets where the comments remain in place but the questions are shuffled randomly (random questions). We make sure that the random questions come from the same test set and language as the original questions. Keep the questions in place and randomize the comments (random comments). Again we shuffle the comments only within test set boundaries. The results in Table TABREF39 show that the performance of the model decreases in both cases, confirming that it learns to take into account both segments. Baseline Experiments ::: How Important are Spelled-Out Targets? Finally we test whether the target really needs to be represented by natural language (e.g. “Do you support X?”). Namely, an alternative is to represent the target with a trainable embedding instead of a question. In order to fit target embeddings smoothly into our architecture, we represent each target type with a different reserved symbol from the M-Bert vocabulary. Segment A is then set to this symbol instead of a natural language question. The results for this experiment are listed in the bottom row of Table TABREF39. An M-Bert model that learns target embeddings instead of encoding a question performs clearly worse in the supervised and cross-lingual settings. From this we conclude that spelled-out natural language questions provide important linguistic detail that can help in stance detection. Discussion The baseline experiments confirm that M-Bert can achieve a reasonable accuracy on x-stance. To put the supervised score into context we list scores that variants of Bert have achieved on other stance detection datasets in Table TABREF46. It seems that the supervised part of x-stance has a similar difficulty as the SemEval-2016 BIBREF0 or MPCHI BIBREF22 datasets on which Bert has previously been evaluated. On the other hand, in the cross-lingual and cross-target settings, the mean score drops by 6–8 percentage points compared to the supervised setting; while zero-shot transfer is possible to a degree, it can still be improved. The additional experiments (Table TABREF39) validate the results and show that the sequence-pair classification approach to stance detection is justified. It is interesting to see what errors the M-Bert model makes. Table TABREF47 presents instances where it predicts the wrong label with a high confidence. These examples indicate that many comments express their stance only on a very implicit level, and thus hint at a potential weakness of the dataset. Because on the voting advice platform the label is explicitly shown to readers in addition to the comments, the comments do not need to express the stance explicitly. Manual annotation could eliminate very implicit samples in a future version of the dataset. However, the sheer size and breadth of the dataset could not realistically be achieved with manual annotation, and, in our view, largely compensates for the implicitness of the texts. Conclusion We have presented a new dataset for political stance detection called x-stance. The dataset extends over a broad range of topics and issues regarding national Swiss politics. This diversity of topics opens up an opportunity to further study multi-target learning. Moreover, being partly Swiss Standard German, partly French and Italian, the dataset promotes a multilingual approach to stance detection. By compiling formal commentary that politicians have written on political questions, we add a new text genre to the field of stance detection. We also propose a question–answer format that allows us to condition stance detection models on a target naturally. Our baseline results with multilingual Bert show that the model has some capability to perform zero-shot transfer to unseen languages and to unseen targets (both within a topic and to unseen topics). However, there is some gap in performance that future work could address. We expect that the x-stance dataset could furthermore be a valuable resource for fields such as argument mining, argument search or topic classification. Acknowledgments This work was funded by the Swiss National Science Foundation (project MUTAMUR; no. 176727). We would like to thank Isabelle Augenstein for helpful feedback. Data Statement ::: Curation rationale In order to study the automatic detection of stances on political issues, questions and candidate responses on the voting advice application smartvote.ch were downloaded. Mainly data pertaining to national-level issues were included to reduce variability. Data Statement ::: Language variety The training set consists of questions and answers in Swiss Standard German and Swiss French (74.1% de-CH; 25.9% fr-CH). The test sets also contain questions and answers in Swiss Italian (67.1% de-CH; 24.7% fr-CH; 8.2% it-CH). The questions have also been translated into English. Data Statement ::: Speaker demographic (answers) Candidates for communal, cantonal or national elections in Switzerland who have filled out an online questionnaire. Age: 18 or older – mixed. Gender: Unknown – mixed. Race/ethnicity: Unknown – mixed. Native language: Unknown – mixed. Socioeconomic status: Unknown – mixed. Different speakers represented: 7581. Presence of disordered speech: Unknown. Data Statement ::: Speech situation The questions were edited and translated by political scientists for a public voting advice website. The answers were written between 2011 and 2020 by the users of the website. Data Statement ::: Text characteristics Questions, answers, arguments and comments regarding political issues.
Did they pefrorm any cross-lingual vs single language evaluation?
612c3675b6c55b60ae6d24265ed8e20f62cb117e
Did Ġthey Ġpe fr orm Ġany Ġcross - ling ual Ġvs Ġsingle Ġlanguage Ġevaluation ?
Yes
<s> Did Ġthey Ġpe fr orm Ġany Ġcross - ling ual Ġvs Ġsingle Ġlanguage Ġevaluation ? </s> Introduction </s> In Ġrecent Ġyears Ġmany Ġdatasets Ġhave Ġbeen Ġcreated Ġfor Ġthe Ġtask Ġof Ġautomated Ġstance Ġdetection , Ġadvancing Ġnatural Ġlanguage Ġunderstanding Ġsystems Ġfor Ġpolitical Ġscience , Ġopinion Ġresearch Ġand Ġother Ġapplication Ġareas . ĠTypically , Ġsuch Ġbenchmarks ĠB IB REF 0 Ġare Ġcomposed Ġof Ġshort Ġpieces Ġof Ġtext Ġcommenting Ġon Ġpoliticians Ġor Ġpublic Ġissues Ġand Ġare Ġmanually Ġannot ated Ġwith Ġtheir Ġstance Ġtowards Ġa Ġtarget Ġentity Ġ( e . g . ĠClimate ĠChange , Ġor ĠTrump ). ĠHowever , Ġthey Ġare Ġlimited Ġin Ġscope Ġon Ġmultiple Ġlevels ĠB IB REF 1 . </s> First Ġof Ġall , Ġit Ġis Ġquestionable Ġhow Ġwell Ġcurrent Ġstance Ġdetection Ġmethods Ġperform Ġin Ġa Ġcross - ling ual Ġsetting , Ġas Ġthe Ġmult ilingual Ġdatasets Ġavailable Ġtoday Ġare Ġrelatively Ġsmall , Ġand Ġspecific Ġto Ġa Ġsingle Ġtarget ĠB IB REF 2 , ĠB IB REF 3 . ĠFurthermore , Ġspecific Ġmodels Ġtend Ġto Ġbe Ġdeveloped Ġfor Ġeach Ġsingle Ġtarget Ġor Ġpair Ġof Ġtargets ĠB IB REF 4 . ĠConcern s Ġhave Ġbeen Ġraised Ġthat Ġcross - target Ġperformance Ġis Ġoften Ġconsiderably Ġlower Ġthan Ġfully Ġsupervised Ġperformance ĠB IB REF 1 . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġa Ġmuch Ġlarger Ġdataset Ġthat Ġcombines Ġmult iling uality Ġand Ġa Ġmultitude Ġof Ġtopics Ġand Ġtargets . Ġx - st ance Ġcomprises Ġmore Ġthan Ġ150 Ġquestions Ġconcerning ĠSwiss Ġpolitics Ġand Ġmore Ġthan Ġ67 k Ġanswers Ġgiven Ġin Ġthe Ġlast Ġdecade Ġby Ġcandidates Ġrunning Ġfor Ġpolitical Ġoffice Ġin ĠSwitzerland . </s> Questions Ġare Ġavailable Ġin Ġfour Ġlanguages : ĠEnglish , ĠSwiss ĠStandard ĠGerman , ĠFrench , Ġand ĠItalian . ĠThe Ġlanguage Ġof Ġa Ġcomment Ġdepends Ġon Ġthe Ġcandidate 's Ġregion Ġof Ġorigin . </s> We Ġhave Ġextracted Ġthe Ġdata Ġfrom Ġthe Ġvoting Ġadvice Ġapplication ĠSmart vote . ĠOn Ġthat Ġplatform , Ġcandidates Ġrespond Ġto Ġquestions Ġmainly Ġin Ġcateg orical Ġform Ġ( yes Ġ/ Ġrather Ġyes Ġ/ Ġrather Ġno Ġ/ Ġno ). ĠThey Ġcan Ġalso Ġsubmit Ġa Ġfree - text Ġcomment Ġin Ġorder Ġto Ġjustify , Ġexplain Ġor Ġdifferentiate Ġtheir Ġcateg orical Ġanswer . ĠAn Ġexample Ġis Ġgiven Ġin ĠFigure ĠFIG REF 1 . </s> We Ġtransform Ġthe Ġdataset Ġinto Ġa Ġstance Ġdetection Ġtask Ġby Ġinterpreting Ġthe Ġquestion Ġas Ġa Ġnatural - language Ġrepresentation Ġof Ġthe Ġtarget , Ġand Ġthe Ġcommentary Ġas Ġthe Ġinput Ġto Ġbe Ġclassified . </s> The Ġdataset Ġis Ġsplit Ġinto Ġa Ġmult ilingual Ġtraining Ġset Ġand Ġinto Ġmultiple Ġtest Ġsets Ġto Ġevaluate Ġzero - shot Ġcross - ling ual Ġand Ġcross - target Ġtransfer . ĠTo Ġprovide Ġa Ġbaseline , Ġwe Ġfine - t une Ġa Ġmult ilingual ĠBert Ġmodel ĠB IB REF 5 Ġon Ġx - st ance . ĠWe Ġshow Ġthat Ġthe Ġbaseline Ġaccuracy Ġis Ġcomparable Ġto Ġprevious Ġstance Ġdetection Ġbenchmarks Ġwhile Ġleaving Ġample Ġroom Ġfor Ġimprovement . ĠIn Ġaddition , Ġmult ilingual ĠBert Ġcan Ġgeneral ize Ġto Ġa Ġdegree Ġboth Ġcross - ling ually Ġand Ġin Ġa Ġcross - target Ġsetting . </s> We Ġhave Ġmade Ġthe Ġdataset Ġand Ġthe Ġcode Ġfor Ġreprodu cing Ġthe Ġbaseline Ġmodel Ġpublicly Ġavailable . </s> Related ĠWork Ġ: :: ĠMult ilingual ĠSt ance ĠDetection </s> In Ġthe Ġcontext Ġof Ġthe ĠI ber E val Ġshared Ġtasks , Ġtwo Ġrelated Ġmult ilingual Ġdatasets Ġhave Ġbeen Ġcreated ĠB IB REF 2 , ĠB IB REF 3 . ĠBoth Ġare Ġa Ġcollection Ġof Ġannot ated ĠSpanish Ġand ĠCatalan Ġtweets . ĠCru cially , Ġthe Ġtweets Ġin Ġboth Ġlanguages Ġfocus Ġon Ġthe Ġsame Ġissue Ġ( Catal an Ġindependence ); Ġgiven Ġthis Ġfact Ġthey Ġare Ġthe Ġfirst Ġtruly Ġmult ilingual Ġstance Ġdetection Ġdatasets Ġknown Ġto Ġus . </s> With Ġregard Ġto Ġthe Ġlanguages Ġcovered Ġby Ġx - st ance , Ġonly Ġmon oling ual Ġdatasets Ġseem Ġto Ġbe Ġavailable . ĠFor ĠFrench , Ġa Ġcollection Ġof Ġtweets Ġon ĠFrench Ġpresidential Ġcandidates Ġhas Ġbeen Ġannot ated Ġwith Ġstance ĠB IB REF 6 . ĠSimilarly , Ġtwo Ġdatasets Ġof ĠItalian Ġtweets Ġon Ġthe Ġoccasion Ġof Ġthe Ġ2016 Ġconstitutional Ġreferendum Ġhave Ġbeen Ġcreated ĠB IB REF 7 , ĠB IB REF 6 . ĠFor ĠGerman , Ġthere Ġis Ġno Ġknown Ġstance Ġdetection Ġdataset , Ġbut Ġan Ġaspect - based Ġsentiment Ġdataset Ġhas Ġbeen Ġcreated Ġfor Ġa ĠGerm E val Ġ2017 Ġshared Ġtask ĠB IB REF 8 . </s> Related ĠWork Ġ: :: ĠMulti - Target ĠSt ance ĠDetection </s> The ĠSem E val - 2016 Ġtask Ġon Ġdetecting Ġstance Ġin Ġtweets ĠB IB REF 9 Ġoffers Ġdata Ġconcerning Ġmultiple Ġtargets Ġ( A the ism , ĠClimate ĠChange , ĠFemin ism , ĠHillary ĠClinton , Ġand ĠAbortion ). ĠIn Ġthe Ġsupervised Ġsubt ask ĠA , Ġparticipants Ġtended Ġto Ġdevelop Ġa Ġtarget - specific Ġmodel Ġfor Ġeach Ġof Ġthose Ġtargets . ĠIn Ġsubt ask ĠB Ġcross - target Ġtransfer Ġto Ġthe Ġtarget ĠâĢ ľ Donald ĠTrump âĢ Ŀ Ġwas Ġtested , Ġfor Ġwhich Ġno Ġannot ated Ġtraining Ġdata Ġwere Ġprovided . ĠWhile Ġthis Ġrequired Ġthe Ġdevelopment Ġof Ġmore Ġuniversal Ġmodels , Ġtheir Ġperformance Ġwas Ġgenerally Ġmuch Ġlower . </s> BI B REF 4 Ġintroduced Ġa Ġmulti - target Ġstance Ġdataset Ġwhich Ġprovides Ġtwo Ġtargets Ġper Ġinstance . ĠFor Ġexample , Ġa Ġmodel Ġdesigned Ġin Ġthis Ġframework Ġis Ġsupposed Ġto Ġsimultaneously Ġclassify Ġa Ġtweet Ġwith Ġregard Ġto ĠClinton Ġand Ġwith Ġregard Ġto ĠTrump . ĠWhile Ġin Ġtheory Ġthe Ġframework Ġallows Ġfor Ġmore Ġthan Ġtwo Ġtargets , Ġit Ġis Ġstill Ġrestricted Ġto Ġa Ġfinite Ġand Ġclearly Ġdefined Ġset Ġof Ġtargets . ĠIt Ġfocuses Ġon Ġmodeling Ġthe Ġdependencies Ġof Ġmultiple Ġtargets Ġwithin Ġthe Ġsame Ġtext Ġsample , Ġwhile Ġour Ġapproach Ġfocuses Ġon Ġlearning Ġstance Ġdetection Ġfrom Ġmany Ġsamples Ġwith Ġmany Ġdifferent Ġtargets . </s> Related ĠWork Ġ: :: ĠRepresent ation ĠLearning Ġfor ĠSt ance ĠDetection </s> In Ġa Ġtarget - specific Ġsetting , ĠB IB REF 10 Ġperform Ġa Ġsystematic Ġevaluation Ġof Ġstance Ġdetection Ġapproaches . ĠThey Ġalso Ġevaluate ĠBert ĠB IB REF 5 Ġand Ġfind Ġthat Ġit Ġconsistently Ġoutper forms Ġprevious Ġapproaches . </s> However , Ġthey Ġonly Ġexperimented Ġwith Ġa Ġsingle - se gment Ġencoding Ġof Ġthe Ġinput , Ġpreventing Ġcross - target Ġtransfer Ġof Ġthe Ġmodel . ĠB IB REF 11 Ġpropose Ġa Ġconditional Ġencoding Ġapproach Ġto Ġencode Ġboth Ġthe Ġtarget Ġand Ġthe Ġtweet Ġas Ġsequences . ĠThey Ġuse Ġa Ġbid irection al ĠL ST M Ġto Ġcondition Ġthe Ġencoding Ġof Ġthe Ġtweets Ġon Ġthe Ġencoding Ġof Ġthe Ġtarget , Ġand Ġthen Ġapply Ġa Ġnon linear Ġprojection Ġon Ġthe Ġcondition ally Ġencoded Ġtweet . ĠThis Ġallows Ġthem Ġto Ġtrain Ġa Ġmodel Ġthat Ġcan Ġgeneral ize Ġto Ġpreviously Ġunseen Ġtargets . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠTask ĠDefinition </s> The Ġinput Ġprovided Ġby Ġx - st ance Ġis Ġtwo - fold : Ġ( A ) Ġa Ġnatural Ġlanguage Ġquestion Ġconcerning Ġa Ġpolitical Ġissue ; Ġ( B ) Ġa Ġnatural Ġlanguage Ġcommentary Ġon Ġa Ġspecific Ġstance Ġtowards Ġthe Ġquestion . </s> The Ġlabel Ġto Ġbe Ġpredicted Ġis Ġeither Ġ` f avor ' Ġor Ġ` against `. ĠThis Ġcorresponds Ġto Ġa Ġstandard Ġestablished Ġby ĠB IB REF 0 . ĠHowever , Ġx - st ance Ġdiffers Ġfrom Ġthat Ġdataset Ġin Ġthat Ġit Ġlacks Ġa Ġ` ne ither ' Ġclass ; Ġall Ġcomments Ġrefer Ġto Ġeither Ġa Ġ` f avor ' Ġor Ġan Ġ` against ` Ġposition . ĠThe Ġtask Ġposed Ġby Ġx - st ance Ġis Ġthus Ġa Ġbinary Ġclassification Ġtask . </s> As Ġan Ġevaluation Ġmetric Ġwe Ġreport Ġthe Ġmacro - average Ġof Ġthe ĠF 1 - score Ġfor Ġ` f avor ' Ġand Ġthe ĠF 1 - score Ġfor Ġ` against ', Ġsimilar Ġto ĠB IB REF 9 . ĠWe Ġuse Ġthis Ġmetric Ġmainly Ġto Ġstrengthen Ġcompar ability Ġwith Ġthe Ġprevious Ġbenchmarks . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠPro ven ance </s> We Ġdownloaded Ġquestions Ġand Ġanswers Ġvia Ġthe ĠSmart vote ĠAPI . ĠThe Ġdownloaded Ġdata Ġcover Ġ175 Ġcommunal , Ġcant onal Ġand Ġnational Ġelections Ġbetween Ġ2011 Ġand Ġ2020 . </s> All Ġcandidates Ġin Ġan Ġelection Ġwho Ġparticipate Ġin ĠSmart vote Ġare Ġasked Ġthe Ġsame Ġset Ġof Ġquestions , Ġbut Ġdepending Ġon Ġthe Ġlocale Ġthey Ġsee Ġtranslated Ġversions Ġof Ġthe Ġquestions . ĠThey Ġcan Ġanswer Ġeach Ġquestion Ġwith Ġeither Ġ` yes ', Ġ` rather Ġyes ', Ġ` rather Ġno ', Ġor Ġ` no '. ĠThey Ġcan Ġsupplement Ġeach Ġanswer Ġwith Ġa Ġcomment Ġof Ġat Ġmost Ġ500 Ġcharacters . </s> The Ġquestions Ġasked Ġon ĠSmart vote Ġhave Ġbeen Ġedited Ġby Ġa Ġteam Ġof Ġpolitical Ġscientists . ĠThey Ġare Ġintended Ġto Ġcover Ġa Ġbroad Ġrange Ġof Ġpolitical Ġissues Ġrelevant Ġat Ġthe Ġtime Ġof Ġthe Ġelection . ĠA Ġdetailed Ġdocumentation Ġof Ġthe Ġdesign Ġof ĠSmart vote Ġand Ġthe Ġediting Ġprocess Ġof Ġthe Ġquestions Ġis Ġprovided Ġby ĠB IB REF 12 . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠPre processing </s> We Ġmerged Ġthe Ġtwo Ġlabels Ġon Ġeach Ġpole Ġinto Ġa Ġsingle Ġlabel : Ġ` yes ' Ġand Ġ` rather Ġyes ' Ġwere Ġcombined Ġinto Ġ` f avor '; Ġ` rather Ġno ', Ġor Ġ` no ' Ġinto Ġ` against `. ĠThis Ġimproves Ġthe Ġconsistency Ġof Ġthe Ġdata Ġand Ġthe Ġcompar ability Ġto Ġprevious Ġstance Ġdetection Ġdatasets . </s> We Ġdid Ġnot Ġfurther Ġpre process Ġthe Ġtext Ġof Ġthe Ġcomments . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠLanguage ĠIdentification </s> As Ġthe ĠAPI Ġdoes Ġnot Ġprovide Ġthe Ġlanguage Ġof Ġcomments , Ġwe Ġemployed Ġa Ġlanguage Ġidentifier Ġto Ġautomatically Ġannot ate Ġthis Ġinformation . ĠWe Ġused Ġthe Ġlang det ect Ġlibrary ĠB IB REF 13 . ĠFor Ġeach Ġrespond er Ġwe Ġclassified Ġall Ġthe Ġcomments Ġjointly , Ġassuming Ġthat Ġresponders Ġdid Ġnot Ġswitch Ġcode Ġduring Ġthe Ġanswering Ġof Ġthe Ġquestionnaire . </s> We Ġapplied Ġthe Ġidentifier Ġin Ġa Ġtwo - step Ġapproach . ĠIn Ġthe Ġfirst Ġrun Ġwe Ġallowed Ġthe Ġidentifier Ġto Ġoutput Ġall Ġ55 Ġlanguages Ġthat Ġit Ġsupports Ġout Ġof Ġthe Ġbox , Ġplus ĠRomans h , Ġthe Ġfourth Ġofficial Ġlanguage Ġin ĠSwitzerland . ĠWe Ġfound Ġthat Ġno ĠRomans h Ġcomments Ġwere Ġdetected Ġand Ġthat Ġall Ġunexpected Ġoutputs Ġwere Ġmis class ifications Ġof ĠGerman , ĠFrench Ġor ĠItalian Ġcomments . ĠWe Ġfurther Ġconcluded Ġthat Ġlittle Ġor Ġno ĠSwiss ĠGerman Ġcomments Ġare Ġin Ġthe Ġdataset : ĠIf Ġthey Ġwere , Ġsome Ġof Ġthem Ġwould Ġhave Ġmanifested Ġthemselves Ġin Ġthe Ġform Ġof Ġmis class ifications Ġ( e . g . Ġas ĠDutch ). </s> In Ġthe Ġsecond Ġrun , Ġdrawing Ġfrom Ġthese Ġconclusions , Ġwe Ġrestricted Ġthe Ġidentifier 's Ġoutput Ġto ĠEnglish , ĠFrench , ĠGerman Ġand ĠItalian . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠFil tering </s> We Ġpre - fil tered Ġthe Ġquestions Ġand Ġanswers Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġdataset . ĠTo Ġkeep Ġthe Ġdomain Ġof Ġthe Ġdata Ġsurvey able , Ġwe Ġset Ġa Ġfocus Ġon Ġnational - level Ġquestions . ĠTherefore , Ġall Ġquestions Ġand Ġcorresponding Ġanswers Ġpertaining Ġto Ġnational Ġelections Ġwere Ġincluded . </s> In Ġthe Ġcontext Ġof Ġcommunal Ġand Ġcant onal Ġelections , Ġcandidates Ġhave Ġanswered Ġboth Ġlocal Ġquestions Ġand Ġa Ġsubset Ġof Ġthe Ġnational Ġquestions . ĠOf Ġthose Ġelections , Ġwe Ġonly Ġconsidered Ġanswers Ġto Ġthe Ġquestions Ġthat Ġalso Ġhad Ġbeen Ġasked Ġin Ġa Ġnational Ġelection . ĠFurthermore , Ġthey Ġwere Ġonly Ġused Ġto Ġaugment Ġthe Ġtraining Ġset Ġwhile Ġthe Ġvalidation Ġand Ġtest Ġsets Ġwere Ġrestricted Ġto Ġanswers Ġfrom Ġnational Ġelections . </s> We Ġdiscarded Ġthe Ġless Ġthan Ġ20 Ġcomments Ġclassified Ġas ĠEnglish . ĠFurthermore , Ġinstances Ġthat Ġmet Ġany Ġof Ġthe Ġfollowing Ġconditions Ġwere Ġfiltered Ġfrom Ġthe Ġdataset : </s> Question Ġis Ġnot Ġa Ġclosed Ġquestion Ġor Ġdoes Ġnot Ġaddress Ġa Ġclearly Ġdefined Ġpolitical Ġissue . </s> No Ġcomment Ġwas Ġsubmitted Ġby Ġthe Ġcandidate Ġor Ġthe Ġcomment Ġis Ġshorter Ġthan Ġ50 Ġcharacters . </s> Comment Ġstarts Ġwith ĠâĢ ľ but âĢ Ŀ Ġor Ġa Ġsimilar Ġindicator Ġthat Ġthe Ġcomment Ġis Ġnot Ġa Ġself - contained Ġstatement . </s> Comment Ġcontains Ġa ĠURL . </s> In Ġtotal , Ġa Ġfifth Ġof Ġthe Ġoriginal Ġcomments Ġwere Ġfiltered Ġout . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠTopics </s> The Ġquestions Ġhave Ġbeen Ġorganized Ġby Ġthe ĠSmart vote Ġeditors Ġinto Ġcategories Ġ( such Ġas ĠâĢ ľ Econom y âĢ Ŀ ). ĠWe Ġfurther Ġconsolidated Ġthe Ġpre - defined Ġcategories Ġinto Ġ12 Ġbroad Ġtopics Ġ( Table ĠT AB REF 7 ). </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠCollection Ġ: :: ĠCompliance </s> The Ġdataset Ġis Ġshared Ġunder Ġa ĠCC ĠBY - NC Ġ4 . 0 Ġlicense . ĠCopyright Ġremains Ġwith Ġwww . smart vote . ch . </s> Given Ġthe Ġsensitive Ġnature Ġof Ġthe Ġdata , Ġwe Ġincrease Ġthe Ġanonymity Ġof Ġthe Ġdata Ġby Ġhashing Ġthe Ġrespondents ' ĠIDs . ĠNo Ġpersonal Ġattributes Ġof Ġthe Ġrespondents , Ġsuch Ġas Ġtheir Ġparty Ġaffiliation , Ġare Ġincluded Ġin Ġthe Ġdataset . ĠWe Ġprovide Ġa Ġdata Ġstatement ĠB IB REF 15 Ġin ĠAppendix ĠSEC REF 8 . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠData ĠSplit </s> We Ġheld Ġout Ġthe Ġtopics ĠâĢ ľ Health care âĢ Ŀ Ġand ĠâĢ ľ Political ĠSystem âĢ Ŀ Ġfrom Ġthe Ġtraining Ġdata Ġand Ġcreated Ġa Ġseparate Ġcross - topic Ġtest Ġset Ġthat Ġcontains Ġthe Ġquestions Ġand Ġanswers Ġrelated Ġto Ġthose Ġtopics . </s> Furthermore , Ġin Ġorder Ġto Ġtest Ġcross - question Ġgeneral ization Ġperformance Ġwithin Ġpreviously Ġseen Ġtopics , Ġwe Ġmanually Ġselected Ġ16 Ġheld - out Ġquestions Ġthat Ġare Ġdistributed Ġover Ġthe Ġremaining Ġ10 Ġtopics . ĠWe Ġselected Ġthe Ġheld - out Ġquestions Ġmanually Ġbecause Ġwe Ġwanted Ġto Ġmake Ġsure Ġthat Ġthey Ġare Ġtruly Ġunseen Ġand Ġthat Ġno Ġparaph r ases Ġof Ġthe Ġquestions Ġare Ġfound Ġin Ġthe Ġtraining Ġset . </s> We Ġdesignated ĠItalian Ġas Ġa Ġtest - only Ġlanguage , Ġsince Ġrelatively Ġfew Ġcomments Ġhave Ġbeen Ġwritten Ġin ĠItalian . ĠFrom Ġthe Ġremaining ĠGerman Ġand ĠFrench Ġdata Ġwe Ġrandomly Ġselected Ġa Ġpercentage Ġof Ġrespondents Ġas Ġvalidation Ġor Ġas Ġtest Ġrespondents . </s> As Ġa Ġresult Ġwe Ġreceived Ġone Ġtraining Ġset , Ġone Ġvalidation Ġset Ġand Ġfour Ġtest Ġsets . ĠThe Ġsizes Ġof Ġthe Ġsets Ġare Ġlisted Ġin ĠTable ĠT AB REF 8 . ĠWe Ġdid Ġnot Ġconsider Ġtest Ġsets Ġthat Ġare Ġcross - ling ual Ġand Ġcross - target Ġat Ġthe Ġsame Ġtime , Ġas Ġthey Ġwould Ġhave Ġbeen Ġtoo Ġsmall Ġto Ġyield Ġsignificant Ġresults . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis </s> Some Ġobservations Ġregarding Ġthe Ġcomposition Ġof Ġx - st ance Ġcan Ġbe Ġmade . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠClass ĠDistribution </s> Figure ĠFIG REF 23 Ġvisual izes Ġthe Ġproportion Ġof Ġ` f avor ' Ġand Ġ` against ` Ġstances Ġfor Ġeach Ġtarget Ġin Ġthe Ġdataset . ĠThe Ġratio Ġdiffers Ġbetween Ġquestions Ġbut Ġis Ġrelatively Ġequally Ġdistributed Ġacross Ġthe Ġtopics . ĠIn Ġparticular , Ġthe Ġquestions Ġbelonging Ġto Ġthe Ġheld - out Ġtopics Ġ( with Ġa Ġ` f avor ' Ġratio Ġof Ġ49 . 4 %) Ġhave Ġa Ġsimilar Ġclass Ġdistribution Ġas Ġthe Ġquestions Ġwithin Ġother Ġtopics Ġ( with Ġa Ġ` f avor ' Ġratio Ġof Ġ50 . 0 %). </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠL ingu istic ĠProperties </s> Not Ġevery Ġquestion Ġis Ġunique ; Ġsome Ġquestions Ġare Ġparaph r ases Ġdescribing Ġthe Ġsame Ġpolitical Ġissue . ĠFor Ġexample , Ġin Ġthe Ġ2015 Ġelection , Ġthe Ġcandidates Ġwere Ġasked : ĠâĢ ľ Should Ġthe Ġconsumption Ġof Ġcannabis Ġas Ġwell Ġas Ġits Ġpossession Ġfor Ġpersonal Ġuse Ġbe Ġlegal ised ? âĢ Ŀ ĠFour Ġyears Ġlater Ġthey Ġwere Ġasked : ĠâĢ ľ Should Ġcannabis Ġuse Ġbe Ġlegalized ? âĢ Ŀ ĠHowever , Ġwe Ġdo Ġnot Ġsee Ġany Ġneed Ġto Ġconsolidate Ġthose Ġdupl icates Ġbecause Ġthey Ġcontribute Ġto Ġthe Ġdiversity Ġof Ġthe Ġtraining Ġdata . </s> We Ġfurther Ġobserve Ġthat Ġwhile Ġsome Ġquestions Ġin Ġthe Ġdataset Ġare Ġquite Ġshort , Ġsome Ġquestions Ġare Ġrather Ġconvoluted . ĠFor Ġexample , Ġa Ġtypical Ġlong Ġquestion Ġreads : </s> Some Ġ1 % Ġof Ġdirect Ġpayments Ġto ĠSwiss Ġagriculture Ġcurrently Ġgo Ġto Ġorganic Ġfarming Ġoperations . ĠShould Ġthis Ġproportion Ġbe Ġincreased Ġat Ġthe Ġexpense Ġof Ġstandard Ġfarming Ġoperations Ġas Ġpart Ġof ĠSwitzerland 's Ġ2014 - 2017 Ġagricultural Ġpolicy ? </s> Such Ġlonger Ġquestions Ġmight Ġbe Ġmore Ġchallenging Ġto Ġprocess Ġsem antically . </s> The Ġx - st ance ĠDat as et Ġ: :: ĠAnalysis Ġ: :: ĠLanguages </s> The Ġx - st ance Ġdataset Ġhas Ġmore ĠGerman Ġsamples Ġthan ĠFrench Ġsamples . ĠThe Ġlanguage Ġratio Ġof Ġabout Ġ3 : 1 Ġis Ġconsistent Ġacross Ġall Ġtraining Ġand Ġtest Ġsets . ĠGiven Ġthe Ġtwo Ġlanguages Ġit Ġis Ġpossible Ġto Ġeither Ġtrain Ġtwo Ġmon oling ual Ġmodels Ġor Ġto Ġtrain Ġa Ġsingle Ġmodel Ġin Ġa Ġmulti - source Ġsetup ĠB IB REF 16 . ĠWe Ġchoose Ġa Ġmulti - source Ġbaseline Ġbecause ĠM - B ert Ġis Ġknown Ġto Ġbenefit Ġfrom Ġmult ilingual Ġtraining Ġdata Ġboth Ġin Ġa Ġsupervised Ġand Ġin Ġa Ġcross - ling ual Ġscenario ĠB IB REF 17 . </s> Bas eline ĠExper iments </s> We Ġevaluate Ġtwo Ġtypes Ġof Ġbas elines Ġto Ġobtain Ġan Ġimpression Ġof Ġthe Ġdifficulty Ġof Ġthe Ġtask . </s> Bas eline ĠExper iments Ġ: :: ĠMajority ĠClass ĠBas elines </s> The Ġfirst Ġpair Ġof Ġbas elines Ġuses Ġthe Ġmost Ġfrequent Ġclass Ġin Ġthe Ġtraining Ġset Ġfor Ġprediction . ĠSpecifically , Ġthe Ġglobal Ġmajority Ġclass Ġbaseline Ġpredicts Ġthe Ġmost Ġfrequent Ġclass Ġacross Ġall Ġtraining Ġtargets Ġwhile Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline Ġpredicts Ġthe Ġclass Ġthat Ġis Ġmost Ġfrequent Ġfor Ġa Ġgiven Ġtarget Ġquestion . ĠThe Ġlatter Ġcan Ġonly Ġbe Ġapplied Ġto Ġthe Ġsupervised Ġtest Ġset . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline </s> Secondly , Ġwe Ġfine - t une Ġmult ilingual ĠBert Ġ( M - B ert ) Ġon Ġthe Ġtask ĠB IB REF 5 Ġwhich Ġhas Ġbeen Ġpret rained Ġjointly Ġin Ġ104 Ġlanguages Ġand Ġhas Ġestablished Ġitself Ġas Ġa Ġstate Ġof Ġthe Ġart Ġfor Ġvarious Ġmult ilingual Ġtasks ĠB IB REF 18 , ĠB IB REF 19 . ĠWithin Ġthe Ġfield Ġof Ġstance Ġdetection , ĠBert Ġcan Ġoutper form Ġboth Ġfeature - based Ġand Ġother Ġneural Ġapproaches Ġin Ġa Ġmon oling ual ĠEnglish Ġsetting ĠB IB REF 10 . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠArchitecture </s> In Ġthe Ġcontext Ġof ĠBert Ġwe Ġinterpret Ġthe Ġx - st ance Ġtask Ġas Ġsequence Ġpair Ġclassification Ġinspired Ġby Ġnatural Ġlanguage Ġinference Ġtasks ĠB IB REF 20 . ĠWe Ġfollow Ġthe Ġprocedure Ġoutlined Ġby ĠB IB REF 5 Ġfor Ġsuch Ġtasks . ĠWe Ġdesignate Ġthe Ġquestion Ġas Ġsegment ĠA Ġand Ġthe Ġcomment Ġas Ġsegment ĠB . ĠThe Ġtwo Ġsegments Ġare Ġseparated Ġwith Ġthe Ġspecial Ġtoken Ġ[ SE P ], Ġand Ġthe Ġspecial Ġtoken Ġ[ CL S ] Ġis Ġprep ended Ġto Ġthe Ġsequence . ĠThe Ġfinal Ġhidden Ġstate Ġcorresponding Ġto Ġ[ CL S ] Ġis Ġthen Ġclassified Ġby Ġa Ġlinear Ġlayer . </s> We Ġfine - t une Ġthe Ġfull Ġmodel Ġwith Ġa Ġcross - ent ropy Ġloss , Ġusing Ġthe ĠAllen N LP Ġlibrary ĠB IB REF 21 Ġas Ġa Ġbasis Ġfor Ġour Ġimplementation . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠTraining </s> We Ġups am pled Ġthe Ġ` f avor ' Ġclass Ġso Ġthat Ġthe Ġtwo Ġclasses Ġare Ġbalanced Ġwhen Ġsum ming Ġover Ġall Ġquestions Ġand Ġtopics . ĠA Ġmaximum Ġsequence Ġlength Ġof Ġ512 Ġsub words Ġand Ġa Ġbatch Ġsize Ġof Ġ16 Ġwas Ġchosen Ġfor Ġall Ġtraining Ġruns . ĠWe Ġthen Ġperformed Ġa Ġgrid Ġsearch Ġwithin Ġthe Ġfollowing Ġrange Ġof Ġhyper param eters Ġbased Ġon Ġthe Ġvalidation Ġaccuracy : </s> Learning Ġrate : Ġ5 e - 5 , Ġ3 e - 5 , Ġ2 e - 5 </s> Number Ġof Ġepoch s : Ġ3 , Ġ4 </s> The Ġgrid Ġsearch Ġwas Ġrepeated Ġindependently Ġfor Ġevery Ġvariant Ġthat Ġwe Ġtested . ĠFurthermore , Ġthe Ġstandard Ġrecommendations Ġfor Ġfine - tun ing ĠBert Ġwere Ġused : ĠAdam Ġwith Ġ${ \ beta Ġ} _ 1 = 0 . 9 $ Ġand Ġ${ \ beta Ġ} _ 2 = 0 . 999 $ ; Ġan ĠL 2 Ġweight Ġdecay Ġof Ġ$ 0 . 01 $ ; Ġa Ġlearning Ġrate Ġwarm up Ġover Ġthe Ġfirst Ġ10 % Ġof Ġthe Ġsteps ; Ġand Ġa Ġlinear Ġdecay Ġof Ġthe Ġlearning Ġrate . ĠA Ġdrop out Ġprobability Ġof Ġ0 . 1 Ġwas Ġset Ġon Ġall Ġlayers . </s> Bas eline ĠExper iments Ġ: :: ĠMult ilingual ĠB ERT ĠBas eline Ġ: :: ĠResults </s> Table ĠT AB REF 36 Ġshows Ġthe Ġresults Ġfor Ġthe Ġcross - ling ual Ġsetting . ĠM - B ert Ġperforms Ġconsistently Ġbetter Ġthan Ġthe Ġmajority Ġclass Ġbas elines . ĠEven Ġthe Ġzero - short Ġperformance Ġin ĠItalian , Ġwhile Ġsignificantly Ġlower Ġthan Ġthe Ġsupervised Ġscores , Ġis Ġmuch Ġbetter Ġthan Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline . </s> Results Ġfor Ġthe Ġcross - target Ġsetting Ġare Ġgiven Ġin ĠTable ĠT AB REF 37 . ĠSimilar Ġto Ġthe Ġcross - ling ual Ġsetting , ĠM - B ert Ġperforms Ġworse Ġin Ġa Ġcross - target Ġsetting Ġbut Ġeasily Ġsurpass es Ġthe Ġmajority Ġclass Ġbas elines . ĠFurthermore , Ġthe Ġcross - question Ġscore Ġof ĠM - B ert Ġis Ġslightly Ġlower Ġthan Ġthe Ġcross - topic Ġscore . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġis ĠCons istent ĠLanguage ? </s> The Ġdefault Ġsetup Ġpreserves Ġhorizontal Ġlanguage Ġconsistency Ġin Ġthat Ġthe Ġlanguage Ġof Ġthe Ġquestions Ġalways Ġmatches Ġthe Ġlanguage Ġof Ġthe Ġcomments . ĠFor Ġexample , Ġthe ĠItalian Ġtest Ġinstances Ġare Ġcombined Ġwith Ġthe ĠItalian Ġversion Ġof Ġthe Ġquestions , Ġeven Ġthough Ġduring Ġtraining Ġthe Ġmodel Ġhas Ġonly Ġever Ġseen Ġthe ĠGerman Ġand ĠFrench Ġversions Ġof Ġthe Ġquestions . </s> An Ġalternative Ġconcept Ġis Ġvertical Ġlanguage Ġconsistency , Ġwhereby Ġthe Ġquestions Ġare Ġconsistently Ġpresented Ġin Ġone Ġlanguage , Ġregardless Ġof Ġthe Ġcomment . ĠTo Ġtest Ġwhether Ġhorizontal Ġor Ġvertical Ġconsistency Ġis Ġmore Ġhelpful , Ġwe Ġtrain Ġand Ġevaluate ĠM - B ert Ġon Ġa Ġdataset Ġvariant Ġwhere Ġall Ġquestions Ġare Ġin Ġtheir ĠEnglish Ġversion . ĠWe Ġchose ĠEnglish Ġas Ġa Ġling ua Ġfr anca Ġbecause Ġit Ġhad Ġthe Ġlargest Ġshare Ġof Ġdata Ġduring Ġthe Ġpret raining Ġof ĠM - B ert . </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 39 . ĠWhile Ġthe Ġeffect Ġis Ġnegligible Ġin Ġmost Ġsettings , Ġthe Ġcross - ling ual Ġperformance Ġclearly Ġincreases Ġwhen Ġall Ġquestions Ġare Ġgiven Ġin ĠEnglish . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġare Ġthe ĠSe gments ? </s> In Ġorder Ġto Ġrule Ġout Ġthat Ġonly Ġthe Ġquestions Ġor Ġonly Ġthe Ġcomments Ġare Ġnecessary Ġto Ġoptim ally Ġsolve Ġthe Ġtask , Ġwe Ġconduct Ġsome Ġadditional Ġexperiments : </s> Only Ġuse Ġa Ġsingle Ġsegment Ġcontaining Ġthe Ġcomment , Ġremoving Ġthe Ġquestions Ġfrom Ġthe Ġtraining Ġand Ġtest Ġdata Ġ( missing Ġquestions ). </s> Only Ġuse Ġthe Ġquestion Ġand Ġremove Ġthe Ġcomment Ġ( missing Ġcomments ). </s> In Ġboth Ġcases Ġthe Ġperformance Ġdecreases Ġacross Ġall Ġevaluation Ġsettings Ġ( Table ĠT AB REF 39 ). ĠThe Ġloss Ġin Ġperformance Ġis Ġmuch Ġhigher Ġwhen Ġcomments Ġare Ġmissing , Ġindicating Ġthat Ġthe Ġcomments Ġcontain Ġthe Ġmost Ġimportant Ġinformation Ġabout Ġstance . ĠAs Ġcan Ġbe Ġexpected , Ġthe Ġscore Ġachieved Ġwithout Ġcomments Ġis Ġonly Ġslightly Ġdifferent Ġfrom Ġthe Ġtarget - wise Ġmajority Ġclass Ġbaseline . </s> But Ġthere Ġis Ġalso Ġa Ġloss Ġin Ġperformance Ġwhen Ġthe Ġquestions Ġare Ġmissing , Ġwhich Ġunder lines Ġthe Ġimportance Ġof Ġpairing Ġboth Ġpieces Ġof Ġtext . ĠThe Ġeffect Ġof Ġmissing Ġquestions Ġis Ġespecially Ġstrong Ġin Ġthe Ġsupervised Ġand Ġcross - ling ual Ġsettings . ĠTo Ġillustrate Ġthis , Ġwe Ġprovide Ġin ĠTable ĠT AB REF 48 Ġsome Ġexamples Ġof Ġcomments Ġthat Ġoccur Ġwith Ġmultiple Ġdifferent Ġtargets Ġin Ġthe Ġtraining Ġset . ĠThose Ġexamples Ġcan Ġexplain Ġwhy Ġthe Ġtarget Ġcan Ġbe Ġessential Ġfor Ġdis amb ig uating Ġa Ġstance Ġdetection Ġproblem . ĠOn Ġthe Ġother Ġhand , Ġthe Ġeffect Ġof Ġo mitting Ġthe Ġquestions Ġis Ġless Ġpronounced Ġin Ġthe Ġcross - target Ġsettings . </s> The Ġabove Ġsingle - se gment Ġexperiments Ġtell Ġus Ġthat Ġboth Ġthe Ġcomment Ġand Ġthe Ġquestion Ġprovide Ġcrucial Ġinformation . ĠBut Ġit Ġis Ġpossible Ġthat Ġthe ĠM - B ert Ġmodel , Ġeven Ġthough Ġtrained Ġon Ġboth Ġsegments , Ġmainly Ġlooks Ġat Ġa Ġsingle Ġsegment Ġat Ġtest Ġtime . ĠTo Ġrule Ġthis Ġout , Ġwe Ġprobe Ġthe Ġmodel Ġwith Ġrandomized Ġdata Ġat Ġtest Ġtime : </s> Test Ġthe Ġmodel Ġon Ġversions Ġof Ġthe Ġtest Ġsets Ġwhere Ġthe Ġcomments Ġremain Ġin Ġplace Ġbut Ġthe Ġquestions Ġare Ġshuff led Ġrandomly Ġ( random Ġquestions ). ĠWe Ġmake Ġsure Ġthat Ġthe Ġrandom Ġquestions Ġcome Ġfrom Ġthe Ġsame Ġtest Ġset Ġand Ġlanguage Ġas Ġthe Ġoriginal Ġquestions . </s> Keep Ġthe Ġquestions Ġin Ġplace Ġand Ġrandom ize Ġthe Ġcomments Ġ( random Ġcomments ). ĠAgain Ġwe Ġshuffle Ġthe Ġcomments Ġonly Ġwithin Ġtest Ġset Ġboundaries . </s> The Ġresults Ġin ĠTable ĠT AB REF 39 Ġshow Ġthat Ġthe Ġperformance Ġof Ġthe Ġmodel Ġdecreases Ġin Ġboth Ġcases , Ġconfirming Ġthat Ġit Ġlearns Ġto Ġtake Ġinto Ġaccount Ġboth Ġsegments . </s> Bas eline ĠExper iments Ġ: :: ĠHow ĠImportant Ġare ĠSpe lled - Out ĠTarg ets ? </s> Finally Ġwe Ġtest Ġwhether Ġthe Ġtarget Ġreally Ġneeds Ġto Ġbe Ġrepresented Ġby Ġnatural Ġlanguage Ġ( e . g . ĠâĢ ľ Do Ġyou Ġsupport ĠX ? âĢ Ŀ ). ĠName ly , Ġan Ġalternative Ġis Ġto Ġrepresent Ġthe Ġtarget Ġwith Ġa Ġtrain able Ġembed ding Ġinstead Ġof Ġa Ġquestion . </s> In Ġorder Ġto Ġfit Ġtarget Ġembed d ings Ġsmoothly Ġinto Ġour Ġarchitecture , Ġwe Ġrepresent Ġeach Ġtarget Ġtype Ġwith Ġa Ġdifferent Ġreserved Ġsymbol Ġfrom Ġthe ĠM - B ert Ġvocabulary . ĠSe gment ĠA Ġis Ġthen Ġset Ġto Ġthis Ġsymbol Ġinstead Ġof Ġa Ġnatural Ġlanguage Ġquestion . </s> The Ġresults Ġfor Ġthis Ġexperiment Ġare Ġlisted Ġin Ġthe Ġbottom Ġrow Ġof ĠTable ĠT AB REF 39 . ĠAn ĠM - B ert Ġmodel Ġthat Ġlearns Ġtarget Ġembed d ings Ġinstead Ġof Ġencoding Ġa Ġquestion Ġperforms Ġclearly Ġworse Ġin Ġthe Ġsupervised Ġand Ġcross - ling ual Ġsettings . ĠFrom Ġthis Ġwe Ġconclude Ġthat Ġspelled - out Ġnatural Ġlanguage Ġquestions Ġprovide Ġimportant Ġlinguistic Ġdetail Ġthat Ġcan Ġhelp Ġin Ġstance Ġdetection . </s> Discussion </s> The Ġbaseline Ġexperiments Ġconfirm Ġthat ĠM - B ert Ġcan Ġachieve Ġa Ġreasonable Ġaccuracy Ġon Ġx - st ance . </s> To Ġput Ġthe Ġsupervised Ġscore Ġinto Ġcontext Ġwe Ġlist Ġscores Ġthat Ġvariants Ġof ĠBert Ġhave Ġachieved Ġon Ġother Ġstance Ġdetection Ġdatasets Ġin ĠTable ĠT AB REF 46 . ĠIt Ġseems Ġthat Ġthe Ġsupervised Ġpart Ġof Ġx - st ance Ġhas Ġa Ġsimilar Ġdifficulty Ġas Ġthe ĠSem E val - 2016 ĠB IB REF 0 Ġor ĠMP CH I ĠB IB REF 22 Ġdatasets Ġon Ġwhich ĠBert Ġhas Ġpreviously Ġbeen Ġevaluated . </s> On Ġthe Ġother Ġhand , Ġin Ġthe Ġcross - ling ual Ġand Ġcross - target Ġsettings , Ġthe Ġmean Ġscore Ġdrops Ġby Ġ6 âĢĵ 8 Ġpercentage Ġpoints Ġcompared Ġto Ġthe Ġsupervised Ġsetting ; Ġwhile Ġzero - shot Ġtransfer Ġis Ġpossible Ġto Ġa Ġdegree , Ġit Ġcan Ġstill Ġbe Ġimproved . </s> The Ġadditional Ġexperiments Ġ( Table ĠT AB REF 39 ) Ġvalidate Ġthe Ġresults Ġand Ġshow Ġthat Ġthe Ġsequence - pair Ġclassification Ġapproach Ġto Ġstance Ġdetection Ġis Ġjustified . </s> It Ġis Ġinteresting Ġto Ġsee Ġwhat Ġerrors Ġthe ĠM - B ert Ġmodel Ġmakes . ĠTable ĠT AB REF 47 Ġpresents Ġinstances Ġwhere Ġit Ġpredicts Ġthe Ġwrong Ġlabel Ġwith Ġa Ġhigh Ġconfidence . ĠThese Ġexamples Ġindicate Ġthat Ġmany Ġcomments Ġexpress Ġtheir Ġstance Ġonly Ġon Ġa Ġvery Ġimplicit Ġlevel , Ġand Ġthus Ġhint Ġat Ġa Ġpotential Ġweakness Ġof Ġthe Ġdataset . ĠBecause Ġon Ġthe Ġvoting Ġadvice Ġplatform Ġthe Ġlabel Ġis Ġexplicitly Ġshown Ġto Ġreaders Ġin Ġaddition Ġto Ġthe Ġcomments , Ġthe Ġcomments Ġdo Ġnot Ġneed Ġto Ġexpress Ġthe Ġstance Ġexplicitly . ĠManual Ġannotation Ġcould Ġeliminate Ġvery Ġimplicit Ġsamples Ġin Ġa Ġfuture Ġversion Ġof Ġthe Ġdataset . ĠHowever , Ġthe Ġsheer Ġsize Ġand Ġbreadth Ġof Ġthe Ġdataset Ġcould Ġnot Ġrealistically Ġbe Ġachieved Ġwith Ġmanual Ġannotation , Ġand , Ġin Ġour Ġview , Ġlargely Ġcompens ates Ġfor Ġthe Ġimplicit ness Ġof Ġthe Ġtexts . </s> Conclusion </s> We Ġhave Ġpresented Ġa Ġnew Ġdataset Ġfor Ġpolitical Ġstance Ġdetection Ġcalled Ġx - st ance . ĠThe Ġdataset Ġextends Ġover Ġa Ġbroad Ġrange Ġof Ġtopics Ġand Ġissues Ġregarding Ġnational ĠSwiss Ġpolitics . ĠThis Ġdiversity Ġof Ġtopics Ġopens Ġup Ġan Ġopportunity Ġto Ġfurther Ġstudy Ġmulti - target Ġlearning . ĠMoreover , Ġbeing Ġpartly ĠSwiss ĠStandard ĠGerman , Ġpartly ĠFrench Ġand ĠItalian , Ġthe Ġdataset Ġpromotes Ġa Ġmult ilingual Ġapproach Ġto Ġstance Ġdetection . </s> By Ġcompiling Ġformal Ġcommentary Ġthat Ġpoliticians Ġhave Ġwritten Ġon Ġpolitical Ġquestions , Ġwe Ġadd Ġa Ġnew Ġtext Ġgenre Ġto Ġthe Ġfield Ġof Ġstance Ġdetection . ĠWe Ġalso Ġpropose Ġa Ġquestion âĢĵ answer Ġformat Ġthat Ġallows Ġus Ġto Ġcondition Ġstance Ġdetection Ġmodels Ġon Ġa Ġtarget Ġnaturally . </s> Our Ġbaseline Ġresults Ġwith Ġmult ilingual ĠBert Ġshow Ġthat Ġthe Ġmodel Ġhas Ġsome Ġcapability Ġto Ġperform Ġzero - shot Ġtransfer Ġto Ġunseen Ġlanguages Ġand Ġto Ġunseen Ġtargets Ġ( both Ġwithin Ġa Ġtopic Ġand Ġto Ġunseen Ġtopics ). ĠHowever , Ġthere Ġis Ġsome Ġgap Ġin Ġperformance Ġthat Ġfuture Ġwork Ġcould Ġaddress . ĠWe Ġexpect Ġthat Ġthe Ġx - st ance Ġdataset Ġcould Ġfurthermore Ġbe Ġa Ġvaluable Ġresource Ġfor Ġfields Ġsuch Ġas Ġargument Ġmining , Ġargument Ġsearch Ġor Ġtopic Ġclassification . </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( project ĠM UT AM UR ; Ġno . Ġ17 67 27 ). ĠWe Ġwould Ġlike Ġto Ġthank ĠIsabel le ĠAug enstein Ġfor Ġhelpful Ġfeedback . </s> Data ĠStatement Ġ: :: ĠC uration Ġrationale </s> In Ġorder Ġto Ġstudy Ġthe Ġautomatic Ġdetection Ġof Ġstances Ġon Ġpolitical Ġissues , Ġquestions Ġand Ġcandidate Ġresponses Ġon Ġthe Ġvoting Ġadvice Ġapplication Ġsmart vote . ch Ġwere Ġdownloaded . ĠMain ly Ġdata Ġpertaining Ġto Ġnational - level Ġissues Ġwere Ġincluded Ġto Ġreduce Ġvariability . </s> Data ĠStatement Ġ: :: ĠLanguage Ġvariety </s> The Ġtraining Ġset Ġconsists Ġof Ġquestions Ġand Ġanswers Ġin ĠSwiss ĠStandard ĠGerman Ġand ĠSwiss ĠFrench Ġ( 74 . 1 % Ġde - CH ; Ġ25 . 9 % Ġfr - CH ). ĠThe Ġtest Ġsets Ġalso Ġcontain Ġquestions Ġand Ġanswers Ġin ĠSwiss ĠItalian Ġ( 67 . 1 % Ġde - CH ; Ġ24 . 7 % Ġfr - CH ; Ġ8 . 2 % Ġit - CH ). ĠThe Ġquestions Ġhave Ġalso Ġbeen Ġtranslated Ġinto ĠEnglish . </s> Data ĠStatement Ġ: :: ĠSpeaker Ġdemographic Ġ( ans w ers ) </s> Cand idates Ġfor Ġcommunal , Ġcant onal Ġor Ġnational Ġelections Ġin ĠSwitzerland Ġwho Ġhave Ġfilled Ġout Ġan Ġonline Ġquestionnaire . </s> Age : Ġ18 Ġor Ġolder ĠâĢĵ Ġmixed . </s> Gender : ĠUnknown ĠâĢĵ Ġmixed . </s> Race / ethnic ity : ĠUnknown ĠâĢĵ Ġmixed . </s> Native Ġlanguage : ĠUnknown ĠâĢĵ Ġmixed . </s> Soc io economic Ġstatus : ĠUnknown ĠâĢĵ Ġmixed . </s> Different Ġspeakers Ġrepresented : Ġ7 581 . </s> Pres ence Ġof Ġdis ordered Ġspeech : ĠUnknown . </s> Data ĠStatement Ġ: :: ĠSpeech Ġsituation </s> The Ġquestions Ġwere Ġedited Ġand Ġtranslated Ġby Ġpolitical Ġscientists Ġfor Ġa Ġpublic Ġvoting Ġadvice Ġwebsite . </s> The Ġanswers Ġwere Ġwritten Ġbetween Ġ2011 Ġand Ġ2020 Ġby Ġthe Ġusers Ġof Ġthe Ġwebsite . </s> Data ĠStatement Ġ: :: ĠText Ġcharacteristics </s> Questions , Ġanswers , Ġarguments Ġand Ġcomments Ġregarding Ġpolitical Ġissues .