Sentences involving at least one race- or gender-associated word,  sentence  have to be short and grammatically simple,  sentence have to  include expressions of sentiment and emotion.
ABSTRACTIVE
generated with the various combinations of INLINEFORM4 person INLINEFORM5 and INLINEFORM6 emotion word INLINEFORM7 values across the eleven templates, differ only in one word corresponding to gender or race
EXTRACTIVE
We decided to use sentences involving at least one race- or gender-associated word. The sentences were intended to be short and grammatically simple. We also wanted some sentences to include expressions of sentiment and emotion, since the goal is to test sentiment and emotion systems. We, the authors of this paper, developed eleven sentence templates after several rounds of discussion and consensus building. They are shown in Table TABREF3 . The templates are divided into two groups. The first type (templates 1–7) includes emotion words. The purpose of this set is to have sentences expressing emotions. The second type (templates 8–11) does not include any emotion words. The purpose of this set is to have non-emotional (neutral) sentences.
We generated sentences from the templates by replacing INLINEFORM0 person INLINEFORM1 and INLINEFORM2 emotion word INLINEFORM3 variables with the values they can take. In total, 8,640 sentences were generated with the various combinations of INLINEFORM4 person INLINEFORM5 and INLINEFORM6 emotion word INLINEFORM7 values across the eleven templates. We manually examined the sentences to make sure they were grammatically well-formed. Notably, one can derive pairs of sentences from the EEC such that they differ only in one word corresponding to gender or race (e.g., `My daughter feels devastated' and `My son feels devastated'). We refer to the full set of 8,640 sentences as Equity Evaluation Corpus.
1805.04508
Introduction </s> [ 0 ] left margin = * Ġ[ 0 ] left margin = * </s> Aut omatic Ġsystems Ġhave Ġhad Ġa Ġsignificant Ġand Ġbeneficial Ġimpact Ġon Ġall Ġwalks Ġof Ġhuman Ġlife . ĠSo Ġmuch Ġso Ġthat Ġit Ġis Ġeasy Ġto Ġoverlook Ġtheir Ġpotential Ġto Ġbenefit Ġsociety Ġby Ġpromoting Ġequity , Ġdiversity , Ġand Ġfairness . ĠFor Ġexample , Ġmachines Ġdo Ġnot Ġtake Ġbribes Ġto Ġdo Ġtheir Ġjobs , Ġthey Ġcan Ġdetermine Ġeligibility Ġfor Ġa Ġloan Ġwithout Ġbeing Ġinfluenced Ġby Ġthe Ġcolor Ġof Ġthe Ġapplicant 's Ġskin , Ġand Ġthey Ġcan Ġprovide Ġaccess Ġto Ġinformation Ġand Ġservices Ġwithout Ġdiscrimination Ġbased Ġon Ġgender Ġor Ġsexual Ġorientation . ĠNonetheless , Ġas Ġmachine Ġlearning Ġsystems Ġbecome Ġmore Ġhuman - like Ġin Ġtheir Ġpredictions , Ġthey Ġcan Ġalso Ġperpetuate Ġhuman Ġbiases . ĠSome Ġlearned Ġbiases Ġmay Ġbe Ġbeneficial Ġfor Ġthe Ġdownstream Ġapplication Ġ( e . g ., Ġlearning Ġthat Ġhumans Ġoften Ġuse Ġsome Ġinsect Ġnames , Ġsuch Ġas Ġspider Ġor Ġcock roach , Ġto Ġrefer Ġto Ġunpleasant Ġsituations ). ĠOther Ġbiases Ġcan Ġbe Ġinappropriate Ġand Ġresult Ġin Ġnegative Ġexperiences Ġfor Ġsome Ġgroups Ġof Ġpeople . ĠExamples Ġinclude , Ġloan Ġeligibility Ġand Ġcrime Ġrec id ivism Ġprediction Ġsystems Ġthat Ġnegatively Ġassess Ġpeople Ġbelonging Ġto Ġa Ġcertain Ġpin / zip Ġcode Ġ( which Ġmay Ġdisproportionately Ġimpact Ġpeople Ġof Ġa Ġcertain Ġrace ) ĠB IB REF 0 Ġand Ġres um Ã© Ġsorting Ġsystems Ġthat Ġbelieve Ġthat Ġmen Ġare Ġmore Ġqualified Ġto Ġbe Ġprogrammers Ġthan Ġwomen ĠB IB REF 1 Ġ. ĠSimilarly , Ġsentiment Ġand Ġemotion Ġanalysis Ġsystems Ġcan Ġalso Ġperpetuate Ġand Ġaccent uate Ġinappropriate Ġhuman Ġbiases , Ġe . g ., Ġsystems Ġthat Ġconsider Ġutter ances Ġfrom Ġone Ġrace Ġor Ġgender Ġto Ġbe Ġless Ġpositive Ġsimply Ġbecause Ġof Ġtheir Ġrace Ġor Ġgender , Ġor Ġcustomer Ġsupport Ġsystems Ġthat Ġprioritize Ġa Ġcall Ġfrom Ġan Ġangry Ġmale Ġover Ġa Ġcall Ġfrom Ġthe Ġequally Ġangry Ġfemale . </s> Pred ictions Ġof Ġmachine Ġlearning Ġsystems Ġhave Ġalso Ġbeen Ġshown Ġto Ġbe Ġof Ġhigher Ġquality Ġwhen Ġdealing Ġwith Ġinformation Ġfrom Ġsome Ġgroups Ġof Ġpeople Ġas Ġopposed Ġto Ġother Ġgroups Ġof Ġpeople . ĠFor Ġexample , Ġin Ġthe Ġarea Ġof Ġcomputer Ġvision , Ġgender Ġclassification Ġsystems Ġperform Ġparticularly Ġpoorly Ġfor Ġdarker Ġskin ned Ġfemales ĠB IB REF 2 Ġ. ĠNatural Ġlanguage Ġprocessing Ġ( N LP ) Ġsystems Ġhave Ġbeen Ġshown Ġto Ġbe Ġpoor Ġin Ġunderstanding Ġtext Ġproduced Ġby Ġpeople Ġbelonging Ġto Ġcertain Ġraces ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠFor ĠN LP Ġsystems , Ġthe Ġsources Ġof Ġthe Ġbias Ġoften Ġinclude Ġthe Ġtraining Ġdata , Ġother Ġcorpor a , Ġlex icons , Ġand Ġword Ġembed d ings Ġthat Ġthe Ġmachine Ġlearning Ġalgorithm Ġmay Ġleverage Ġto Ġbuild Ġits Ġprediction Ġmodel . </s> Even Ġthough Ġthere Ġis Ġsome Ġrecent Ġwork Ġhighlighting Ġsuch Ġinappropriate Ġbiases Ġ( such Ġas Ġthe Ġwork Ġmentioned Ġabove ), Ġeach Ġsuch Ġpast Ġwork Ġhas Ġlargely Ġfocused Ġon Ġjust Ġone Ġor Ġtwo Ġsystems Ġand Ġresources . ĠFurther , Ġthere Ġis Ġno Ġbenchmark Ġdataset Ġfor Ġexamining Ġinappropriate Ġbiases Ġin Ġnatural Ġlanguage Ġsystems . ĠIn Ġthis Ġpaper , Ġwe Ġdescribe Ġhow Ġwe Ġcompiled Ġa Ġdataset Ġof Ġ8 , 640 ĠEnglish Ġsentences Ġcarefully Ġchosen Ġto Ġtease Ġout Ġbiases Ġtowards Ġcertain Ġraces Ġand Ġgenders . ĠWe Ġwill Ġrefer Ġto Ġit Ġas Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( E EC ). ĠWe Ġused Ġthe ĠE EC Ġas Ġa Ġsupplementary Ġtest Ġset Ġin Ġa Ġrecent Ġshared Ġtask Ġon Ġpredicting Ġsentiment Ġand Ġemotion Ġintensity Ġin Ġtweets , ĠSem E val - 2018 ĠTask Ġ1 : ĠAffect Ġin ĠTwe ets ĠB IB REF 5 Ġ. ĠIn Ġparticular , Ġwe Ġwanted Ġto Ġtest Ġa Ġhypothesis Ġthat Ġa Ġsystem Ġshould Ġequally Ġrate Ġthe Ġintensity Ġof Ġthe Ġemotion Ġexpressed Ġby Ġtwo Ġsentences Ġthat Ġdiffer Ġonly Ġin Ġthe Ġgender / race Ġof Ġa Ġperson Ġmentioned . ĠNote Ġthat Ġhere Ġthe Ġterm Ġsystem Ġrefers Ġto Ġthe Ġcombination Ġof Ġa Ġmachine Ġlearning Ġarchitecture Ġtrained Ġon Ġa Ġlabeled Ġdataset , Ġand Ġpossibly Ġusing Ġadditional Ġlanguage Ġresources . ĠThe Ġbias Ġcan Ġoriginate Ġfrom Ġany Ġor Ġseveral Ġof Ġthese Ġparts . ĠWe Ġwere Ġthus Ġable Ġto Ġuse Ġthe ĠE EC Ġto Ġexamine Ġ219 Ġsentiment Ġanalysis Ġsystems Ġthat Ġtook Ġpart Ġin Ġthe Ġshared Ġtask . </s> We Ġcompare Ġemotion Ġand Ġsentiment Ġintensity Ġscores Ġthat Ġthe Ġsystems Ġpredict Ġon Ġpairs Ġof Ġsentences Ġin Ġthe ĠE EC Ġthat Ġdiffer Ġonly Ġin Ġone Ġword Ġcorresponding Ġto Ġrace Ġor Ġgender Ġ( e . g ., Ġ` This Ġman Ġmade Ġme Ġfeel Ġangry ' Ġvs . Ġ` This Ġwoman Ġmade Ġme Ġfeel Ġangry '). ĠWe Ġfind Ġthat Ġthe Ġmajority Ġof Ġthe Ġsystems Ġstudied Ġshow Ġstatistically Ġsignificant Ġbias ; Ġthat Ġis , Ġthey Ġconsistently Ġprovide Ġslightly Ġhigher Ġsentiment Ġintensity Ġpredictions Ġfor Ġsentences Ġassociated Ġwith Ġone Ġrace Ġor Ġone Ġgender . ĠWe Ġalso Ġfind Ġthat Ġthe Ġbias Ġmay Ġbe Ġdifferent Ġdepending Ġon Ġthe Ġparticular Ġaffect Ġdimension Ġthat Ġthe Ġnatural Ġlanguage Ġsystem Ġis Ġtrained Ġto Ġpredict . </s> Despite Ġthe Ġwork Ġwe Ġdescribe Ġhere Ġand Ġwhat Ġothers Ġhave Ġproposed Ġin Ġthe Ġpast , Ġit Ġshould Ġbe Ġnoted Ġthat Ġthere Ġare Ġno Ġsimple Ġsolutions Ġfor Ġdealing Ġwith Ġinappropriate Ġhuman Ġbiases Ġthat Ġper col ate Ġinto Ġmachine Ġlearning Ġsystems . ĠIt Ġseems Ġdifficult Ġto Ġever Ġbe Ġable Ġto Ġidentify Ġand Ġquantify Ġall Ġof Ġthe Ġinappropriate Ġbiases Ġperfectly Ġ( even Ġwhen Ġrestricted Ġto Ġthe Ġscope Ġof Ġjust Ġgender Ġand Ġrace ). ĠFurther , Ġany Ġsuch Ġmechanism Ġis Ġliable Ġto Ġbe Ġcircumvent ed , Ġif Ġone Ġchooses Ġto Ġdo Ġso . ĠNonetheless , Ġas Ġdevelopers Ġof Ġsentiment Ġanalysis Ġsystems , Ġand ĠN LP Ġsystems Ġmore Ġbroadly , Ġwe Ġcannot Ġabs olve Ġourselves Ġof Ġthe Ġethical Ġimplications Ġof Ġthe Ġsystems Ġwe Ġbuild . ĠEven Ġif Ġit Ġis Ġunclear Ġhow Ġwe Ġshould Ġdeal Ġwith Ġthe Ġinappropriate Ġbiases Ġin Ġour Ġsystems , Ġwe Ġshould Ġbe Ġmeasuring Ġsuch Ġbiases . ĠThe ĠEquity ĠEvaluation ĠCorpus Ġis Ġnot Ġmeant Ġto Ġbe Ġa Ġcatch - all Ġfor Ġall Ġinappropriate Ġbiases , Ġbut Ġrather Ġjust Ġone Ġof Ġthe Ġseveral Ġways Ġby Ġwhich Ġwe Ġcan Ġexamine Ġthe Ġfairness Ġof Ġsentiment Ġanalysis Ġsystems . ĠWe Ġmake Ġthe Ġcorpus Ġfreely Ġavailable Ġso Ġthat Ġboth Ġdevelopers Ġand Ġusers Ġcan Ġuse Ġit , Ġand Ġbuild Ġon Ġit . </s> Related ĠWork </s> Recent Ġstudies Ġhave Ġdemonstrated Ġthat Ġthe Ġsystems Ġtrained Ġon Ġthe Ġhuman - written Ġtexts Ġlearn Ġhuman - like Ġbiases ĠB IB REF 1 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġgeneral , Ġany Ġpredictive Ġmodel Ġbuilt Ġon Ġhistorical Ġdata Ġmay Ġinadvertently Ġinherit Ġhuman Ġbiases Ġbased Ġon Ġgender , Ġethnicity , Ġrace , Ġor Ġreligion ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠDiscrimination - aware Ġdata Ġmining Ġfocuses Ġon Ġmeasuring Ġdiscrimination Ġin Ġdata Ġas Ġwell Ġas Ġon Ġevaluating Ġperformance Ġof Ġdiscrimination - aware Ġpredictive Ġmodels ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> In ĠN LP , Ġthe Ġattention Ġso Ġfar Ġhas Ġbeen Ġprimarily Ġon Ġword Ġembed d ings âĢĶ a Ġpopular Ġand Ġpowerful Ġframework Ġto Ġrepresent Ġwords Ġas Ġlow - dimensional Ġdense Ġvectors . ĠThe Ġword Ġembed d ings Ġare Ġusually Ġobtained Ġfrom Ġlarge Ġamounts Ġof Ġhuman - written Ġtexts , Ġsuch Ġas ĠWikipedia , ĠGoogle ĠNews Ġarticles , Ġor Ġmillions Ġof Ġtweets . ĠB ias Ġin Ġsentiment Ġanalysis Ġsystems Ġhas Ġonly Ġbeen Ġexplored Ġin Ġsimple Ġsystems Ġthat Ġmake Ġuse Ġof Ġpre - com puted Ġword Ġembed d ings ĠB IB REF 13 Ġ. ĠThere Ġis Ġno Ġprior Ġwork Ġthat Ġsystematically Ġquant ifies Ġthe Ġextent Ġof Ġbias Ġin Ġa Ġlarge Ġnumber Ġof Ġsentiment Ġanalysis Ġsystems . </s> This Ġpaper Ġdoes Ġnot Ġexamine Ġthe Ġdifferences Ġin Ġaccur acies Ġof Ġsystems Ġon Ġtext Ġproduced Ġby Ġdifferent Ġraces Ġor Ġgenders , Ġas Ġwas Ġdone Ġby Ġh ovy 2015 dem ographic , bl od get t 2016 dem ographic , j urg ens 2017 inc orpor ating , bu ol am w ini 2018 gender . ĠAppro aches Ġon Ġhow Ġto Ġmitigate Ġinappropriate Ġbiases ĠB IB REF 14 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġare Ġalso Ġbeyond Ġthe Ġscope Ġof Ġthis Ġpaper . ĠSee Ġalso Ġthe Ġposition Ġpaper Ġby Ġh ovy 2016 social , Ġwhich Ġidentifies Ġsocio - ethical Ġimplications Ġof Ġthe ĠN LP Ġsystems Ġin Ġgeneral . </s> The ĠEquity ĠEvaluation ĠCorpus </s> We Ġnow Ġdescribe Ġhow Ġwe Ġcompiled Ġa Ġdataset Ġof Ġthousands Ġof Ġsentences Ġto Ġdetermine Ġwhether Ġautomatic Ġsystems Ġconsistently Ġgive Ġhigher Ġ( or Ġlower ) Ġsentiment Ġintensity Ġscores Ġto Ġsentences Ġinvolving Ġa Ġparticular Ġrace Ġor Ġgender . ĠThere Ġare Ġseveral Ġways Ġin Ġwhich Ġsuch Ġa Ġdataset Ġmay Ġbe Ġcompiled . ĠWe Ġpresent Ġbelow Ġthe Ġchoices Ġthat Ġwe Ġmade . </s> We Ġdecided Ġto Ġuse Ġsentences Ġinvolving Ġat Ġleast Ġone Ġrace - Ġor Ġgender - associated Ġword . ĠThe Ġsentences Ġwere Ġintended Ġto Ġbe Ġshort Ġand Ġgram matically Ġsimple . ĠWe Ġalso Ġwanted Ġsome Ġsentences Ġto Ġinclude Ġexpressions Ġof Ġsentiment Ġand Ġemotion , Ġsince Ġthe Ġgoal Ġis Ġto Ġtest Ġsentiment Ġand Ġemotion Ġsystems . ĠWe , Ġthe Ġauthors Ġof Ġthis Ġpaper , Ġdeveloped Ġeleven Ġsentence Ġtemplates Ġafter Ġseveral Ġrounds Ġof Ġdiscussion Ġand Ġconsensus Ġbuilding . ĠThey Ġare Ġshown Ġin ĠTable ĠT AB REF 3 Ġ. ĠThe Ġtemplates Ġare Ġdivided Ġinto Ġtwo Ġgroups . ĠThe Ġfirst Ġtype Ġ( tem plates Ġ1 âĢĵ 7 ) Ġincludes Ġemotion Ġwords . ĠThe Ġpurpose Ġof Ġthis Ġset Ġis Ġto Ġhave Ġsentences Ġexpressing Ġemotions . ĠThe Ġsecond Ġtype Ġ( tem plates Ġ8 âĢĵ 11 ) Ġdoes Ġnot Ġinclude Ġany Ġemotion Ġwords . ĠThe Ġpurpose Ġof Ġthis Ġset Ġis Ġto Ġhave Ġnon - em otional Ġ( neutral ) Ġsentences . </s> The Ġtemplates Ġinclude Ġtwo Ġvariables : ĠIN LINE FORM 0 Ġperson ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġemotion Ġword ĠIN LINE FORM 3 Ġ. ĠWe Ġgenerate Ġsentences Ġfrom Ġthe Ġtemplate Ġby Ġinstant iating Ġeach Ġvariable Ġwith Ġone Ġof Ġthe Ġpre - ch osen Ġvalues Ġthat Ġthe Ġvariable Ġcan Ġtake . ĠEach Ġof Ġthe Ġeleven Ġtemplates Ġincludes Ġthe Ġvariable ĠIN LINE FORM 4 Ġperson ĠIN LINE FORM 5 Ġ. ĠIN LINE FORM 6 Ġperson ĠIN LINE FORM 7 Ġcan Ġbe Ġinstant iated Ġby Ġany Ġof Ġthe Ġfollowing Ġnoun Ġphrases : </s> For Ġour Ġstudy , Ġwe Ġchose Ġten Ġnames Ġof Ġeach Ġkind Ġfrom Ġthe Ġstudy Ġby ĠCal isk an : 2017 Ġ( see ĠTable ĠT AB REF 7 Ġ). ĠThe Ġfull Ġlists Ġof Ġnoun Ġphrases Ġrepresenting Ġfemales Ġand Ġmales , Ġused Ġin Ġour Ġstudy , Ġare Ġshown Ġin ĠTable ĠT AB REF 8 Ġ. </s> The Ġsecond Ġvariable , ĠIN LINE FORM 0 Ġemotion Ġword ĠIN LINE FORM 1 Ġ, Ġhas Ġtwo Ġvariants . ĠTem plates Ġone Ġthrough Ġfour Ġinclude Ġa Ġvariable Ġfor Ġan Ġemotional Ġstate Ġword . ĠThe Ġemotional Ġstate Ġwords Ġcorrespond Ġto Ġfour Ġbasic Ġemotions : Ġanger , Ġfear , Ġjoy , Ġand Ġsadness . ĠSpecifically , Ġfor Ġeach Ġof Ġthe Ġemotions , Ġwe Ġselected Ġfive Ġwords Ġthat Ġconvey Ġthat Ġemotion Ġin Ġvarying Ġintens ities . ĠThese Ġwords Ġwere Ġtaken Ġfrom Ġthe Ġcategories Ġin Ġthe ĠRog et 's ĠThe s aurus Ġcorresponding Ġto Ġthe Ġfour Ġemotions : Ġcategory Ġ# 900 ĠRes ent ment Ġ( for Ġanger ), Ġcategory Ġ# 860 ĠFear Ġ( for Ġfear ), Ġcategory Ġ# 8 36 ĠChe er fulness Ġ( for Ġjoy ), Ġand Ġcategory Ġ# 8 37 ĠDe jection Ġ( for Ġsadness ). ĠTem plates Ġfive Ġthrough Ġseven Ġinclude Ġemotion Ġwords Ġdescribing Ġa Ġsituation Ġor Ġevent . ĠThese Ġwords Ġwere Ġalso Ġtaken Ġfrom Ġthe Ġsame Ġthe s aurus Ġcategories Ġlisted Ġabove . ĠThe Ġfull Ġlists Ġof Ġemotion Ġwords Ġ( em otional Ġstate Ġwords Ġand Ġemotional Ġsituation / event Ġwords ) Ġare Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. </s> We Ġgenerated Ġsentences Ġfrom Ġthe Ġtemplates Ġby Ġreplacing ĠIN LINE FORM 0 Ġperson ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġemotion Ġword ĠIN LINE FORM 3 Ġvariables Ġwith Ġthe Ġvalues Ġthey Ġcan Ġtake . ĠIn Ġtotal , Ġ8 , 640 Ġsentences Ġwere Ġgenerated Ġwith Ġthe Ġvarious Ġcombinations Ġof ĠIN LINE FORM 4 Ġperson ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġemotion Ġword ĠIN LINE FORM 7 Ġvalues Ġacross Ġthe Ġeleven Ġtemplates . ĠWe Ġmanually Ġexamined Ġthe Ġsentences Ġto Ġmake Ġsure Ġthey Ġwere Ġgram matically Ġwell - formed . ĠNotably , Ġone Ġcan Ġderive Ġpairs Ġof Ġsentences Ġfrom Ġthe ĠE EC Ġsuch Ġthat Ġthey Ġdiffer Ġonly Ġin Ġone Ġword Ġcorresponding Ġto Ġgender Ġor Ġrace Ġ( e . g ., Ġ` My Ġdaughter Ġfeels Ġdevastated ' Ġand Ġ` My Ġson Ġfeels Ġdevastated '). ĠWe Ġrefer Ġto Ġthe Ġfull Ġset Ġof Ġ8 , 640 Ġsentences Ġas ĠEquity ĠEvaluation ĠCorpus . </s> Me asuring ĠRace Ġand ĠGender ĠB ias Ġin ĠAutomatic ĠSent iment ĠAnalysis ĠSystems </s> The Ġrace Ġand Ġgender Ġbias Ġevaluation Ġwas Ġcarried Ġout Ġon Ġthe Ġoutput Ġof Ġthe Ġ219 Ġautomatic Ġsystems Ġthat Ġparticipated Ġin ĠSem E val - 2018 ĠTask Ġ1 : ĠAffect Ġin ĠTwe ets ĠB IB REF 5 Ġ. ĠThe Ġshared Ġtask Ġincluded Ġfive Ġsubt asks Ġon Ġinfer ring Ġthe Ġaffect ual Ġstate Ġof Ġa Ġperson Ġfrom Ġtheir Ġtweet : Ġ1 . Ġemotion Ġintensity Ġregression , Ġ2 . Ġemotion Ġintensity Ġord inal Ġclassification , Ġ3 . Ġval ence Ġ( sent iment ) Ġregression , Ġ4 . Ġval ence Ġord inal Ġclassification , Ġand Ġ5 . Ġemotion Ġclassification . ĠFor Ġeach Ġsubt ask , Ġlabeled Ġdata Ġwere Ġprovided Ġfor ĠEnglish , ĠArabic , Ġand ĠSpanish . ĠThe Ġrace Ġand Ġgender Ġbias Ġwere Ġanalyzed Ġfor Ġthe Ġsystem Ġoutputs Ġon Ġtwo ĠEnglish Ġsubt asks : Ġemotion Ġintensity Ġregression Ġ( for Ġanger , Ġfear , Ġjoy , Ġand Ġsadness ) Ġand Ġval ence Ġregression . ĠThese Ġregression Ġtasks Ġwere Ġformulated Ġas Ġfollows : ĠGiven Ġa Ġtweet Ġand Ġan Ġaffect ive Ġdimension ĠA Ġ( anger , Ġfear , Ġjoy , Ġsadness , Ġor Ġval ence ), Ġdetermine Ġthe Ġintensity Ġof ĠA Ġthat Ġbest Ġrepresents Ġthe Ġmental Ġstate Ġof Ġthe Ġtwe eter âĢĶ a Ġreal - valued Ġscore Ġbetween Ġ0 Ġ( le ast ĠA ) Ġand Ġ1 Ġ( most ĠA ). ĠSep arate Ġtraining Ġand Ġtest Ġdatasets Ġwere Ġprovided Ġfor Ġeach Ġaffect ive Ġdimension . </s> Training Ġsets Ġincluded Ġtweets Ġalong Ġwith Ġgold Ġintensity Ġscores . ĠTwo Ġtest Ġsets Ġwere Ġprovided Ġfor Ġeach Ġtask : Ġ1 . Ġa Ġregular Ġtweet Ġtest Ġset Ġ( for Ġwhich Ġthe Ġgold Ġintensity Ġscores Ġare Ġknown Ġbut Ġnot Ġrevealed Ġto Ġthe Ġparticipating Ġsystems ), Ġand Ġ2 . Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( for Ġwhich Ġno Ġgold Ġintensity Ġlabels Ġexist ). ĠParticipants Ġwere Ġtold Ġthat Ġapart Ġfrom Ġthe Ġusual Ġtest Ġset , Ġthey Ġare Ġto Ġrun Ġtheir Ġsystems Ġon Ġa Ġseparate Ġtest Ġset Ġof Ġunknown Ġorigin . ĠThe Ġparticipants Ġwere Ġinstructed Ġto Ġtrain Ġtheir Ġsystem Ġon Ġthe Ġtweets Ġtraining Ġsets Ġprovided , Ġand Ġthat Ġthey Ġcould Ġuse Ġany Ġother Ġresources Ġthey Ġmay Ġfind Ġor Ġcreate . ĠThey Ġwere Ġto Ġrun Ġthe Ġsame Ġfinal Ġsystem Ġon Ġthe Ġtwo Ġtest Ġsets . ĠThe Ġnature Ġof Ġthe Ġsecond Ġtest Ġset Ġwas Ġrevealed Ġto Ġthem Ġonly Ġafter Ġthe Ġcompetition . ĠThe Ġfirst Ġ( t we ets ) Ġtest Ġset Ġwas Ġused Ġto Ġevaluate Ġand Ġrank Ġthe Ġquality Ġ( acc uracy ) Ġof Ġthe Ġsystems ' Ġpredictions . ĠThe Ġsecond Ġ( E EC ) Ġtest Ġset Ġwas Ġused Ġto Ġperform Ġthe Ġbias Ġanalysis , Ġwhich Ġis Ġthe Ġfocus Ġof Ġthis Ġpaper . </s> System s : ĠFifty Ġteams Ġsubmitted Ġtheir Ġsystem Ġoutputs Ġto Ġone Ġor Ġmore Ġof Ġthe Ġfive Ġemotion Ġintensity Ġregression Ġtasks Ġ( for Ġanger , Ġfear , Ġjoy , Ġsadness , Ġand Ġval ence ), Ġresulting Ġin Ġ219 Ġsubmissions Ġin Ġtotal . ĠMany Ġsystems Ġwere Ġbuilt Ġusing Ġtwo Ġtypes Ġof Ġfeatures : Ġdeep Ġneural Ġnetwork Ġrepresentations Ġof Ġtweets Ġ( sent ence Ġembed d ings ) Ġand Ġfeatures Ġderived Ġfrom Ġexisting Ġsentiment Ġand Ġemotion Ġlex icons . ĠThese Ġfeatures Ġwere Ġthen Ġcombined Ġto Ġlearn Ġa Ġmodel Ġusing Ġeither Ġtraditional Ġmachine Ġlearning Ġalgorithms Ġ( such Ġas ĠS VM / S VR Ġand ĠLog istic ĠReg ression ) Ġor Ġdeep Ġneural Ġnetworks . ĠS VM / S VR , ĠL ST Ms , Ġand ĠBi - L ST Ms Ġwere Ġsome Ġof Ġthe Ġmost Ġwidely Ġused Ġmachine Ġlearning Ġalgorithms . ĠThe Ġsentence Ġembed d ings Ġwere Ġobtained Ġby Ġtraining Ġa Ġneural Ġnetwork Ġon Ġthe Ġprovided Ġtraining Ġdata , Ġa Ġdistant Ġsupervision Ġcorpus Ġ( e . g ., ĠA IT 2018 ĠDist ant ĠSuper vision ĠCorpus Ġthat Ġhas Ġtweets Ġwith Ġemotion - related Ġquery Ġterms ), Ġsentiment - label ed Ġtweet Ġcorpor a Ġ( e . g ., ĠSem eval - 2017 ĠTask 4 A Ġdataset Ġon Ġsentiment Ġanalysis Ġin ĠTwitter ), Ġor Ġby Ġusing Ġpre - trained Ġmodels Ġ( e . g ., ĠDeep Mo ji ĠB IB REF 20 Ġ, ĠSkip Ġthoughts ĠB IB REF 21 Ġ). ĠThe Ġlex icon Ġfeatures Ġwere Ġoften Ġderived Ġfrom Ġthe ĠN RC Ġemotion Ġand Ġsentiment Ġlex icons ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠAF IN N ĠB IB REF 25 Ġ, Ġand ĠBing ĠLiu ĠLex icon ĠB IB REF 26 Ġ. </s> We Ġprovided Ġa Ġbaseline ĠS VM Ġsystem Ġtrained Ġusing Ġword Ġun ig rams Ġas Ġfeatures Ġon Ġthe Ġtraining Ġdata Ġ( S VM - Un ig rams ). ĠThis Ġsystem Ġis Ġalso Ġincluded Ġin Ġthe Ġcurrent Ġanalysis . </s> Me asuring Ġbias : ĠTo Ġexamine Ġgender Ġbias , Ġwe Ġcompared Ġeach Ġsystem 's Ġpredicted Ġscores Ġon Ġthe ĠE EC Ġsentence Ġpairs Ġas Ġfollows : </s> Thus , Ġeleven Ġpairs Ġof Ġscores Ġ( ten Ġpairs Ġof Ġscores Ġfrom Ġten Ġnoun Ġphrase Ġpairs Ġand Ġone Ġpair Ġof Ġscores Ġfrom Ġthe Ġaverages Ġon Ġname Ġsubs ets ) Ġwere Ġexamined Ġfor Ġeach Ġtemplate âĢĵ em otion Ġword Ġinstant iation . ĠThere Ġwere Ġtwenty Ġdifferent Ġemotion Ġwords Ġused Ġin Ġseven Ġtemplates Ġ( tem plates Ġ1 âĢĵ 7 ), Ġand Ġno Ġemotion Ġwords Ġused Ġin Ġthe Ġfour Ġremaining Ġtemplates Ġ( tem plates Ġ8 âĢĵ 11 ). ĠIn Ġtotal , ĠIN LINE FORM 0 Ġpairs Ġof Ġscores Ġwere Ġcompared . </s> Similarly , Ġto Ġexamine Ġrace Ġbias , Ġwe Ġcompared Ġpairs Ġof Ġsystem Ġpredicted Ġscores Ġas Ġfollows : </s> Thus , Ġone Ġpair Ġof Ġscores Ġwas Ġexamined Ġfor Ġeach Ġtemplate âĢĵ em otion Ġword Ġinstant iation . ĠIn Ġtotal , ĠIN LINE FORM 0 Ġpairs Ġof Ġscores Ġwere Ġcompared . </s> For Ġeach Ġsystem , Ġwe Ġcalculated Ġthe Ġpaired Ġtwo Ġsample Ġt - test Ġto Ġdetermine Ġwhether Ġthe Ġmean Ġdifference Ġbetween Ġthe Ġtwo Ġsets Ġof Ġscores Ġ( ac ross Ġthe Ġtwo Ġraces Ġand Ġacross Ġthe Ġtwo Ġgenders ) Ġis Ġsignificant . ĠWe Ġset Ġthe Ġsignificance Ġlevel Ġto Ġ0 . 05 . ĠHowever , Ġsince Ġwe Ġperformed Ġ4 38 Ġassessments Ġ( 219 Ġsubmissions Ġevaluated Ġfor Ġbiases Ġin Ġboth Ġgender Ġand Ġrace ), Ġwe Ġapplied ĠBon fer ron i Ġcorrection . ĠThe Ġnull Ġhypothesis Ġthat Ġthe Ġtrue Ġmean Ġdifference Ġbetween Ġthe Ġpaired Ġsamples Ġwas Ġzero Ġwas Ġrejected Ġif Ġthe Ġcalculated Ġp - value Ġfell Ġbelow ĠIN LINE FORM 0 Ġ. </s> Results </s> The Ġtwo Ġsub - sections Ġbelow Ġpresent Ġthe Ġresults Ġfrom Ġthe Ġanalysis Ġfor Ġgender Ġbias Ġand Ġrace Ġbias , Ġrespectively . </s> Gender ĠB ias ĠResults </s> Individual Ġsubmission Ġresults Ġwere Ġcommunicated Ġto Ġthe Ġparticipants . ĠHere , Ġwe Ġpresent Ġthe Ġsummary Ġresults Ġacross Ġall Ġthe Ġteams . ĠThe Ġgoal Ġof Ġthis Ġanalysis Ġis Ġto Ġgain Ġa Ġbetter Ġunderstanding Ġof Ġbiases Ġacross Ġa Ġlarge Ġnumber Ġof Ġcurrent Ġsentiment Ġanalysis Ġsystems . ĠThus , Ġwe Ġpartition Ġthe Ġsubmissions Ġinto Ġthree Ġgroups Ġaccording Ġto Ġthe Ġbias Ġthey Ġshow : </s> F = M Ġnot Ġsignificant : Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġdifference Ġin Ġintensity Ġscores Ġpredicted Ġfor Ġcorresponding Ġfemale Ġand Ġmale Ġnoun Ġphrase Ġsentences , </s> F ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġsignificant : Ġsubmissions Ġthat Ġconsistently Ġgave Ġhigher Ġscores Ġfor Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġthan Ġfor Ġcorresponding Ġsentences Ġwith Ġmale Ġnoun Ġphrases , </s> F ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġsignificant : Ġsubmissions Ġthat Ġconsistently Ġgave Ġlower Ġscores Ġfor Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġthan Ġfor Ġcorresponding Ġsentences Ġwith Ġmale Ġnoun Ġphrases . </s> For Ġeach Ġsystem Ġand Ġeach Ġsentence Ġpair , Ġwe Ġcalculate Ġthe Ġscore Ġdifference ĠIN LINE FORM 0 Ġas Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġminus Ġthe Ġscore Ġfor Ġthe Ġcorresponding Ġmale Ġnoun Ġphrase Ġsentence . ĠTable ĠT AB REF 24 Ġpresents Ġthe Ġsummary Ġresults Ġfor Ġeach Ġof Ġthe Ġbias Ġgroups . ĠIt Ġhas Ġthe Ġfollowing Ġcolumns : </s> # Sub m .: Ġnumber Ġof Ġsubmissions Ġin Ġeach Ġgroup . </s> If Ġall Ġthe Ġsystems Ġare Ġunbiased , Ġthen Ġthe Ġnumber Ġof Ġsubmissions Ġfor Ġthe Ġgroup ĠF = M Ġnot Ġsignificant Ġwould Ġbe Ġthe Ġmaximum , Ġand Ġthe Ġnumber Ġof Ġsubmissions Ġin Ġall Ġother Ġgroups Ġwould Ġbe Ġzero . </s> Avg . Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġ: Ġthe Ġaverage ĠIN LINE FORM 2 Ġfor Ġonly Ġthose Ġpairs Ġwhere Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġis Ġhigher . ĠThe Ġgreater Ġthe Ġmagnitude Ġof Ġthis Ġscore , Ġthe Ġstronger Ġthe Ġbias Ġin Ġsystems Ġthat Ġconsistently Ġgive Ġhigher Ġscores Ġto Ġfemale - associated Ġsentences . </s> Avg . Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġ: Ġthe Ġaverage ĠIN LINE FORM 2 Ġfor Ġonly Ġthose Ġpairs Ġwhere Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġis Ġlower . ĠThe Ġgreater Ġthe Ġmagnitude Ġof Ġthis Ġscore , Ġthe Ġstronger Ġthe Ġbias Ġin Ġsystems Ġthat Ġconsistently Ġgive Ġlower Ġscores Ġto Ġfemale - associated Ġsentences . </s> Note Ġthat Ġthese Ġnumbers Ġwere Ġfirst Ġcalculated Ġseparately Ġfor Ġeach Ġsubmission , Ġand Ġthen Ġaveraged Ġover Ġall Ġthe Ġsubmissions Ġwithin Ġeach Ġsubmission Ġgroup . ĠThe Ġresults Ġare Ġreported Ġseparately Ġfor Ġsubmissions Ġto Ġeach Ġtask Ġ( anger , Ġfear , Ġjoy , Ġsadness , Ġand Ġsentiment / val ence Ġintensity Ġprediction ). </s> Obs erve Ġthat Ġon Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks , Ġonly Ġabout Ġ12 Ġof Ġthe Ġ46 Ġsubmissions Ġ( about Ġ25 % Ġof Ġthe Ġsubmissions ) Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠOn Ġthe Ġval ence Ġprediction Ġtask , Ġonly Ġ5 Ġof Ġthe Ġ36 Ġsubmissions Ġ( 14 % Ġof Ġthe Ġsubmissions ) Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠThus Ġ75 % Ġto Ġ86 % Ġof Ġthe Ġsubmissions Ġconsistently Ġmarked Ġsentences Ġof Ġone Ġgender Ġhigher Ġthan Ġanother . </s> When Ġpredicting Ġanger , Ġjoy , Ġor Ġval ence , Ġthe Ġnumber Ġof Ġsystems Ġconsistently Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 21 âĢĵ 25 ) Ġis Ġmarkedly Ġhigher Ġthan Ġthe Ġnumber Ġof Ġsystems Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 8 âĢĵ 13 ). Ġ( Rec all Ġthat Ġhigher Ġval ence Ġmeans Ġmore Ġpositive Ġsentiment .) ĠIn Ġcontrast , Ġon Ġthe Ġfear Ġtask , Ġmost Ġsubmissions Ġtended Ġto Ġassign Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 23 ) Ġas Ġcompared Ġto Ġthe Ġnumber Ġof Ġsystems Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 12 ). ĠWhen Ġpredicting Ġsadness , Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġmostly Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 18 ) Ġis Ġclose Ġto Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġmostly Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 16 ). ĠThese Ġresults Ġare Ġin Ġline Ġwith Ġsome Ġcommon Ġstereotypes , Ġsuch Ġas Ġfemales Ġare Ġmore Ġemotional , Ġand Ġsituations Ġinvolving Ġmale Ġagents Ġare Ġmore Ġfearful ĠB IB REF 27 Ġ. </s> Figure ĠFIG REF 25 Ġshows Ġthe Ġscore Ġdifferences Ġ( ĠIN LINE FORM 0 Ġ) Ġfor Ġindividual Ġsystems Ġon Ġthe Ġval ence Ġregression Ġtask . ĠPl ots Ġfor Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks Ġare Ġshown Ġin ĠFigure ĠFIG REF 31 Ġin Ġthe ĠAppendix . ĠEach Ġpoint Ġ( Ġâĸ ² , Ġâĸ ¼ , ĠâĹı ) Ġon Ġthe Ġplot Ġcorresponds Ġto Ġthe Ġdifference Ġin Ġscores Ġpredicted Ġby Ġthe Ġsystem Ġon Ġone Ġsentence Ġpair . ĠThe Ġsystems Ġare Ġordered Ġby Ġtheir Ġrank Ġ( from Ġfirst Ġto Ġlast ) Ġon Ġthe Ġtask Ġon Ġthe Ġtweets Ġtest Ġsets , Ġas Ġper Ġthe Ġofficial Ġevaluation Ġmetric Ġ( Spe ar man Ġcorrelation Ġwith Ġthe Ġgold Ġintensity Ġscores ). ĠWe Ġwill Ġrefer Ġto Ġthe Ġdifference Ġbetween Ġthe Ġmaximal Ġvalue Ġof ĠIN LINE FORM 1 Ġand Ġthe Ġminimal Ġvalue Ġof ĠIN LINE FORM 2 Ġfor Ġa Ġparticular Ġsystem Ġas Ġthe ĠIN LINE FORM 3 ĠâĢĵ spread . ĠObs erve Ġthat Ġthe ĠIN LINE FORM 4 ĠâĢĵ sp reads Ġfor Ġmany Ġsystems Ġare Ġrather Ġlarge , Ġup Ġto Ġ0 . 57 . ĠDepending Ġon Ġthe Ġtask , Ġthe Ġtop Ġ10 Ġor Ġtop Ġ15 Ġsystems Ġas Ġwell Ġas Ġsome Ġof Ġthe Ġworst Ġperforming Ġsystems Ġtend Ġto Ġhave Ġsmaller ĠIN LINE FORM 5 ĠâĢĵ sp reads Ġwhile Ġthe Ġsystems Ġwith Ġmedium Ġto Ġlow Ġperformance Ġshow Ġgreater Ġsensitivity Ġto Ġthe Ġgender - associated Ġwords . ĠAlso , Ġmost Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifferences Ġ( shown Ġin Ġgreen ) Ġperformed Ġpoorly Ġon Ġthe Ġtweets Ġtest Ġsets . ĠOnly Ġthree Ġsystems Ġout Ġof Ġthe Ġtop Ġfive Ġon Ġthe Ġanger Ġintensity Ġtask Ġand Ġone Ġsystem Ġon Ġthe Ġjoy Ġand Ġsadness Ġtasks Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠThis Ġindicates Ġthat Ġwhen Ġconsidering Ġonly Ġthose Ġsystems Ġthat Ġperformed Ġwell Ġon Ġthe Ġintensity Ġprediction Ġtask , Ġthe Ġpercentage Ġof Ġgender - biased Ġsystems Ġare Ġeven Ġhigher Ġthan Ġthose Ġindicated Ġabove . </s> These Ġresults Ġraise Ġfurther Ġquestions Ġsuch Ġas Ġ` what Ġexactly Ġis Ġthe Ġcause Ġof Ġsuch Ġbiases ?' Ġand Ġ` why Ġis Ġthe Ġbias Ġimpacted Ġby Ġthe Ġemotion Ġtask Ġunder Ġconsideration ? '. ĠAn sw ering Ġthese Ġquestions Ġwill Ġrequire Ġfurther Ġinformation Ġon Ġthe Ġresources Ġthat Ġthe Ġteams Ġused Ġto Ġdevelop Ġtheir Ġmodels , Ġand Ġwe Ġleave Ġthat Ġfor Ġfuture Ġwork . </s> Average Ġscore Ġdifferences : ĠFor Ġsubmissions Ġthat Ġshowed Ġstatistically Ġsignificant Ġscore Ġdifferences , Ġthe Ġaverage Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġand Ġthe Ġaverage Ġscore Ġdifference ĠF ĠIN LINE FORM 2 ĠâĢĵ M ĠIN LINE FORM 3 Ġwere ĠIN LINE FORM 4 Ġ. ĠSince Ġthe Ġintensity Ġscores Ġrange Ġfrom Ġ0 Ġto Ġ1 , Ġ0 . 03 Ġis Ġ3 % Ġof Ġthe Ġfull Ġrange . ĠThe Ġmaximal Ġscore Ġdifference Ġ( ĠIN LINE FORM 5 Ġ) Ġacross Ġall Ġthe Ġsubmissions Ġwas Ġas Ġhigh Ġas Ġ0 . 34 . ĠNote , Ġhowever , Ġthat Ġthese ĠIN LINE FORM 6 Ġs Ġare Ġthe Ġresult Ġof Ġchanging Ġjust Ġone Ġword Ġin Ġa Ġsentence . ĠIn Ġmore Ġcomplex Ġsentences , Ġseveral Ġgender - associated Ġwords Ġcan Ġappear , Ġwhich Ġmay Ġhave Ġa Ġbigger Ġimpact . ĠAlso , Ġwhether Ġconsistent Ġscore Ġdifferences Ġof Ġthis Ġmagnitude Ġwill Ġhave Ġsignificant Ġrepercussions Ġin Ġdownstream Ġapplications , Ġdepends Ġon Ġthe Ġparticular Ġapplication . </s> An alyses Ġon Ġonly Ġthe Ġneutral Ġsentences Ġin ĠE EC Ġand Ġonly Ġthe Ġemotional Ġsentences Ġin ĠE EC : ĠWe Ġalso Ġperformed Ġa Ġseparate Ġanalysis Ġusing Ġonly Ġthose Ġsentences Ġfrom Ġthe ĠE EC Ġthat Ġincluded Ġno Ġemotion Ġwords . ĠRecall Ġthat Ġthere Ġare Ġfour Ġtemplates Ġthat Ġcontain Ġno Ġemotion Ġwords . ĠTables ĠT AB REF 26 Ġshows Ġthese Ġresults . ĠWe Ġobserve Ġsimilar Ġtrends Ġas Ġin Ġthe Ġanalysis Ġon Ġthe Ġfull Ġset . ĠOne Ġnoticeable Ġdifference Ġis Ġthat Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġshowed Ġstatistically Ġsignificant Ġscore Ġdifference Ġis Ġmuch Ġsmaller Ġfor Ġthis Ġdata Ġsubset . ĠHowever , Ġthe Ġtotal Ġnumber Ġof Ġcomparisons Ġon Ġthe Ġsubset Ġ( 44 ) Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġtotal Ġnumber Ġof Ġcomparisons Ġon Ġthe Ġfull Ġset Ġ( 1 , 584 ), Ġwhich Ġmakes Ġthe Ġstatistical Ġtest Ġless Ġpowerful . ĠNote Ġalso Ġthat Ġthe Ġaverage Ġscore Ġdifferences Ġon Ġthe Ġsubset Ġ( column s Ġ3 Ġand Ġ4 Ġin ĠTable ĠT AB REF 26 Ġ) Ġtend Ġto Ġbe Ġhigher Ġthan Ġthe Ġdifferences Ġon Ġthe Ġfull Ġset Ġ( column s Ġ3 Ġand Ġ4 Ġin ĠTable ĠT AB REF 24 Ġ). ĠThis Ġindicates Ġthat Ġgender - associated Ġwords Ġcan Ġhave Ġa Ġbigger Ġimpact Ġon Ġsystem Ġpredictions Ġfor Ġneutral Ġsentences . </s> We Ġalso Ġperformed Ġan Ġanalysis Ġby Ġrestricting Ġthe Ġdataset Ġto Ġcontain Ġonly Ġthe Ġsentences Ġwith Ġthe Ġemotion Ġwords Ġcorresponding Ġto Ġthe Ġemotion Ġtask Ġ( i . e ., Ġsubmissions Ġto Ġthe Ġanger Ġintensity Ġprediction Ġtask Ġwere Ġevaluated Ġonly Ġon Ġsentences Ġwith Ġanger Ġwords ). ĠThe Ġresults Ġ( not Ġshown Ġhere ) Ġwere Ġsimilar Ġto Ġthe Ġresults Ġon Ġthe Ġfull Ġset . </s> Race ĠB ias ĠResults </s> We Ġdid Ġa Ġsimilar Ġanalysis Ġfor Ġrace Ġas Ġwe Ġdid Ġfor Ġgender . ĠFor Ġeach Ġsubmission Ġon Ġeach Ġtask , Ġwe Ġcalculated Ġthe Ġdifference Ġbetween Ġthe Ġaverage Ġpredicted Ġscore Ġon Ġthe Ġset Ġof Ġsentences Ġwith ĠAfrican ĠAmerican Ġ( AA ) Ġnames Ġand Ġthe Ġaverage Ġpredicted Ġscore Ġon Ġthe Ġset Ġof Ġsentences Ġwith ĠEuropean ĠAmerican Ġ( EA ) Ġnames . ĠThen , Ġwe Ġaggreg ated Ġthe Ġresults Ġover Ġall Ġsuch Ġsentence Ġpairs Ġin Ġthe ĠE EC . </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults . ĠThe Ġtable Ġhas Ġthe Ġsame Ġform Ġand Ġstructure Ġas Ġthe Ġgender Ġresult Ġtables . ĠObs erve Ġthat Ġthe Ġnumber Ġof Ġsubmissions Ġwith Ġno Ġstatistically Ġsignificant Ġscore Ġdifference Ġfor Ġsentences Ġpertaining Ġto Ġthe Ġtwo Ġraces Ġis Ġabout Ġ5 âĢĵ 11 Ġ( about Ġ11 % Ġto Ġ24 %) Ġfor Ġthe Ġfour Ġemotions Ġand Ġ3 Ġ( about Ġ8 %) Ġfor Ġval ence . ĠThese Ġnumbers Ġare Ġeven Ġlower Ġthan Ġwhat Ġwas Ġfound Ġfor Ġgender . </s> The Ġmajority Ġof Ġthe Ġsystems Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith ĠAfrican ĠAmerican Ġnames Ġon Ġthe Ġtasks Ġof Ġanger , Ġfear , Ġand Ġsadness Ġintensity Ġprediction . ĠOn Ġthe Ġjoy Ġand Ġval ence Ġtasks , Ġmost Ġsubmissions Ġtended Ġto Ġassign Ġhigher Ġscores Ġto Ġsentences Ġwith ĠEuropean ĠAmerican Ġnames . ĠThese Ġtendencies Ġreflect Ġsome Ġcommon Ġstereotypes Ġthat Ġassociate ĠAfrican ĠAmericans Ġwith Ġmore Ġnegative Ġemotions ĠB IB REF 28 Ġ. </s> Figure ĠFIG REF 28 Ġshows Ġthe Ġscore Ġdifferences Ġfor Ġindividual Ġsystems Ġon Ġrace Ġsentence Ġpairs Ġon Ġthe Ġval ence Ġregression Ġtask . ĠPl ots Ġfor Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks Ġare Ġshown Ġin ĠFigure ĠFIG REF 32 Ġin Ġthe ĠAppendix . ĠHere , Ġthe ĠIN LINE FORM 0 ĠâĢĵ sp reads Ġare Ġsmaller Ġthan Ġon Ġthe Ġgender Ġsentence Ġpairs âĢĶ from Ġ0 Ġto Ġ0 . 15 . ĠAs Ġin Ġthe Ġgender Ġanalysis , Ġon Ġthe Ġval ence Ġtask Ġthe Ġtop Ġ13 Ġsystems Ġas Ġwell Ġas Ġsome Ġof Ġthe Ġworst Ġperforming Ġsystems Ġhave Ġsmaller ĠIN LINE FORM 1 ĠâĢĵ spread Ġwhile Ġthe Ġsystems Ġwith Ġmedium Ġto Ġlow Ġperformance Ġshow Ġgreater Ġsensitivity Ġto Ġthe Ġrace - associated Ġnames . ĠHowever , Ġwe Ġdo Ġnot Ġobserve Ġthe Ġsame Ġpattern Ġin Ġthe Ġemotion Ġintensity Ġtasks . ĠAlso , Ġsimilar Ġto Ġthe Ġgender Ġanalysis , Ġmost Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifferences Ġobtained Ġlower Ġscores Ġon Ġthe Ġtweets Ġtest Ġsets . ĠOnly Ġone Ġsystem Ġout Ġof Ġthe Ġtop Ġfive Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference Ġon Ġthe Ġanger Ġand Ġfear Ġintensity Ġtasks , Ġand Ġnone Ġon Ġthe Ġother Ġtasks . ĠOnce Ġagain , Ġjust Ġas Ġin Ġthe Ġcase Ġof Ġgender , Ġthis Ġraises Ġquestions Ġof Ġthe Ġexact Ġcauses Ġof Ġsuch Ġbiases . ĠWe Ġhope Ġto Ġexplore Ġthis Ġin Ġfuture Ġwork . </s> Discussion </s> As Ġmentioned Ġin Ġthe Ġintroduction , Ġbias Ġcan Ġoriginate Ġfrom Ġany Ġor Ġseveral Ġparts Ġof Ġa Ġsystem : Ġthe Ġlabeled Ġand Ġunl abel ed Ġdatasets Ġused Ġto Ġlearn Ġdifferent Ġparts Ġof Ġthe Ġmodel , Ġthe Ġlanguage Ġresources Ġused Ġ( e . g ., Ġpre - trained Ġword Ġembed d ings , Ġlex icons ), Ġthe Ġlearning Ġmethod Ġused Ġ( al gorithm , Ġfeatures , Ġparameters ), Ġetc . ĠIn Ġour Ġanalysis , Ġwe Ġfound Ġsystems Ġtrained Ġusing Ġa Ġvariety Ġof Ġalgorithms Ġ( traditional Ġas Ġwell Ġas Ġdeep Ġneural Ġnetworks ) Ġand Ġa Ġvariety Ġof Ġlanguage Ġresources Ġshowing Ġgender Ġand Ġrace Ġbiases . ĠFurther Ġexperiments Ġmay Ġtease Ġout Ġthe Ġextent Ġof Ġbias Ġin Ġeach Ġof Ġthese Ġparts . </s> We Ġalso Ġanalyzed Ġthe Ġoutput Ġof Ġour Ġbaseline ĠS VM Ġsystem Ġtrained Ġusing Ġword Ġun ig rams Ġ( S VM - Un ig rams ). ĠThe Ġsystem Ġdoes Ġnot Ġuse Ġany Ġlanguage Ġresources Ġother Ġthan Ġthe Ġtraining Ġdata . ĠWe Ġobserve Ġthat Ġthis Ġbaseline Ġsystem Ġalso Ġshows Ġsmall Ġbias Ġin Ġgender Ġand Ġrace . ĠThe ĠIN LINE FORM 0 Ġ- sp reads Ġfor Ġthis Ġsystem Ġwere Ġquite Ġsmall : Ġ0 . 09 Ġto Ġ0 . 2 Ġon Ġthe Ġgender Ġsentence Ġpairs Ġand Ġless Ġthan Ġ0 . 002 Ġon Ġthe Ġrace Ġsentence Ġpairs . ĠThe Ġpredicted Ġintensity Ġscores Ġtended Ġto Ġbe Ġhigher Ġon Ġthe Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġthan Ġon Ġthe Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġfor Ġthe Ġtasks Ġof Ġanger , Ġfear , Ġand Ġsadness Ġintensity Ġprediction . ĠThis Ġtendency Ġwas Ġreversed Ġon Ġthe Ġtask Ġof Ġval ence Ġprediction . ĠOn Ġthe Ġrace Ġsentence Ġpairs , Ġthe Ġsystem Ġpredicted Ġhigher Ġintensity Ġscores Ġon Ġthe Ġsentences Ġwith ĠEuropean ĠAmerican Ġnames Ġfor Ġall Ġfour Ġemotion Ġintensity Ġprediction Ġtasks , Ġand Ġon Ġthe Ġsentences Ġwith ĠAfrican ĠAmerican Ġnames Ġfor Ġthe Ġtask Ġof Ġval ence Ġprediction . ĠThis Ġindicates Ġthat Ġthe Ġtraining Ġdata Ġcontains Ġsome Ġbiases Ġ( in Ġthe Ġform Ġof Ġsome Ġun ig rams Ġassociated Ġwith Ġa Ġparticular Ġgender Ġor Ġrace Ġtending Ġto Ġappear Ġin Ġtweets Ġlabeled Ġwith Ġcertain Ġemotions ). ĠThe Ġlabeled Ġdatasets Ġfor Ġthe Ġshared Ġtask Ġwere Ġcreated Ġusing Ġa Ġfairly Ġstandard Ġapproach : Ġpolling ĠTwitter Ġwith Ġtask - related Ġquery Ġterms Ġ( in Ġthis Ġcase , Ġemotion Ġwords ) Ġand Ġthen Ġmanually Ġannot ating Ġthe Ġtweets Ġwith Ġtask - specific Ġlabels . ĠThe ĠS VM - Un ig ram Ġbias Ġresults Ġshow Ġthat Ġdata Ġcollected Ġby Ġdistant Ġsupervision Ġcan Ġbe Ġa Ġsource Ġof Ġbias . ĠHowever , Ġit Ġshould Ġbe Ġnoted Ġthat Ġdifferent Ġlearning Ġmethods Ġin Ġcombination Ġwith Ġdifferent Ġlanguage Ġresources Ġcan Ġaccent uate , Ġreverse , Ġor Ġmask Ġthe Ġbias Ġpresent Ġin Ġthe Ġtraining Ġdata Ġto Ġdifferent Ġdegrees . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġcreated Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( E EC ), Ġwhich Ġconsists Ġof Ġ8 , 640 Ġsentences Ġspecifically Ġchosen Ġto Ġtease Ġout Ġgender Ġand Ġrace Ġbiases Ġin Ġnatural Ġlanguage Ġprocessing Ġsystems . ĠWe Ġused Ġthe ĠE EC Ġto Ġanalyze Ġ219 ĠN LP Ġsystems Ġthat Ġparticipated Ġin Ġa Ġrecent Ġinternational Ġshared Ġtask Ġon Ġpredicting Ġsentiment Ġand Ġemotion Ġintensity . ĠWe Ġfound Ġthat Ġmore Ġthan Ġ75 % Ġof Ġthe Ġsystems Ġtend Ġto Ġmark Ġsentences Ġinvolving Ġone Ġgender / race Ġwith Ġhigher Ġintensity Ġscores Ġthan Ġthe Ġsentences Ġinvolving Ġthe Ġother Ġgender / race . ĠWe Ġfound Ġsuch Ġbiases Ġto Ġbe Ġmore Ġwidely Ġprevalent Ġfor Ġrace Ġthan Ġfor Ġgender . ĠWe Ġalso Ġfound Ġthat Ġthe Ġbias Ġcan Ġbe Ġdifferent Ġdepending Ġon Ġthe Ġparticular Ġaffect Ġdimension Ġinvolved . </s> We Ġfound Ġthe Ġscore Ġdifferences Ġacross Ġgenders Ġand Ġacross Ġraces Ġto Ġbe Ġsomewhat Ġsmall Ġon Ġaverage Ġ( ĠIN LINE FORM 0 Ġ, Ġwhich Ġis ĠIN LINE FORM 1 Ġof Ġthe Ġ0 Ġto Ġ1 Ġscore Ġrange ). ĠHowever , Ġfor Ġsome Ġsystems Ġthe Ġscore Ġdifferences Ġreached Ġas Ġhigh Ġas Ġ0 . 34 Ġ( 34 %). ĠWhat Ġimpact Ġa Ġconsistent Ġbias , Ġeven Ġwith Ġan Ġaverage Ġmagnitude ĠIN LINE FORM 2 Ġ, Ġmight Ġhave Ġin Ġdownstream Ġapplications Ġmerits Ġfurther Ġinvestigation . </s> We Ġplan Ġto Ġextend Ġthe ĠE EC Ġwith Ġsentences Ġassociated Ġwith Ġcountry Ġnames , Ġprofessions Ġ( e . g ., Ġdoctors , Ġpolice Ġofficers , Ġjan itors , Ġteachers , Ġetc .), Ġfields Ġof Ġstudy Ġ( e . g ., Ġarts Ġvs . Ġsciences ), Ġas Ġwell Ġas Ġraces Ġ( e . g ., ĠAsian , Ġmixed , Ġetc .) Ġand Ġgenders Ġ( e . g ., Ġag ender , Ġand ro gy ne , Ġtrans , Ġqueer , Ġetc .) Ġnot Ġincluded Ġin Ġthe Ġcurrent Ġstudy . ĠWe Ġcan Ġthen Ġuse Ġthe Ġcorpus Ġto Ġexamine Ġbiases Ġacross Ġeach Ġof Ġthose Ġvariables Ġas Ġwell . ĠWe Ġare Ġalso Ġinterested Ġin Ġexploring Ġwhich Ġsystems Ġ( or Ġwhat Ġtechniques ) Ġaccent uate Ġinappropriate Ġbiases Ġin Ġthe Ġdata Ġand Ġwhich Ġsystems Ġmitigate Ġsuch Ġbiases . ĠFinally , Ġwe Ġare Ġinterested Ġin Ġexploring Ġhow Ġthe Ġquality Ġof Ġsentiment Ġanalysis Ġpredictions Ġvaries Ġwhen Ġapplied Ġto Ġtext Ġproduced Ġby Ġdifferent Ġdemographic Ġgroups , Ġsuch Ġas Ġpeople Ġof Ġdifferent Ġraces , Ġgenders , Ġand Ġethnic ities . </s> The ĠEquity ĠEvaluation ĠCorpus Ġand Ġthe Ġproposed Ġmethodology Ġto Ġexamine Ġbias Ġare Ġnot Ġmeant Ġto Ġbe Ġcomprehensive . ĠHowever , Ġusing Ġseveral Ġapproaches Ġand Ġdatasets Ġsuch Ġas Ġthe Ġone Ġproposed Ġhere Ġcan Ġbring Ġabout Ġa Ġmore Ġthorough Ġexamination Ġof Ġinappropriate Ġbiases Ġin Ġmodern Ġmachine Ġlearning Ġsystems . </s> App endix </s> Fig ures ĠFIG REF 31 Ġand ĠFIG REF 32 Ġshow Ġbox Ġplots Ġof Ġthe Ġscore Ġdifferences Ġfor Ġeach Ġsystem Ġon Ġthe Ġfour Ġemotion Ġintensity Ġregression Ġtasks Ġon Ġthe Ġgender Ġand Ġrace Ġsentence Ġpairs , Ġrespectively . ĠEach Ġpoint Ġon Ġa Ġplot Ġcorresponds Ġto Ġthe Ġdifference Ġin Ġscores Ġpredicted Ġby Ġthe Ġsystem Ġon Ġone Ġsentence Ġpair . ĠThe Ġsystems Ġare Ġordered Ġby Ġtheir Ġperformance Ġrank Ġ( from Ġfirst Ġto Ġlast ) Ġon Ġthe Ġtask Ġas Ġper Ġthe Ġofficial Ġevaluation Ġmetric Ġon Ġthe Ġtweets Ġtest Ġsets .
Introduction [0]leftmargin=* [0]leftmargin=* Automatic systems have had a significant and beneficial impact on all walks of human life. So much so that it is easy to overlook their potential to benefit society by promoting equity, diversity, and fairness. For example, machines do not take bribes to do their jobs, they can determine eligibility for a loan without being influenced by the color of the applicant's skin, and they can provide access to information and services without discrimination based on gender or sexual orientation. Nonetheless, as machine learning systems become more human-like in their predictions, they can also perpetuate human biases. Some learned biases may be beneficial for the downstream application (e.g., learning that humans often use some insect names, such as spider or cockroach, to refer to unpleasant situations). Other biases can be inappropriate and result in negative experiences for some groups of people. Examples include, loan eligibility and crime recidivism prediction systems that negatively assess people belonging to a certain pin/zip code (which may disproportionately impact people of a certain race) BIBREF0 and resumé sorting systems that believe that men are more qualified to be programmers than women BIBREF1 . Similarly, sentiment and emotion analysis systems can also perpetuate and accentuate inappropriate human biases, e.g., systems that consider utterances from one race or gender to be less positive simply because of their race or gender, or customer support systems that prioritize a call from an angry male over a call from the equally angry female. Predictions of machine learning systems have also been shown to be of higher quality when dealing with information from some groups of people as opposed to other groups of people. For example, in the area of computer vision, gender classification systems perform particularly poorly for darker skinned females BIBREF2 . Natural language processing (NLP) systems have been shown to be poor in understanding text produced by people belonging to certain races BIBREF3 , BIBREF4 . For NLP systems, the sources of the bias often include the training data, other corpora, lexicons, and word embeddings that the machine learning algorithm may leverage to build its prediction model. Even though there is some recent work highlighting such inappropriate biases (such as the work mentioned above), each such past work has largely focused on just one or two systems and resources. Further, there is no benchmark dataset for examining inappropriate biases in natural language systems. In this paper, we describe how we compiled a dataset of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders. We will refer to it as the Equity Evaluation Corpus (EEC). We used the EEC as a supplementary test set in a recent shared task on predicting sentiment and emotion intensity in tweets, SemEval-2018 Task 1: Affect in Tweets BIBREF5 . In particular, we wanted to test a hypothesis that a system should equally rate the intensity of the emotion expressed by two sentences that differ only in the gender/race of a person mentioned. Note that here the term system refers to the combination of a machine learning architecture trained on a labeled dataset, and possibly using additional language resources. The bias can originate from any or several of these parts. We were thus able to use the EEC to examine 219 sentiment analysis systems that took part in the shared task. We compare emotion and sentiment intensity scores that the systems predict on pairs of sentences in the EEC that differ only in one word corresponding to race or gender (e.g., `This man made me feel angry' vs. `This woman made me feel angry'). We find that the majority of the systems studied show statistically significant bias; that is, they consistently provide slightly higher sentiment intensity predictions for sentences associated with one race or one gender. We also find that the bias may be different depending on the particular affect dimension that the natural language system is trained to predict. Despite the work we describe here and what others have proposed in the past, it should be noted that there are no simple solutions for dealing with inappropriate human biases that percolate into machine learning systems. It seems difficult to ever be able to identify and quantify all of the inappropriate biases perfectly (even when restricted to the scope of just gender and race). Further, any such mechanism is liable to be circumvented, if one chooses to do so. Nonetheless, as developers of sentiment analysis systems, and NLP systems more broadly, we cannot absolve ourselves of the ethical implications of the systems we build. Even if it is unclear how we should deal with the inappropriate biases in our systems, we should be measuring such biases. The Equity Evaluation Corpus is not meant to be a catch-all for all inappropriate biases, but rather just one of the several ways by which we can examine the fairness of sentiment analysis systems. We make the corpus freely available so that both developers and users can use it, and build on it. Related Work Recent studies have demonstrated that the systems trained on the human-written texts learn human-like biases BIBREF1 , BIBREF6 . In general, any predictive model built on historical data may inadvertently inherit human biases based on gender, ethnicity, race, or religion BIBREF7 , BIBREF8 . Discrimination-aware data mining focuses on measuring discrimination in data as well as on evaluating performance of discrimination-aware predictive models BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . In NLP, the attention so far has been primarily on word embeddings—a popular and powerful framework to represent words as low-dimensional dense vectors. The word embeddings are usually obtained from large amounts of human-written texts, such as Wikipedia, Google News articles, or millions of tweets. Bias in sentiment analysis systems has only been explored in simple systems that make use of pre-computed word embeddings BIBREF13 . There is no prior work that systematically quantifies the extent of bias in a large number of sentiment analysis systems. This paper does not examine the differences in accuracies of systems on text produced by different races or genders, as was done by hovy2015demographic,blodgett2016demographic,jurgens2017incorporating,buolamwini2018gender. Approaches on how to mitigate inappropriate biases BIBREF14 , BIBREF1 , BIBREF15 , BIBREF16 , BIBREF13 , BIBREF17 , BIBREF18 are also beyond the scope of this paper. See also the position paper by hovy2016social, which identifies socio-ethical implications of the NLP systems in general. The Equity Evaluation Corpus We now describe how we compiled a dataset of thousands of sentences to determine whether automatic systems consistently give higher (or lower) sentiment intensity scores to sentences involving a particular race or gender. There are several ways in which such a dataset may be compiled. We present below the choices that we made. We decided to use sentences involving at least one race- or gender-associated word. The sentences were intended to be short and grammatically simple. We also wanted some sentences to include expressions of sentiment and emotion, since the goal is to test sentiment and emotion systems. We, the authors of this paper, developed eleven sentence templates after several rounds of discussion and consensus building. They are shown in Table TABREF3 . The templates are divided into two groups. The first type (templates 1–7) includes emotion words. The purpose of this set is to have sentences expressing emotions. The second type (templates 8–11) does not include any emotion words. The purpose of this set is to have non-emotional (neutral) sentences. The templates include two variables: INLINEFORM0 person INLINEFORM1 and INLINEFORM2 emotion word INLINEFORM3 . We generate sentences from the template by instantiating each variable with one of the pre-chosen values that the variable can take. Each of the eleven templates includes the variable INLINEFORM4 person INLINEFORM5 . INLINEFORM6 person INLINEFORM7 can be instantiated by any of the following noun phrases: For our study, we chose ten names of each kind from the study by Caliskan:2017 (see Table TABREF7 ). The full lists of noun phrases representing females and males, used in our study, are shown in Table TABREF8 . The second variable, INLINEFORM0 emotion word INLINEFORM1 , has two variants. Templates one through four include a variable for an emotional state word. The emotional state words correspond to four basic emotions: anger, fear, joy, and sadness. Specifically, for each of the emotions, we selected five words that convey that emotion in varying intensities. These words were taken from the categories in the Roget's Thesaurus corresponding to the four emotions: category #900 Resentment (for anger), category #860 Fear (for fear), category #836 Cheerfulness (for joy), and category #837 Dejection (for sadness). Templates five through seven include emotion words describing a situation or event. These words were also taken from the same thesaurus categories listed above. The full lists of emotion words (emotional state words and emotional situation/event words) are shown in Table TABREF10 . We generated sentences from the templates by replacing INLINEFORM0 person INLINEFORM1 and INLINEFORM2 emotion word INLINEFORM3 variables with the values they can take. In total, 8,640 sentences were generated with the various combinations of INLINEFORM4 person INLINEFORM5 and INLINEFORM6 emotion word INLINEFORM7 values across the eleven templates. We manually examined the sentences to make sure they were grammatically well-formed. Notably, one can derive pairs of sentences from the EEC such that they differ only in one word corresponding to gender or race (e.g., `My daughter feels devastated' and `My son feels devastated'). We refer to the full set of 8,640 sentences as Equity Evaluation Corpus. Measuring Race and Gender Bias in Automatic Sentiment Analysis Systems The race and gender bias evaluation was carried out on the output of the 219 automatic systems that participated in SemEval-2018 Task 1: Affect in Tweets BIBREF5 . The shared task included five subtasks on inferring the affectual state of a person from their tweet: 1. emotion intensity regression, 2. emotion intensity ordinal classification, 3. valence (sentiment) regression, 4. valence ordinal classification, and 5. emotion classification. For each subtask, labeled data were provided for English, Arabic, and Spanish. The race and gender bias were analyzed for the system outputs on two English subtasks: emotion intensity regression (for anger, fear, joy, and sadness) and valence regression. These regression tasks were formulated as follows: Given a tweet and an affective dimension A (anger, fear, joy, sadness, or valence), determine the intensity of A that best represents the mental state of the tweeter—a real-valued score between 0 (least A) and 1 (most A). Separate training and test datasets were provided for each affective dimension. Training sets included tweets along with gold intensity scores. Two test sets were provided for each task: 1. a regular tweet test set (for which the gold intensity scores are known but not revealed to the participating systems), and 2. the Equity Evaluation Corpus (for which no gold intensity labels exist). Participants were told that apart from the usual test set, they are to run their systems on a separate test set of unknown origin. The participants were instructed to train their system on the tweets training sets provided, and that they could use any other resources they may find or create. They were to run the same final system on the two test sets. The nature of the second test set was revealed to them only after the competition. The first (tweets) test set was used to evaluate and rank the quality (accuracy) of the systems' predictions. The second (EEC) test set was used to perform the bias analysis, which is the focus of this paper. Systems: Fifty teams submitted their system outputs to one or more of the five emotion intensity regression tasks (for anger, fear, joy, sadness, and valence), resulting in 219 submissions in total. Many systems were built using two types of features: deep neural network representations of tweets (sentence embeddings) and features derived from existing sentiment and emotion lexicons. These features were then combined to learn a model using either traditional machine learning algorithms (such as SVM/SVR and Logistic Regression) or deep neural networks. SVM/SVR, LSTMs, and Bi-LSTMs were some of the most widely used machine learning algorithms. The sentence embeddings were obtained by training a neural network on the provided training data, a distant supervision corpus (e.g., AIT2018 Distant Supervision Corpus that has tweets with emotion-related query terms), sentiment-labeled tweet corpora (e.g., Semeval-2017 Task4A dataset on sentiment analysis in Twitter), or by using pre-trained models (e.g., DeepMoji BIBREF20 , Skip thoughts BIBREF21 ). The lexicon features were often derived from the NRC emotion and sentiment lexicons BIBREF22 , BIBREF23 , BIBREF24 , AFINN BIBREF25 , and Bing Liu Lexicon BIBREF26 . We provided a baseline SVM system trained using word unigrams as features on the training data (SVM-Unigrams). This system is also included in the current analysis. Measuring bias: To examine gender bias, we compared each system's predicted scores on the EEC sentence pairs as follows: Thus, eleven pairs of scores (ten pairs of scores from ten noun phrase pairs and one pair of scores from the averages on name subsets) were examined for each template–emotion word instantiation. There were twenty different emotion words used in seven templates (templates 1–7), and no emotion words used in the four remaining templates (templates 8–11). In total, INLINEFORM0 pairs of scores were compared. Similarly, to examine race bias, we compared pairs of system predicted scores as follows: Thus, one pair of scores was examined for each template–emotion word instantiation. In total, INLINEFORM0 pairs of scores were compared. For each system, we calculated the paired two sample t-test to determine whether the mean difference between the two sets of scores (across the two races and across the two genders) is significant. We set the significance level to 0.05. However, since we performed 438 assessments (219 submissions evaluated for biases in both gender and race), we applied Bonferroni correction. The null hypothesis that the true mean difference between the paired samples was zero was rejected if the calculated p-value fell below INLINEFORM0 . Results The two sub-sections below present the results from the analysis for gender bias and race bias, respectively. Gender Bias Results Individual submission results were communicated to the participants. Here, we present the summary results across all the teams. The goal of this analysis is to gain a better understanding of biases across a large number of current sentiment analysis systems. Thus, we partition the submissions into three groups according to the bias they show: F=M not significant: submissions that showed no statistically significant difference in intensity scores predicted for corresponding female and male noun phrase sentences, F INLINEFORM0 –M INLINEFORM1 significant: submissions that consistently gave higher scores for sentences with female noun phrases than for corresponding sentences with male noun phrases, F INLINEFORM0 –M INLINEFORM1 significant: submissions that consistently gave lower scores for sentences with female noun phrases than for corresponding sentences with male noun phrases. For each system and each sentence pair, we calculate the score difference INLINEFORM0 as the score for the female noun phrase sentence minus the score for the corresponding male noun phrase sentence. Table TABREF24 presents the summary results for each of the bias groups. It has the following columns: #Subm.: number of submissions in each group. If all the systems are unbiased, then the number of submissions for the group F=M not significant would be the maximum, and the number of submissions in all other groups would be zero. Avg. score difference F INLINEFORM0 –M INLINEFORM1 : the average INLINEFORM2 for only those pairs where the score for the female noun phrase sentence is higher. The greater the magnitude of this score, the stronger the bias in systems that consistently give higher scores to female-associated sentences. Avg. score difference F INLINEFORM0 –M INLINEFORM1 : the average INLINEFORM2 for only those pairs where the score for the female noun phrase sentence is lower. The greater the magnitude of this score, the stronger the bias in systems that consistently give lower scores to female-associated sentences. Note that these numbers were first calculated separately for each submission, and then averaged over all the submissions within each submission group. The results are reported separately for submissions to each task (anger, fear, joy, sadness, and sentiment/valence intensity prediction). Observe that on the four emotion intensity prediction tasks, only about 12 of the 46 submissions (about 25% of the submissions) showed no statistically significant score difference. On the valence prediction task, only 5 of the 36 submissions (14% of the submissions) showed no statistically significant score difference. Thus 75% to 86% of the submissions consistently marked sentences of one gender higher than another. When predicting anger, joy, or valence, the number of systems consistently giving higher scores to sentences with female noun phrases (21–25) is markedly higher than the number of systems giving higher scores to sentences with male noun phrases (8–13). (Recall that higher valence means more positive sentiment.) In contrast, on the fear task, most submissions tended to assign higher scores to sentences with male noun phrases (23) as compared to the number of systems giving higher scores to sentences with female noun phrases (12). When predicting sadness, the number of submissions that mostly assigned higher scores to sentences with female noun phrases (18) is close to the number of submissions that mostly assigned higher scores to sentences with male noun phrases (16). These results are in line with some common stereotypes, such as females are more emotional, and situations involving male agents are more fearful BIBREF27 . Figure FIGREF25 shows the score differences ( INLINEFORM0 ) for individual systems on the valence regression task. Plots for the four emotion intensity prediction tasks are shown in Figure FIGREF31 in the Appendix. Each point ( ▲, ▼, ●) on the plot corresponds to the difference in scores predicted by the system on one sentence pair. The systems are ordered by their rank (from first to last) on the task on the tweets test sets, as per the official evaluation metric (Spearman correlation with the gold intensity scores). We will refer to the difference between the maximal value of INLINEFORM1 and the minimal value of INLINEFORM2 for a particular system as the INLINEFORM3 –spread. Observe that the INLINEFORM4 –spreads for many systems are rather large, up to 0.57. Depending on the task, the top 10 or top 15 systems as well as some of the worst performing systems tend to have smaller INLINEFORM5 –spreads while the systems with medium to low performance show greater sensitivity to the gender-associated words. Also, most submissions that showed no statistically significant score differences (shown in green) performed poorly on the tweets test sets. Only three systems out of the top five on the anger intensity task and one system on the joy and sadness tasks showed no statistically significant score difference. This indicates that when considering only those systems that performed well on the intensity prediction task, the percentage of gender-biased systems are even higher than those indicated above. These results raise further questions such as `what exactly is the cause of such biases?' and `why is the bias impacted by the emotion task under consideration?'. Answering these questions will require further information on the resources that the teams used to develop their models, and we leave that for future work. Average score differences: For submissions that showed statistically significant score differences, the average score difference F INLINEFORM0 –M INLINEFORM1 and the average score difference F INLINEFORM2 –M INLINEFORM3 were INLINEFORM4 . Since the intensity scores range from 0 to 1, 0.03 is 3% of the full range. The maximal score difference ( INLINEFORM5 ) across all the submissions was as high as 0.34. Note, however, that these INLINEFORM6 s are the result of changing just one word in a sentence. In more complex sentences, several gender-associated words can appear, which may have a bigger impact. Also, whether consistent score differences of this magnitude will have significant repercussions in downstream applications, depends on the particular application. Analyses on only the neutral sentences in EEC and only the emotional sentences in EEC: We also performed a separate analysis using only those sentences from the EEC that included no emotion words. Recall that there are four templates that contain no emotion words. Tables TABREF26 shows these results. We observe similar trends as in the analysis on the full set. One noticeable difference is that the number of submissions that showed statistically significant score difference is much smaller for this data subset. However, the total number of comparisons on the subset (44) is much smaller than the total number of comparisons on the full set (1,584), which makes the statistical test less powerful. Note also that the average score differences on the subset (columns 3 and 4 in Table TABREF26 ) tend to be higher than the differences on the full set (columns 3 and 4 in Table TABREF24 ). This indicates that gender-associated words can have a bigger impact on system predictions for neutral sentences. We also performed an analysis by restricting the dataset to contain only the sentences with the emotion words corresponding to the emotion task (i.e., submissions to the anger intensity prediction task were evaluated only on sentences with anger words). The results (not shown here) were similar to the results on the full set. Race Bias Results We did a similar analysis for race as we did for gender. For each submission on each task, we calculated the difference between the average predicted score on the set of sentences with African American (AA) names and the average predicted score on the set of sentences with European American (EA) names. Then, we aggregated the results over all such sentence pairs in the EEC. Table TABREF29 shows the results. The table has the same form and structure as the gender result tables. Observe that the number of submissions with no statistically significant score difference for sentences pertaining to the two races is about 5–11 (about 11% to 24%) for the four emotions and 3 (about 8%) for valence. These numbers are even lower than what was found for gender. The majority of the systems assigned higher scores to sentences with African American names on the tasks of anger, fear, and sadness intensity prediction. On the joy and valence tasks, most submissions tended to assign higher scores to sentences with European American names. These tendencies reflect some common stereotypes that associate African Americans with more negative emotions BIBREF28 . Figure FIGREF28 shows the score differences for individual systems on race sentence pairs on the valence regression task. Plots for the four emotion intensity prediction tasks are shown in Figure FIGREF32 in the Appendix. Here, the INLINEFORM0 –spreads are smaller than on the gender sentence pairs—from 0 to 0.15. As in the gender analysis, on the valence task the top 13 systems as well as some of the worst performing systems have smaller INLINEFORM1 –spread while the systems with medium to low performance show greater sensitivity to the race-associated names. However, we do not observe the same pattern in the emotion intensity tasks. Also, similar to the gender analysis, most submissions that showed no statistically significant score differences obtained lower scores on the tweets test sets. Only one system out of the top five showed no statistically significant score difference on the anger and fear intensity tasks, and none on the other tasks. Once again, just as in the case of gender, this raises questions of the exact causes of such biases. We hope to explore this in future work. Discussion As mentioned in the introduction, bias can originate from any or several parts of a system: the labeled and unlabeled datasets used to learn different parts of the model, the language resources used (e.g., pre-trained word embeddings, lexicons), the learning method used (algorithm, features, parameters), etc. In our analysis, we found systems trained using a variety of algorithms (traditional as well as deep neural networks) and a variety of language resources showing gender and race biases. Further experiments may tease out the extent of bias in each of these parts. We also analyzed the output of our baseline SVM system trained using word unigrams (SVM-Unigrams). The system does not use any language resources other than the training data. We observe that this baseline system also shows small bias in gender and race. The INLINEFORM0 -spreads for this system were quite small: 0.09 to 0.2 on the gender sentence pairs and less than 0.002 on the race sentence pairs. The predicted intensity scores tended to be higher on the sentences with male noun phrases than on the sentences with female noun phrases for the tasks of anger, fear, and sadness intensity prediction. This tendency was reversed on the task of valence prediction. On the race sentence pairs, the system predicted higher intensity scores on the sentences with European American names for all four emotion intensity prediction tasks, and on the sentences with African American names for the task of valence prediction. This indicates that the training data contains some biases (in the form of some unigrams associated with a particular gender or race tending to appear in tweets labeled with certain emotions). The labeled datasets for the shared task were created using a fairly standard approach: polling Twitter with task-related query terms (in this case, emotion words) and then manually annotating the tweets with task-specific labels. The SVM-Unigram bias results show that data collected by distant supervision can be a source of bias. However, it should be noted that different learning methods in combination with different language resources can accentuate, reverse, or mask the bias present in the training data to different degrees. Conclusions and Future Work We created the Equity Evaluation Corpus (EEC), which consists of 8,640 sentences specifically chosen to tease out gender and race biases in natural language processing systems. We used the EEC to analyze 219 NLP systems that participated in a recent international shared task on predicting sentiment and emotion intensity. We found that more than 75% of the systems tend to mark sentences involving one gender/race with higher intensity scores than the sentences involving the other gender/race. We found such biases to be more widely prevalent for race than for gender. We also found that the bias can be different depending on the particular affect dimension involved. We found the score differences across genders and across races to be somewhat small on average ( INLINEFORM0 , which is INLINEFORM1 of the 0 to 1 score range). However, for some systems the score differences reached as high as 0.34 (34%). What impact a consistent bias, even with an average magnitude INLINEFORM2 , might have in downstream applications merits further investigation. We plan to extend the EEC with sentences associated with country names, professions (e.g., doctors, police officers, janitors, teachers, etc.), fields of study (e.g., arts vs. sciences), as well as races (e.g., Asian, mixed, etc.) and genders (e.g., agender, androgyne, trans, queer, etc.) not included in the current study. We can then use the corpus to examine biases across each of those variables as well. We are also interested in exploring which systems (or what techniques) accentuate inappropriate biases in the data and which systems mitigate such biases. Finally, we are interested in exploring how the quality of sentiment analysis predictions varies when applied to text produced by different demographic groups, such as people of different races, genders, and ethnicities. The Equity Evaluation Corpus and the proposed methodology to examine bias are not meant to be comprehensive. However, using several approaches and datasets such as the one proposed here can bring about a more thorough examination of inappropriate biases in modern machine learning systems. Appendix Figures FIGREF31 and FIGREF32 show box plots of the score differences for each system on the four emotion intensity regression tasks on the gender and race sentence pairs, respectively. Each point on a plot corresponds to the difference in scores predicted by the system on one sentence pair. The systems are ordered by their performance rank (from first to last) on the task as per the official evaluation metric on the tweets test sets.
What criteria are used to select the 8,640 English sentences?
0f12dc077fe8e5b95ca9163cea1dd17195c96929
What Ġcriteria Ġare Ġused Ġto Ġselect Ġthe Ġ8 , 640 ĠEnglish Ġsentences ?
Sent ences Ġinvolving Ġat Ġleast Ġone Ġrace - Ġor Ġgender - associated Ġword , Ġ Ġsentence Ġ Ġhave Ġto Ġbe Ġshort Ġand Ġgram matically Ġsimple , Ġ Ġsentence Ġhave Ġto Ġ Ġinclude Ġexpressions Ġof Ġsentiment Ġand Ġemotion .
<s> What Ġcriteria Ġare Ġused Ġto Ġselect Ġthe Ġ8 , 640 ĠEnglish Ġsentences ? </s> Introduction </s> [ 0 ] left margin = * Ġ[ 0 ] left margin = * </s> Aut omatic Ġsystems Ġhave Ġhad Ġa Ġsignificant Ġand Ġbeneficial Ġimpact Ġon Ġall Ġwalks Ġof Ġhuman Ġlife . ĠSo Ġmuch Ġso Ġthat Ġit Ġis Ġeasy Ġto Ġoverlook Ġtheir Ġpotential Ġto Ġbenefit Ġsociety Ġby Ġpromoting Ġequity , Ġdiversity , Ġand Ġfairness . ĠFor Ġexample , Ġmachines Ġdo Ġnot Ġtake Ġbribes Ġto Ġdo Ġtheir Ġjobs , Ġthey Ġcan Ġdetermine Ġeligibility Ġfor Ġa Ġloan Ġwithout Ġbeing Ġinfluenced Ġby Ġthe Ġcolor Ġof Ġthe Ġapplicant 's Ġskin , Ġand Ġthey Ġcan Ġprovide Ġaccess Ġto Ġinformation Ġand Ġservices Ġwithout Ġdiscrimination Ġbased Ġon Ġgender Ġor Ġsexual Ġorientation . ĠNonetheless , Ġas Ġmachine Ġlearning Ġsystems Ġbecome Ġmore Ġhuman - like Ġin Ġtheir Ġpredictions , Ġthey Ġcan Ġalso Ġperpetuate Ġhuman Ġbiases . ĠSome Ġlearned Ġbiases Ġmay Ġbe Ġbeneficial Ġfor Ġthe Ġdownstream Ġapplication Ġ( e . g ., Ġlearning Ġthat Ġhumans Ġoften Ġuse Ġsome Ġinsect Ġnames , Ġsuch Ġas Ġspider Ġor Ġcock roach , Ġto Ġrefer Ġto Ġunpleasant Ġsituations ). ĠOther Ġbiases Ġcan Ġbe Ġinappropriate Ġand Ġresult Ġin Ġnegative Ġexperiences Ġfor Ġsome Ġgroups Ġof Ġpeople . ĠExamples Ġinclude , Ġloan Ġeligibility Ġand Ġcrime Ġrec id ivism Ġprediction Ġsystems Ġthat Ġnegatively Ġassess Ġpeople Ġbelonging Ġto Ġa Ġcertain Ġpin / zip Ġcode Ġ( which Ġmay Ġdisproportionately Ġimpact Ġpeople Ġof Ġa Ġcertain Ġrace ) ĠB IB REF 0 Ġand Ġres um Ã© Ġsorting Ġsystems Ġthat Ġbelieve Ġthat Ġmen Ġare Ġmore Ġqualified Ġto Ġbe Ġprogrammers Ġthan Ġwomen ĠB IB REF 1 Ġ. ĠSimilarly , Ġsentiment Ġand Ġemotion Ġanalysis Ġsystems Ġcan Ġalso Ġperpetuate Ġand Ġaccent uate Ġinappropriate Ġhuman Ġbiases , Ġe . g ., Ġsystems Ġthat Ġconsider Ġutter ances Ġfrom Ġone Ġrace Ġor Ġgender Ġto Ġbe Ġless Ġpositive Ġsimply Ġbecause Ġof Ġtheir Ġrace Ġor Ġgender , Ġor Ġcustomer Ġsupport Ġsystems Ġthat Ġprioritize Ġa Ġcall Ġfrom Ġan Ġangry Ġmale Ġover Ġa Ġcall Ġfrom Ġthe Ġequally Ġangry Ġfemale . </s> Pred ictions Ġof Ġmachine Ġlearning Ġsystems Ġhave Ġalso Ġbeen Ġshown Ġto Ġbe Ġof Ġhigher Ġquality Ġwhen Ġdealing Ġwith Ġinformation Ġfrom Ġsome Ġgroups Ġof Ġpeople Ġas Ġopposed Ġto Ġother Ġgroups Ġof Ġpeople . ĠFor Ġexample , Ġin Ġthe Ġarea Ġof Ġcomputer Ġvision , Ġgender Ġclassification Ġsystems Ġperform Ġparticularly Ġpoorly Ġfor Ġdarker Ġskin ned Ġfemales ĠB IB REF 2 Ġ. ĠNatural Ġlanguage Ġprocessing Ġ( N LP ) Ġsystems Ġhave Ġbeen Ġshown Ġto Ġbe Ġpoor Ġin Ġunderstanding Ġtext Ġproduced Ġby Ġpeople Ġbelonging Ġto Ġcertain Ġraces ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠFor ĠN LP Ġsystems , Ġthe Ġsources Ġof Ġthe Ġbias Ġoften Ġinclude Ġthe Ġtraining Ġdata , Ġother Ġcorpor a , Ġlex icons , Ġand Ġword Ġembed d ings Ġthat Ġthe Ġmachine Ġlearning Ġalgorithm Ġmay Ġleverage Ġto Ġbuild Ġits Ġprediction Ġmodel . </s> Even Ġthough Ġthere Ġis Ġsome Ġrecent Ġwork Ġhighlighting Ġsuch Ġinappropriate Ġbiases Ġ( such Ġas Ġthe Ġwork Ġmentioned Ġabove ), Ġeach Ġsuch Ġpast Ġwork Ġhas Ġlargely Ġfocused Ġon Ġjust Ġone Ġor Ġtwo Ġsystems Ġand Ġresources . ĠFurther , Ġthere Ġis Ġno Ġbenchmark Ġdataset Ġfor Ġexamining Ġinappropriate Ġbiases Ġin Ġnatural Ġlanguage Ġsystems . ĠIn Ġthis Ġpaper , Ġwe Ġdescribe Ġhow Ġwe Ġcompiled Ġa Ġdataset Ġof Ġ8 , 640 ĠEnglish Ġsentences Ġcarefully Ġchosen Ġto Ġtease Ġout Ġbiases Ġtowards Ġcertain Ġraces Ġand Ġgenders . ĠWe Ġwill Ġrefer Ġto Ġit Ġas Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( E EC ). ĠWe Ġused Ġthe ĠE EC Ġas Ġa Ġsupplementary Ġtest Ġset Ġin Ġa Ġrecent Ġshared Ġtask Ġon Ġpredicting Ġsentiment Ġand Ġemotion Ġintensity Ġin Ġtweets , ĠSem E val - 2018 ĠTask Ġ1 : ĠAffect Ġin ĠTwe ets ĠB IB REF 5 Ġ. ĠIn Ġparticular , Ġwe Ġwanted Ġto Ġtest Ġa Ġhypothesis Ġthat Ġa Ġsystem Ġshould Ġequally Ġrate Ġthe Ġintensity Ġof Ġthe Ġemotion Ġexpressed Ġby Ġtwo Ġsentences Ġthat Ġdiffer Ġonly Ġin Ġthe Ġgender / race Ġof Ġa Ġperson Ġmentioned . ĠNote Ġthat Ġhere Ġthe Ġterm Ġsystem Ġrefers Ġto Ġthe Ġcombination Ġof Ġa Ġmachine Ġlearning Ġarchitecture Ġtrained Ġon Ġa Ġlabeled Ġdataset , Ġand Ġpossibly Ġusing Ġadditional Ġlanguage Ġresources . ĠThe Ġbias Ġcan Ġoriginate Ġfrom Ġany Ġor Ġseveral Ġof Ġthese Ġparts . ĠWe Ġwere Ġthus Ġable Ġto Ġuse Ġthe ĠE EC Ġto Ġexamine Ġ219 Ġsentiment Ġanalysis Ġsystems Ġthat Ġtook Ġpart Ġin Ġthe Ġshared Ġtask . </s> We Ġcompare Ġemotion Ġand Ġsentiment Ġintensity Ġscores Ġthat Ġthe Ġsystems Ġpredict Ġon Ġpairs Ġof Ġsentences Ġin Ġthe ĠE EC Ġthat Ġdiffer Ġonly Ġin Ġone Ġword Ġcorresponding Ġto Ġrace Ġor Ġgender Ġ( e . g ., Ġ` This Ġman Ġmade Ġme Ġfeel Ġangry ' Ġvs . Ġ` This Ġwoman Ġmade Ġme Ġfeel Ġangry '). ĠWe Ġfind Ġthat Ġthe Ġmajority Ġof Ġthe Ġsystems Ġstudied Ġshow Ġstatistically Ġsignificant Ġbias ; Ġthat Ġis , Ġthey Ġconsistently Ġprovide Ġslightly Ġhigher Ġsentiment Ġintensity Ġpredictions Ġfor Ġsentences Ġassociated Ġwith Ġone Ġrace Ġor Ġone Ġgender . ĠWe Ġalso Ġfind Ġthat Ġthe Ġbias Ġmay Ġbe Ġdifferent Ġdepending Ġon Ġthe Ġparticular Ġaffect Ġdimension Ġthat Ġthe Ġnatural Ġlanguage Ġsystem Ġis Ġtrained Ġto Ġpredict . </s> Despite Ġthe Ġwork Ġwe Ġdescribe Ġhere Ġand Ġwhat Ġothers Ġhave Ġproposed Ġin Ġthe Ġpast , Ġit Ġshould Ġbe Ġnoted Ġthat Ġthere Ġare Ġno Ġsimple Ġsolutions Ġfor Ġdealing Ġwith Ġinappropriate Ġhuman Ġbiases Ġthat Ġper col ate Ġinto Ġmachine Ġlearning Ġsystems . ĠIt Ġseems Ġdifficult Ġto Ġever Ġbe Ġable Ġto Ġidentify Ġand Ġquantify Ġall Ġof Ġthe Ġinappropriate Ġbiases Ġperfectly Ġ( even Ġwhen Ġrestricted Ġto Ġthe Ġscope Ġof Ġjust Ġgender Ġand Ġrace ). ĠFurther , Ġany Ġsuch Ġmechanism Ġis Ġliable Ġto Ġbe Ġcircumvent ed , Ġif Ġone Ġchooses Ġto Ġdo Ġso . ĠNonetheless , Ġas Ġdevelopers Ġof Ġsentiment Ġanalysis Ġsystems , Ġand ĠN LP Ġsystems Ġmore Ġbroadly , Ġwe Ġcannot Ġabs olve Ġourselves Ġof Ġthe Ġethical Ġimplications Ġof Ġthe Ġsystems Ġwe Ġbuild . ĠEven Ġif Ġit Ġis Ġunclear Ġhow Ġwe Ġshould Ġdeal Ġwith Ġthe Ġinappropriate Ġbiases Ġin Ġour Ġsystems , Ġwe Ġshould Ġbe Ġmeasuring Ġsuch Ġbiases . ĠThe ĠEquity ĠEvaluation ĠCorpus Ġis Ġnot Ġmeant Ġto Ġbe Ġa Ġcatch - all Ġfor Ġall Ġinappropriate Ġbiases , Ġbut Ġrather Ġjust Ġone Ġof Ġthe Ġseveral Ġways Ġby Ġwhich Ġwe Ġcan Ġexamine Ġthe Ġfairness Ġof Ġsentiment Ġanalysis Ġsystems . ĠWe Ġmake Ġthe Ġcorpus Ġfreely Ġavailable Ġso Ġthat Ġboth Ġdevelopers Ġand Ġusers Ġcan Ġuse Ġit , Ġand Ġbuild Ġon Ġit . </s> Related ĠWork </s> Recent Ġstudies Ġhave Ġdemonstrated Ġthat Ġthe Ġsystems Ġtrained Ġon Ġthe Ġhuman - written Ġtexts Ġlearn Ġhuman - like Ġbiases ĠB IB REF 1 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġgeneral , Ġany Ġpredictive Ġmodel Ġbuilt Ġon Ġhistorical Ġdata Ġmay Ġinadvertently Ġinherit Ġhuman Ġbiases Ġbased Ġon Ġgender , Ġethnicity , Ġrace , Ġor Ġreligion ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠDiscrimination - aware Ġdata Ġmining Ġfocuses Ġon Ġmeasuring Ġdiscrimination Ġin Ġdata Ġas Ġwell Ġas Ġon Ġevaluating Ġperformance Ġof Ġdiscrimination - aware Ġpredictive Ġmodels ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> In ĠN LP , Ġthe Ġattention Ġso Ġfar Ġhas Ġbeen Ġprimarily Ġon Ġword Ġembed d ings âĢĶ a Ġpopular Ġand Ġpowerful Ġframework Ġto Ġrepresent Ġwords Ġas Ġlow - dimensional Ġdense Ġvectors . ĠThe Ġword Ġembed d ings Ġare Ġusually Ġobtained Ġfrom Ġlarge Ġamounts Ġof Ġhuman - written Ġtexts , Ġsuch Ġas ĠWikipedia , ĠGoogle ĠNews Ġarticles , Ġor Ġmillions Ġof Ġtweets . ĠB ias Ġin Ġsentiment Ġanalysis Ġsystems Ġhas Ġonly Ġbeen Ġexplored Ġin Ġsimple Ġsystems Ġthat Ġmake Ġuse Ġof Ġpre - com puted Ġword Ġembed d ings ĠB IB REF 13 Ġ. ĠThere Ġis Ġno Ġprior Ġwork Ġthat Ġsystematically Ġquant ifies Ġthe Ġextent Ġof Ġbias Ġin Ġa Ġlarge Ġnumber Ġof Ġsentiment Ġanalysis Ġsystems . </s> This Ġpaper Ġdoes Ġnot Ġexamine Ġthe Ġdifferences Ġin Ġaccur acies Ġof Ġsystems Ġon Ġtext Ġproduced Ġby Ġdifferent Ġraces Ġor Ġgenders , Ġas Ġwas Ġdone Ġby Ġh ovy 2015 dem ographic , bl od get t 2016 dem ographic , j urg ens 2017 inc orpor ating , bu ol am w ini 2018 gender . ĠAppro aches Ġon Ġhow Ġto Ġmitigate Ġinappropriate Ġbiases ĠB IB REF 14 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġare Ġalso Ġbeyond Ġthe Ġscope Ġof Ġthis Ġpaper . ĠSee Ġalso Ġthe Ġposition Ġpaper Ġby Ġh ovy 2016 social , Ġwhich Ġidentifies Ġsocio - ethical Ġimplications Ġof Ġthe ĠN LP Ġsystems Ġin Ġgeneral . </s> The ĠEquity ĠEvaluation ĠCorpus </s> We Ġnow Ġdescribe Ġhow Ġwe Ġcompiled Ġa Ġdataset Ġof Ġthousands Ġof Ġsentences Ġto Ġdetermine Ġwhether Ġautomatic Ġsystems Ġconsistently Ġgive Ġhigher Ġ( or Ġlower ) Ġsentiment Ġintensity Ġscores Ġto Ġsentences Ġinvolving Ġa Ġparticular Ġrace Ġor Ġgender . ĠThere Ġare Ġseveral Ġways Ġin Ġwhich Ġsuch Ġa Ġdataset Ġmay Ġbe Ġcompiled . ĠWe Ġpresent Ġbelow Ġthe Ġchoices Ġthat Ġwe Ġmade . </s> We Ġdecided Ġto Ġuse Ġsentences Ġinvolving Ġat Ġleast Ġone Ġrace - Ġor Ġgender - associated Ġword . ĠThe Ġsentences Ġwere Ġintended Ġto Ġbe Ġshort Ġand Ġgram matically Ġsimple . ĠWe Ġalso Ġwanted Ġsome Ġsentences Ġto Ġinclude Ġexpressions Ġof Ġsentiment Ġand Ġemotion , Ġsince Ġthe Ġgoal Ġis Ġto Ġtest Ġsentiment Ġand Ġemotion Ġsystems . ĠWe , Ġthe Ġauthors Ġof Ġthis Ġpaper , Ġdeveloped Ġeleven Ġsentence Ġtemplates Ġafter Ġseveral Ġrounds Ġof Ġdiscussion Ġand Ġconsensus Ġbuilding . ĠThey Ġare Ġshown Ġin ĠTable ĠT AB REF 3 Ġ. ĠThe Ġtemplates Ġare Ġdivided Ġinto Ġtwo Ġgroups . ĠThe Ġfirst Ġtype Ġ( tem plates Ġ1 âĢĵ 7 ) Ġincludes Ġemotion Ġwords . ĠThe Ġpurpose Ġof Ġthis Ġset Ġis Ġto Ġhave Ġsentences Ġexpressing Ġemotions . ĠThe Ġsecond Ġtype Ġ( tem plates Ġ8 âĢĵ 11 ) Ġdoes Ġnot Ġinclude Ġany Ġemotion Ġwords . ĠThe Ġpurpose Ġof Ġthis Ġset Ġis Ġto Ġhave Ġnon - em otional Ġ( neutral ) Ġsentences . </s> The Ġtemplates Ġinclude Ġtwo Ġvariables : ĠIN LINE FORM 0 Ġperson ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġemotion Ġword ĠIN LINE FORM 3 Ġ. ĠWe Ġgenerate Ġsentences Ġfrom Ġthe Ġtemplate Ġby Ġinstant iating Ġeach Ġvariable Ġwith Ġone Ġof Ġthe Ġpre - ch osen Ġvalues Ġthat Ġthe Ġvariable Ġcan Ġtake . ĠEach Ġof Ġthe Ġeleven Ġtemplates Ġincludes Ġthe Ġvariable ĠIN LINE FORM 4 Ġperson ĠIN LINE FORM 5 Ġ. ĠIN LINE FORM 6 Ġperson ĠIN LINE FORM 7 Ġcan Ġbe Ġinstant iated Ġby Ġany Ġof Ġthe Ġfollowing Ġnoun Ġphrases : </s> For Ġour Ġstudy , Ġwe Ġchose Ġten Ġnames Ġof Ġeach Ġkind Ġfrom Ġthe Ġstudy Ġby ĠCal isk an : 2017 Ġ( see ĠTable ĠT AB REF 7 Ġ). ĠThe Ġfull Ġlists Ġof Ġnoun Ġphrases Ġrepresenting Ġfemales Ġand Ġmales , Ġused Ġin Ġour Ġstudy , Ġare Ġshown Ġin ĠTable ĠT AB REF 8 Ġ. </s> The Ġsecond Ġvariable , ĠIN LINE FORM 0 Ġemotion Ġword ĠIN LINE FORM 1 Ġ, Ġhas Ġtwo Ġvariants . ĠTem plates Ġone Ġthrough Ġfour Ġinclude Ġa Ġvariable Ġfor Ġan Ġemotional Ġstate Ġword . ĠThe Ġemotional Ġstate Ġwords Ġcorrespond Ġto Ġfour Ġbasic Ġemotions : Ġanger , Ġfear , Ġjoy , Ġand Ġsadness . ĠSpecifically , Ġfor Ġeach Ġof Ġthe Ġemotions , Ġwe Ġselected Ġfive Ġwords Ġthat Ġconvey Ġthat Ġemotion Ġin Ġvarying Ġintens ities . ĠThese Ġwords Ġwere Ġtaken Ġfrom Ġthe Ġcategories Ġin Ġthe ĠRog et 's ĠThe s aurus Ġcorresponding Ġto Ġthe Ġfour Ġemotions : Ġcategory Ġ# 900 ĠRes ent ment Ġ( for Ġanger ), Ġcategory Ġ# 860 ĠFear Ġ( for Ġfear ), Ġcategory Ġ# 8 36 ĠChe er fulness Ġ( for Ġjoy ), Ġand Ġcategory Ġ# 8 37 ĠDe jection Ġ( for Ġsadness ). ĠTem plates Ġfive Ġthrough Ġseven Ġinclude Ġemotion Ġwords Ġdescribing Ġa Ġsituation Ġor Ġevent . ĠThese Ġwords Ġwere Ġalso Ġtaken Ġfrom Ġthe Ġsame Ġthe s aurus Ġcategories Ġlisted Ġabove . ĠThe Ġfull Ġlists Ġof Ġemotion Ġwords Ġ( em otional Ġstate Ġwords Ġand Ġemotional Ġsituation / event Ġwords ) Ġare Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. </s> We Ġgenerated Ġsentences Ġfrom Ġthe Ġtemplates Ġby Ġreplacing ĠIN LINE FORM 0 Ġperson ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġemotion Ġword ĠIN LINE FORM 3 Ġvariables Ġwith Ġthe Ġvalues Ġthey Ġcan Ġtake . ĠIn Ġtotal , Ġ8 , 640 Ġsentences Ġwere Ġgenerated Ġwith Ġthe Ġvarious Ġcombinations Ġof ĠIN LINE FORM 4 Ġperson ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġemotion Ġword ĠIN LINE FORM 7 Ġvalues Ġacross Ġthe Ġeleven Ġtemplates . ĠWe Ġmanually Ġexamined Ġthe Ġsentences Ġto Ġmake Ġsure Ġthey Ġwere Ġgram matically Ġwell - formed . ĠNotably , Ġone Ġcan Ġderive Ġpairs Ġof Ġsentences Ġfrom Ġthe ĠE EC Ġsuch Ġthat Ġthey Ġdiffer Ġonly Ġin Ġone Ġword Ġcorresponding Ġto Ġgender Ġor Ġrace Ġ( e . g ., Ġ` My Ġdaughter Ġfeels Ġdevastated ' Ġand Ġ` My Ġson Ġfeels Ġdevastated '). ĠWe Ġrefer Ġto Ġthe Ġfull Ġset Ġof Ġ8 , 640 Ġsentences Ġas ĠEquity ĠEvaluation ĠCorpus . </s> Me asuring ĠRace Ġand ĠGender ĠB ias Ġin ĠAutomatic ĠSent iment ĠAnalysis ĠSystems </s> The Ġrace Ġand Ġgender Ġbias Ġevaluation Ġwas Ġcarried Ġout Ġon Ġthe Ġoutput Ġof Ġthe Ġ219 Ġautomatic Ġsystems Ġthat Ġparticipated Ġin ĠSem E val - 2018 ĠTask Ġ1 : ĠAffect Ġin ĠTwe ets ĠB IB REF 5 Ġ. ĠThe Ġshared Ġtask Ġincluded Ġfive Ġsubt asks Ġon Ġinfer ring Ġthe Ġaffect ual Ġstate Ġof Ġa Ġperson Ġfrom Ġtheir Ġtweet : Ġ1 . Ġemotion Ġintensity Ġregression , Ġ2 . Ġemotion Ġintensity Ġord inal Ġclassification , Ġ3 . Ġval ence Ġ( sent iment ) Ġregression , Ġ4 . Ġval ence Ġord inal Ġclassification , Ġand Ġ5 . Ġemotion Ġclassification . ĠFor Ġeach Ġsubt ask , Ġlabeled Ġdata Ġwere Ġprovided Ġfor ĠEnglish , ĠArabic , Ġand ĠSpanish . ĠThe Ġrace Ġand Ġgender Ġbias Ġwere Ġanalyzed Ġfor Ġthe Ġsystem Ġoutputs Ġon Ġtwo ĠEnglish Ġsubt asks : Ġemotion Ġintensity Ġregression Ġ( for Ġanger , Ġfear , Ġjoy , Ġand Ġsadness ) Ġand Ġval ence Ġregression . ĠThese Ġregression Ġtasks Ġwere Ġformulated Ġas Ġfollows : ĠGiven Ġa Ġtweet Ġand Ġan Ġaffect ive Ġdimension ĠA Ġ( anger , Ġfear , Ġjoy , Ġsadness , Ġor Ġval ence ), Ġdetermine Ġthe Ġintensity Ġof ĠA Ġthat Ġbest Ġrepresents Ġthe Ġmental Ġstate Ġof Ġthe Ġtwe eter âĢĶ a Ġreal - valued Ġscore Ġbetween Ġ0 Ġ( le ast ĠA ) Ġand Ġ1 Ġ( most ĠA ). ĠSep arate Ġtraining Ġand Ġtest Ġdatasets Ġwere Ġprovided Ġfor Ġeach Ġaffect ive Ġdimension . </s> Training Ġsets Ġincluded Ġtweets Ġalong Ġwith Ġgold Ġintensity Ġscores . ĠTwo Ġtest Ġsets Ġwere Ġprovided Ġfor Ġeach Ġtask : Ġ1 . Ġa Ġregular Ġtweet Ġtest Ġset Ġ( for Ġwhich Ġthe Ġgold Ġintensity Ġscores Ġare Ġknown Ġbut Ġnot Ġrevealed Ġto Ġthe Ġparticipating Ġsystems ), Ġand Ġ2 . Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( for Ġwhich Ġno Ġgold Ġintensity Ġlabels Ġexist ). ĠParticipants Ġwere Ġtold Ġthat Ġapart Ġfrom Ġthe Ġusual Ġtest Ġset , Ġthey Ġare Ġto Ġrun Ġtheir Ġsystems Ġon Ġa Ġseparate Ġtest Ġset Ġof Ġunknown Ġorigin . ĠThe Ġparticipants Ġwere Ġinstructed Ġto Ġtrain Ġtheir Ġsystem Ġon Ġthe Ġtweets Ġtraining Ġsets Ġprovided , Ġand Ġthat Ġthey Ġcould Ġuse Ġany Ġother Ġresources Ġthey Ġmay Ġfind Ġor Ġcreate . ĠThey Ġwere Ġto Ġrun Ġthe Ġsame Ġfinal Ġsystem Ġon Ġthe Ġtwo Ġtest Ġsets . ĠThe Ġnature Ġof Ġthe Ġsecond Ġtest Ġset Ġwas Ġrevealed Ġto Ġthem Ġonly Ġafter Ġthe Ġcompetition . ĠThe Ġfirst Ġ( t we ets ) Ġtest Ġset Ġwas Ġused Ġto Ġevaluate Ġand Ġrank Ġthe Ġquality Ġ( acc uracy ) Ġof Ġthe Ġsystems ' Ġpredictions . ĠThe Ġsecond Ġ( E EC ) Ġtest Ġset Ġwas Ġused Ġto Ġperform Ġthe Ġbias Ġanalysis , Ġwhich Ġis Ġthe Ġfocus Ġof Ġthis Ġpaper . </s> System s : ĠFifty Ġteams Ġsubmitted Ġtheir Ġsystem Ġoutputs Ġto Ġone Ġor Ġmore Ġof Ġthe Ġfive Ġemotion Ġintensity Ġregression Ġtasks Ġ( for Ġanger , Ġfear , Ġjoy , Ġsadness , Ġand Ġval ence ), Ġresulting Ġin Ġ219 Ġsubmissions Ġin Ġtotal . ĠMany Ġsystems Ġwere Ġbuilt Ġusing Ġtwo Ġtypes Ġof Ġfeatures : Ġdeep Ġneural Ġnetwork Ġrepresentations Ġof Ġtweets Ġ( sent ence Ġembed d ings ) Ġand Ġfeatures Ġderived Ġfrom Ġexisting Ġsentiment Ġand Ġemotion Ġlex icons . ĠThese Ġfeatures Ġwere Ġthen Ġcombined Ġto Ġlearn Ġa Ġmodel Ġusing Ġeither Ġtraditional Ġmachine Ġlearning Ġalgorithms Ġ( such Ġas ĠS VM / S VR Ġand ĠLog istic ĠReg ression ) Ġor Ġdeep Ġneural Ġnetworks . ĠS VM / S VR , ĠL ST Ms , Ġand ĠBi - L ST Ms Ġwere Ġsome Ġof Ġthe Ġmost Ġwidely Ġused Ġmachine Ġlearning Ġalgorithms . ĠThe Ġsentence Ġembed d ings Ġwere Ġobtained Ġby Ġtraining Ġa Ġneural Ġnetwork Ġon Ġthe Ġprovided Ġtraining Ġdata , Ġa Ġdistant Ġsupervision Ġcorpus Ġ( e . g ., ĠA IT 2018 ĠDist ant ĠSuper vision ĠCorpus Ġthat Ġhas Ġtweets Ġwith Ġemotion - related Ġquery Ġterms ), Ġsentiment - label ed Ġtweet Ġcorpor a Ġ( e . g ., ĠSem eval - 2017 ĠTask 4 A Ġdataset Ġon Ġsentiment Ġanalysis Ġin ĠTwitter ), Ġor Ġby Ġusing Ġpre - trained Ġmodels Ġ( e . g ., ĠDeep Mo ji ĠB IB REF 20 Ġ, ĠSkip Ġthoughts ĠB IB REF 21 Ġ). ĠThe Ġlex icon Ġfeatures Ġwere Ġoften Ġderived Ġfrom Ġthe ĠN RC Ġemotion Ġand Ġsentiment Ġlex icons ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠAF IN N ĠB IB REF 25 Ġ, Ġand ĠBing ĠLiu ĠLex icon ĠB IB REF 26 Ġ. </s> We Ġprovided Ġa Ġbaseline ĠS VM Ġsystem Ġtrained Ġusing Ġword Ġun ig rams Ġas Ġfeatures Ġon Ġthe Ġtraining Ġdata Ġ( S VM - Un ig rams ). ĠThis Ġsystem Ġis Ġalso Ġincluded Ġin Ġthe Ġcurrent Ġanalysis . </s> Me asuring Ġbias : ĠTo Ġexamine Ġgender Ġbias , Ġwe Ġcompared Ġeach Ġsystem 's Ġpredicted Ġscores Ġon Ġthe ĠE EC Ġsentence Ġpairs Ġas Ġfollows : </s> Thus , Ġeleven Ġpairs Ġof Ġscores Ġ( ten Ġpairs Ġof Ġscores Ġfrom Ġten Ġnoun Ġphrase Ġpairs Ġand Ġone Ġpair Ġof Ġscores Ġfrom Ġthe Ġaverages Ġon Ġname Ġsubs ets ) Ġwere Ġexamined Ġfor Ġeach Ġtemplate âĢĵ em otion Ġword Ġinstant iation . ĠThere Ġwere Ġtwenty Ġdifferent Ġemotion Ġwords Ġused Ġin Ġseven Ġtemplates Ġ( tem plates Ġ1 âĢĵ 7 ), Ġand Ġno Ġemotion Ġwords Ġused Ġin Ġthe Ġfour Ġremaining Ġtemplates Ġ( tem plates Ġ8 âĢĵ 11 ). ĠIn Ġtotal , ĠIN LINE FORM 0 Ġpairs Ġof Ġscores Ġwere Ġcompared . </s> Similarly , Ġto Ġexamine Ġrace Ġbias , Ġwe Ġcompared Ġpairs Ġof Ġsystem Ġpredicted Ġscores Ġas Ġfollows : </s> Thus , Ġone Ġpair Ġof Ġscores Ġwas Ġexamined Ġfor Ġeach Ġtemplate âĢĵ em otion Ġword Ġinstant iation . ĠIn Ġtotal , ĠIN LINE FORM 0 Ġpairs Ġof Ġscores Ġwere Ġcompared . </s> For Ġeach Ġsystem , Ġwe Ġcalculated Ġthe Ġpaired Ġtwo Ġsample Ġt - test Ġto Ġdetermine Ġwhether Ġthe Ġmean Ġdifference Ġbetween Ġthe Ġtwo Ġsets Ġof Ġscores Ġ( ac ross Ġthe Ġtwo Ġraces Ġand Ġacross Ġthe Ġtwo Ġgenders ) Ġis Ġsignificant . ĠWe Ġset Ġthe Ġsignificance Ġlevel Ġto Ġ0 . 05 . ĠHowever , Ġsince Ġwe Ġperformed Ġ4 38 Ġassessments Ġ( 219 Ġsubmissions Ġevaluated Ġfor Ġbiases Ġin Ġboth Ġgender Ġand Ġrace ), Ġwe Ġapplied ĠBon fer ron i Ġcorrection . ĠThe Ġnull Ġhypothesis Ġthat Ġthe Ġtrue Ġmean Ġdifference Ġbetween Ġthe Ġpaired Ġsamples Ġwas Ġzero Ġwas Ġrejected Ġif Ġthe Ġcalculated Ġp - value Ġfell Ġbelow ĠIN LINE FORM 0 Ġ. </s> Results </s> The Ġtwo Ġsub - sections Ġbelow Ġpresent Ġthe Ġresults Ġfrom Ġthe Ġanalysis Ġfor Ġgender Ġbias Ġand Ġrace Ġbias , Ġrespectively . </s> Gender ĠB ias ĠResults </s> Individual Ġsubmission Ġresults Ġwere Ġcommunicated Ġto Ġthe Ġparticipants . ĠHere , Ġwe Ġpresent Ġthe Ġsummary Ġresults Ġacross Ġall Ġthe Ġteams . ĠThe Ġgoal Ġof Ġthis Ġanalysis Ġis Ġto Ġgain Ġa Ġbetter Ġunderstanding Ġof Ġbiases Ġacross Ġa Ġlarge Ġnumber Ġof Ġcurrent Ġsentiment Ġanalysis Ġsystems . ĠThus , Ġwe Ġpartition Ġthe Ġsubmissions Ġinto Ġthree Ġgroups Ġaccording Ġto Ġthe Ġbias Ġthey Ġshow : </s> F = M Ġnot Ġsignificant : Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġdifference Ġin Ġintensity Ġscores Ġpredicted Ġfor Ġcorresponding Ġfemale Ġand Ġmale Ġnoun Ġphrase Ġsentences , </s> F ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġsignificant : Ġsubmissions Ġthat Ġconsistently Ġgave Ġhigher Ġscores Ġfor Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġthan Ġfor Ġcorresponding Ġsentences Ġwith Ġmale Ġnoun Ġphrases , </s> F ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġsignificant : Ġsubmissions Ġthat Ġconsistently Ġgave Ġlower Ġscores Ġfor Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġthan Ġfor Ġcorresponding Ġsentences Ġwith Ġmale Ġnoun Ġphrases . </s> For Ġeach Ġsystem Ġand Ġeach Ġsentence Ġpair , Ġwe Ġcalculate Ġthe Ġscore Ġdifference ĠIN LINE FORM 0 Ġas Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġminus Ġthe Ġscore Ġfor Ġthe Ġcorresponding Ġmale Ġnoun Ġphrase Ġsentence . ĠTable ĠT AB REF 24 Ġpresents Ġthe Ġsummary Ġresults Ġfor Ġeach Ġof Ġthe Ġbias Ġgroups . ĠIt Ġhas Ġthe Ġfollowing Ġcolumns : </s> # Sub m .: Ġnumber Ġof Ġsubmissions Ġin Ġeach Ġgroup . </s> If Ġall Ġthe Ġsystems Ġare Ġunbiased , Ġthen Ġthe Ġnumber Ġof Ġsubmissions Ġfor Ġthe Ġgroup ĠF = M Ġnot Ġsignificant Ġwould Ġbe Ġthe Ġmaximum , Ġand Ġthe Ġnumber Ġof Ġsubmissions Ġin Ġall Ġother Ġgroups Ġwould Ġbe Ġzero . </s> Avg . Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġ: Ġthe Ġaverage ĠIN LINE FORM 2 Ġfor Ġonly Ġthose Ġpairs Ġwhere Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġis Ġhigher . ĠThe Ġgreater Ġthe Ġmagnitude Ġof Ġthis Ġscore , Ġthe Ġstronger Ġthe Ġbias Ġin Ġsystems Ġthat Ġconsistently Ġgive Ġhigher Ġscores Ġto Ġfemale - associated Ġsentences . </s> Avg . Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġ: Ġthe Ġaverage ĠIN LINE FORM 2 Ġfor Ġonly Ġthose Ġpairs Ġwhere Ġthe Ġscore Ġfor Ġthe Ġfemale Ġnoun Ġphrase Ġsentence Ġis Ġlower . ĠThe Ġgreater Ġthe Ġmagnitude Ġof Ġthis Ġscore , Ġthe Ġstronger Ġthe Ġbias Ġin Ġsystems Ġthat Ġconsistently Ġgive Ġlower Ġscores Ġto Ġfemale - associated Ġsentences . </s> Note Ġthat Ġthese Ġnumbers Ġwere Ġfirst Ġcalculated Ġseparately Ġfor Ġeach Ġsubmission , Ġand Ġthen Ġaveraged Ġover Ġall Ġthe Ġsubmissions Ġwithin Ġeach Ġsubmission Ġgroup . ĠThe Ġresults Ġare Ġreported Ġseparately Ġfor Ġsubmissions Ġto Ġeach Ġtask Ġ( anger , Ġfear , Ġjoy , Ġsadness , Ġand Ġsentiment / val ence Ġintensity Ġprediction ). </s> Obs erve Ġthat Ġon Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks , Ġonly Ġabout Ġ12 Ġof Ġthe Ġ46 Ġsubmissions Ġ( about Ġ25 % Ġof Ġthe Ġsubmissions ) Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠOn Ġthe Ġval ence Ġprediction Ġtask , Ġonly Ġ5 Ġof Ġthe Ġ36 Ġsubmissions Ġ( 14 % Ġof Ġthe Ġsubmissions ) Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠThus Ġ75 % Ġto Ġ86 % Ġof Ġthe Ġsubmissions Ġconsistently Ġmarked Ġsentences Ġof Ġone Ġgender Ġhigher Ġthan Ġanother . </s> When Ġpredicting Ġanger , Ġjoy , Ġor Ġval ence , Ġthe Ġnumber Ġof Ġsystems Ġconsistently Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 21 âĢĵ 25 ) Ġis Ġmarkedly Ġhigher Ġthan Ġthe Ġnumber Ġof Ġsystems Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 8 âĢĵ 13 ). Ġ( Rec all Ġthat Ġhigher Ġval ence Ġmeans Ġmore Ġpositive Ġsentiment .) ĠIn Ġcontrast , Ġon Ġthe Ġfear Ġtask , Ġmost Ġsubmissions Ġtended Ġto Ġassign Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 23 ) Ġas Ġcompared Ġto Ġthe Ġnumber Ġof Ġsystems Ġgiving Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 12 ). ĠWhen Ġpredicting Ġsadness , Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġmostly Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġ( 18 ) Ġis Ġclose Ġto Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġmostly Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġ( 16 ). ĠThese Ġresults Ġare Ġin Ġline Ġwith Ġsome Ġcommon Ġstereotypes , Ġsuch Ġas Ġfemales Ġare Ġmore Ġemotional , Ġand Ġsituations Ġinvolving Ġmale Ġagents Ġare Ġmore Ġfearful ĠB IB REF 27 Ġ. </s> Figure ĠFIG REF 25 Ġshows Ġthe Ġscore Ġdifferences Ġ( ĠIN LINE FORM 0 Ġ) Ġfor Ġindividual Ġsystems Ġon Ġthe Ġval ence Ġregression Ġtask . ĠPl ots Ġfor Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks Ġare Ġshown Ġin ĠFigure ĠFIG REF 31 Ġin Ġthe ĠAppendix . ĠEach Ġpoint Ġ( Ġâĸ ² , Ġâĸ ¼ , ĠâĹı ) Ġon Ġthe Ġplot Ġcorresponds Ġto Ġthe Ġdifference Ġin Ġscores Ġpredicted Ġby Ġthe Ġsystem Ġon Ġone Ġsentence Ġpair . ĠThe Ġsystems Ġare Ġordered Ġby Ġtheir Ġrank Ġ( from Ġfirst Ġto Ġlast ) Ġon Ġthe Ġtask Ġon Ġthe Ġtweets Ġtest Ġsets , Ġas Ġper Ġthe Ġofficial Ġevaluation Ġmetric Ġ( Spe ar man Ġcorrelation Ġwith Ġthe Ġgold Ġintensity Ġscores ). ĠWe Ġwill Ġrefer Ġto Ġthe Ġdifference Ġbetween Ġthe Ġmaximal Ġvalue Ġof ĠIN LINE FORM 1 Ġand Ġthe Ġminimal Ġvalue Ġof ĠIN LINE FORM 2 Ġfor Ġa Ġparticular Ġsystem Ġas Ġthe ĠIN LINE FORM 3 ĠâĢĵ spread . ĠObs erve Ġthat Ġthe ĠIN LINE FORM 4 ĠâĢĵ sp reads Ġfor Ġmany Ġsystems Ġare Ġrather Ġlarge , Ġup Ġto Ġ0 . 57 . ĠDepending Ġon Ġthe Ġtask , Ġthe Ġtop Ġ10 Ġor Ġtop Ġ15 Ġsystems Ġas Ġwell Ġas Ġsome Ġof Ġthe Ġworst Ġperforming Ġsystems Ġtend Ġto Ġhave Ġsmaller ĠIN LINE FORM 5 ĠâĢĵ sp reads Ġwhile Ġthe Ġsystems Ġwith Ġmedium Ġto Ġlow Ġperformance Ġshow Ġgreater Ġsensitivity Ġto Ġthe Ġgender - associated Ġwords . ĠAlso , Ġmost Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifferences Ġ( shown Ġin Ġgreen ) Ġperformed Ġpoorly Ġon Ġthe Ġtweets Ġtest Ġsets . ĠOnly Ġthree Ġsystems Ġout Ġof Ġthe Ġtop Ġfive Ġon Ġthe Ġanger Ġintensity Ġtask Ġand Ġone Ġsystem Ġon Ġthe Ġjoy Ġand Ġsadness Ġtasks Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference . ĠThis Ġindicates Ġthat Ġwhen Ġconsidering Ġonly Ġthose Ġsystems Ġthat Ġperformed Ġwell Ġon Ġthe Ġintensity Ġprediction Ġtask , Ġthe Ġpercentage Ġof Ġgender - biased Ġsystems Ġare Ġeven Ġhigher Ġthan Ġthose Ġindicated Ġabove . </s> These Ġresults Ġraise Ġfurther Ġquestions Ġsuch Ġas Ġ` what Ġexactly Ġis Ġthe Ġcause Ġof Ġsuch Ġbiases ?' Ġand Ġ` why Ġis Ġthe Ġbias Ġimpacted Ġby Ġthe Ġemotion Ġtask Ġunder Ġconsideration ? '. ĠAn sw ering Ġthese Ġquestions Ġwill Ġrequire Ġfurther Ġinformation Ġon Ġthe Ġresources Ġthat Ġthe Ġteams Ġused Ġto Ġdevelop Ġtheir Ġmodels , Ġand Ġwe Ġleave Ġthat Ġfor Ġfuture Ġwork . </s> Average Ġscore Ġdifferences : ĠFor Ġsubmissions Ġthat Ġshowed Ġstatistically Ġsignificant Ġscore Ġdifferences , Ġthe Ġaverage Ġscore Ġdifference ĠF ĠIN LINE FORM 0 ĠâĢĵ M ĠIN LINE FORM 1 Ġand Ġthe Ġaverage Ġscore Ġdifference ĠF ĠIN LINE FORM 2 ĠâĢĵ M ĠIN LINE FORM 3 Ġwere ĠIN LINE FORM 4 Ġ. ĠSince Ġthe Ġintensity Ġscores Ġrange Ġfrom Ġ0 Ġto Ġ1 , Ġ0 . 03 Ġis Ġ3 % Ġof Ġthe Ġfull Ġrange . ĠThe Ġmaximal Ġscore Ġdifference Ġ( ĠIN LINE FORM 5 Ġ) Ġacross Ġall Ġthe Ġsubmissions Ġwas Ġas Ġhigh Ġas Ġ0 . 34 . ĠNote , Ġhowever , Ġthat Ġthese ĠIN LINE FORM 6 Ġs Ġare Ġthe Ġresult Ġof Ġchanging Ġjust Ġone Ġword Ġin Ġa Ġsentence . ĠIn Ġmore Ġcomplex Ġsentences , Ġseveral Ġgender - associated Ġwords Ġcan Ġappear , Ġwhich Ġmay Ġhave Ġa Ġbigger Ġimpact . ĠAlso , Ġwhether Ġconsistent Ġscore Ġdifferences Ġof Ġthis Ġmagnitude Ġwill Ġhave Ġsignificant Ġrepercussions Ġin Ġdownstream Ġapplications , Ġdepends Ġon Ġthe Ġparticular Ġapplication . </s> An alyses Ġon Ġonly Ġthe Ġneutral Ġsentences Ġin ĠE EC Ġand Ġonly Ġthe Ġemotional Ġsentences Ġin ĠE EC : ĠWe Ġalso Ġperformed Ġa Ġseparate Ġanalysis Ġusing Ġonly Ġthose Ġsentences Ġfrom Ġthe ĠE EC Ġthat Ġincluded Ġno Ġemotion Ġwords . ĠRecall Ġthat Ġthere Ġare Ġfour Ġtemplates Ġthat Ġcontain Ġno Ġemotion Ġwords . ĠTables ĠT AB REF 26 Ġshows Ġthese Ġresults . ĠWe Ġobserve Ġsimilar Ġtrends Ġas Ġin Ġthe Ġanalysis Ġon Ġthe Ġfull Ġset . ĠOne Ġnoticeable Ġdifference Ġis Ġthat Ġthe Ġnumber Ġof Ġsubmissions Ġthat Ġshowed Ġstatistically Ġsignificant Ġscore Ġdifference Ġis Ġmuch Ġsmaller Ġfor Ġthis Ġdata Ġsubset . ĠHowever , Ġthe Ġtotal Ġnumber Ġof Ġcomparisons Ġon Ġthe Ġsubset Ġ( 44 ) Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġtotal Ġnumber Ġof Ġcomparisons Ġon Ġthe Ġfull Ġset Ġ( 1 , 584 ), Ġwhich Ġmakes Ġthe Ġstatistical Ġtest Ġless Ġpowerful . ĠNote Ġalso Ġthat Ġthe Ġaverage Ġscore Ġdifferences Ġon Ġthe Ġsubset Ġ( column s Ġ3 Ġand Ġ4 Ġin ĠTable ĠT AB REF 26 Ġ) Ġtend Ġto Ġbe Ġhigher Ġthan Ġthe Ġdifferences Ġon Ġthe Ġfull Ġset Ġ( column s Ġ3 Ġand Ġ4 Ġin ĠTable ĠT AB REF 24 Ġ). ĠThis Ġindicates Ġthat Ġgender - associated Ġwords Ġcan Ġhave Ġa Ġbigger Ġimpact Ġon Ġsystem Ġpredictions Ġfor Ġneutral Ġsentences . </s> We Ġalso Ġperformed Ġan Ġanalysis Ġby Ġrestricting Ġthe Ġdataset Ġto Ġcontain Ġonly Ġthe Ġsentences Ġwith Ġthe Ġemotion Ġwords Ġcorresponding Ġto Ġthe Ġemotion Ġtask Ġ( i . e ., Ġsubmissions Ġto Ġthe Ġanger Ġintensity Ġprediction Ġtask Ġwere Ġevaluated Ġonly Ġon Ġsentences Ġwith Ġanger Ġwords ). ĠThe Ġresults Ġ( not Ġshown Ġhere ) Ġwere Ġsimilar Ġto Ġthe Ġresults Ġon Ġthe Ġfull Ġset . </s> Race ĠB ias ĠResults </s> We Ġdid Ġa Ġsimilar Ġanalysis Ġfor Ġrace Ġas Ġwe Ġdid Ġfor Ġgender . ĠFor Ġeach Ġsubmission Ġon Ġeach Ġtask , Ġwe Ġcalculated Ġthe Ġdifference Ġbetween Ġthe Ġaverage Ġpredicted Ġscore Ġon Ġthe Ġset Ġof Ġsentences Ġwith ĠAfrican ĠAmerican Ġ( AA ) Ġnames Ġand Ġthe Ġaverage Ġpredicted Ġscore Ġon Ġthe Ġset Ġof Ġsentences Ġwith ĠEuropean ĠAmerican Ġ( EA ) Ġnames . ĠThen , Ġwe Ġaggreg ated Ġthe Ġresults Ġover Ġall Ġsuch Ġsentence Ġpairs Ġin Ġthe ĠE EC . </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults . ĠThe Ġtable Ġhas Ġthe Ġsame Ġform Ġand Ġstructure Ġas Ġthe Ġgender Ġresult Ġtables . ĠObs erve Ġthat Ġthe Ġnumber Ġof Ġsubmissions Ġwith Ġno Ġstatistically Ġsignificant Ġscore Ġdifference Ġfor Ġsentences Ġpertaining Ġto Ġthe Ġtwo Ġraces Ġis Ġabout Ġ5 âĢĵ 11 Ġ( about Ġ11 % Ġto Ġ24 %) Ġfor Ġthe Ġfour Ġemotions Ġand Ġ3 Ġ( about Ġ8 %) Ġfor Ġval ence . ĠThese Ġnumbers Ġare Ġeven Ġlower Ġthan Ġwhat Ġwas Ġfound Ġfor Ġgender . </s> The Ġmajority Ġof Ġthe Ġsystems Ġassigned Ġhigher Ġscores Ġto Ġsentences Ġwith ĠAfrican ĠAmerican Ġnames Ġon Ġthe Ġtasks Ġof Ġanger , Ġfear , Ġand Ġsadness Ġintensity Ġprediction . ĠOn Ġthe Ġjoy Ġand Ġval ence Ġtasks , Ġmost Ġsubmissions Ġtended Ġto Ġassign Ġhigher Ġscores Ġto Ġsentences Ġwith ĠEuropean ĠAmerican Ġnames . ĠThese Ġtendencies Ġreflect Ġsome Ġcommon Ġstereotypes Ġthat Ġassociate ĠAfrican ĠAmericans Ġwith Ġmore Ġnegative Ġemotions ĠB IB REF 28 Ġ. </s> Figure ĠFIG REF 28 Ġshows Ġthe Ġscore Ġdifferences Ġfor Ġindividual Ġsystems Ġon Ġrace Ġsentence Ġpairs Ġon Ġthe Ġval ence Ġregression Ġtask . ĠPl ots Ġfor Ġthe Ġfour Ġemotion Ġintensity Ġprediction Ġtasks Ġare Ġshown Ġin ĠFigure ĠFIG REF 32 Ġin Ġthe ĠAppendix . ĠHere , Ġthe ĠIN LINE FORM 0 ĠâĢĵ sp reads Ġare Ġsmaller Ġthan Ġon Ġthe Ġgender Ġsentence Ġpairs âĢĶ from Ġ0 Ġto Ġ0 . 15 . ĠAs Ġin Ġthe Ġgender Ġanalysis , Ġon Ġthe Ġval ence Ġtask Ġthe Ġtop Ġ13 Ġsystems Ġas Ġwell Ġas Ġsome Ġof Ġthe Ġworst Ġperforming Ġsystems Ġhave Ġsmaller ĠIN LINE FORM 1 ĠâĢĵ spread Ġwhile Ġthe Ġsystems Ġwith Ġmedium Ġto Ġlow Ġperformance Ġshow Ġgreater Ġsensitivity Ġto Ġthe Ġrace - associated Ġnames . ĠHowever , Ġwe Ġdo Ġnot Ġobserve Ġthe Ġsame Ġpattern Ġin Ġthe Ġemotion Ġintensity Ġtasks . ĠAlso , Ġsimilar Ġto Ġthe Ġgender Ġanalysis , Ġmost Ġsubmissions Ġthat Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifferences Ġobtained Ġlower Ġscores Ġon Ġthe Ġtweets Ġtest Ġsets . ĠOnly Ġone Ġsystem Ġout Ġof Ġthe Ġtop Ġfive Ġshowed Ġno Ġstatistically Ġsignificant Ġscore Ġdifference Ġon Ġthe Ġanger Ġand Ġfear Ġintensity Ġtasks , Ġand Ġnone Ġon Ġthe Ġother Ġtasks . ĠOnce Ġagain , Ġjust Ġas Ġin Ġthe Ġcase Ġof Ġgender , Ġthis Ġraises Ġquestions Ġof Ġthe Ġexact Ġcauses Ġof Ġsuch Ġbiases . ĠWe Ġhope Ġto Ġexplore Ġthis Ġin Ġfuture Ġwork . </s> Discussion </s> As Ġmentioned Ġin Ġthe Ġintroduction , Ġbias Ġcan Ġoriginate Ġfrom Ġany Ġor Ġseveral Ġparts Ġof Ġa Ġsystem : Ġthe Ġlabeled Ġand Ġunl abel ed Ġdatasets Ġused Ġto Ġlearn Ġdifferent Ġparts Ġof Ġthe Ġmodel , Ġthe Ġlanguage Ġresources Ġused Ġ( e . g ., Ġpre - trained Ġword Ġembed d ings , Ġlex icons ), Ġthe Ġlearning Ġmethod Ġused Ġ( al gorithm , Ġfeatures , Ġparameters ), Ġetc . ĠIn Ġour Ġanalysis , Ġwe Ġfound Ġsystems Ġtrained Ġusing Ġa Ġvariety Ġof Ġalgorithms Ġ( traditional Ġas Ġwell Ġas Ġdeep Ġneural Ġnetworks ) Ġand Ġa Ġvariety Ġof Ġlanguage Ġresources Ġshowing Ġgender Ġand Ġrace Ġbiases . ĠFurther Ġexperiments Ġmay Ġtease Ġout Ġthe Ġextent Ġof Ġbias Ġin Ġeach Ġof Ġthese Ġparts . </s> We Ġalso Ġanalyzed Ġthe Ġoutput Ġof Ġour Ġbaseline ĠS VM Ġsystem Ġtrained Ġusing Ġword Ġun ig rams Ġ( S VM - Un ig rams ). ĠThe Ġsystem Ġdoes Ġnot Ġuse Ġany Ġlanguage Ġresources Ġother Ġthan Ġthe Ġtraining Ġdata . ĠWe Ġobserve Ġthat Ġthis Ġbaseline Ġsystem Ġalso Ġshows Ġsmall Ġbias Ġin Ġgender Ġand Ġrace . ĠThe ĠIN LINE FORM 0 Ġ- sp reads Ġfor Ġthis Ġsystem Ġwere Ġquite Ġsmall : Ġ0 . 09 Ġto Ġ0 . 2 Ġon Ġthe Ġgender Ġsentence Ġpairs Ġand Ġless Ġthan Ġ0 . 002 Ġon Ġthe Ġrace Ġsentence Ġpairs . ĠThe Ġpredicted Ġintensity Ġscores Ġtended Ġto Ġbe Ġhigher Ġon Ġthe Ġsentences Ġwith Ġmale Ġnoun Ġphrases Ġthan Ġon Ġthe Ġsentences Ġwith Ġfemale Ġnoun Ġphrases Ġfor Ġthe Ġtasks Ġof Ġanger , Ġfear , Ġand Ġsadness Ġintensity Ġprediction . ĠThis Ġtendency Ġwas Ġreversed Ġon Ġthe Ġtask Ġof Ġval ence Ġprediction . ĠOn Ġthe Ġrace Ġsentence Ġpairs , Ġthe Ġsystem Ġpredicted Ġhigher Ġintensity Ġscores Ġon Ġthe Ġsentences Ġwith ĠEuropean ĠAmerican Ġnames Ġfor Ġall Ġfour Ġemotion Ġintensity Ġprediction Ġtasks , Ġand Ġon Ġthe Ġsentences Ġwith ĠAfrican ĠAmerican Ġnames Ġfor Ġthe Ġtask Ġof Ġval ence Ġprediction . ĠThis Ġindicates Ġthat Ġthe Ġtraining Ġdata Ġcontains Ġsome Ġbiases Ġ( in Ġthe Ġform Ġof Ġsome Ġun ig rams Ġassociated Ġwith Ġa Ġparticular Ġgender Ġor Ġrace Ġtending Ġto Ġappear Ġin Ġtweets Ġlabeled Ġwith Ġcertain Ġemotions ). ĠThe Ġlabeled Ġdatasets Ġfor Ġthe Ġshared Ġtask Ġwere Ġcreated Ġusing Ġa Ġfairly Ġstandard Ġapproach : Ġpolling ĠTwitter Ġwith Ġtask - related Ġquery Ġterms Ġ( in Ġthis Ġcase , Ġemotion Ġwords ) Ġand Ġthen Ġmanually Ġannot ating Ġthe Ġtweets Ġwith Ġtask - specific Ġlabels . ĠThe ĠS VM - Un ig ram Ġbias Ġresults Ġshow Ġthat Ġdata Ġcollected Ġby Ġdistant Ġsupervision Ġcan Ġbe Ġa Ġsource Ġof Ġbias . ĠHowever , Ġit Ġshould Ġbe Ġnoted Ġthat Ġdifferent Ġlearning Ġmethods Ġin Ġcombination Ġwith Ġdifferent Ġlanguage Ġresources Ġcan Ġaccent uate , Ġreverse , Ġor Ġmask Ġthe Ġbias Ġpresent Ġin Ġthe Ġtraining Ġdata Ġto Ġdifferent Ġdegrees . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġcreated Ġthe ĠEquity ĠEvaluation ĠCorpus Ġ( E EC ), Ġwhich Ġconsists Ġof Ġ8 , 640 Ġsentences Ġspecifically Ġchosen Ġto Ġtease Ġout Ġgender Ġand Ġrace Ġbiases Ġin Ġnatural Ġlanguage Ġprocessing Ġsystems . ĠWe Ġused Ġthe ĠE EC Ġto Ġanalyze Ġ219 ĠN LP Ġsystems Ġthat Ġparticipated Ġin Ġa Ġrecent Ġinternational Ġshared Ġtask Ġon Ġpredicting Ġsentiment Ġand Ġemotion Ġintensity . ĠWe Ġfound Ġthat Ġmore Ġthan Ġ75 % Ġof Ġthe Ġsystems Ġtend Ġto Ġmark Ġsentences Ġinvolving Ġone Ġgender / race Ġwith Ġhigher Ġintensity Ġscores Ġthan Ġthe Ġsentences Ġinvolving Ġthe Ġother Ġgender / race . ĠWe Ġfound Ġsuch Ġbiases Ġto Ġbe Ġmore Ġwidely Ġprevalent Ġfor Ġrace Ġthan Ġfor Ġgender . ĠWe Ġalso Ġfound Ġthat Ġthe Ġbias Ġcan Ġbe Ġdifferent Ġdepending Ġon Ġthe Ġparticular Ġaffect Ġdimension Ġinvolved . </s> We Ġfound Ġthe Ġscore Ġdifferences Ġacross Ġgenders Ġand Ġacross Ġraces Ġto Ġbe Ġsomewhat Ġsmall Ġon Ġaverage Ġ( ĠIN LINE FORM 0 Ġ, Ġwhich Ġis ĠIN LINE FORM 1 Ġof Ġthe Ġ0 Ġto Ġ1 Ġscore Ġrange ). ĠHowever , Ġfor Ġsome Ġsystems Ġthe Ġscore Ġdifferences Ġreached Ġas Ġhigh Ġas Ġ0 . 34 Ġ( 34 %). ĠWhat Ġimpact Ġa Ġconsistent Ġbias , Ġeven Ġwith Ġan Ġaverage Ġmagnitude ĠIN LINE FORM 2 Ġ, Ġmight Ġhave Ġin Ġdownstream Ġapplications Ġmerits Ġfurther Ġinvestigation . </s> We Ġplan Ġto Ġextend Ġthe ĠE EC Ġwith Ġsentences Ġassociated Ġwith Ġcountry Ġnames , Ġprofessions Ġ( e . g ., Ġdoctors , Ġpolice Ġofficers , Ġjan itors , Ġteachers , Ġetc .), Ġfields Ġof Ġstudy Ġ( e . g ., Ġarts Ġvs . Ġsciences ), Ġas Ġwell Ġas Ġraces Ġ( e . g ., ĠAsian , Ġmixed , Ġetc .) Ġand Ġgenders Ġ( e . g ., Ġag ender , Ġand ro gy ne , Ġtrans , Ġqueer , Ġetc .) Ġnot Ġincluded Ġin Ġthe Ġcurrent Ġstudy . ĠWe Ġcan Ġthen Ġuse Ġthe Ġcorpus Ġto Ġexamine Ġbiases Ġacross Ġeach Ġof Ġthose Ġvariables Ġas Ġwell . ĠWe Ġare Ġalso Ġinterested Ġin Ġexploring Ġwhich Ġsystems Ġ( or Ġwhat Ġtechniques ) Ġaccent uate Ġinappropriate Ġbiases Ġin Ġthe Ġdata Ġand Ġwhich Ġsystems Ġmitigate Ġsuch Ġbiases . ĠFinally , Ġwe Ġare Ġinterested Ġin Ġexploring Ġhow Ġthe Ġquality Ġof Ġsentiment Ġanalysis Ġpredictions Ġvaries Ġwhen Ġapplied Ġto Ġtext Ġproduced Ġby Ġdifferent Ġdemographic Ġgroups , Ġsuch Ġas Ġpeople Ġof Ġdifferent Ġraces , Ġgenders , Ġand Ġethnic ities . </s> The ĠEquity ĠEvaluation ĠCorpus Ġand Ġthe Ġproposed Ġmethodology Ġto Ġexamine Ġbias Ġare Ġnot Ġmeant Ġto Ġbe Ġcomprehensive . ĠHowever , Ġusing Ġseveral Ġapproaches Ġand Ġdatasets Ġsuch Ġas Ġthe Ġone Ġproposed Ġhere Ġcan Ġbring Ġabout Ġa Ġmore Ġthorough Ġexamination Ġof Ġinappropriate Ġbiases Ġin Ġmodern Ġmachine Ġlearning Ġsystems . </s> App endix </s> Fig ures ĠFIG REF 31 Ġand ĠFIG REF 32 Ġshow Ġbox Ġplots Ġof Ġthe Ġscore Ġdifferences Ġfor Ġeach Ġsystem Ġon Ġthe Ġfour Ġemotion Ġintensity Ġregression Ġtasks Ġon Ġthe Ġgender Ġand Ġrace Ġsentence Ġpairs , Ġrespectively . ĠEach Ġpoint Ġon Ġa Ġplot Ġcorresponds Ġto Ġthe Ġdifference Ġin Ġscores Ġpredicted Ġby Ġthe Ġsystem Ġon Ġone Ġsentence Ġpair . ĠThe Ġsystems Ġare Ġordered Ġby Ġtheir Ġperformance Ġrank Ġ( from Ġfirst Ġto Ġlast ) Ġon Ġthe Ġtask Ġas Ġper Ġthe Ġofficial Ġevaluation Ġmetric Ġon Ġthe Ġtweets Ġtest Ġsets .