create fair systems
EXTRACTIVE
 broadcast recordings are also a valuable source of data for the speech processing community, recent works uncovering gender bias in several natural language processing (NLP) tools
EXTRACTIVE
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of “Fairness by Design" proposed by BIBREF31.
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community. Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3. Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies. This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8.
1908.08717
Introduction </s> In Ġrecent Ġyears , Ġgender Ġhas Ġbecome Ġa Ġhot Ġtopic Ġwithin Ġthe Ġpolitical , Ġsocietal Ġand Ġresearch Ġspheres . ĠNumerous Ġstudies Ġhave Ġbeen Ġconducted Ġin Ġorder Ġto Ġevaluate Ġthe Ġpresence Ġof Ġwomen Ġin Ġmedia , Ġoften Ġrevealing Ġtheir Ġunder - represent ation , Ġsuch Ġas Ġthe ĠGlobal ĠMedia ĠMonitoring ĠProject ĠB IB REF 0 . ĠIn Ġthe ĠFrench Ġcontext , Ġthe ĠC SA ĠB IB REF 1 Ġproduces Ġa Ġreport Ġon Ġgender Ġrepresentation Ġin Ġmedia Ġon Ġa Ġyearly Ġbasis . ĠThe Ġ2017 Ġreport Ġshows Ġthat Ġwomen Ġrepresent Ġ40 % Ġof ĠFrench Ġmedia Ġspeakers , Ġwith Ġa Ġsignificant Ġdrop Ġduring Ġhigh - aud ience Ġhours Ġ( 6 : 00 - 8 : 00 pm ) Ġreaching Ġa Ġvalue Ġof Ġonly Ġ29 %. ĠAnother Ġlarge Ġscale Ġstudy Ġconfirmed Ġthis Ġtrend Ġwith Ġan Ġautomatic Ġanalysis Ġof Ġgender Ġin ĠFrench Ġaud iov is ual s Ġstreams , Ġhighlighting Ġa Ġhuge Ġvariation Ġacross Ġtype Ġof Ġshows ĠB IB REF 2 . </s> Besides Ġthe Ġsocial Ġimpact Ġof Ġgender Ġrepresentation , Ġbroadcast Ġrecordings Ġare Ġalso Ġa Ġvaluable Ġsource Ġof Ġdata Ġfor Ġthe Ġspeech Ġprocessing Ġcommunity . ĠIndeed , Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems Ġrequire Ġlarge Ġamount Ġof Ġannot ated Ġspeech Ġdata Ġto Ġbe Ġefficiently Ġtrained , Ġwhich Ġleaves Ġus Ġfacing Ġthe Ġemerging Ġconcern Ġabout Ġthe Ġfact Ġthat Ġ" AI Ġartifacts Ġtend Ġto Ġreflect Ġthe Ġgoals , Ġknowledge Ġand Ġexperience Ġof Ġtheir Ġcreators " ĠB IB REF 3 . ĠSince Ġwe Ġknow Ġthat Ġwomen Ġare Ġunder - represented Ġin Ġmedia Ġand Ġthat Ġthe ĠAI Ġdiscipline Ġhas Ġretained Ġa Ġmale - oriented Ġfocus ĠB IB REF 4 , Ġwe Ġcan Ġlegitimately Ġwonder Ġabout Ġthe Ġimpact Ġof Ġusing Ġsuch Ġdata Ġas Ġa Ġtraining Ġset Ġfor ĠAS R Ġtechnologies . ĠThis Ġconcern Ġis Ġstrengthened Ġby Ġthe Ġrecent Ġworks Ġuncover ing Ġgender Ġbias Ġin Ġseveral Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtools Ġsuch Ġas ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 . </s> In Ġthis Ġpaper , Ġwe Ġfirst Ġhighlight Ġthe Ġimportance Ġof ĠTV Ġand Ġradio Ġbroadcast Ġas Ġa Ġsource Ġof Ġdata Ġfor ĠAS R , Ġand Ġthe Ġpotential Ġimpact Ġit Ġcan Ġhave . ĠWe Ġthen Ġperform Ġa Ġstatistical Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġa Ġdata Ġset Ġcomposed Ġof Ġfour Ġstate - of - the - art Ġcorpor a Ġof ĠFrench Ġbroadcast , Ġwidely Ġused Ġwithin Ġthe Ġspeech Ġcommunity . ĠFinally Ġwe Ġquestion Ġthe Ġimpact Ġof Ġsuch Ġa Ġrepresentation Ġon Ġthe Ġsystems Ġdeveloped Ġon Ġthis Ġdata , Ġthrough Ġthe Ġperspective Ġof Ġan ĠAS R Ġsystem . </s> From Ġgender Ġrepresentation Ġin Ġdata Ġto Ġgender Ġbias Ġin ĠAI Ġ: :: ĠOn Ġthe Ġimportance Ġof Ġdata </s> The Ġever Ġgrowing Ġuse Ġof Ġmachine Ġlearning Ġin Ġscience Ġhas Ġbeen Ġenabled Ġby Ġseveral Ġprogresses Ġamong Ġwhich Ġthe Ġexponential Ġgrowth Ġof Ġdata Ġavailable . ĠThe Ġquality Ġof Ġa Ġsystem Ġnow Ġdepends Ġmostly Ġon Ġthe Ġquality Ġand Ġquantity Ġof Ġthe Ġdata Ġit Ġhas Ġbeen Ġtrained Ġon . ĠIf Ġit Ġdoes Ġnot Ġdiscard Ġthe Ġimportance Ġof Ġan Ġappropriate Ġarchitecture , Ġit Ġreaff irms Ġthe Ġfact Ġthat Ġrich Ġand Ġlarge Ġcorpor a Ġare Ġa Ġvaluable Ġresource . ĠCorpor a Ġare Ġresearch Ġcontributions Ġwhich Ġdo Ġnot Ġonly Ġallow Ġto Ġsave Ġand Ġobserve Ġcertain Ġphenomena Ġor Ġvalidate Ġa Ġhypothesis Ġor Ġmodel , Ġbut Ġare Ġalso Ġa Ġmandatory Ġpart Ġof Ġthe Ġtechnology Ġdevelopment . ĠThis Ġtrend Ġis Ġnotably Ġobservable Ġwithin Ġthe ĠN LP Ġfield , Ġwhere Ġindustrial Ġtechnologies , Ġsuch Ġas ĠApple , ĠAmazon Ġor ĠGoogle Ġvocal Ġassistants Ġnow Ġreach Ġhigh Ġperformance Ġlevel Ġpartly Ġdue Ġto Ġthe Ġamount Ġof Ġdata Ġpossessed Ġby Ġthese Ġcompanies ĠB IB REF 9 . </s> Sur prisingly , Ġas Ġdata Ġis Ġsaid Ġto Ġbe ĠâĢ ľ the Ġnew Ġoil ", Ġfew Ġdata Ġsets Ġare Ġavailable Ġfor ĠAS R Ġsystems . ĠThe Ġbest Ġknown Ġare Ġcorpor a Ġlike ĠTIM IT ĠB IB REF 10 , ĠSwitch board ĠB IB REF 11 Ġor ĠFisher ĠB IB REF 12 Ġwhich Ġdate Ġback Ġto Ġthe Ġearly Ġ1990 s . ĠThe Ġsc arc eness Ġof Ġavailable Ġcorpor a Ġis Ġjustified Ġby Ġthe Ġfact Ġthat Ġgathering Ġand Ġannot ating Ġaudio Ġdata Ġis Ġcostly Ġboth Ġin Ġterms Ġof Ġmoney Ġand Ġtime . ĠTelephone Ġconversations Ġand Ġbroadcast Ġrecordings Ġhave Ġbeen Ġthe Ġprimary Ġsource Ġof Ġspontaneous Ġspeech Ġused . ĠOut Ġof Ġall Ġthe Ġ130 Ġaudio Ġresources Ġproposed Ġby ĠL DC Ġto Ġtrain Ġautomatic Ġspeech Ġrecognition Ġsystems Ġin ĠEnglish , Ġapproximately Ġ14 % Ġof Ġthem Ġare Ġbased Ġon Ġbroadcast Ġnews Ġand Ġconversation . ĠFor ĠFrench Ġspeech Ġtechnologies , Ġfour Ġcorpor a Ġcontaining Ġradio Ġand ĠTV Ġbroadcast Ġare Ġthe Ġmost Ġwidely Ġused : ĠEST ER 1 ĠB IB REF 13 , ĠEST ER 2 ĠB IB REF 14 , ĠET APE ĠB IB REF 15 Ġand ĠRE P ERE ĠB IB REF 16 . ĠThese Ġfour Ġcorpor a Ġhave Ġbeen Ġbuilt Ġalongside Ġevaluation Ġcampaigns Ġand Ġare Ġstill , Ġto Ġour Ġknowledge , Ġthe Ġlargest ĠFrench Ġones Ġof Ġtheir Ġtype Ġavailable Ġto Ġdate . </s> From Ġgender Ġrepresentation Ġin Ġdata Ġto Ġgender Ġbias Ġin ĠAI Ġ: :: ĠFrom Ġdata Ġto Ġbias </s> The Ġgender Ġissue Ġhas Ġreturned Ġto Ġthe Ġforefront Ġof Ġthe Ġmedia Ġscene Ġin Ġrecent Ġyears Ġand Ġwith Ġthe Ġemergence Ġof ĠAI Ġtechnologies Ġin Ġour Ġdaily Ġlives , Ġgender Ġbias Ġhas Ġbecome Ġa Ġscientific Ġtopic Ġthat Ġresearchers Ġare Ġjust Ġbeginning Ġto Ġaddress . ĠSeveral Ġstudies Ġrevealed Ġthe Ġexistence Ġof Ġgender Ġbias Ġin ĠAI Ġtechnologies Ġsuch Ġas Ġface Ġrecognition Ġ( Gender Sh ades ĠB IB REF 17 ), ĠN LP Ġ( word Ġembed d ings ĠB IB REF 5 Ġand Ġsemantics ĠB IB REF 6 ) Ġand Ġmachine Ġtranslation Ġ( BI B REF 18 , ĠB IB REF 7 ). ĠThe Ġimpact Ġof Ġthe Ġtraining Ġdata Ġused Ġwithin Ġthese Ġdeep - learning Ġalgorithms Ġis Ġtherefore Ġquestioned . </s> B ias Ġcan Ġbe Ġfound Ġat Ġdifferent Ġlevels Ġas Ġpointed Ġout Ġby ĠB IB REF 19 . ĠB IB REF 20 Ġdefines Ġbias Ġas Ġa Ġskew Ġthat Ġproduces Ġa Ġtype Ġof Ġharm . ĠShe Ġdistinguishes Ġtwo Ġtypes Ġof Ġharms Ġthat Ġare Ġallocation Ġharm Ġand Ġrepresentation Ġharm . ĠThe Ġallocation Ġharm Ġoccurs Ġwhen Ġa Ġsystem Ġis Ġperforming Ġbetter Ġor Ġworse Ġfor Ġa Ġcertain Ġgroup Ġwhile Ġrepresent ational Ġharm Ġcontributes Ġto Ġthe Ġperpet uation Ġof Ġstereotypes . ĠBoth Ġtypes Ġof Ġharm Ġare Ġthe Ġresults Ġof Ġbias Ġin Ġmachine Ġlearning Ġthat Ġoften Ġcomes Ġfrom Ġthe Ġdata Ġsystems Ġare Ġtrained Ġon . ĠDis par ities Ġin Ġrepresentation Ġin Ġour Ġsocial Ġstructures Ġis Ġcaptured Ġand Ġreflected Ġby Ġthe Ġtraining Ġdata , Ġthrough Ġstatistical Ġpatterns . ĠThe ĠGender Sh ades Ġstudy Ġis Ġa Ġstriking Ġexample Ġof Ġwhat Ġdata Ġdisparity Ġand Ġlack Ġof Ġrepresentation Ġcan Ġproduce : Ġthe Ġauthors Ġtested Ġseveral Ġgender Ġrecognition Ġmodules Ġused Ġby Ġfacial Ġrecognition Ġtools Ġand Ġfound Ġdifference Ġin Ġerror - rate Ġas Ġhigh Ġas Ġ34 Ġpercentage Ġpoints Ġbetween Ġrecognition Ġof Ġwhite Ġmale Ġand Ġblack Ġfemale Ġfaces . ĠThe Ġscarce Ġpresence Ġof Ġwomen Ġand Ġcolored Ġpeople Ġin Ġtraining Ġset Ġresulted Ġin Ġbias Ġin Ġperformance Ġtowards Ġthese Ġtwo Ġcategories , Ġwith Ġa Ġstrong Ġintersection al Ġbias . ĠAs Ġwritten Ġby ĠB IB REF 21 Ġ" A Ġdata Ġset Ġmay Ġhave Ġmany Ġmillions Ġof Ġpieces Ġof Ġdata , Ġbut Ġthis Ġdoes Ġnot Ġmean Ġit Ġis Ġrandom Ġor Ġrepresentative . ĠTo Ġmake Ġstatistical Ġclaims Ġabout Ġa Ġdata Ġset , Ġwe Ġneed Ġto Ġknow Ġwhere Ġdata Ġis Ġcoming Ġfrom ; Ġit Ġis Ġsimilarly Ġimportant Ġto Ġknow Ġand Ġaccount Ġfor Ġthe Ġweaknesses Ġin Ġthat Ġdata ." Ġ( p . 668 ). </s> Regarding ĠAS R Ġtechnology , Ġlittle Ġwork Ġhas Ġexplored Ġthe Ġpresence Ġof Ġgender Ġbias Ġwithin Ġthe Ġsystems Ġand Ġno Ġconsensus Ġhas Ġbeen Ġreached . ĠB IB REF 22 Ġfound Ġthat Ġspeech Ġrecogn izers Ġperform Ġbetter Ġon Ġfemale Ġvoice Ġon Ġa Ġbroadcast Ġnews Ġand Ġtelephone Ġcorpus . ĠThey Ġproposed Ġseveral Ġexplanations Ġto Ġthis Ġobservation , Ġsuch Ġas Ġthe Ġlarger Ġpresence Ġof Ġnon - professional Ġmale Ġspeech Ġin Ġthe Ġbroadcast Ġdata , Ġimplying Ġa Ġless Ġprepared Ġspeech Ġfor Ġthese Ġspeakers Ġor Ġa Ġmore Ġnormative Ġlanguage Ġand Ġstandard Ġpronunciation Ġfor Ġwomen Ġlinked Ġto Ġthe Ġtraditional Ġrole Ġof Ġwomen Ġin Ġlanguage Ġacquisition Ġand Ġeducation . ĠThe Ġsame Ġtrend Ġwas Ġobserved Ġby ĠB IB REF 23 . ĠMore Ġrecently , ĠB IB REF 24 Ġdiscovered Ġa Ġgender Ġbias Ġwithin ĠYouTube 's Ġautomatic Ġcaption ing Ġsystem Ġbut Ġthis Ġbias Ġwas Ġnot Ġobserved Ġin Ġa Ġsecond Ġstudy Ġevaluating ĠBing ĠSpeech Ġsystem Ġand ĠYouTube ĠAutomatic ĠCapt ions Ġon Ġa Ġlarger Ġdata Ġset ĠB IB REF 8 . ĠHowever Ġrace Ġand Ġdialect Ġbias Ġwere Ġfound . ĠGeneral ĠAmerican Ġspeakers Ġand Ġwhite Ġspeakers Ġhad Ġthe Ġlowest Ġerror Ġrate Ġfor Ġboth Ġsystems . ĠIf Ġthe Ġbetter Ġperformance Ġon ĠGeneral ĠAmerican Ġspeakers Ġcould Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthey Ġare Ġall Ġvoice Ġprofessionals , Ġproducing Ġclear Ġand Ġarticulated Ġspeech , Ġbut Ġno Ġexplanation Ġis Ġprovided Ġfor Ġbiases Ġtowards Ġnon - white Ġspeakers . </s> Gender Ġbias Ġin ĠAS R Ġtechnology Ġis Ġstill Ġan Ġopen Ġresearch Ġquestion Ġas Ġno Ġclear Ġanswer Ġhas Ġbeen Ġreached Ġso Ġfar . ĠIt Ġseems Ġthat Ġmany Ġparameters Ġare Ġto Ġtake Ġinto Ġaccount Ġto Ġachieve Ġa Ġgeneral Ġagreement . ĠAs Ġwe Ġestablished Ġthe Ġimportance Ġof ĠTV Ġand Ġradio Ġbroadcast Ġas Ġa Ġsource Ġof Ġdata Ġfor ĠAS R , Ġand Ġthe Ġpotential Ġimpact Ġit Ġcan Ġhave , Ġthe Ġfollowing Ġcontent Ġof Ġthis Ġpaper Ġis Ġstructured Ġas Ġthis : Ġwe Ġfirst Ġdescribe Ġstatistically Ġthe Ġgender Ġrepresentation Ġof Ġa Ġdata Ġset Ġcomposed Ġof Ġfour Ġstate - of - the - art Ġcorpor a Ġof ĠFrench Ġbroadcast , Ġwidely Ġused Ġwithin Ġthe Ġspeech Ġcommunity , Ġintroducing Ġthe Ġnotion Ġof Ġspeaker 's Ġrole Ġto Ġrefine Ġour Ġanalysis Ġin Ġterms Ġof Ġvoice Ġprofessionalism . ĠWe Ġthen Ġquestion Ġthe Ġimpact Ġof Ġsuch Ġa Ġrepresentation Ġon Ġa ĠAS R Ġsystem Ġtrained Ġon Ġthese Ġdata . ĠB IB REF 25 </s> Method ology </s> This Ġsection Ġis Ġorganized Ġas Ġfollows : Ġwe Ġfirst Ġpresent Ġthe Ġdata Ġwe Ġare Ġworking Ġon . ĠIn Ġa Ġsecond Ġtime Ġwe Ġexplain Ġhow Ġwe Ġproceed Ġto Ġdescribe Ġthe Ġgender Ġrepresentation Ġin Ġour Ġcorpus Ġand Ġintroduce Ġthe Ġnotion Ġof Ġspeaker 's Ġrole . ĠThe Ġthird Ġsubsection Ġintroduces Ġthe ĠAS R Ġsystem Ġand Ġmetrics Ġused Ġto Ġevaluate Ġgender Ġbias Ġin Ġperformance . </s> Method ology Ġ: :: ĠData Ġpresentation </s> Our Ġdata Ġconsists Ġof Ġtwo Ġsets Ġused Ġto Ġtrain Ġand Ġevaluate Ġour Ġautomatic Ġspeech Ġrecognition Ġsystem . ĠFour Ġmajor Ġevaluation Ġcampaigns Ġhave Ġenabled Ġthe Ġcreation Ġof Ġwide Ġcorpor a Ġof ĠFrench Ġbroadcast Ġspeech : ĠEST ER 1 ĠB IB REF 13 , ĠEST ER 2 ĠB IB REF 14 , ĠET APE ĠB IB REF 15 Ġand ĠRE P ERE ĠB IB REF 16 . ĠThese Ġfour Ġcollections Ġcontain Ġradio Ġand / or ĠTV Ġbroadcasts Ġaired Ġbetween Ġ1998 Ġand Ġ2013 Ġwhich Ġare Ġused Ġby Ġmost Ġacademic Ġresearchers Ġin ĠAS R . ĠShow Ġduration Ġvaries Ġbetween Ġ10 min Ġand Ġan Ġhour . ĠAs Ġyears Ġwent Ġby Ġand Ġspeech Ġprocessing Ġresearch Ġwas Ġprogressing , Ġthe Ġdifficulty Ġof Ġthe Ġtasks Ġaugmented Ġand Ġthe Ġcontent Ġof Ġthese Ġevaluation Ġcorpor a Ġchanged . ĠEST ER 1 Ġand ĠEST ER 2 Ġmainly Ġcontain Ġprepared Ġspeech Ġsuch Ġas Ġbroadcast Ġnews , Ġwhereas ĠET APE Ġand ĠRE P ERE Ġconsists Ġalso Ġof Ġdebates Ġand Ġentertainment Ġshows , Ġspontaneous Ġspeech Ġintroducing Ġmore Ġdifficulty Ġin Ġits Ġrecognition . </s> Our Ġtraining Ġset Ġcontains Ġ27 , 08 5 Ġspeech Ġutter ances Ġproduced Ġby Ġ2 , 506 Ġspeakers , Ġaccounting Ġfor Ġapproximately Ġ100 Ġhours Ġof Ġspeech . ĠOur Ġevaluation Ġset Ġcontains Ġ74 , 0 64 Ġspeech Ġutter ances Ġproduced Ġby Ġ1 , 268 Ġspeakers Ġfor Ġa Ġtotal Ġof Ġ70 Ġhours Ġof Ġspeech . ĠTraining Ġdata Ġby Ġshow , Ġmedium Ġand Ġspeech Ġtype Ġis Ġsummarized Ġin ĠTable Ġand Ġevaluation Ġdata Ġin ĠTable Ġ. ĠEvaluation Ġdata Ġhas Ġa Ġhigher Ġvariety Ġof Ġshows Ġwith Ġboth Ġprepared Ġ( P ) Ġand Ġspontaneous Ġ( S ) Ġspeech Ġtype Ġ( ac cent ed Ġspeech Ġfrom ĠAfrican Ġradio Ġbroadcast Ġis Ġalso Ġincluded Ġin Ġthe Ġevaluation Ġset ). </s> Method ology Ġ: :: ĠMethod ology Ġfor Ġdescriptive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata </s> We Ġfirst Ġdescribe Ġthe Ġgender Ġrepresentation Ġin Ġtraining Ġdata . ĠGender Ġrepresentation Ġis Ġmeasured Ġin Ġterms Ġof Ġnumber Ġof Ġspeakers , Ġnumber Ġof Ġutter ances Ġ( or Ġspeech Ġturns ), Ġand Ġturn Ġlengths Ġ( desc ript ive Ġstatistics Ġare Ġgiven Ġin ĠSection ĠSEC REF 16 ). ĠEach Ġspeech Ġturn Ġwas Ġmapped Ġto Ġits Ġspeaker Ġin Ġorder Ġto Ġassociate Ġit Ġwith Ġa Ġgender . </s> As Ġpointed Ġout Ġby Ġthe ĠC SA Ġreport ĠB IB REF 1 , Ġwomen Ġpresence Ġtends Ġto Ġbe Ġmarginal Ġwithin Ġthe Ġhigh - aud ience Ġhours , Ġshowing Ġthat Ġwomen Ġare Ġrepresented Ġbut Ġless Ġthan Ġmen Ġand Ġwithin Ġcertain Ġgiven Ġconditions . ĠIt Ġis Ġclear Ġthat Ġa Ġsmall Ġnumber Ġof Ġspeakers Ġis Ġresponsible Ġfor Ġa Ġlarge Ġnumber Ġof Ġspeech Ġturns . ĠMost Ġof Ġthese Ġspeakers Ġare Ġjournalists , Ġpoliticians , Ġpresent ers Ġand Ġsuch , Ġwho Ġare Ġrepresentative Ġof Ġa Ġshow . ĠTherefore , Ġwe Ġintroduce Ġthe Ġnotion Ġof Ġspeaker 's Ġrole Ġto Ġrefine Ġour Ġexploration Ġof Ġgender Ġdisparity , Ġfollowing Ġstudies Ġwhich Ġquant ified Ġwomen 's Ġpresence Ġin Ġterms Ġof Ġrole . ĠWithin Ġour Ġwork , Ġwe Ġdefine Ġthe Ġnotion Ġof Ġspeaker Ġrole Ġby Ġtwo Ġcriteria Ġspecifying Ġthe Ġspeaker 's Ġon - air Ġpresence , Ġnamely Ġthe Ġnumber Ġof Ġspeech Ġturns Ġand Ġthe Ġcumulative Ġduration Ġof Ġhis Ġor Ġher Ġspeaking Ġtime Ġin Ġa Ġshow . ĠBased Ġon Ġthe Ġavailable Ġspeech Ġtranscript ions Ġand Ġmeta - data , Ġwe Ġcompute Ġfor Ġeach Ġspeaker Ġthe Ġnumber Ġof Ġspeech Ġturns Ġuttered Ġas Ġwell Ġas Ġtheir Ġtotal Ġlength . ĠWe Ġthen Ġuse Ġthe Ġfollowing Ġcriteria Ġto Ġdefine Ġspeaker 's Ġrole : Ġa Ġspeaker Ġis Ġconsidered Ġas Ġspeaking Ġoften Ġ( respect ively Ġseldom ) Ġif Ġhe / she Ġaccum ulates Ġa Ġtotal Ġof Ġturns Ġhigher Ġ( respect ively Ġlower ) Ġthan Ġ1 % Ġof Ġthe Ġtotal Ġnumber Ġof Ġspeech Ġturns Ġin Ġa Ġgiven Ġshow . ĠThe Ġsame Ġprocess Ġis Ġapplied Ġto Ġidentify Ġspeakers Ġtalking Ġfor Ġa Ġlong Ġperiod Ġfrom Ġthose Ġwho Ġdo Ġnot . ĠWe Ġend Ġup Ġwith Ġtwo Ġsalient Ġroles Ġcalled ĠAnch ors Ġand ĠPun ct ual Ġspeakers : </s> the ĠAnch or Ġspeakers Ġ( A ) Ġare Ġabove Ġthe Ġthreshold Ġof Ġ1 % Ġfor Ġboth Ġcriteria , Ġmeaning Ġthey Ġare Ġintervening Ġoften Ġand Ġfor Ġa Ġlong Ġtime Ġthus Ġholding Ġan Ġimportant Ġplace Ġin Ġinteraction ; </s> the ĠPun ct ual Ġspeakers Ġ( PS ) Ġon Ġthe Ġcontrary Ġare Ġbelow Ġthe Ġthreshold Ġof Ġ1 % Ġfor Ġboth Ġthe Ġtotal Ġnumber Ġof Ġturns Ġand Ġthe Ġtotal Ġspeech Ġtime . </s> These Ġroles Ġare Ġdefined Ġat Ġthe Ġshow Ġlevel . ĠThey Ġcould Ġbe Ġroughly Ġassimil ated Ġto Ġthe Ġcategor ization ĠâĢ ľ host / gu est âĢ Ŀ Ġin Ġradio Ġand ĠTV Ġshows . ĠAnch ors Ġcould Ġbe Ġdescribed Ġas Ġprofessional Ġspeakers , Ġproducing Ġmostly Ġprepared Ġspeech , Ġwhereas ĠPun ct ual Ġspeakers Ġare Ġmore Ġlikely Ġto Ġbe ĠâĢ ľ every day Ġpeople ". ĠThe Ġconcept Ġof Ġspeaker 's Ġrole Ġmakes Ġsense Ġat Ġboth Ġsoc iological Ġand Ġtechnical Ġlevels . ĠAn ĠAnch or Ġspeaker Ġis Ġmore Ġlikely Ġto Ġbe Ġknown Ġfrom Ġthe Ġaudience Ġ( soc iety ), Ġbut Ġhe Ġor Ġshe Ġwill Ġalso Ġlikely Ġhave Ġa Ġprofessional Ġ( clear ) Ġway Ġof Ġspeaking Ġ( as Ġmentioned Ġby ĠB IB REF 22 Ġand ĠB IB REF 8 ), Ġas Ġwell Ġas Ġa Ġhigh Ġnumber Ġof Ġutter ances , Ġaugment ing Ġthe Ġamount Ġof Ġdata Ġavailable Ġfor Ġa Ġgiven Ġgender Ġcategory . </s> Method ology Ġ: :: ĠGender Ġbias Ġevaluation Ġprocedure Ġof Ġan ĠAS R Ġsystem Ġperformance Ġ: :: ĠAS R Ġsystem </s> The ĠAS R Ġsystem Ġused Ġin Ġthis Ġwork Ġis Ġdescribed Ġin ĠB IB REF 25 . ĠIt Ġuses Ġthe ĠK AL DI Ġtool kit ĠB IB REF 26 , Ġfollowing Ġa Ġstandard ĠK aldi Ġrecipe . ĠThe Ġacoustic Ġmodel Ġis Ġbased Ġon Ġa Ġhybrid ĠH MM - D NN Ġarchitecture Ġand Ġtrained Ġon Ġthe Ġdata Ġsummarized Ġin ĠTable Ġ. ĠAc oustic Ġtraining Ġdata Ġcorrespond Ġto Ġ100 h Ġof Ġnon - sp ont aneous Ġspeech Ġtype Ġ( mostly Ġbroadcast Ġnews ) Ġcoming Ġfrom Ġboth Ġradio Ġand ĠTV Ġshows . ĠA Ġ5 - gram Ġlanguage Ġmodel Ġis Ġtrained Ġfrom Ġseveral ĠFrench Ġcorpor a Ġ( 3 , 323 M Ġwords Ġin Ġtotal ) Ġusing ĠSR IL M Ġtool kit ĠB IB REF 27 . ĠThe Ġpronunciation Ġmodel Ġis Ġdeveloped Ġusing Ġthe Ġlex ical Ġresource ĠBD LE X ĠB IB REF 28 Ġas Ġwell Ġas Ġautomatic Ġgrap he me - to - ph on eme Ġ( G 2 P ) Ġtranscription Ġto Ġfind Ġpronunciation Ġvariants Ġof Ġour Ġvocabulary Ġ( limited Ġto Ġ80 K ). ĠIt Ġis Ġimportant Ġto Ġre - spec ify Ġhere , Ġfor Ġfurther Ġanalysis , Ġthat Ġour ĠK aldi Ġpipeline Ġfollows Ġspeaker Ġadaptive Ġtraining Ġ( S AT ) Ġwhere Ġwe Ġtrain Ġand Ġdecode Ġusing Ġspeaker Ġadapted Ġfeatures Ġ( f M LL R - adapt ed Ġfeatures ) Ġin Ġper - spe aker Ġmode . ĠIt Ġis Ġwell Ġknown Ġthat Ġspeaker Ġadaptation Ġacts Ġas Ġan Ġeffective Ġprocedure Ġto Ġreduce Ġmismatch Ġbetween Ġtraining Ġand Ġevaluation Ġconditions ĠB IB REF 29 , ĠB IB REF 26 . </s> Method ology Ġ: :: ĠGender Ġbias Ġevaluation Ġprocedure Ġof Ġan ĠAS R Ġsystem Ġperformance Ġ: :: ĠEvaluation </s> Word ĠError ĠRate Ġ( WER ) Ġis Ġa Ġcommon Ġmetric Ġto Ġevaluate ĠAS R Ġperformance . ĠIt Ġis Ġmeasured Ġas Ġthe Ġsum Ġof Ġerrors Ġ( insert ions , Ġdelet ions Ġand Ġsubstit utions ) Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġwords Ġin Ġthe Ġreference Ġtranscription . ĠAs Ġwe Ġare Ġinvestigating Ġthe Ġimpact Ġon Ġperformance Ġof Ġspeaker 's Ġgender Ġand Ġrole , Ġwe Ġcomputed Ġthe ĠW ER Ġfor Ġeach Ġspeaker Ġat Ġthe Ġepisode Ġ( show Ġoccurrence ) Ġlevel . ĠAnaly zing Ġat Ġsuch Ġgran ularity Ġallows Ġus Ġto Ġavoid Ġlarge ĠW ER Ġvariation Ġthat Ġcould Ġbe Ġobserved Ġat Ġutter ance Ġlevel Ġ( especially Ġfor Ġshort Ġspeech Ġturns ) Ġbut Ġalso Ġmakes Ġpossible Ġto Ġget Ġseveral ĠW ER Ġvalues Ġfor Ġa Ġgiven Ġspeaker , Ġone Ġfor Ġeach Ġoccurrence Ġof Ġa Ġshow Ġin Ġwhich Ġhe / she Ġappears Ġon . ĠSpeaker 's Ġgender Ġwas Ġprovided Ġby Ġthe Ġmeta - data Ġand Ġrole Ġwas Ġobtained Ġusing Ġthe Ġcriteria Ġfrom ĠSection ĠSEC REF 6 Ġcomputed Ġfor Ġeach Ġshow . ĠThis Ġenables Ġus Ġto Ġanalyze Ġour Ġresults Ġacross Ġgender Ġand Ġrole Ġcategories Ġwhich Ġwas Ġdone Ġusing ĠWil co xon Ġrank Ġsum Ġtests Ġalso Ġcalled ĠMann - Whit ney ĠU Ġtest Ġ( with Ġ$\ alpha Ġ$ = Ġ0 . 001 ) ĠB IB REF 30 . ĠThe Ġchoice Ġof Ġa ĠWil co xon Ġrank Ġsum Ġtest Ġand Ġnot Ġthe Ġcommonly Ġused Ġt - test Ġis Ġmotivated Ġby Ġthe Ġnon - norm ality Ġof Ġour Ġdata . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠGender Ġrepresentation </s> As Ġexpected , Ġwe Ġobserve Ġa Ġdisparity Ġin Ġterms Ġof Ġgender Ġrepresentation Ġin Ġour Ġdata Ġ( see ĠTable Ġ). ĠWomen Ġrepresent Ġ33 . 16 % Ġof Ġthe Ġspeakers , Ġconfirming Ġthe Ġfigures Ġgiven Ġby Ġthe ĠGM MP Ġreport ĠB IB REF 0 . ĠHowever , Ġit Ġis Ġworth Ġnoticing Ġthat Ġwomen Ġaccount Ġfor Ġonly Ġ22 . 57 % Ġof Ġthe Ġtotal Ġspeech Ġtime , Ġwhich Ġleads Ġus Ġto Ġconclude Ġthat Ġwomen Ġalso Ġspeak Ġless Ġthan Ġmen . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠSpeaker 's Ġrole Ġrepresentation </s> Table Ġpresents Ġroles ' Ġrepresentation Ġin Ġtraining Ġdata Ġand Ġshows Ġthat Ġdespite Ġthe Ġsmall Ġnumber Ġof ĠAnch or Ġspeakers Ġin Ġour Ġdata Ġ( 3 . 79 %), Ġthey Ġnevertheless Ġconcentrate Ġ35 . 71 Ġ% Ġof Ġthe Ġtotal Ġspeech Ġtime . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠRole Ġand Ġgender Ġinteraction </s> When Ġcrossing Ġboth Ġparameters , Ġwe Ġcan Ġobserve Ġthat Ġthe Ġgender Ġdistribution Ġis Ġnot Ġconstant Ġthroughout Ġroles . ĠWomen Ġrepresent Ġ29 . 47 % Ġof Ġthe Ġspeakers Ġwithin Ġthe ĠAnch or Ġcategory , Ġeven Ġless Ġthan Ġamong Ġthe ĠPun ct ual Ġspeakers . ĠTheir Ġpercentage Ġof Ġspeech Ġis Ġalso Ġsmaller . ĠWhen Ġcalculating Ġthe Ġaverage Ġspeech Ġtime Ġuttered Ġby Ġa Ġfemale ĠAnch or , Ġwe Ġobtain Ġa Ġvalue Ġof Ġ15 . 9 Ġmin Ġagainst Ġ25 . 2 Ġmin Ġfor Ġa Ġmale ĠAnch or , Ġwhich Ġsuggests Ġthat Ġeven Ġwithin Ġthe ĠAnch or Ġcategory Ġmen Ġtend Ġto Ġspeak Ġmore . ĠThis Ġconfirms Ġthe Ġexistence Ġof Ġgender Ġdisparities Ġwithin ĠFrench Ġmedia . ĠIt Ġcorrobor ates Ġwith Ġthe Ġanalysis Ġof Ġthe ĠC SA ĠB IB REF 1 , Ġwhich Ġshows Ġthat Ġwomen Ġwere Ġless Ġpresent Ġduring Ġhigh - aud ience Ġhours . ĠOur Ġstudy Ġshows Ġthat Ġthey Ġare Ġalso Ġless Ġpresent Ġin Ġimportant Ġroles . ĠThese Ġresults Ġlegitimate Ġour Ġinitial Ġquestioning Ġon Ġthe Ġimpact Ġof Ġgender Ġbalance Ġon ĠAS R Ġperformance Ġtrained Ġon Ġbroadcast Ġrecordings . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠImpact Ġof Ġgender Ġon ĠW ER </s> As Ġexplained Ġin ĠSection ĠSEC REF 13 , ĠW ER Ġis Ġthe Ġsum Ġof Ġerrors Ġdivided Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġtranscription Ġreference . ĠThe Ġhigher Ġthe ĠW ER , Ġthe Ġpoorer Ġthe Ġsystem Ġperformance . ĠOur Ġ70 h Ġevaluation Ġdata Ġcontains Ġa Ġlarge Ġamount Ġof Ġspontaneous Ġspeech Ġand Ġis Ġvery Ġchallenging Ġfor Ġthe ĠAS R Ġsystem Ġtrained Ġon Ġprepared Ġspeech : Ġwe Ġobserve Ġan Ġoverall Ġaverage ĠW ER Ġof Ġ42 . 9 % Ġfor Ġwomen Ġand Ġ34 . 3 % Ġfor Ġmen . ĠThis Ġdifference Ġof ĠW ER Ġbetween Ġmen Ġand Ġwomen Ġis Ġstatistically Ġsignificant Ġ( med ( M ) Ġ= Ġ25 %; Ġmed ( F ) Ġ= Ġ29 %; ĠU Ġ= Ġ7 09 040 ; Ġp - value Ġ< Ġ0 . 001 ). </s> However , Ġwhen Ġobserving Ġgender Ġdifferences Ġacross Ġshows , Ġno Ġclear Ġtrend Ġcan Ġbe Ġidentified , Ġas Ġshown Ġin ĠFigure ĠFIG REF 21 . ĠFor Ġshows Ġlike ĠAfrica 1 ĠInf os Ġor ĠLa ĠPlace Ġdu ĠVillage , Ġwe Ġfind Ġan Ġaverage ĠW ER Ġlower Ġfor Ġwomen Ġthan Ġfor Ġmen , Ġwhile Ġthe Ġtrend Ġis Ġreversed Ġfor Ġshows Ġsuch Ġas ĠUn ĠTem ps Ġde ĠPau ch on Ġor ĠLe ĠMas que Ġet Ġla ĠPl ume . ĠThe Ġdisparity Ġof Ġthe Ġresults Ġdepending Ġon Ġthe Ġshow Ġleads Ġus Ġto Ġbelieve Ġthat Ġother Ġfactors Ġmay Ġbe Ġentangled Ġwithin Ġthe Ġobserved Ġphenomenon . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠImpact Ġof Ġrole Ġon ĠW ER </s> Spe aker 's Ġrole Ġseems Ġto Ġhave Ġan Ġimpact Ġon ĠW ER : Ġwe Ġobtain Ġan Ġaverage ĠW ER Ġof Ġ30 . 8 % Ġfor Ġthe ĠAnch or Ġspeakers Ġand Ġ42 . 23 % Ġfor Ġthe ĠPun ct ual Ġspeakers . ĠThis Ġdifference Ġis Ġstatistically Ġsignificant Ġwith Ġa Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 14 } $ Ġ( med ( A ) Ġ= Ġ21 %; Ġmed ( P ) Ġ= Ġ31 %; ĠU Ġ= Ġ540 , 430 ; Ġp - value Ġ< Ġ0 . 001 ) Ġ. </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠRole Ġand Ġgender Ġinteraction </s> Figure ĠFIG REF 25 Ġpresents Ġthe ĠW ER Ġdistribution Ġ( WER Ġbeing Ġobtained Ġfor Ġeach Ġspeaker Ġin Ġa Ġshow Ġoccurrence ) Ġaccording Ġto Ġthe Ġspeaker 's Ġrole Ġand Ġgender . ĠIt Ġis Ġworth Ġnoticing Ġthat Ġthe Ġgender Ġdifference Ġis Ġonly Ġsignificant Ġwithin Ġthe ĠPun ct ual Ġspeakers Ġgroup . ĠThe Ġaverage ĠW ER Ġis Ġof Ġ49 . 04 % Ġfor Ġthe Ġwomen Ġand Ġ38 . 56 % Ġfor Ġthe Ġmen Ġwith Ġa Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 6 } $ Ġ( med ( F ) Ġ= Ġ39 %; Ġmed ( M ) Ġ= Ġ29 %; ĠU Ġ= Ġ251 , 450 ; Ġp - value Ġ< Ġ0 . 001 ), Ġwhereas Ġit Ġis Ġjust Ġa Ġtrend Ġbetween Ġmale Ġand Ġfemale ĠAnch ors Ġ( med ( F ) Ġ= Ġ21 %; Ġmed ( M ) Ġ= Ġ21 %; ĠU Ġ= Ġ116 , 230 ; Ġp - value Ġ= Ġ0 . 173 ). ĠThis Ġcould Ġbe Ġexplained Ġby Ġthe Ġquantity Ġof Ġdata Ġavailable Ġper Ġspeaker . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠSpeech Ġtype Ġas Ġa Ġthird Ġentangled Ġfactor ? </s> In Ġorder Ġto Ġtry Ġto Ġexplain Ġthe Ġobserved Ġvariation Ġin Ġour Ġresults Ġdepending Ġon Ġshows Ġand Ġgender Ġ( Figure ĠFIG REF 21 ), Ġwe Ġadd Ġthe Ġnotion Ġof Ġspeech Ġtype Ġto Ġshed Ġsome Ġlight Ġon Ġour Ġresults . ĠB IB REF 22 Ġand ĠB IB REF 24 Ġsuggested Ġthat Ġthe Ġspeaker Ġprofessionalism , Ġassociated Ġwith Ġclear Ġand Ġhyper - art ic ulated Ġspeech Ġcould Ġbe Ġan Ġexplaining Ġfactor Ġfor Ġbetter Ġperformance . </s> Based Ġon Ġour Ġcategor ization Ġin Ġprepared Ġspeech Ġ( mostly Ġnews Ġreports ) Ġand Ġspontaneous Ġspeech Ġ( mostly Ġdebates Ġand Ġentertainment Ġshows ), Ġwe Ġcross Ġthis Ġparameter Ġin Ġour Ġperformance Ġanalysis . ĠAs Ġshown Ġon ĠFigure ĠFIG REF 26 , Ġthese Ġresults Ġconfirm Ġthe Ġinherent Ġchallenge Ġof Ġspontaneous Ġspeech Ġcompared Ġto Ġprepared Ġspeech . ĠW ER Ġscores Ġare Ġsimilar Ġbetween Ġmen Ġand Ġwomen Ġwhen Ġconsidering Ġprepared Ġspeech Ġ( med ( F ) Ġ= Ġ18 %; Ġmed ( M ) Ġ= Ġ21 %; ĠU Ġ= Ġ217 , 160 ; Ġp - value Ġ= Ġ0 . 005 ) Ġwhereas Ġthey Ġare Ġworse Ġfor Ġwomen Ġ( 61 . 29 %) Ġthan Ġfor Ġmen Ġ( 46 . 51 %) Ġwith Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 14 } $ Ġfor Ġthe Ġspontaneous Ġspeech Ġtype Ġ( med ( F ) Ġ= Ġ61 %; Ġmed ( M ) Ġ= Ġ37 %; ĠU Ġ= Ġ153 , 580 ; Ġp - value Ġ< Ġ0 . 001 ). </s> Discussion </s> We Ġfind Ġa Ġclear Ġdisparity Ġin Ġterms Ġof Ġwomen Ġpresence Ġand Ġspeech Ġquantity Ġin ĠFrench Ġmedia . ĠOur Ġdata Ġbeing Ġrecorded Ġbetween Ġ1998 Ġand Ġ2013 , Ġwe Ġcan Ġexpect Ġthis Ġdisparity Ġto Ġbe Ġsmaller Ġon Ġmore Ġrecent Ġbroadcast Ġrecordings , Ġespecially Ġsince Ġthe ĠFrench Ġgovernment Ġdisplays Ġefforts Ġtoward Ġparity Ġin Ġmedia Ġrepresentation . ĠOne Ġcan Ġalso Ġargue Ġthat Ġeven Ġif Ġour Ġanalysis Ġwas Ġconducted Ġon Ġa Ġlarge Ġamount Ġof Ġdata Ġit Ġdoes Ġnot Ġreach Ġthe Ġexhaust iveness Ġof Ġlarge - scale Ġstudies Ġsuch Ġas Ġthe Ġone Ġof ĠB IB REF 2 . ĠNonetheless Ġit Ġdoes Ġnot Ġaffect Ġthe Ġrelevance Ġof Ġour Ġfindings , Ġbecause Ġif Ġreal - world Ġgender Ġrepresentation Ġmight Ġbe Ġmore Ġbalanced Ġtoday , Ġthese Ġcorpor a Ġare Ġstill Ġused Ġas Ġtraining Ġdata Ġfor ĠAI Ġsystems . </s> The Ġperformance Ġdifference Ġacross Ġgender Ġwe Ġobserved Ġcorrobor ates Ġ( on Ġa Ġlarger Ġquantity Ġand Ġvariety Ġof Ġlanguage Ġdata Ġproduced Ġby Ġmore Ġthan Ġ2400 Ġspeakers ) Ġthe Ġresults Ġobtained Ġby ĠB IB REF 24 Ġon Ġisolated Ġwords Ġrecognition . ĠHowever Ġthe Ġfollowing Ġstudy Ġon Ġread Ġspeech Ġdoes Ġnot Ġreplicate Ġthese Ġresults . ĠYet Ġa Ġperformance Ġdegradation Ġis Ġobserved Ġacross Ġdialect Ġand Ġrace ĠB IB REF 8 . ĠB IB REF 22 Ġfound Ġlower ĠW ER Ġfor Ġwomen Ġthan Ġmen Ġon Ġbroadcast Ġnews Ġand Ġconvers ational Ġtelephone Ġspeech Ġfor Ġboth ĠEnglish Ġand ĠFrench . ĠThe Ġauthors Ġsuggest Ġthat Ġgender Ġstereotypes Ġassociated Ġwith Ġwomen Ġrole Ġin Ġeducation Ġand Ġlanguage Ġacquisition Ġinduce Ġa Ġmore Ġnormative Ġel oc ution . ĠWe Ġobserved Ġthat Ġthe Ġhigher Ġthe Ġdegree Ġof Ġnorm ativity Ġof Ġspeech Ġthe Ġsmaller Ġthe Ġgender Ġdifference . ĠNo Ġsignificant Ġgender Ġbias Ġis Ġobserved Ġfor Ġprepared Ġspeech Ġnor Ġwithin Ġthe ĠAnch or Ġcategory . ĠEven Ġif Ġwe Ġdo Ġnot Ġfind Ġsimilar Ġresults Ġwith Ġlower ĠW ER Ġfor Ġwomen Ġthan Ġmen , Ġwe Ġobtained Ġa Ġmedian ĠW ER Ġsmaller Ġfor Ġwomen Ġon Ġprepared Ġspeech Ġand Ġequal Ġto Ġthe Ġmale Ġmedian ĠW ER Ġfor Ġthe ĠAnch or Ġspeakers . </s> Another Ġexplanation Ġcould Ġbe Ġthe Ġuse Ġof Ġadaptation Ġwithin Ġthe Ġpipeline . ĠMost Ġbroadcast Ġprograms Ġtranscription Ġsystems Ġhave Ġa Ġspeaker Ġadaptation Ġstep Ġwithin Ġtheir Ġdecoding Ġpipeline , Ġwhich Ġis Ġthe Ġcase Ġfor Ġour Ġsystem . ĠAn ĠAnch or Ġspeaker Ġintervening Ġmore Ġoften Ġwould Ġhave Ġa Ġlarger Ġquantity Ġof Ġdata Ġto Ġrealize Ġsuch Ġadaptation Ġof Ġthe Ġacoustic Ġmodel . ĠOn Ġthe Ġcontrary , ĠPun ct ual Ġspeakers Ġwho Ġappear Ġscarcely Ġin Ġthe Ġdata Ġare Ġnot Ġprovided Ġwith Ġthe Ġsame Ġamount Ġof Ġadaptation Ġdata . ĠHence Ġwe Ġcan Ġhypothes ize Ġthat Ġgender Ġperformance Ġdifference Ġobserved Ġfor ĠPun ct ual Ġspeakers Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġfemale Ġspeech Ġis Ġfurther Ġfrom Ġthe Ġ( initial Ġnon - adapt ed ) Ġacoustic Ġmodel Ġas Ġit Ġwas Ġtrained Ġon Ġun balanced Ġdata Ġ( as Ġshown Ġin ĠTable Ġ). ĠConsidering Ġthat ĠPun ct ual Ġspeakers Ġrepresent Ġ92 . 78 % Ġof Ġthe Ġspeakers , Ġthis Ġexplains Ġwhy Ġgender Ġdifference Ġis Ġsignificant Ġover Ġour Ġentire Ġdata Ġset . ĠA Ġway Ġto Ġconfirm Ġour Ġhypothesis Ġwould Ġbe Ġto Ġreproduce Ġour Ġanalysis Ġon ĠW ER Ġvalues Ġobtained Ġwithout Ġusing Ġspeaker Ġadapted Ġfeatures Ġat Ġthe Ġdecoding Ġstep . </s> When Ġdecoding Ġprepared Ġspeech Ġ( hen ce Ġsimilar Ġto Ġthe Ġtraining Ġdata ), Ġno Ġsignificant Ġdifference Ġis Ġfound Ġin ĠW ER Ġbetween Ġmen Ġand Ġwomen , Ġrevealing Ġthat Ġthe Ġspeaker Ġadaptation Ġstep Ġcould Ġbe Ġsufficient Ġto Ġreach Ġsame Ġperformance Ġfor Ġboth Ġgenders . ĠBut Ġwhen Ġdecoding Ġmore Ġspontaneous Ġspeech , Ġthere Ġis Ġa Ġmismatch Ġwith Ġthe Ġinitial Ġacoustic Ġmodel Ġ( trained Ġon Ġprepared Ġspeech ). ĠConsequently , Ġthe Ġspeaker Ġadaptation Ġstep Ġmight Ġnot Ġbe Ġenough Ġto Ġrecover Ġgood ĠAS R Ġperformance , Ġespecially Ġfor Ġwomen Ġfor Ġwhom Ġless Ġadaptation Ġdata Ġis Ġavailable Ġ( see ĠSection Ġ4 . 2 . 3 ). </s> Conclusion </s> This Ġpaper Ġhas Ġinvestigated Ġgender Ġbias Ġin ĠAS R Ġperformance Ġthrough Ġthe Ġfollowing Ġresearch Ġquestions : Ġi ) Ġwhat Ġis Ġthe Ġproportion Ġof Ġmen Ġand Ġwomen Ġin ĠFrench Ġradio Ġand ĠTV Ġmedia Ġdata Ġ? Ġii ) Ġwhat Ġis Ġthe Ġimpact Ġof Ġthe Ġobserved Ġdisparity Ġon ĠAS R Ġperformance Ġ? Ġiii ) Ġis Ġthis Ġas Ġsimple Ġas Ġa Ġproblem Ġof Ġgender Ġproportion Ġin Ġthe Ġtraining Ġdata Ġor Ġare Ġother Ġfactors Ġentangled Ġ? ĠOur Ġcontributions Ġare Ġthe Ġfollowing : </s> Desc ript ive Ġanalysis Ġof Ġthe Ġbroadcast Ġdata Ġused Ġto Ġtrain Ġour ĠAS R Ġsystem Ġconfirms Ġthe Ġalready Ġknown Ġdisparity , Ġwhere Ġ65 % Ġof Ġthe Ġspeakers Ġare Ġmen , Ġspeaking Ġmore Ġthan Ġ75 % Ġof Ġthe Ġtime . </s> When Ġinvestigating ĠW ER Ġscores Ġaccording Ġto Ġgender , Ġspeaker 's Ġrole Ġand Ġspeech Ġtype , Ġhuge Ġvariations Ġare Ġobserved . ĠWe Ġconclude Ġthat Ġgender Ġis Ġclearly Ġa Ġfactor Ġof Ġvariation Ġin ĠAS R Ġperformance , Ġwith Ġa ĠW ER Ġincrease Ġof Ġ24 % Ġfor Ġwomen Ġcompared Ġto Ġmen , Ġexhibiting Ġa Ġclear Ġgender Ġbias . </s> Gender Ġbias Ġvaries Ġacross Ġspeaker 's Ġrole Ġand Ġspeech Ġspont ane ity Ġlevel . ĠPerformance Ġfor ĠPun ct ual Ġspeakers Ġrespectively Ġspontaneous Ġspeech Ġseems Ġto Ġreinforce Ġthis Ġgender Ġbias Ġwith Ġa ĠW ER Ġincrease Ġof Ġ27 . 2 % Ġrespectively Ġ31 . 8 % Ġbetween Ġmale Ġand Ġfemale Ġspeakers . </s> We Ġfound Ġthat Ġan ĠAS R Ġsystem Ġtrained Ġon Ġun balanced Ġdata Ġregarding Ġgender Ġproduces Ġgender Ġbias Ġperformance . ĠTherefore , Ġin Ġorder Ġto Ġcreate Ġfair Ġsystems Ġit Ġis Ġnecessary Ġto Ġtake Ġinto Ġaccount Ġthe Ġrepresentation Ġproblems Ġin Ġsociety Ġthat Ġare Ġgoing Ġto Ġbe Ġencaps ulated Ġin Ġthe Ġdata . ĠUnderstanding Ġhow Ġwomen Ġunder - represent ation Ġin Ġbroadcast Ġdata Ġcan Ġlead Ġto Ġbias Ġin ĠAS R Ġperformances Ġis Ġthe Ġkey Ġto Ġprevent Ġre - im plement ing Ġand Ġreinforcing Ġdiscrimination Ġalready Ġexisting Ġin Ġour Ġsocieties . ĠThis Ġis Ġin Ġline Ġwith Ġthe Ġconcept Ġof ĠâĢ ľ Fair ness Ġby ĠDesign " Ġproposed Ġby ĠB IB REF 31 . </s> Gender , Ġrace , Ġreligion , Ġnationality Ġare Ġall Ġcharacteristics Ġthat Ġwe Ġdeem Ġunfair Ġto Ġclassify Ġon , Ġand Ġthese Ġethical Ġstand points Ġneeds Ġto Ġbe Ġtaken Ġinto Ġaccount Ġin Ġsystems ' Ġdesign . ĠCharacter istics Ġthat Ġare Ġnot Ġconsidered Ġas Ġrelevant Ġin Ġa Ġgiven Ġtask Ġcan Ġbe Ġencaps ulated Ġin Ġdata Ġnonetheless , Ġand Ġlead Ġto Ġbias Ġperformance . ĠBeing Ġaware Ġof Ġthe Ġdemographic Ġske ws Ġour Ġdata Ġset Ġmight Ġcontain Ġis Ġa Ġfirst Ġstep Ġto Ġtrack Ġthe Ġlife Ġcycle Ġof Ġa Ġtraining Ġdata Ġset Ġand Ġa Ġnecessary Ġstep Ġto Ġcontrol Ġthe Ġtools Ġwe Ġdevelop .
Introduction In recent years, gender has become a hot topic within the political, societal and research spheres. Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0. In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis. The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%. Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2. Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community. Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3. Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies. This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8. In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have. We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community. Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system. From gender representation in data to gender bias in AI ::: On the importance of data The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available. The quality of a system now depends mostly on the quality and quantity of the data it has been trained on. If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource. Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development. This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9. Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems. The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s. The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time. Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used. Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation. For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date. From gender representation in data to gender bias in AI ::: From data to bias The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address. Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7). The impact of the training data used within these deep-learning algorithms is therefore questioned. Bias can be found at different levels as pointed out by BIBREF19. BIBREF20 defines bias as a skew that produces a type of harm. She distinguishes two types of harms that are allocation harm and representation harm. The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes. Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on. Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns. The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces. The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias. As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative. To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data." (p.668). Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached. BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus. They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education. The same trend was observed by BIBREF23. More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8. However race and dialect bias were found. General American speakers and white speakers had the lowest error rate for both systems. If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers. Gender bias in ASR technology is still an open research question as no clear answer has been reached so far. It seems that many parameters are to take into account to achieve a general agreement. As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism. We then question the impact of such a representation on a ASR system trained on these data. BIBREF25 Methodology This section is organized as follows: we first present the data we are working on. In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role. The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance. Methodology ::: Data presentation Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition. Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech. Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech. Training data by show, medium and speech type is summarized in Table and evaluation data in Table . Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set). Methodology ::: Methodology for descriptive analysis of gender representation in training data We first describe the gender representation in training data. Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16). Each speech turn was mapped to its speaker in order to associate it with a gender. As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions. It is clear that a small number of speakers is responsible for a large number of speech turns. Most of these speakers are journalists, politicians, presenters and such, who are representative of a show. Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role. Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show. Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length. We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show. The same process is applied to identify speakers talking for a long period from those who do not. We end up with two salient roles called Anchors and Punctual speakers: the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction; the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time. These roles are defined at the show level. They could be roughly assimilated to the categorization “host/guest” in radio and TV shows. Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people". The concept of speaker's role makes sense at both sociological and technical levels. An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category. Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system The ASR system used in this work is described in BIBREF25. It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe. The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table . Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows. A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27. The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K). It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode. It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26. Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation Word Error Rate (WER) is a common metric to evaluate ASR performance. It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription. As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level. Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on. Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show. This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0.001) BIBREF30. The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data. Results ::: Descriptive analysis of gender representation in training data ::: Gender representation As expected, we observe a disparity in terms of gender representation in our data (see Table ). Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. However, it is worth noticing that women account for only 22.57% of the total speech time, which leads us to conclude that women also speak less than men. Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3.79%), they nevertheless concentrate 35.71 % of the total speech time. Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction When crossing both parameters, we can observe that the gender distribution is not constant throughout roles. Women represent 29.47% of the speakers within the Anchor category, even less than among the Punctual speakers. Their percentage of speech is also smaller. When calculating the average speech time uttered by a female Anchor, we obtain a value of 15.9 min against 25.2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more. This confirms the existence of gender disparities within French media. It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours. Our study shows that they are also less present in important roles. These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings. Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference. The higher the WER, the poorer the system performance. Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42.9% for women and 34.3% for men. This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0.001). However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21. For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume. The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon. Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER Speaker's role seems to have an impact on WER: we obtain an average WER of 30.8% for the Anchor speakers and 42.23% for the Punctual speakers. This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0.001) . Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender. It is worth noticing that the gender difference is only significant within the Punctual speakers group. The average WER is of 49.04% for the women and 38.56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0.001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0.173). This could be explained by the quantity of data available per speaker. Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor? In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results. BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance. Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis. As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech. WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0.005) whereas they are worse for women (61.29%) than for men (46.51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0.001). Discussion We find a clear disparity in terms of women presence and speech quantity in French media. Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation. One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2. Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems. The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition. However the following study on read speech does not replicate these results. Yet a performance degradation is observed across dialect and race BIBREF8. BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French. The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution. We observed that the higher the degree of normativity of speech the smaller the gender difference. No significant gender bias is observed for prepared speech nor within the Anchor category. Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers. Another explanation could be the use of adaptation within the pipeline. Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system. An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model. On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data. Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table ). Considering that Punctual speakers represent 92.78% of the speakers, this explains why gender difference is significant over our entire data set. A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step. When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders. But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech). Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4.2.3). Conclusion This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following: Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time. When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed. We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias. Gender bias varies across speaker's role and speech spontaneity level. Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27.2% respectively 31.8% between male and female speakers. We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of “Fairness by Design" proposed by BIBREF31. Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design. Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance. Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop.
What is the goal of investigating NLP gender bias specifically in the news broadcast domain and Anchor role?
f85f2a532e7e700d9f8f9c09cd08d4e47b87bdd3
What Ġis Ġthe Ġgoal Ġof Ġinvestigating ĠN LP Ġgender Ġbias Ġspecifically Ġin Ġthe Ġnews Ġbroadcast Ġdomain Ġand ĠAnch or Ġrole ?
create Ġfair Ġsystems
<s> What Ġis Ġthe Ġgoal Ġof Ġinvestigating ĠN LP Ġgender Ġbias Ġspecifically Ġin Ġthe Ġnews Ġbroadcast Ġdomain Ġand ĠAnch or Ġrole ? </s> Introduction </s> In Ġrecent Ġyears , Ġgender Ġhas Ġbecome Ġa Ġhot Ġtopic Ġwithin Ġthe Ġpolitical , Ġsocietal Ġand Ġresearch Ġspheres . ĠNumerous Ġstudies Ġhave Ġbeen Ġconducted Ġin Ġorder Ġto Ġevaluate Ġthe Ġpresence Ġof Ġwomen Ġin Ġmedia , Ġoften Ġrevealing Ġtheir Ġunder - represent ation , Ġsuch Ġas Ġthe ĠGlobal ĠMedia ĠMonitoring ĠProject ĠB IB REF 0 . ĠIn Ġthe ĠFrench Ġcontext , Ġthe ĠC SA ĠB IB REF 1 Ġproduces Ġa Ġreport Ġon Ġgender Ġrepresentation Ġin Ġmedia Ġon Ġa Ġyearly Ġbasis . ĠThe Ġ2017 Ġreport Ġshows Ġthat Ġwomen Ġrepresent Ġ40 % Ġof ĠFrench Ġmedia Ġspeakers , Ġwith Ġa Ġsignificant Ġdrop Ġduring Ġhigh - aud ience Ġhours Ġ( 6 : 00 - 8 : 00 pm ) Ġreaching Ġa Ġvalue Ġof Ġonly Ġ29 %. ĠAnother Ġlarge Ġscale Ġstudy Ġconfirmed Ġthis Ġtrend Ġwith Ġan Ġautomatic Ġanalysis Ġof Ġgender Ġin ĠFrench Ġaud iov is ual s Ġstreams , Ġhighlighting Ġa Ġhuge Ġvariation Ġacross Ġtype Ġof Ġshows ĠB IB REF 2 . </s> Besides Ġthe Ġsocial Ġimpact Ġof Ġgender Ġrepresentation , Ġbroadcast Ġrecordings Ġare Ġalso Ġa Ġvaluable Ġsource Ġof Ġdata Ġfor Ġthe Ġspeech Ġprocessing Ġcommunity . ĠIndeed , Ġautomatic Ġspeech Ġrecognition Ġ( AS R ) Ġsystems Ġrequire Ġlarge Ġamount Ġof Ġannot ated Ġspeech Ġdata Ġto Ġbe Ġefficiently Ġtrained , Ġwhich Ġleaves Ġus Ġfacing Ġthe Ġemerging Ġconcern Ġabout Ġthe Ġfact Ġthat Ġ" AI Ġartifacts Ġtend Ġto Ġreflect Ġthe Ġgoals , Ġknowledge Ġand Ġexperience Ġof Ġtheir Ġcreators " ĠB IB REF 3 . ĠSince Ġwe Ġknow Ġthat Ġwomen Ġare Ġunder - represented Ġin Ġmedia Ġand Ġthat Ġthe ĠAI Ġdiscipline Ġhas Ġretained Ġa Ġmale - oriented Ġfocus ĠB IB REF 4 , Ġwe Ġcan Ġlegitimately Ġwonder Ġabout Ġthe Ġimpact Ġof Ġusing Ġsuch Ġdata Ġas Ġa Ġtraining Ġset Ġfor ĠAS R Ġtechnologies . ĠThis Ġconcern Ġis Ġstrengthened Ġby Ġthe Ġrecent Ġworks Ġuncover ing Ġgender Ġbias Ġin Ġseveral Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtools Ġsuch Ġas ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 . </s> In Ġthis Ġpaper , Ġwe Ġfirst Ġhighlight Ġthe Ġimportance Ġof ĠTV Ġand Ġradio Ġbroadcast Ġas Ġa Ġsource Ġof Ġdata Ġfor ĠAS R , Ġand Ġthe Ġpotential Ġimpact Ġit Ġcan Ġhave . ĠWe Ġthen Ġperform Ġa Ġstatistical Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġa Ġdata Ġset Ġcomposed Ġof Ġfour Ġstate - of - the - art Ġcorpor a Ġof ĠFrench Ġbroadcast , Ġwidely Ġused Ġwithin Ġthe Ġspeech Ġcommunity . ĠFinally Ġwe Ġquestion Ġthe Ġimpact Ġof Ġsuch Ġa Ġrepresentation Ġon Ġthe Ġsystems Ġdeveloped Ġon Ġthis Ġdata , Ġthrough Ġthe Ġperspective Ġof Ġan ĠAS R Ġsystem . </s> From Ġgender Ġrepresentation Ġin Ġdata Ġto Ġgender Ġbias Ġin ĠAI Ġ: :: ĠOn Ġthe Ġimportance Ġof Ġdata </s> The Ġever Ġgrowing Ġuse Ġof Ġmachine Ġlearning Ġin Ġscience Ġhas Ġbeen Ġenabled Ġby Ġseveral Ġprogresses Ġamong Ġwhich Ġthe Ġexponential Ġgrowth Ġof Ġdata Ġavailable . ĠThe Ġquality Ġof Ġa Ġsystem Ġnow Ġdepends Ġmostly Ġon Ġthe Ġquality Ġand Ġquantity Ġof Ġthe Ġdata Ġit Ġhas Ġbeen Ġtrained Ġon . ĠIf Ġit Ġdoes Ġnot Ġdiscard Ġthe Ġimportance Ġof Ġan Ġappropriate Ġarchitecture , Ġit Ġreaff irms Ġthe Ġfact Ġthat Ġrich Ġand Ġlarge Ġcorpor a Ġare Ġa Ġvaluable Ġresource . ĠCorpor a Ġare Ġresearch Ġcontributions Ġwhich Ġdo Ġnot Ġonly Ġallow Ġto Ġsave Ġand Ġobserve Ġcertain Ġphenomena Ġor Ġvalidate Ġa Ġhypothesis Ġor Ġmodel , Ġbut Ġare Ġalso Ġa Ġmandatory Ġpart Ġof Ġthe Ġtechnology Ġdevelopment . ĠThis Ġtrend Ġis Ġnotably Ġobservable Ġwithin Ġthe ĠN LP Ġfield , Ġwhere Ġindustrial Ġtechnologies , Ġsuch Ġas ĠApple , ĠAmazon Ġor ĠGoogle Ġvocal Ġassistants Ġnow Ġreach Ġhigh Ġperformance Ġlevel Ġpartly Ġdue Ġto Ġthe Ġamount Ġof Ġdata Ġpossessed Ġby Ġthese Ġcompanies ĠB IB REF 9 . </s> Sur prisingly , Ġas Ġdata Ġis Ġsaid Ġto Ġbe ĠâĢ ľ the Ġnew Ġoil ", Ġfew Ġdata Ġsets Ġare Ġavailable Ġfor ĠAS R Ġsystems . ĠThe Ġbest Ġknown Ġare Ġcorpor a Ġlike ĠTIM IT ĠB IB REF 10 , ĠSwitch board ĠB IB REF 11 Ġor ĠFisher ĠB IB REF 12 Ġwhich Ġdate Ġback Ġto Ġthe Ġearly Ġ1990 s . ĠThe Ġsc arc eness Ġof Ġavailable Ġcorpor a Ġis Ġjustified Ġby Ġthe Ġfact Ġthat Ġgathering Ġand Ġannot ating Ġaudio Ġdata Ġis Ġcostly Ġboth Ġin Ġterms Ġof Ġmoney Ġand Ġtime . ĠTelephone Ġconversations Ġand Ġbroadcast Ġrecordings Ġhave Ġbeen Ġthe Ġprimary Ġsource Ġof Ġspontaneous Ġspeech Ġused . ĠOut Ġof Ġall Ġthe Ġ130 Ġaudio Ġresources Ġproposed Ġby ĠL DC Ġto Ġtrain Ġautomatic Ġspeech Ġrecognition Ġsystems Ġin ĠEnglish , Ġapproximately Ġ14 % Ġof Ġthem Ġare Ġbased Ġon Ġbroadcast Ġnews Ġand Ġconversation . ĠFor ĠFrench Ġspeech Ġtechnologies , Ġfour Ġcorpor a Ġcontaining Ġradio Ġand ĠTV Ġbroadcast Ġare Ġthe Ġmost Ġwidely Ġused : ĠEST ER 1 ĠB IB REF 13 , ĠEST ER 2 ĠB IB REF 14 , ĠET APE ĠB IB REF 15 Ġand ĠRE P ERE ĠB IB REF 16 . ĠThese Ġfour Ġcorpor a Ġhave Ġbeen Ġbuilt Ġalongside Ġevaluation Ġcampaigns Ġand Ġare Ġstill , Ġto Ġour Ġknowledge , Ġthe Ġlargest ĠFrench Ġones Ġof Ġtheir Ġtype Ġavailable Ġto Ġdate . </s> From Ġgender Ġrepresentation Ġin Ġdata Ġto Ġgender Ġbias Ġin ĠAI Ġ: :: ĠFrom Ġdata Ġto Ġbias </s> The Ġgender Ġissue Ġhas Ġreturned Ġto Ġthe Ġforefront Ġof Ġthe Ġmedia Ġscene Ġin Ġrecent Ġyears Ġand Ġwith Ġthe Ġemergence Ġof ĠAI Ġtechnologies Ġin Ġour Ġdaily Ġlives , Ġgender Ġbias Ġhas Ġbecome Ġa Ġscientific Ġtopic Ġthat Ġresearchers Ġare Ġjust Ġbeginning Ġto Ġaddress . ĠSeveral Ġstudies Ġrevealed Ġthe Ġexistence Ġof Ġgender Ġbias Ġin ĠAI Ġtechnologies Ġsuch Ġas Ġface Ġrecognition Ġ( Gender Sh ades ĠB IB REF 17 ), ĠN LP Ġ( word Ġembed d ings ĠB IB REF 5 Ġand Ġsemantics ĠB IB REF 6 ) Ġand Ġmachine Ġtranslation Ġ( BI B REF 18 , ĠB IB REF 7 ). ĠThe Ġimpact Ġof Ġthe Ġtraining Ġdata Ġused Ġwithin Ġthese Ġdeep - learning Ġalgorithms Ġis Ġtherefore Ġquestioned . </s> B ias Ġcan Ġbe Ġfound Ġat Ġdifferent Ġlevels Ġas Ġpointed Ġout Ġby ĠB IB REF 19 . ĠB IB REF 20 Ġdefines Ġbias Ġas Ġa Ġskew Ġthat Ġproduces Ġa Ġtype Ġof Ġharm . ĠShe Ġdistinguishes Ġtwo Ġtypes Ġof Ġharms Ġthat Ġare Ġallocation Ġharm Ġand Ġrepresentation Ġharm . ĠThe Ġallocation Ġharm Ġoccurs Ġwhen Ġa Ġsystem Ġis Ġperforming Ġbetter Ġor Ġworse Ġfor Ġa Ġcertain Ġgroup Ġwhile Ġrepresent ational Ġharm Ġcontributes Ġto Ġthe Ġperpet uation Ġof Ġstereotypes . ĠBoth Ġtypes Ġof Ġharm Ġare Ġthe Ġresults Ġof Ġbias Ġin Ġmachine Ġlearning Ġthat Ġoften Ġcomes Ġfrom Ġthe Ġdata Ġsystems Ġare Ġtrained Ġon . ĠDis par ities Ġin Ġrepresentation Ġin Ġour Ġsocial Ġstructures Ġis Ġcaptured Ġand Ġreflected Ġby Ġthe Ġtraining Ġdata , Ġthrough Ġstatistical Ġpatterns . ĠThe ĠGender Sh ades Ġstudy Ġis Ġa Ġstriking Ġexample Ġof Ġwhat Ġdata Ġdisparity Ġand Ġlack Ġof Ġrepresentation Ġcan Ġproduce : Ġthe Ġauthors Ġtested Ġseveral Ġgender Ġrecognition Ġmodules Ġused Ġby Ġfacial Ġrecognition Ġtools Ġand Ġfound Ġdifference Ġin Ġerror - rate Ġas Ġhigh Ġas Ġ34 Ġpercentage Ġpoints Ġbetween Ġrecognition Ġof Ġwhite Ġmale Ġand Ġblack Ġfemale Ġfaces . ĠThe Ġscarce Ġpresence Ġof Ġwomen Ġand Ġcolored Ġpeople Ġin Ġtraining Ġset Ġresulted Ġin Ġbias Ġin Ġperformance Ġtowards Ġthese Ġtwo Ġcategories , Ġwith Ġa Ġstrong Ġintersection al Ġbias . ĠAs Ġwritten Ġby ĠB IB REF 21 Ġ" A Ġdata Ġset Ġmay Ġhave Ġmany Ġmillions Ġof Ġpieces Ġof Ġdata , Ġbut Ġthis Ġdoes Ġnot Ġmean Ġit Ġis Ġrandom Ġor Ġrepresentative . ĠTo Ġmake Ġstatistical Ġclaims Ġabout Ġa Ġdata Ġset , Ġwe Ġneed Ġto Ġknow Ġwhere Ġdata Ġis Ġcoming Ġfrom ; Ġit Ġis Ġsimilarly Ġimportant Ġto Ġknow Ġand Ġaccount Ġfor Ġthe Ġweaknesses Ġin Ġthat Ġdata ." Ġ( p . 668 ). </s> Regarding ĠAS R Ġtechnology , Ġlittle Ġwork Ġhas Ġexplored Ġthe Ġpresence Ġof Ġgender Ġbias Ġwithin Ġthe Ġsystems Ġand Ġno Ġconsensus Ġhas Ġbeen Ġreached . ĠB IB REF 22 Ġfound Ġthat Ġspeech Ġrecogn izers Ġperform Ġbetter Ġon Ġfemale Ġvoice Ġon Ġa Ġbroadcast Ġnews Ġand Ġtelephone Ġcorpus . ĠThey Ġproposed Ġseveral Ġexplanations Ġto Ġthis Ġobservation , Ġsuch Ġas Ġthe Ġlarger Ġpresence Ġof Ġnon - professional Ġmale Ġspeech Ġin Ġthe Ġbroadcast Ġdata , Ġimplying Ġa Ġless Ġprepared Ġspeech Ġfor Ġthese Ġspeakers Ġor Ġa Ġmore Ġnormative Ġlanguage Ġand Ġstandard Ġpronunciation Ġfor Ġwomen Ġlinked Ġto Ġthe Ġtraditional Ġrole Ġof Ġwomen Ġin Ġlanguage Ġacquisition Ġand Ġeducation . ĠThe Ġsame Ġtrend Ġwas Ġobserved Ġby ĠB IB REF 23 . ĠMore Ġrecently , ĠB IB REF 24 Ġdiscovered Ġa Ġgender Ġbias Ġwithin ĠYouTube 's Ġautomatic Ġcaption ing Ġsystem Ġbut Ġthis Ġbias Ġwas Ġnot Ġobserved Ġin Ġa Ġsecond Ġstudy Ġevaluating ĠBing ĠSpeech Ġsystem Ġand ĠYouTube ĠAutomatic ĠCapt ions Ġon Ġa Ġlarger Ġdata Ġset ĠB IB REF 8 . ĠHowever Ġrace Ġand Ġdialect Ġbias Ġwere Ġfound . ĠGeneral ĠAmerican Ġspeakers Ġand Ġwhite Ġspeakers Ġhad Ġthe Ġlowest Ġerror Ġrate Ġfor Ġboth Ġsystems . ĠIf Ġthe Ġbetter Ġperformance Ġon ĠGeneral ĠAmerican Ġspeakers Ġcould Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthey Ġare Ġall Ġvoice Ġprofessionals , Ġproducing Ġclear Ġand Ġarticulated Ġspeech , Ġbut Ġno Ġexplanation Ġis Ġprovided Ġfor Ġbiases Ġtowards Ġnon - white Ġspeakers . </s> Gender Ġbias Ġin ĠAS R Ġtechnology Ġis Ġstill Ġan Ġopen Ġresearch Ġquestion Ġas Ġno Ġclear Ġanswer Ġhas Ġbeen Ġreached Ġso Ġfar . ĠIt Ġseems Ġthat Ġmany Ġparameters Ġare Ġto Ġtake Ġinto Ġaccount Ġto Ġachieve Ġa Ġgeneral Ġagreement . ĠAs Ġwe Ġestablished Ġthe Ġimportance Ġof ĠTV Ġand Ġradio Ġbroadcast Ġas Ġa Ġsource Ġof Ġdata Ġfor ĠAS R , Ġand Ġthe Ġpotential Ġimpact Ġit Ġcan Ġhave , Ġthe Ġfollowing Ġcontent Ġof Ġthis Ġpaper Ġis Ġstructured Ġas Ġthis : Ġwe Ġfirst Ġdescribe Ġstatistically Ġthe Ġgender Ġrepresentation Ġof Ġa Ġdata Ġset Ġcomposed Ġof Ġfour Ġstate - of - the - art Ġcorpor a Ġof ĠFrench Ġbroadcast , Ġwidely Ġused Ġwithin Ġthe Ġspeech Ġcommunity , Ġintroducing Ġthe Ġnotion Ġof Ġspeaker 's Ġrole Ġto Ġrefine Ġour Ġanalysis Ġin Ġterms Ġof Ġvoice Ġprofessionalism . ĠWe Ġthen Ġquestion Ġthe Ġimpact Ġof Ġsuch Ġa Ġrepresentation Ġon Ġa ĠAS R Ġsystem Ġtrained Ġon Ġthese Ġdata . ĠB IB REF 25 </s> Method ology </s> This Ġsection Ġis Ġorganized Ġas Ġfollows : Ġwe Ġfirst Ġpresent Ġthe Ġdata Ġwe Ġare Ġworking Ġon . ĠIn Ġa Ġsecond Ġtime Ġwe Ġexplain Ġhow Ġwe Ġproceed Ġto Ġdescribe Ġthe Ġgender Ġrepresentation Ġin Ġour Ġcorpus Ġand Ġintroduce Ġthe Ġnotion Ġof Ġspeaker 's Ġrole . ĠThe Ġthird Ġsubsection Ġintroduces Ġthe ĠAS R Ġsystem Ġand Ġmetrics Ġused Ġto Ġevaluate Ġgender Ġbias Ġin Ġperformance . </s> Method ology Ġ: :: ĠData Ġpresentation </s> Our Ġdata Ġconsists Ġof Ġtwo Ġsets Ġused Ġto Ġtrain Ġand Ġevaluate Ġour Ġautomatic Ġspeech Ġrecognition Ġsystem . ĠFour Ġmajor Ġevaluation Ġcampaigns Ġhave Ġenabled Ġthe Ġcreation Ġof Ġwide Ġcorpor a Ġof ĠFrench Ġbroadcast Ġspeech : ĠEST ER 1 ĠB IB REF 13 , ĠEST ER 2 ĠB IB REF 14 , ĠET APE ĠB IB REF 15 Ġand ĠRE P ERE ĠB IB REF 16 . ĠThese Ġfour Ġcollections Ġcontain Ġradio Ġand / or ĠTV Ġbroadcasts Ġaired Ġbetween Ġ1998 Ġand Ġ2013 Ġwhich Ġare Ġused Ġby Ġmost Ġacademic Ġresearchers Ġin ĠAS R . ĠShow Ġduration Ġvaries Ġbetween Ġ10 min Ġand Ġan Ġhour . ĠAs Ġyears Ġwent Ġby Ġand Ġspeech Ġprocessing Ġresearch Ġwas Ġprogressing , Ġthe Ġdifficulty Ġof Ġthe Ġtasks Ġaugmented Ġand Ġthe Ġcontent Ġof Ġthese Ġevaluation Ġcorpor a Ġchanged . ĠEST ER 1 Ġand ĠEST ER 2 Ġmainly Ġcontain Ġprepared Ġspeech Ġsuch Ġas Ġbroadcast Ġnews , Ġwhereas ĠET APE Ġand ĠRE P ERE Ġconsists Ġalso Ġof Ġdebates Ġand Ġentertainment Ġshows , Ġspontaneous Ġspeech Ġintroducing Ġmore Ġdifficulty Ġin Ġits Ġrecognition . </s> Our Ġtraining Ġset Ġcontains Ġ27 , 08 5 Ġspeech Ġutter ances Ġproduced Ġby Ġ2 , 506 Ġspeakers , Ġaccounting Ġfor Ġapproximately Ġ100 Ġhours Ġof Ġspeech . ĠOur Ġevaluation Ġset Ġcontains Ġ74 , 0 64 Ġspeech Ġutter ances Ġproduced Ġby Ġ1 , 268 Ġspeakers Ġfor Ġa Ġtotal Ġof Ġ70 Ġhours Ġof Ġspeech . ĠTraining Ġdata Ġby Ġshow , Ġmedium Ġand Ġspeech Ġtype Ġis Ġsummarized Ġin ĠTable Ġand Ġevaluation Ġdata Ġin ĠTable Ġ. ĠEvaluation Ġdata Ġhas Ġa Ġhigher Ġvariety Ġof Ġshows Ġwith Ġboth Ġprepared Ġ( P ) Ġand Ġspontaneous Ġ( S ) Ġspeech Ġtype Ġ( ac cent ed Ġspeech Ġfrom ĠAfrican Ġradio Ġbroadcast Ġis Ġalso Ġincluded Ġin Ġthe Ġevaluation Ġset ). </s> Method ology Ġ: :: ĠMethod ology Ġfor Ġdescriptive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata </s> We Ġfirst Ġdescribe Ġthe Ġgender Ġrepresentation Ġin Ġtraining Ġdata . ĠGender Ġrepresentation Ġis Ġmeasured Ġin Ġterms Ġof Ġnumber Ġof Ġspeakers , Ġnumber Ġof Ġutter ances Ġ( or Ġspeech Ġturns ), Ġand Ġturn Ġlengths Ġ( desc ript ive Ġstatistics Ġare Ġgiven Ġin ĠSection ĠSEC REF 16 ). ĠEach Ġspeech Ġturn Ġwas Ġmapped Ġto Ġits Ġspeaker Ġin Ġorder Ġto Ġassociate Ġit Ġwith Ġa Ġgender . </s> As Ġpointed Ġout Ġby Ġthe ĠC SA Ġreport ĠB IB REF 1 , Ġwomen Ġpresence Ġtends Ġto Ġbe Ġmarginal Ġwithin Ġthe Ġhigh - aud ience Ġhours , Ġshowing Ġthat Ġwomen Ġare Ġrepresented Ġbut Ġless Ġthan Ġmen Ġand Ġwithin Ġcertain Ġgiven Ġconditions . ĠIt Ġis Ġclear Ġthat Ġa Ġsmall Ġnumber Ġof Ġspeakers Ġis Ġresponsible Ġfor Ġa Ġlarge Ġnumber Ġof Ġspeech Ġturns . ĠMost Ġof Ġthese Ġspeakers Ġare Ġjournalists , Ġpoliticians , Ġpresent ers Ġand Ġsuch , Ġwho Ġare Ġrepresentative Ġof Ġa Ġshow . ĠTherefore , Ġwe Ġintroduce Ġthe Ġnotion Ġof Ġspeaker 's Ġrole Ġto Ġrefine Ġour Ġexploration Ġof Ġgender Ġdisparity , Ġfollowing Ġstudies Ġwhich Ġquant ified Ġwomen 's Ġpresence Ġin Ġterms Ġof Ġrole . ĠWithin Ġour Ġwork , Ġwe Ġdefine Ġthe Ġnotion Ġof Ġspeaker Ġrole Ġby Ġtwo Ġcriteria Ġspecifying Ġthe Ġspeaker 's Ġon - air Ġpresence , Ġnamely Ġthe Ġnumber Ġof Ġspeech Ġturns Ġand Ġthe Ġcumulative Ġduration Ġof Ġhis Ġor Ġher Ġspeaking Ġtime Ġin Ġa Ġshow . ĠBased Ġon Ġthe Ġavailable Ġspeech Ġtranscript ions Ġand Ġmeta - data , Ġwe Ġcompute Ġfor Ġeach Ġspeaker Ġthe Ġnumber Ġof Ġspeech Ġturns Ġuttered Ġas Ġwell Ġas Ġtheir Ġtotal Ġlength . ĠWe Ġthen Ġuse Ġthe Ġfollowing Ġcriteria Ġto Ġdefine Ġspeaker 's Ġrole : Ġa Ġspeaker Ġis Ġconsidered Ġas Ġspeaking Ġoften Ġ( respect ively Ġseldom ) Ġif Ġhe / she Ġaccum ulates Ġa Ġtotal Ġof Ġturns Ġhigher Ġ( respect ively Ġlower ) Ġthan Ġ1 % Ġof Ġthe Ġtotal Ġnumber Ġof Ġspeech Ġturns Ġin Ġa Ġgiven Ġshow . ĠThe Ġsame Ġprocess Ġis Ġapplied Ġto Ġidentify Ġspeakers Ġtalking Ġfor Ġa Ġlong Ġperiod Ġfrom Ġthose Ġwho Ġdo Ġnot . ĠWe Ġend Ġup Ġwith Ġtwo Ġsalient Ġroles Ġcalled ĠAnch ors Ġand ĠPun ct ual Ġspeakers : </s> the ĠAnch or Ġspeakers Ġ( A ) Ġare Ġabove Ġthe Ġthreshold Ġof Ġ1 % Ġfor Ġboth Ġcriteria , Ġmeaning Ġthey Ġare Ġintervening Ġoften Ġand Ġfor Ġa Ġlong Ġtime Ġthus Ġholding Ġan Ġimportant Ġplace Ġin Ġinteraction ; </s> the ĠPun ct ual Ġspeakers Ġ( PS ) Ġon Ġthe Ġcontrary Ġare Ġbelow Ġthe Ġthreshold Ġof Ġ1 % Ġfor Ġboth Ġthe Ġtotal Ġnumber Ġof Ġturns Ġand Ġthe Ġtotal Ġspeech Ġtime . </s> These Ġroles Ġare Ġdefined Ġat Ġthe Ġshow Ġlevel . ĠThey Ġcould Ġbe Ġroughly Ġassimil ated Ġto Ġthe Ġcategor ization ĠâĢ ľ host / gu est âĢ Ŀ Ġin Ġradio Ġand ĠTV Ġshows . ĠAnch ors Ġcould Ġbe Ġdescribed Ġas Ġprofessional Ġspeakers , Ġproducing Ġmostly Ġprepared Ġspeech , Ġwhereas ĠPun ct ual Ġspeakers Ġare Ġmore Ġlikely Ġto Ġbe ĠâĢ ľ every day Ġpeople ". ĠThe Ġconcept Ġof Ġspeaker 's Ġrole Ġmakes Ġsense Ġat Ġboth Ġsoc iological Ġand Ġtechnical Ġlevels . ĠAn ĠAnch or Ġspeaker Ġis Ġmore Ġlikely Ġto Ġbe Ġknown Ġfrom Ġthe Ġaudience Ġ( soc iety ), Ġbut Ġhe Ġor Ġshe Ġwill Ġalso Ġlikely Ġhave Ġa Ġprofessional Ġ( clear ) Ġway Ġof Ġspeaking Ġ( as Ġmentioned Ġby ĠB IB REF 22 Ġand ĠB IB REF 8 ), Ġas Ġwell Ġas Ġa Ġhigh Ġnumber Ġof Ġutter ances , Ġaugment ing Ġthe Ġamount Ġof Ġdata Ġavailable Ġfor Ġa Ġgiven Ġgender Ġcategory . </s> Method ology Ġ: :: ĠGender Ġbias Ġevaluation Ġprocedure Ġof Ġan ĠAS R Ġsystem Ġperformance Ġ: :: ĠAS R Ġsystem </s> The ĠAS R Ġsystem Ġused Ġin Ġthis Ġwork Ġis Ġdescribed Ġin ĠB IB REF 25 . ĠIt Ġuses Ġthe ĠK AL DI Ġtool kit ĠB IB REF 26 , Ġfollowing Ġa Ġstandard ĠK aldi Ġrecipe . ĠThe Ġacoustic Ġmodel Ġis Ġbased Ġon Ġa Ġhybrid ĠH MM - D NN Ġarchitecture Ġand Ġtrained Ġon Ġthe Ġdata Ġsummarized Ġin ĠTable Ġ. ĠAc oustic Ġtraining Ġdata Ġcorrespond Ġto Ġ100 h Ġof Ġnon - sp ont aneous Ġspeech Ġtype Ġ( mostly Ġbroadcast Ġnews ) Ġcoming Ġfrom Ġboth Ġradio Ġand ĠTV Ġshows . ĠA Ġ5 - gram Ġlanguage Ġmodel Ġis Ġtrained Ġfrom Ġseveral ĠFrench Ġcorpor a Ġ( 3 , 323 M Ġwords Ġin Ġtotal ) Ġusing ĠSR IL M Ġtool kit ĠB IB REF 27 . ĠThe Ġpronunciation Ġmodel Ġis Ġdeveloped Ġusing Ġthe Ġlex ical Ġresource ĠBD LE X ĠB IB REF 28 Ġas Ġwell Ġas Ġautomatic Ġgrap he me - to - ph on eme Ġ( G 2 P ) Ġtranscription Ġto Ġfind Ġpronunciation Ġvariants Ġof Ġour Ġvocabulary Ġ( limited Ġto Ġ80 K ). ĠIt Ġis Ġimportant Ġto Ġre - spec ify Ġhere , Ġfor Ġfurther Ġanalysis , Ġthat Ġour ĠK aldi Ġpipeline Ġfollows Ġspeaker Ġadaptive Ġtraining Ġ( S AT ) Ġwhere Ġwe Ġtrain Ġand Ġdecode Ġusing Ġspeaker Ġadapted Ġfeatures Ġ( f M LL R - adapt ed Ġfeatures ) Ġin Ġper - spe aker Ġmode . ĠIt Ġis Ġwell Ġknown Ġthat Ġspeaker Ġadaptation Ġacts Ġas Ġan Ġeffective Ġprocedure Ġto Ġreduce Ġmismatch Ġbetween Ġtraining Ġand Ġevaluation Ġconditions ĠB IB REF 29 , ĠB IB REF 26 . </s> Method ology Ġ: :: ĠGender Ġbias Ġevaluation Ġprocedure Ġof Ġan ĠAS R Ġsystem Ġperformance Ġ: :: ĠEvaluation </s> Word ĠError ĠRate Ġ( WER ) Ġis Ġa Ġcommon Ġmetric Ġto Ġevaluate ĠAS R Ġperformance . ĠIt Ġis Ġmeasured Ġas Ġthe Ġsum Ġof Ġerrors Ġ( insert ions , Ġdelet ions Ġand Ġsubstit utions ) Ġdivided Ġby Ġthe Ġtotal Ġnumber Ġof Ġwords Ġin Ġthe Ġreference Ġtranscription . ĠAs Ġwe Ġare Ġinvestigating Ġthe Ġimpact Ġon Ġperformance Ġof Ġspeaker 's Ġgender Ġand Ġrole , Ġwe Ġcomputed Ġthe ĠW ER Ġfor Ġeach Ġspeaker Ġat Ġthe Ġepisode Ġ( show Ġoccurrence ) Ġlevel . ĠAnaly zing Ġat Ġsuch Ġgran ularity Ġallows Ġus Ġto Ġavoid Ġlarge ĠW ER Ġvariation Ġthat Ġcould Ġbe Ġobserved Ġat Ġutter ance Ġlevel Ġ( especially Ġfor Ġshort Ġspeech Ġturns ) Ġbut Ġalso Ġmakes Ġpossible Ġto Ġget Ġseveral ĠW ER Ġvalues Ġfor Ġa Ġgiven Ġspeaker , Ġone Ġfor Ġeach Ġoccurrence Ġof Ġa Ġshow Ġin Ġwhich Ġhe / she Ġappears Ġon . ĠSpeaker 's Ġgender Ġwas Ġprovided Ġby Ġthe Ġmeta - data Ġand Ġrole Ġwas Ġobtained Ġusing Ġthe Ġcriteria Ġfrom ĠSection ĠSEC REF 6 Ġcomputed Ġfor Ġeach Ġshow . ĠThis Ġenables Ġus Ġto Ġanalyze Ġour Ġresults Ġacross Ġgender Ġand Ġrole Ġcategories Ġwhich Ġwas Ġdone Ġusing ĠWil co xon Ġrank Ġsum Ġtests Ġalso Ġcalled ĠMann - Whit ney ĠU Ġtest Ġ( with Ġ$\ alpha Ġ$ = Ġ0 . 001 ) ĠB IB REF 30 . ĠThe Ġchoice Ġof Ġa ĠWil co xon Ġrank Ġsum Ġtest Ġand Ġnot Ġthe Ġcommonly Ġused Ġt - test Ġis Ġmotivated Ġby Ġthe Ġnon - norm ality Ġof Ġour Ġdata . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠGender Ġrepresentation </s> As Ġexpected , Ġwe Ġobserve Ġa Ġdisparity Ġin Ġterms Ġof Ġgender Ġrepresentation Ġin Ġour Ġdata Ġ( see ĠTable Ġ). ĠWomen Ġrepresent Ġ33 . 16 % Ġof Ġthe Ġspeakers , Ġconfirming Ġthe Ġfigures Ġgiven Ġby Ġthe ĠGM MP Ġreport ĠB IB REF 0 . ĠHowever , Ġit Ġis Ġworth Ġnoticing Ġthat Ġwomen Ġaccount Ġfor Ġonly Ġ22 . 57 % Ġof Ġthe Ġtotal Ġspeech Ġtime , Ġwhich Ġleads Ġus Ġto Ġconclude Ġthat Ġwomen Ġalso Ġspeak Ġless Ġthan Ġmen . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠSpeaker 's Ġrole Ġrepresentation </s> Table Ġpresents Ġroles ' Ġrepresentation Ġin Ġtraining Ġdata Ġand Ġshows Ġthat Ġdespite Ġthe Ġsmall Ġnumber Ġof ĠAnch or Ġspeakers Ġin Ġour Ġdata Ġ( 3 . 79 %), Ġthey Ġnevertheless Ġconcentrate Ġ35 . 71 Ġ% Ġof Ġthe Ġtotal Ġspeech Ġtime . </s> Results Ġ: :: ĠDes cript ive Ġanalysis Ġof Ġgender Ġrepresentation Ġin Ġtraining Ġdata Ġ: :: ĠRole Ġand Ġgender Ġinteraction </s> When Ġcrossing Ġboth Ġparameters , Ġwe Ġcan Ġobserve Ġthat Ġthe Ġgender Ġdistribution Ġis Ġnot Ġconstant Ġthroughout Ġroles . ĠWomen Ġrepresent Ġ29 . 47 % Ġof Ġthe Ġspeakers Ġwithin Ġthe ĠAnch or Ġcategory , Ġeven Ġless Ġthan Ġamong Ġthe ĠPun ct ual Ġspeakers . ĠTheir Ġpercentage Ġof Ġspeech Ġis Ġalso Ġsmaller . ĠWhen Ġcalculating Ġthe Ġaverage Ġspeech Ġtime Ġuttered Ġby Ġa Ġfemale ĠAnch or , Ġwe Ġobtain Ġa Ġvalue Ġof Ġ15 . 9 Ġmin Ġagainst Ġ25 . 2 Ġmin Ġfor Ġa Ġmale ĠAnch or , Ġwhich Ġsuggests Ġthat Ġeven Ġwithin Ġthe ĠAnch or Ġcategory Ġmen Ġtend Ġto Ġspeak Ġmore . ĠThis Ġconfirms Ġthe Ġexistence Ġof Ġgender Ġdisparities Ġwithin ĠFrench Ġmedia . ĠIt Ġcorrobor ates Ġwith Ġthe Ġanalysis Ġof Ġthe ĠC SA ĠB IB REF 1 , Ġwhich Ġshows Ġthat Ġwomen Ġwere Ġless Ġpresent Ġduring Ġhigh - aud ience Ġhours . ĠOur Ġstudy Ġshows Ġthat Ġthey Ġare Ġalso Ġless Ġpresent Ġin Ġimportant Ġroles . ĠThese Ġresults Ġlegitimate Ġour Ġinitial Ġquestioning Ġon Ġthe Ġimpact Ġof Ġgender Ġbalance Ġon ĠAS R Ġperformance Ġtrained Ġon Ġbroadcast Ġrecordings . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠImpact Ġof Ġgender Ġon ĠW ER </s> As Ġexplained Ġin ĠSection ĠSEC REF 13 , ĠW ER Ġis Ġthe Ġsum Ġof Ġerrors Ġdivided Ġby Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġtranscription Ġreference . ĠThe Ġhigher Ġthe ĠW ER , Ġthe Ġpoorer Ġthe Ġsystem Ġperformance . ĠOur Ġ70 h Ġevaluation Ġdata Ġcontains Ġa Ġlarge Ġamount Ġof Ġspontaneous Ġspeech Ġand Ġis Ġvery Ġchallenging Ġfor Ġthe ĠAS R Ġsystem Ġtrained Ġon Ġprepared Ġspeech : Ġwe Ġobserve Ġan Ġoverall Ġaverage ĠW ER Ġof Ġ42 . 9 % Ġfor Ġwomen Ġand Ġ34 . 3 % Ġfor Ġmen . ĠThis Ġdifference Ġof ĠW ER Ġbetween Ġmen Ġand Ġwomen Ġis Ġstatistically Ġsignificant Ġ( med ( M ) Ġ= Ġ25 %; Ġmed ( F ) Ġ= Ġ29 %; ĠU Ġ= Ġ7 09 040 ; Ġp - value Ġ< Ġ0 . 001 ). </s> However , Ġwhen Ġobserving Ġgender Ġdifferences Ġacross Ġshows , Ġno Ġclear Ġtrend Ġcan Ġbe Ġidentified , Ġas Ġshown Ġin ĠFigure ĠFIG REF 21 . ĠFor Ġshows Ġlike ĠAfrica 1 ĠInf os Ġor ĠLa ĠPlace Ġdu ĠVillage , Ġwe Ġfind Ġan Ġaverage ĠW ER Ġlower Ġfor Ġwomen Ġthan Ġfor Ġmen , Ġwhile Ġthe Ġtrend Ġis Ġreversed Ġfor Ġshows Ġsuch Ġas ĠUn ĠTem ps Ġde ĠPau ch on Ġor ĠLe ĠMas que Ġet Ġla ĠPl ume . ĠThe Ġdisparity Ġof Ġthe Ġresults Ġdepending Ġon Ġthe Ġshow Ġleads Ġus Ġto Ġbelieve Ġthat Ġother Ġfactors Ġmay Ġbe Ġentangled Ġwithin Ġthe Ġobserved Ġphenomenon . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠImpact Ġof Ġrole Ġon ĠW ER </s> Spe aker 's Ġrole Ġseems Ġto Ġhave Ġan Ġimpact Ġon ĠW ER : Ġwe Ġobtain Ġan Ġaverage ĠW ER Ġof Ġ30 . 8 % Ġfor Ġthe ĠAnch or Ġspeakers Ġand Ġ42 . 23 % Ġfor Ġthe ĠPun ct ual Ġspeakers . ĠThis Ġdifference Ġis Ġstatistically Ġsignificant Ġwith Ġa Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 14 } $ Ġ( med ( A ) Ġ= Ġ21 %; Ġmed ( P ) Ġ= Ġ31 %; ĠU Ġ= Ġ540 , 430 ; Ġp - value Ġ< Ġ0 . 001 ) Ġ. </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠRole Ġand Ġgender Ġinteraction </s> Figure ĠFIG REF 25 Ġpresents Ġthe ĠW ER Ġdistribution Ġ( WER Ġbeing Ġobtained Ġfor Ġeach Ġspeaker Ġin Ġa Ġshow Ġoccurrence ) Ġaccording Ġto Ġthe Ġspeaker 's Ġrole Ġand Ġgender . ĠIt Ġis Ġworth Ġnoticing Ġthat Ġthe Ġgender Ġdifference Ġis Ġonly Ġsignificant Ġwithin Ġthe ĠPun ct ual Ġspeakers Ġgroup . ĠThe Ġaverage ĠW ER Ġis Ġof Ġ49 . 04 % Ġfor Ġthe Ġwomen Ġand Ġ38 . 56 % Ġfor Ġthe Ġmen Ġwith Ġa Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 6 } $ Ġ( med ( F ) Ġ= Ġ39 %; Ġmed ( M ) Ġ= Ġ29 %; ĠU Ġ= Ġ251 , 450 ; Ġp - value Ġ< Ġ0 . 001 ), Ġwhereas Ġit Ġis Ġjust Ġa Ġtrend Ġbetween Ġmale Ġand Ġfemale ĠAnch ors Ġ( med ( F ) Ġ= Ġ21 %; Ġmed ( M ) Ġ= Ġ21 %; ĠU Ġ= Ġ116 , 230 ; Ġp - value Ġ= Ġ0 . 173 ). ĠThis Ġcould Ġbe Ġexplained Ġby Ġthe Ġquantity Ġof Ġdata Ġavailable Ġper Ġspeaker . </s> Results Ġ: :: ĠPerformance Ġ( WER ) Ġanalysis Ġon Ġevaluation Ġdata Ġ: :: ĠSpeech Ġtype Ġas Ġa Ġthird Ġentangled Ġfactor ? </s> In Ġorder Ġto Ġtry Ġto Ġexplain Ġthe Ġobserved Ġvariation Ġin Ġour Ġresults Ġdepending Ġon Ġshows Ġand Ġgender Ġ( Figure ĠFIG REF 21 ), Ġwe Ġadd Ġthe Ġnotion Ġof Ġspeech Ġtype Ġto Ġshed Ġsome Ġlight Ġon Ġour Ġresults . ĠB IB REF 22 Ġand ĠB IB REF 24 Ġsuggested Ġthat Ġthe Ġspeaker Ġprofessionalism , Ġassociated Ġwith Ġclear Ġand Ġhyper - art ic ulated Ġspeech Ġcould Ġbe Ġan Ġexplaining Ġfactor Ġfor Ġbetter Ġperformance . </s> Based Ġon Ġour Ġcategor ization Ġin Ġprepared Ġspeech Ġ( mostly Ġnews Ġreports ) Ġand Ġspontaneous Ġspeech Ġ( mostly Ġdebates Ġand Ġentertainment Ġshows ), Ġwe Ġcross Ġthis Ġparameter Ġin Ġour Ġperformance Ġanalysis . ĠAs Ġshown Ġon ĠFigure ĠFIG REF 26 , Ġthese Ġresults Ġconfirm Ġthe Ġinherent Ġchallenge Ġof Ġspontaneous Ġspeech Ġcompared Ġto Ġprepared Ġspeech . ĠW ER Ġscores Ġare Ġsimilar Ġbetween Ġmen Ġand Ġwomen Ġwhen Ġconsidering Ġprepared Ġspeech Ġ( med ( F ) Ġ= Ġ18 %; Ġmed ( M ) Ġ= Ġ21 %; ĠU Ġ= Ġ217 , 160 ; Ġp - value Ġ= Ġ0 . 005 ) Ġwhereas Ġthey Ġare Ġworse Ġfor Ġwomen Ġ( 61 . 29 %) Ġthan Ġfor Ġmen Ġ( 46 . 51 %) Ġwith Ġp - value Ġsmaller Ġthan Ġ$ 10 ^{ - 14 } $ Ġfor Ġthe Ġspontaneous Ġspeech Ġtype Ġ( med ( F ) Ġ= Ġ61 %; Ġmed ( M ) Ġ= Ġ37 %; ĠU Ġ= Ġ153 , 580 ; Ġp - value Ġ< Ġ0 . 001 ). </s> Discussion </s> We Ġfind Ġa Ġclear Ġdisparity Ġin Ġterms Ġof Ġwomen Ġpresence Ġand Ġspeech Ġquantity Ġin ĠFrench Ġmedia . ĠOur Ġdata Ġbeing Ġrecorded Ġbetween Ġ1998 Ġand Ġ2013 , Ġwe Ġcan Ġexpect Ġthis Ġdisparity Ġto Ġbe Ġsmaller Ġon Ġmore Ġrecent Ġbroadcast Ġrecordings , Ġespecially Ġsince Ġthe ĠFrench Ġgovernment Ġdisplays Ġefforts Ġtoward Ġparity Ġin Ġmedia Ġrepresentation . ĠOne Ġcan Ġalso Ġargue Ġthat Ġeven Ġif Ġour Ġanalysis Ġwas Ġconducted Ġon Ġa Ġlarge Ġamount Ġof Ġdata Ġit Ġdoes Ġnot Ġreach Ġthe Ġexhaust iveness Ġof Ġlarge - scale Ġstudies Ġsuch Ġas Ġthe Ġone Ġof ĠB IB REF 2 . ĠNonetheless Ġit Ġdoes Ġnot Ġaffect Ġthe Ġrelevance Ġof Ġour Ġfindings , Ġbecause Ġif Ġreal - world Ġgender Ġrepresentation Ġmight Ġbe Ġmore Ġbalanced Ġtoday , Ġthese Ġcorpor a Ġare Ġstill Ġused Ġas Ġtraining Ġdata Ġfor ĠAI Ġsystems . </s> The Ġperformance Ġdifference Ġacross Ġgender Ġwe Ġobserved Ġcorrobor ates Ġ( on Ġa Ġlarger Ġquantity Ġand Ġvariety Ġof Ġlanguage Ġdata Ġproduced Ġby Ġmore Ġthan Ġ2400 Ġspeakers ) Ġthe Ġresults Ġobtained Ġby ĠB IB REF 24 Ġon Ġisolated Ġwords Ġrecognition . ĠHowever Ġthe Ġfollowing Ġstudy Ġon Ġread Ġspeech Ġdoes Ġnot Ġreplicate Ġthese Ġresults . ĠYet Ġa Ġperformance Ġdegradation Ġis Ġobserved Ġacross Ġdialect Ġand Ġrace ĠB IB REF 8 . ĠB IB REF 22 Ġfound Ġlower ĠW ER Ġfor Ġwomen Ġthan Ġmen Ġon Ġbroadcast Ġnews Ġand Ġconvers ational Ġtelephone Ġspeech Ġfor Ġboth ĠEnglish Ġand ĠFrench . ĠThe Ġauthors Ġsuggest Ġthat Ġgender Ġstereotypes Ġassociated Ġwith Ġwomen Ġrole Ġin Ġeducation Ġand Ġlanguage Ġacquisition Ġinduce Ġa Ġmore Ġnormative Ġel oc ution . ĠWe Ġobserved Ġthat Ġthe Ġhigher Ġthe Ġdegree Ġof Ġnorm ativity Ġof Ġspeech Ġthe Ġsmaller Ġthe Ġgender Ġdifference . ĠNo Ġsignificant Ġgender Ġbias Ġis Ġobserved Ġfor Ġprepared Ġspeech Ġnor Ġwithin Ġthe ĠAnch or Ġcategory . ĠEven Ġif Ġwe Ġdo Ġnot Ġfind Ġsimilar Ġresults Ġwith Ġlower ĠW ER Ġfor Ġwomen Ġthan Ġmen , Ġwe Ġobtained Ġa Ġmedian ĠW ER Ġsmaller Ġfor Ġwomen Ġon Ġprepared Ġspeech Ġand Ġequal Ġto Ġthe Ġmale Ġmedian ĠW ER Ġfor Ġthe ĠAnch or Ġspeakers . </s> Another Ġexplanation Ġcould Ġbe Ġthe Ġuse Ġof Ġadaptation Ġwithin Ġthe Ġpipeline . ĠMost Ġbroadcast Ġprograms Ġtranscription Ġsystems Ġhave Ġa Ġspeaker Ġadaptation Ġstep Ġwithin Ġtheir Ġdecoding Ġpipeline , Ġwhich Ġis Ġthe Ġcase Ġfor Ġour Ġsystem . ĠAn ĠAnch or Ġspeaker Ġintervening Ġmore Ġoften Ġwould Ġhave Ġa Ġlarger Ġquantity Ġof Ġdata Ġto Ġrealize Ġsuch Ġadaptation Ġof Ġthe Ġacoustic Ġmodel . ĠOn Ġthe Ġcontrary , ĠPun ct ual Ġspeakers Ġwho Ġappear Ġscarcely Ġin Ġthe Ġdata Ġare Ġnot Ġprovided Ġwith Ġthe Ġsame Ġamount Ġof Ġadaptation Ġdata . ĠHence Ġwe Ġcan Ġhypothes ize Ġthat Ġgender Ġperformance Ġdifference Ġobserved Ġfor ĠPun ct ual Ġspeakers Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġfemale Ġspeech Ġis Ġfurther Ġfrom Ġthe Ġ( initial Ġnon - adapt ed ) Ġacoustic Ġmodel Ġas Ġit Ġwas Ġtrained Ġon Ġun balanced Ġdata Ġ( as Ġshown Ġin ĠTable Ġ). ĠConsidering Ġthat ĠPun ct ual Ġspeakers Ġrepresent Ġ92 . 78 % Ġof Ġthe Ġspeakers , Ġthis Ġexplains Ġwhy Ġgender Ġdifference Ġis Ġsignificant Ġover Ġour Ġentire Ġdata Ġset . ĠA Ġway Ġto Ġconfirm Ġour Ġhypothesis Ġwould Ġbe Ġto Ġreproduce Ġour Ġanalysis Ġon ĠW ER Ġvalues Ġobtained Ġwithout Ġusing Ġspeaker Ġadapted Ġfeatures Ġat Ġthe Ġdecoding Ġstep . </s> When Ġdecoding Ġprepared Ġspeech Ġ( hen ce Ġsimilar Ġto Ġthe Ġtraining Ġdata ), Ġno Ġsignificant Ġdifference Ġis Ġfound Ġin ĠW ER Ġbetween Ġmen Ġand Ġwomen , Ġrevealing Ġthat Ġthe Ġspeaker Ġadaptation Ġstep Ġcould Ġbe Ġsufficient Ġto Ġreach Ġsame Ġperformance Ġfor Ġboth Ġgenders . ĠBut Ġwhen Ġdecoding Ġmore Ġspontaneous Ġspeech , Ġthere Ġis Ġa Ġmismatch Ġwith Ġthe Ġinitial Ġacoustic Ġmodel Ġ( trained Ġon Ġprepared Ġspeech ). ĠConsequently , Ġthe Ġspeaker Ġadaptation Ġstep Ġmight Ġnot Ġbe Ġenough Ġto Ġrecover Ġgood ĠAS R Ġperformance , Ġespecially Ġfor Ġwomen Ġfor Ġwhom Ġless Ġadaptation Ġdata Ġis Ġavailable Ġ( see ĠSection Ġ4 . 2 . 3 ). </s> Conclusion </s> This Ġpaper Ġhas Ġinvestigated Ġgender Ġbias Ġin ĠAS R Ġperformance Ġthrough Ġthe Ġfollowing Ġresearch Ġquestions : Ġi ) Ġwhat Ġis Ġthe Ġproportion Ġof Ġmen Ġand Ġwomen Ġin ĠFrench Ġradio Ġand ĠTV Ġmedia Ġdata Ġ? Ġii ) Ġwhat Ġis Ġthe Ġimpact Ġof Ġthe Ġobserved Ġdisparity Ġon ĠAS R Ġperformance Ġ? Ġiii ) Ġis Ġthis Ġas Ġsimple Ġas Ġa Ġproblem Ġof Ġgender Ġproportion Ġin Ġthe Ġtraining Ġdata Ġor Ġare Ġother Ġfactors Ġentangled Ġ? ĠOur Ġcontributions Ġare Ġthe Ġfollowing : </s> Desc ript ive Ġanalysis Ġof Ġthe Ġbroadcast Ġdata Ġused Ġto Ġtrain Ġour ĠAS R Ġsystem Ġconfirms Ġthe Ġalready Ġknown Ġdisparity , Ġwhere Ġ65 % Ġof Ġthe Ġspeakers Ġare Ġmen , Ġspeaking Ġmore Ġthan Ġ75 % Ġof Ġthe Ġtime . </s> When Ġinvestigating ĠW ER Ġscores Ġaccording Ġto Ġgender , Ġspeaker 's Ġrole Ġand Ġspeech Ġtype , Ġhuge Ġvariations Ġare Ġobserved . ĠWe Ġconclude Ġthat Ġgender Ġis Ġclearly Ġa Ġfactor Ġof Ġvariation Ġin ĠAS R Ġperformance , Ġwith Ġa ĠW ER Ġincrease Ġof Ġ24 % Ġfor Ġwomen Ġcompared Ġto Ġmen , Ġexhibiting Ġa Ġclear Ġgender Ġbias . </s> Gender Ġbias Ġvaries Ġacross Ġspeaker 's Ġrole Ġand Ġspeech Ġspont ane ity Ġlevel . ĠPerformance Ġfor ĠPun ct ual Ġspeakers Ġrespectively Ġspontaneous Ġspeech Ġseems Ġto Ġreinforce Ġthis Ġgender Ġbias Ġwith Ġa ĠW ER Ġincrease Ġof Ġ27 . 2 % Ġrespectively Ġ31 . 8 % Ġbetween Ġmale Ġand Ġfemale Ġspeakers . </s> We Ġfound Ġthat Ġan ĠAS R Ġsystem Ġtrained Ġon Ġun balanced Ġdata Ġregarding Ġgender Ġproduces Ġgender Ġbias Ġperformance . ĠTherefore , Ġin Ġorder Ġto Ġcreate Ġfair Ġsystems Ġit Ġis Ġnecessary Ġto Ġtake Ġinto Ġaccount Ġthe Ġrepresentation Ġproblems Ġin Ġsociety Ġthat Ġare Ġgoing Ġto Ġbe Ġencaps ulated Ġin Ġthe Ġdata . ĠUnderstanding Ġhow Ġwomen Ġunder - represent ation Ġin Ġbroadcast Ġdata Ġcan Ġlead Ġto Ġbias Ġin ĠAS R Ġperformances Ġis Ġthe Ġkey Ġto Ġprevent Ġre - im plement ing Ġand Ġreinforcing Ġdiscrimination Ġalready Ġexisting Ġin Ġour Ġsocieties . ĠThis Ġis Ġin Ġline Ġwith Ġthe Ġconcept Ġof ĠâĢ ľ Fair ness Ġby ĠDesign " Ġproposed Ġby ĠB IB REF 31 . </s> Gender , Ġrace , Ġreligion , Ġnationality Ġare Ġall Ġcharacteristics Ġthat Ġwe Ġdeem Ġunfair Ġto Ġclassify Ġon , Ġand Ġthese Ġethical Ġstand points Ġneeds Ġto Ġbe Ġtaken Ġinto Ġaccount Ġin Ġsystems ' Ġdesign . ĠCharacter istics Ġthat Ġare Ġnot Ġconsidered Ġas Ġrelevant Ġin Ġa Ġgiven Ġtask Ġcan Ġbe Ġencaps ulated Ġin Ġdata Ġnonetheless , Ġand Ġlead Ġto Ġbias Ġperformance . ĠBeing Ġaware Ġof Ġthe Ġdemographic Ġske ws Ġour Ġdata Ġset Ġmight Ġcontain Ġis Ġa Ġfirst Ġstep Ġto Ġtrack Ġthe Ġlife Ġcycle Ġof Ġa Ġtraining Ġdata Ġset Ġand Ġa Ġnecessary Ġstep Ġto Ġcontrol Ġthe Ġtools Ġwe Ġdevelop .