---
address: |
    TALP Research Center, Universitat Politècnica de Catalunya, Barcelona\
    $^*$ DFKI GmBH and Saarland University, Saarbrücken\
    \
bibliography:
- 'lrec2020W-xample.bib'
- 'paper.bib'
title: '[GeBioToolkit]{}: Automatic Extraction of Gender-Balanced Multilingual Corpus of Wikipedia Biographies\'
---

Introduction
============

Gender biases are present in many natural language processing applications [@costajussa:2019]. This comes as an undesired characteristic of deep learning architectures where their outputs seem to reflect demographic asymmetries [@pratesEtAl:2018]. This is of course not due to the architecture itself but to the data used to train a system. Recent research is being devoted to correct the asymmetries mainly by data augmentation techniques in fields such as coreference resolution [@rudingerEtAl:2018; @zhaoEtAl:2018; @websterEtAl:2018] or abusive language detection [@parkEtAl:2018]. Test sets have been created in those cases, but we are not aware of any test set available for machine translation (MT).

From another side, machine translation either neural, statistical or rule-based, usually operates in a sentence-by-sentence basis. However, when translating consistently a document, surrounding sentences may have valuable information. The translation of pronouns, verb tenses and even content words might depend on other fragments within a document. This affects also the translation of the gender markers, specially when translating from a language without these marks (e.g. English) into a language with them (e.g. Catalan). Test sets at sentence level are not useful to evaluate these phenomena. But in the time of claims of human parity in MT, all these phenomena are crucial, and document-level evaluation is needed in order to discriminate among systems [@lubliEtAl:2018]. Beyond these gender-balanced and document-level needs, the rise of multilingual neural machine translation and the lack of multi-way parallel corpus that can evaluate its abilities (e.g. zero-shot), motivates the creation of new multilingual corpora. In turn, this motivation prompts the development of software to automatically create such corpora.

In order to create a *[GeBioToolkit]{} *that is able to systematically extract multilingual parallel corpus at sentence level and with document-level information, we rely on Wikipedia, a free online multilingual encyclopedia written by volunteers. The toolkit is customizable for languages and gender-balance. We take advantage of Wikipedia multilinguality to extract a corpus of biographies, being each biography a document available in all the selected languages. However, given the bias in the representation of males and females also in Wikipedia [@bammanSmith:2014] and, to deliver a balanced set, we need to identify and select a subset of documents just after the toolkit performs the extraction of parallel sentences so it can assure parity. Note that our document-level extraction is consistent (all sentences in a document belong to the same personality) but the documents do not keep coherence anymore, since we are removing some sentences within that document. In our experiments in English–Spanish–Catalan, [GeBioToolkit]{} is able to extract parallel sentences with 87.5% accuracy according to a human evaluation.

Besides providing the tool, we also manually post-edited a small subset of the English (en), Spanish (es) and Catalan (ca) outputs to provide a test dataset which is ready-to-use for machine translation applications; to our knowledge the first gender-balanced dataset extracted from real texts in the area: *[GeBioCorpus]{}*. With these three languages we aim to cover two linguistic families (Germanic and Romance) which differ in morphology and specifically in gender inflection. The choice in languages is intended to allow the evaluation of machine translation outputs in three different settings: distant morphologies for a high-resourced language pair (English–Spanish) and a low-resourced pair (English–Catalan), and closely related languages (Spanish–Catalan). The latter responds the challenge recently initiated in the WMT International Evaluation[^1]. We used native/fluent speakers to provide post-editions on the final multilingual set in the three languages.

The rest of the paper is organized as follows. Section \[s:related\] describes some available multilingual datasets used for machine translation evaluation, related work on parallel sentence extraction from Wikipedia and a brief mention to general research lines on gender bias in NLP. Section \[sec:toolkit\] describes the architecture of the tool and Section \[s:corpus\] the methodology, evaluation and characteristics of the extracted corpora. Finally, Section \[s:conclusions\] summarizes the work and points at several future extensions of [GeBioToolkit]{}.

Related Work {#s:related}
============

There are several multilingual parallel datasets available to evaluate MT outputs. The corpora covering more languages are JRC-Acquis (Acquis Communautaire) and the TED talks corpus. Arab-Acquis [@arabAcquis:2017] is a multilingual dataset for 22 European languages plus Arabic with more than 12,000 sentences coming from JRC-Acquis, that is, a collection of legislative texts of the European Union. In order to make the test set equal in all the languages, only sentences that are parallel simultaneously in the 22 languages were extracted [@koehnEtAl:2009] and, therefore, the document structure of the data is lost.

The Web Inventory of Transcribed and Translated Talks, WIT$^{3(}$[^2]$^{)}$, includes English subtitles from TED talks and their translations currently in 109 languages. Parallel corpora are extracted for several pairs [@cettoloEtAl:2012] and test sets are annually prepared for the International Workshop on Spoken Language Translation (IWSLT) evaluation campaigns. Test sets exist for all the pairs among German, English, Italian, Dutch and Romanian; and from English to Arabic, Basque, Chinese, Czech, Dutch, Farsi, French, German, Hebrew, Italian, Japanese, Korean, Polish, Portuguese-Brazil, Romanian, Russian, Slovak, Slovenian, Spanish, Thai, Turkish and Vietnamese. In this case the whole talks are aligned at sentence level, so, the document structure is kept but the set is not equivalent in all the languages. Similarly, the news translation task at the annual workshops and conferences on statistical machine translation (WMT) distributes collections of parallel news to their participants. Test sets have been created over the years for pairs including English into Chinese, Czech, Estonian, Finnish, French, German, Hindi, Hungarian, Kazakh, Latvian, Spanish, Romanian, Russian and Turkish. Again, the document structure is kept in the dataset but, in general, the set is not equivalent in all the languages. Wikipedia is widely used in natural language processing and it is an excellent resource for multilingual tasks, parallel sentence extraction being among them. However, we do not know of any multilingual corpus of biographies extracted from the resource. On the monolingual side, one can find WikiBiography[^3], a corpus of 1,200 biographies in German with automatic annotations of PoS, lemmas, syntactic dependencies, anaphora, discourse connectives, classified named entities and temporal expressions. The authors in also extract 927,403 biographies in this case from the English Wikipedia. The set is pre-processed in order to learn event classes in biographies.

Regarding the automatic extraction of parallel corpora, Wikipedia has been traditionally used as a resource. In , the authors extract parallel sentences based on the available metadata in Wikipedia texts. Both and extracted parallel sentences by translating the articles into a common language and consider those sentences with a high translation quality to be parallel. The ACCURAT project [@Stefanescuetal:12; @Skadinaetal:12] also devoted efforts in parallel sentence mining in Wikipedia. Later, used the combination of cross-lingual similarity measures to extract domain specific parallel sentences. The most recent initiative is the so-called LASER [@artetxeSchwenk:2019], which relies on vector representations of sentences to extract similar pairs. This toolkit has been used to extract the WikiMatrix corpus [@DBLP:journals/corr/abs-1907-05791] which contains 135 million parallel sentences for 1,620 different language pairs in 85 different languages.

As far as we are concerned, there is no gender-balanced dataset for machine translation, except for the artificial gold standard created for English–Spanish [@font:2019]. However, there are a notable number of works towards doing research in the area: from balancing data sets in monolingual tasks [@websterEtAl:2018; @rudingerEtAl:2018] to evaluating gender bias in several tasks [@basta-etal-2019-evaluating; @stanovsky-etal-2019-evaluating].

![image](diagrama5.png){width="75.00000%"}

[GeBioToolkit]{} {#sec:toolkit}
================

Base Architecture {#ss:method}
-----------------

To extract the corpus previously introduced, we develop [GeBioToolkit]{}. The tool retrieves (multi-)parallel sentences for any Wikipedia category and for any arbitrary number of languages at the same time. In its default configuration, the toolkit retrieves gender-balanced corpora. [GeBioToolkit]{} is composed by three blocks. The first block, corpus extractor, provides a layer to transform, collect and select entries in the desired languages. The second block, corpus alignment, finds the parallel sentences within a text and provides a quality check of the parallel sentences given a few restrictions. The third block, gender classifier, includes a filtering module which classifies the gender of the entry and outputs the final parallel corpus. The gender detection functionality can be activated or deactivated at will, allowing the tool to be used in a more general context, where gender-balanced data is either non-relevant or not needed. Figure \[fig:gebiocorpus\] depicts its architecture.

The tool requires three inputs: [([*i*]{}) ]{}a list of the desired languages, [([*ii*]{}) ]{}the dump files for the languages[^4] and [([*iii*]{}) ]{}a list of the articles’ titles belonging to the category to extract (currently in English). For our purpose of gathering a corpus of biographies, we retrieve the list of articles that belong to the “living people” category by using the PetScan tool[^5]. The **corpus extractor** module starts by looking for the equivalent articles to those input in the other languages via the Wikipedia interlanguage links. The module also retrieves the multilingual titles providing a dictionary of {english\_entry\_title: \[language, title\]} that is used on the file extraction and file selection modules. File extraction retrieves then the selected entries in the previous dictionary from the Wikipedia dump. [GeBioToolkit]{} uses a modified version of the *wikiextractor*[^6] software to retrieve and store the different Wikipedia entries from each language. Finally, file selection generates a dictionary similar to the one obtained before, but it only stores the entries for which the files were successfully retrieved. The **corpus alignment** module makes use of the text and dictionaries retrieved in the previous step and the LASER toolkit[^7]. LASER (Language-Agnostic SEntence Representations) allows to obtain sentence embeddings through a multilingual sentence encoder [@DBLP:journals/corr/abs-1907-05791]. Translations can be found then as close pairs (tuples) in the multilingual semantic space. In order to perform the parallel sentence extraction, we follow the margin-based criterion introduced in . The margin between two candidate sentences $x$ and $y$ is defined as the ratio between the cosine distance between the two embedded sentences, and the average cosine similarity of its nearest neighbors in both directions:

$margin(x,y) = \frac{\cos{(x,y)}}{ \sum\limits_{\substack{z\in NN_k(x)}} \frac{\cos(x,z)}{2k} + \sum\limits_{\substack{z\in NN_k(y)}} \frac{\cos(y,z)}{2k}},$

where $NN_k(x)$ denotes the $k$ unique nearest neighbors of $x$ in the other language, $NN_k(y)$ the same for $y$. To extract parallel sentences on more than two languages, let us say $i$ languages, we use a greedy approach with a pivot language $L_1$. We detect all the parallel sentences in the pairs $L_1$–$L_i$ and then extract the intersection of sentences between the language pairs.

Gender Detection Module {#ss:gender}
-----------------------

The previous two blocks implement a general approach to parallel sentence extraction following a similar methodology as that used to extract the WikiMatrix corpus. But for our purpose, we need to specifically deal with gender bias. Wikipedia is known to have a gender bias in its content. Depending on the language and year of the study, the percentage of biographies of women with respect to the total of biographies ranges from a 13% to a 23% [@reagleRhue:2011; @bammanSmith:2014; @wagnerEtAl:2016]. And it is not only this, but also men and women are characterised differently [@Graells-GarridoEtAl:2015; @wagnerEtAl:2016], showing for instance more man in sports and woman in divorces [@reagleRhue:2011].

To allow extracting a gender-balanced dataset, we detect the gender automatically and filter files in order to have 50% of articles for each gender. Following , the gender of the article is extracted as that corresponding to the maximum number of gendered pronouns (i.e., he and she in English) mentioned in the article. According to their results, this method of gender inference is overwhelmingly accurate, in a random test set of 500 articles, it achieved 100% precision with 97.6% recall (12 articles had no pronominal mentions and so gender was not assigned).

Cleaning Module {#ss:postpro}
---------------

We analyse the accuracy of the extractions in Section \[ss:eval\], but a simple visual inspection already shows a frequent source of noise in the extracted data that can be easily removed with a post-processing step. Some of the extracted sentence pairs include information in one language that is lacking in the other one(s). For example, the sentence in Spanish *“Mahrez se casó con su novia inglesa en 2015 y tuvieron una hija ese mismo año.”* is aligned with the sentence in English *“Mahrez married his English girlfriend Rita Johal in 2015.”*, where the Spanish segment *“y tuvieron una hija ese mismo año.”* is not present in English. To avoid this, we filter out sentence pairs with large length ratio (filter samples in which one of the sentences is at least 20% longer than the others). Such filtering can be generalised to distant language pairs by estimating language-dependent length factors [@pouliquen:2003] or considering more elaborated corpus filtering techniques [@koehn-etal-2019-findings].

File Restructure Module {#ss:file}
-----------------------

Finally, the extracted parallel sentences are written as an xml file using a document-level mark-up. As said before, output documents do not preserve coherence any more, but document-level characteristics such as lexical consistency are kept and the mark-up allows use the information.

Table \[tab:example2\] shows an excerpt of the output file for English and Spanish. For each original biography, we keep the English title as document ID, the original Wikipedia ID, the extracted gender, and in our clean test set (see Section \[s:corpus\]) we also add the topic or category of each document. The `docid` field links documents among languages.

[GeBioCorpus]{} {#s:corpus}
===============

We use the [GeBioToolkit]{} presented in the above section to extract a multilingual parallel corpus balanced in gender for English, Spanish and Catalan. The motivation for extracting this specific dataset is the variation in morphology in English (Germanic language) as compared to Spanish and Catalan (Romance languages). The variation in morphology is one of the most relevant challenges when solving biases in gender, see examples in [@pratesEtAl:2018; @font:2019].

![image](amount_of_samples4.png){width="100.00000%"}

![image](union7.png){width="95.00000%"}

  *[GeBioCorpus]{}-v1*                                                  
  ---------------------- ---------- -------- -------- -------- -------- --------
                         F          M        F        M        F        M
  Documents              2287       2741     2287     2741     2287     2741
  Sentences              [8000]{}   8000     8000     8000     8000     8000
  Average sent/doc       3.5        2.9      3.5      2.9      3.5      2.9
  Words                  228.9k     235.3k   230.1k   236.0k   240.9k   245.7k
  Average words/doc      56.9       51.1     56.3     47.3     59.8     53.3
  Vocabulary             24.3k      24.8k    27.1k    27.6k    27.6k    28.0k
  *[GeBioCorpus]{}-v2*                                                  
                         F          M        F        M        F        M
  Documents              257        339      257      339      257      339
  Sentences              1000       1000     1000     1000     1000     1000
  Average sent/doc       3.9        2.9      3.9      2.9      3.9      2.9
  Words                  28.7k      27.5k    28.9k    28.0k    30.1k    29.4k
  Average words/doc      61.6       46.6     61.4     46.4     65.4     49.7
  Vocabulary             6.1k       6.0k     6.5k     6.5k     6.7k     6.6k

Data Statistics {#ss:statistics}
---------------

We extract all the articles belonging to the category “living people” in the 40 largest Wikipedia editions to have a rough idea of the number of biographies available per language. Figure \[fig:evollang\] shows the amount of these articles for the 20 languages with the largest number of biographies. The edition with the most available entries is the English one with 922,120 entries. The Spanish Wikipedia contains 148,445 entries (6th largest one) and the Catalan edition 40,983 (20th largest one). All three are highlighted in Figure \[fig:evollang\]. Even if the Catalan Wikipedia is not as big as the English and Spanish ones, there is a noticeable amount of comparable articles between Spanish and Catalan which translates into significant number of parallel sentences — extracted in WikiMatrix 3,377k sentences for en–es, 1,580k sentences for ca–es and 210k sentences for en–ca from the full editions.

[GeBioToolkit]{} extracts multilingually aligned parallel sentences, so it is interesting to study also the union of languages. The more languages involved, the lesser number or comparable articles one will obtain. Figure \[fig:fileslang\] shows the number of articles per sets of languages and broken-down by gender. The total number of documents decays when starting with English and Arabic (set with only 2 languages) and one incrementally adds French, German, Italian, Spanish, Polish, Russian, Portuguese, Japanese and Dutch. With all languages (set with 11 languages), the number of comparable documents results in 17,285, out of which 13,676 are male bios and 3,609 are female bios. These numbers reconfirm the results by [@reagleRhue:2011; @bammanSmith:2014; @wagnerEtAl:2016] on the underrepresentation of women in Wikipedia. Note that by doing the union of different languages and the intersection of its articles, the amount of documents decays quite abruptly, but the percentual difference between man and woman remains close to be constant.

When we consider English, Spanish and Catalan, we retrieve 31,413 bibliographies. [GeBioToolkit]{} selects 81,405 sentences from them, 53,389 sentences are related to male bibliographies and 28,016 to female bibliographies. 47.5% of the male sentences are removed to obtain a gender-balanced corpus. Then, [GeBioToolkit]{} performs the length-based cleaning, which filters the corpus down to 16,679 sentences on male bibliographies and 10,730 sentences on female bibliographies. As output, we provide two versions of the corpus. [GeBioCorpus]{}-v1 contains 16,000 sentences as extracted from Wikipedia without any manual post-edition. [GeBioCorpus]{}-v2 contains 2,000 sentences with manual post-edition as explained in Section \[ss:cats\]. One set excludes the other in addition to excluding other sentences (1730 for female) that we plan to post-edit in the future. See statistics on number of documents, sentences, words and vocabulary broken-down per language and gender in Table \[tab:statistics\]. Table \[tab:example3\] provides some examples of sentences from [GeBioCorpus]{}-v2.

Manual Post-Editing and Categorization {#ss:cats}
--------------------------------------

After the extraction of the complete corpus, and in order to provide a high-quality evaluation corpus, we ask native/fluent speakers to post-edit 2,000 sentences belonging to both female and male documents. Annotators were asked to either edit segments to make them parallel (if differences were small) or to delete segments that were too different. Also during the process, annotators were asked to provide a topic for the profession of each personality. Categories are based on the Wikipedia list of occupations[^8], which includes: (C1) Healthcare and medicine, (C2) Arts, (C3) Business, (C4) Industrial and manufacturing, (C5) Law enforcement, social movements and armed forces, (C6) Science, technology and education, (C7) Politics, (C8) Religion, and (C9) Sports. This corresponds to the field “topic” added in the header of each document as seen in Table \[tab:example2\]. Figure \[fig:categories\] shows the distribution of the 9 topics by gender. Note that the most unbalanced topics are C5 (Law enforcement, social movements and armed forces), C8 (Religion) and C9 (Sports). We report other statistics for this test set in Table \[tab:statistics\].

Human Evaluation {#ss:eval}
----------------

Finally, we perform a human evaluation on the quality of [GeBioCorpus]{}-v1, that is, the data as extracted by [GeBioToolkit]{}  without any post-edition. For this, we randomly select 50 sentences in the three languages (English, Spanish and Catalan) and present each tuple to 7 different native/fluent speakers. These annotators were asked to score a tuple with 1 if the three sentences convey the same meaning and 0 otherwise. On average, evaluators gave 87.5% accuracy and when computing the majority vote among the evaluators accuracy reached 96%. We computed Fleiss’ kappa [@fleiss1971mns] as a measure for the interannotator agreement which resulted in 0.67, which is considered a substantial agreement[^9]. On average, our [GeBioToolkit]{} plus filtering by length has 87.5% accuracy.

Conclusions {#s:conclusions}
===========

This paper presents two main contributions. From one side, [GeBioToolkit]{}, based on LASER, allows to automatically extract multilingual parallel corpora at sentence-level which can be customized in number of languages and balanced in gender. Document-level information is kept in the corpus and each document is tagged with the ID of the original Wikipedia article, the language and the gender of the person is referring to. From another side, we provide a multilingual corpus of biographies in English, Spanish and Catalan, [GeBioCorpus]{}. Two versions of this corpus are presented. The first version, [GeBioCorpus]{}-v1, contains 16k sentences which have been directly extracted using [GeBioToolkit]{}. This version of the corpus is used to make an evaluation of the quality of the extractions produced by our tool. A manual evaluation shows that the accuracy in the multilingual parallel sentences is 87.5%. The second version of this corpus, [GeBioCorpus]{}-v2, contains 2k sentences which have been post-edited by native speakers and categorized with type of profession. In this case, and in addition to the automatic tags of each document (ID, language and gender), each document is tagged with an occupation category. This labels allows to split the corpus in different subsets to evaluate domain-specific translations and gender accuracy for instance.

As future improvements to the tool, we will remove the dependence on PetScan —or any external tool to obtain lists of articles—, and extract all the necessary information from the input Wikipedia dump. We will also study the viability of using Wikidata information instead of the most frequent pronoun in a text in order to classify the gender of an article.

Both [GeBioToolkit]{} and [GeBioCorpus]{} will be made available in github during the review process of this paper[^10].

Acknowledgments {#acknowledgments .unnumbered}
===============

The authors want to thank Jordi Armengol, Magdalena Biesialska, Casimiro Carrino, Noe Casas, Guillem Cortès, Carlos Escolano, Gerard Gallego and Bardia Rafieian for their help in postediting the [GeBioCorpus]{}, Eduardo Lipperheide, Anna Li, Ayyoub Marroun, Laura Termes, Adrià Garcia, Alba Salomon and Clara Grenzner for the manual evaluation and Melissa Steiner, Ekaterina Lapshinova-Koltunski and Pauline Krielke for all the support and fruitful discussions.

This work is supported in part by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the postdoctoral senior grant Ramón y Cajal, the contract TEC2015-69266-P (MINECO/FEDER,EU) and the contract PCIN-2017-079 (AEI/MINECO). It is also partially funded by the German Federal Ministry of Education and Research under the funding code 01IW17001 (Deeplee). Responsibility for the content of this publication is with the authors.

Bibliographical References
==========================

[^1]: <http://www.statmt.org/wmt19/similar.html>

[^2]: <https://wit3.fbk.eu/>

[^3]: <https://www.h-its.org/en/research/nlp/wikibiography-corpus/>

[^4]: Dumps can be downloaded from\
    <https://dumps.wikimedia.org/>

[^5]: <https://petscan.wmflabs.org/>

[^6]: <https://github.com/attardi/wikiextractor>

[^7]: <https://github.com/facebookresearch/LASER>

[^8]: <https://en.wikipedia.org/wiki/Lists_of_occupations>

[^9]: <https://en.wikipedia.org/wiki/Fleiss%27_kappa>

[^10]: <https://github.com/PLXIV/Gebiotoolkit>
