Juhani Luotolahti, Jenna Kanerva, Jouni Luoma, Valtteri Skantsi, Sampo Pyysalo, Veronika Laippala, Filip Ginter &ndash; University of Turku, Finland; University of Oulu, Finland
Finnish Internet Parsebank
We present a Finnish web corpus with multiple text sources and rich additional annotations. The corpus is based in large parts on a dedicated Internet crawl, supplementing data from the Common Crawl initiative and the Finnish Wikipedia. The size of the corpus is 6.2 billion tokens from 9.5 million source documents. The text is enriched with morphological analyses, word lemmas, dependency trees, named entities and text register (genre) identification. Paragraph-level scores of an n-gram language model, as well as paragraph duplication rate in each document are provided, allowing for further filtering of the dataset by the end user. Thanks to changes in the 2023 Finnish copyright legislation, the corpus is openly available for research purposes, and can also be accessed through the NoSketchEngine concordance tool and the dep search dependency tree query tool, all at https://turkunlp.org/finnish nlp.html.
3.1 Data sources ¶ Our corpus is based on three primary data sources: Finnish Wikipedia, Common Crawl, and a custom web-crawl. [...] The Common Crawl dataset includes both plain text and raw HTML files, at the time without language metadata. We employed a language detection step using CLD3 as the language detector and MapReduce to download only the Finnish-language plaintext from the Amazon cloud service that hosts Common Crawl. As shown in Table2, this resulted in only a moderate amount of new data (3.2GB deduplicated text) ontop of Wikipedia (1.5GB deduplicated text). ¶ Consequently, we conducted a dedicated web crawl using the SpiderLing webcrawler (Suchomel & Pomikálek,2012). This web crawler is specifically designed forcollecting monolingual plaintext web corpora. It comprises a web crawling engine, atrigram-based language detector, and a boilerplate remover called Justext, which isresponsible for extracting plain text. Moreover, the crawler is lightweight and easyto run. The crawl was seeded with the list of all domain names in the.fi top-level domain, as well as the URLs of all Finnish text pages we gathered from CommonCrawl in the previous step. The crawl was carried out between 2014 and 2016. ¶ The final sizes of text obtained from the three sources are summarized in Table2, which shows that the dedicated webcrawl constitutes by far the largest portion of the final corpus. Note that in the newer versions of Common Crawl, a considerably stronger emphasis is placed on multilingual coverage, and the benefit of a dedicated webcrawl might be smaller but very unlikely to vanish entirely.