Hynek Kydlíček, Jindřich Libovický &ndash; Univerzita Karlova, Czech Republic
A Dataset and Strong Baselines for Classification of Czech News Texts
Pre-trained models for Czech Natural Language Processing are often evaluated on purely linguistic tasks (POS tagging, parsing, NER) and relatively simple classification tasks such as sentiment classification or article classification from a single news source. As an alternative, we present CZEch~NEws~Classification~dataset (CZE-NEC), one of the largest Czech classification datasets, composed of news articles from various sources spanning over twenty years, which allows a more rigorous evaluation of such models. We define four classification tasks: news source, news category, inferred author's gender, and day of the week. To verify the task difficulty, we conducted a human evaluation, which revealed that human performance lags behind strong machine-learning baselines built upon pre-trained transformer models. Furthermore, we show that language-specific pre-trained encoder analysis outperforms selected commercially available large-scale generative language models.
We create the CZE-NEC by crawling Czech news websites from CommonCrawl (§ 2.1) and use the available metadata to define classification tasks (§ 2.3). [...] We have collected the news stories text from the following six Czech online news providers: SeznamZprávy.cz, iRozhlas.cz, Novinky.cz, Deník.cz, iDnes.cz, and Aktuálně.cz. Instead of crawling the pages directly, we used the CommonCrawl archive to extract the articles.