Matyas Bohacek, Michal Bravansky, Filip Trhlík, Václav Moravec &ndash; Faculty of Social Sciences, Charles University, Prague, Czech Republic; Gymnasium of Johannes Kepler, Prague, Czech Republic; University College London, United Kingdom
Czech-ing the News: Article Trustworthiness Dataset for Czech
We present the Verifee dataset: a multimodal dataset of news articles with fine-grained trustworthiness annotations. We bring a diverse set of researchers from social, media, and computer sciences aboard to study this interdisciplinary problem holistically and develop a detailed methodology that assesses the texts through the lens of editorial transparency, journalist conventions, and objective reporting while penalizing manipulative techniques. We collect over 10,000 annotated articles from 60 Czech online news sources. Each item is categorized into one of the 4 proposed classes on the credibility spectrum {--} ranging from entirely trustworthy articles to deceptive ones {--} and annotated of its manipulative attributes. We fine-tune prominent sequence-to-sequence language models for the trustworthiness classification task on our dataset and report the best F-1 score of 0.53. We open-source the dataset, annotation methodology, and annotators{'} instructions in full length at https://www.verifee.ai/research/ to enable easy build-up work.
Initially, we assembled nearly 94, 000 articles by scraping URLs of 60 Czech news sources² obtained from Common Crawl³. These sources included mainstream journalistic websites, tabloids, independent news outlets, and websites that are part of the disinformation ecosystem (Štˇetka et al., 2021), capturing the full scope of journalistic content in the Czech Republic. [...] We applied multiple filters and balancing mechanisms based on text length and topics to mitigate deficiencies caused by inherent flaws in Common Crawl, which reduced the dataset’s size from 94, 000 to 10, 197 items. This way, we also ensured that the data is as representative of the Czech news ecosystem and as diverse as possible.