TITLE: The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild 

ABSTRACT: This paper presents a new training dataset for automatic genre identification
GINCO, which is based on 1,125 crawled Slovenian web documents that consist of
650 thousand words. Each document was manually annotated for genre with a new
annotation schema that builds upon existing schemata, having primarily clarity
of labels and inter-annotator agreement in mind. The dataset consists of
various challenges related to web-based data, such as machine translated
content, encoding errors, multiple contents presented in one document etc.,
enabling evaluation of classifiers in realistic conditions. The initial machine
learning experiments on the dataset show that (1) pre-Transformer models are
drastically less able to model the phenomena, with macro F1 metrics ranging
around 0.22, while Transformer-based models achieve scores of around 0.58, and
(2) multilingual Transformer models work as well on the task as the monolingual
models that were previously proven to be superior to multilingual models on
standard NLP tasks.
This paper presents a new training dataset for automatic genre identification
GINCO, which is based on 1,125 crawled Slovenian web documents that consist of
650 thousand words. Each document was manually annotated for genre with a new
annotation schema that builds upon existing schemata, having primarily clarity
of labels and inter-annotator agreement in mind. The dataset consists of
various challenges related to web-based data, such as machine translated
content, encoding errors, multiple contents presented in one document etc.,
enabling evaluation of classifiers in realistic conditions. The initial machine
learning experiments on the dataset show that (1) pre-Transformer models are
drastically less able to model the phenomena, with macro F1 metrics ranging
around 0.22, while Transformer-based models achieve scores of around 0.58, and
(2) multilingual Transformer models work as well on the task as the monolingual
models that were previously proven to be superior to multilingual models on
standard NLP tasks.
The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild
http://arxiv.org/abs/2201.03857v1