DIALOG-22 RuATD Generated Text Detection

Text Generation Models (TGMs) succeed in creating text that matches human language style reasonably well. Detectors that can distinguish between TGM-generated text and human-written ones play an important role in preventing abuse of TGM. In this paper, we describe our pipeline for the two DIALOG-22 RuATD tasks: detecting generated text (binary task) and classification of which model was used to generate text (multiclass task). We achieved 1st place on the binary classification task with an accuracy score of 0.82995 on the private test set and 4th place on the multiclass classification task with an accuracy score of 0.62856 on the private test set. We proposed an ensemble method of different pre-trained models based on the attention mechanism.


Introduction
As the language neural nets got better at generating texts, it's getting harder and harder to distinguish the human-written text from generated one. So manual detection of these texts got impossible. For that reason, it's desirable to build a system that can automatically detect generated text.
The proposed system will use various features extracted from the text such as length, punctuation, word choice etc. To determine whether the text is human-generated or not. The accuracy of this system can be improved by using machine learning algorithms which will learn how humans generate texts and then use those features for detection purpose.
There are many ways to build such a system, but probably the most reliable one is based on machine learning algorithms. These algorithms can be trained on a large number of examples -both humangenerated and computer-generated texts. After being trained, they should be able to identify which texts are computer-generated with high accuracy.
This approach already works in other areas, such as spam detection. Some early experiments have shown promising results and indicate that this approach works well for the detection of the generated text.
The DIALOG-22 shared task of RuATD 2022 had 2 tracks for binary classification and multiclass classification. In this report, we will describe the data, how we handled it, the models we used, and the ensembling technique. For the binary classification task , we frame the generated text detection problem as follows: given a piece of text , label it as either human-written or machine-generated = { , }.

Datasets
Provided datasets offer the train and test splits. The part of the set was annotated automatically by different generative models. Various language models were fine-tuned on different tasks: machine translation, paraphrasing, summarization, simplification, and unconditional text generation -are used to generate texts. The texts written by a human were collected from open sources from different domains. (Shamardina et al., 2022b), (Shamardina et al., 2022c).

Related Works
In this section, we will discuss various methods for detecting machine-generated texts.
Over the past years, many approaches appeared for generated text detection. The latest works are usually based on using transformer-based models. Either fine-tuning the proposed task or using probabilities distribution to make decision-based on them (Ippolito et al., 2019). Here we list some examples of different methods: • First, we calculate the mean likelihood over all machine-generated sequences, then the mean of human-written ones. If the likelihood according to some language model is closer to the machinegenerated mean likelihood, then we consider it as generated text (Solaiman et al., 2019); • In GLTR (Gehrmann et al., 2019) described a method using a language model to compute the probability distribution of the next word given the text sequence. For each sequence position, we get the likelihood of the ground-truth next word within this list. Then these ranks are displayed on a histogram. Based on the distribution of bins, we can decide if this text is generated or not.
• Bert fine-tuning on the classification task. Having a label of text if it's machine-generated or not, we can fine-tune the language model to predict (Solaiman et al., 2019) • Also, there is possible to use human-machine collaboration. Real or Fake Tool provides a game-like interface for humans to decide at what point the text begins to be written by a computer (Dugan et al., 2020). More approaches are described in the survey (Jawahar et al., 2020).

Experiment Setup
In this section, we present the experiment configurations we use to solve binary and multiclass tasks.

Data preproccessing
We decided not to perform any preprocessing of the text itself. Regarding the data split: • Binary classification. We concatenated the train set with the validation set. This new concatenated dataset gave us an option to perform a 5-fold cross-validation; • Multiclass classification. We took the data as is without changing anything.

Models
All models we used for fine-tuning we got from transformers library (Wolf et al., 2020). Here we will describe these models: • sberbank-ai/sbert_large_nlu_ru (Antuhov, 2020), sberbank-ai/ruBert-large, DeepPavlov/rubert-base-cased are BERT models (Devlin et al., 2018). Sber-ai models were fine-tuned on a closed dataset collected by their research group (Sber, 2022). DeepPavlov's RuBERT was fine-tuned on the Russian part of Wikipedia and news data. • IlyaGusev/mbart_ru_sum_gazeta is a mBART model fine-tuned on the dataset for summarization of Russian news (Gusev, 2020). The original MBart model was pretrained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text (Liu et al., 2020). • MoritzLaurer/mDeBERTa-v3-base-mnli-xnli (He et al., 2021) This multilingual model is suitable for multilingual zero-shot classification. The original model was pre-trained by Microsoft on the CC100 multilingual dataset (Wenzek et al., 2020). This model was fine-tuned on the XNLI development dataset and the MNLI train dataset. The XNLI development set consists of translated texts for each of the 15 languages (Conneau et al., 2018). • DeepPavlov/xlm-roberta-large-en-ru-mnli is an XLM-RoBERTa model (Conneau et al., 2019) which was fine-tuned on the english-russian part of the MNLI (Williams et al., 2017) dataset.

Binary classification (with ensembling technique)
Five chosen models were used in the experiement: sberbank-ai/sbert_large_nlu_ru, sberbankai/ruBert-large, IlyaGusev/mbart_ru_sum_gazeta , MoritzLaurer/mDeBERTa-v3-base-mnli-xnli, DeepPavlov/xlm-roberta-large-en-ru-mnli Let's describe steps for training each of these models: 1. We split our training dataset into non-overlapping 5-folds and performed cross-validation ( Figure  1); 2. for each validation fold we predicted the target and as a result, we got out-of-fold predictions for training data; 3. for the test set, we predicted the target from every 5 models from cross-validation and averaged the result which is marked with red; As a meta-model, we choose Logistic Regression, which was trained on out-of-fold predictions. Then we predict the final results for the test set.

Multiclass classification
For the multiclass classification we chose these models: DeepPavlov/rubert-base-cased, DeepPavlov/xlm-roberta-large-en-ru, IlyaGusev/mbart_ru_sum_gazeta. We fine-tune these models without cross-validation only on provided train set. Table 3 shows each model accuracy score on binary classification task. The multiclass classification accuracy is shown in Table 4. On both tasks, the best performing single model was DeepPavlov/xlmroberta-large-en-ru-mnli. We managed to ensemble models in the binary task, so the ensemble of models showed the best accuracy.

Conclusion
In this paper, we described our pipeline for the DIALOG-22 RuATD challenge. Our solution achieved 1st place in binary classification using ensembling techniques and 4th place for the multiclass classification task using only a single model. However, the proposed solution requires a lot of computational power, so it cannot be used in real-time systems to detect generated texts. But it gives us an understanding that we still need to upgrade methods to distinguish generated texts from human-written ones.