Sequence-to-Sequence Lexical Normalization with Multilingual Transformers

Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP models when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multi-lingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.


Introduction
Social media is a pervasive part of our modern lives and provides us with a rich source of information and insight into human behaviour. Usergenerated content has been a valuable resource for the research community, especially in the form of text, but it is notoriously noisy and non-standard. Models that operate on social media posts go beyond marketing and advertisement applications, and have the potential to impact real human lives through, for instance, detecting loneliness (Guntuku et al., 2019), stress (Winata et al., 2018), life satisfaction (Yang and Srinivasan, 2016), suicidal ideation (Matero et al., 2019;Cao et al., 2019), and mental health problems such as depression (Yates et al., 2017;Bucur et al., 2021a;Tadesse et al., 2019) and PTSD (Coppersmith et al., 2014;Amir et al., 2019).
Outside of a formal setting, users communicate freely in text form, resorting to abbreviations, slang or plain spelling mistakes or typos. Eisenstein (2013) further explored bad language on social media, in the sense of language that defies our expectation of good spelling, vocabulary and syntax. He identified several underlying factors for the cause of non-standard text: user illiteracy, length limits imposed by social media sites (i.e. Twitter), text input affordances (i.e. standard mobile keyboards or predictive entry), pragmatics (emoticons/emoji, abbreviations and expressive lengthening), and a social component. Nguyen et al. (2021) further explored the latter, concluding that some types of non-standard text have strong social meaning, and normalization could induce a loss of meaning.
However, it is well known that for most benchmark tasks, noisy/non-standard text has proven to be a real problem to NLP models, such as BERT (Kumar et al., 2020), trained on clean or curated data, but fine-tuned on tasks with noisy and inconsistent format.
To overcome this predicament, Eisenstein (2013) proposes two possible approaches: either domain adaptation or normalization. While domain adaptation is not specific to natural language processing, text normalization and cleaning have always been a central part of any modern text processing pipeline. Text normalization is the process of adapting an input text to a more standard form. It has proven to be effective in increasing performance on tasks such as POS tagging (van der Goot and Çetinoglu, 2021), dependency parsing (van der Goot, 2019a) and sentiment analysis (Mandal and Nanmaran, 2018). Naturally, most text normalization pipelines are based on supervised models, which require carefully annotated data. However, annotating a large corpus of text in multiple languages is often cumbersome and expensive, and some approaches rely on synthetically generating corrupted text (Dekker and van der Goot, 2020;Ma, 2019).
Commonly, approaches are based on word-level normalization. One of the most prominent methods is MoNoise (van der Goot and van Noord, 2017), in which the text correction pipeline is similar to a classic ranked retrieval. However, MoNoise operates at the individual word and uses a spelling correction module and a word embedding module. While word embeddings can be made to account for a specific sentence context, it is mostly discarded.
Different from current methods, we aim to perform text normalization at a sentence level. This approach has several advantages, compared to word or subword methods: i) it can be naturally framed as a sequence-to-sequence type problem, ii) it is more straightforward, as it requires only one module, as opposed to a multi-stage pipeline (i.e. complex candidate generation and ranking), and iii) the same model can be trained on multiple languages at the same time, without increasing in size and computational processing.
In this edition of The Workshop on Noisy Usergenerated Text (W-NUT), organizers propose the shared task of multilingual lexical normalization 1 , in which participants are required to perform lexical normalization on 12 different languages (van der Goot et al., 2021a).
As such, we use the state-of-the-art multilingual sequence-to-sequence transformer model mBART (Tang et al., 2020) and fine-tune it for our task. mBART is one of the first models that can be fine-tuned simultaneously on multiple languages without performance loss. We show that framing text normalization as a neural machine translation problem is a viable method for text normalization, improving performance on extrinsic, downstream tasks compared to models that operate on raw, unprocessed social media text. We made the code publicly available on github. 2

Related Work
The W-NUT workshop hosted a shared task on lexical normalization of user-generated content from English tweets in its first edition (Baldwin et al., 2015a). The task received from the competing teams two categories of submissions, from constrained (using only the training data provided by the organizers) and unconstrained systems (using other publicly available data or tools).
The best model, from Jin (2015), generated candidates from the most similar canonical forms from the training data evaluated with the Jaccard Index. A random forest classifier was used to predict the suitable canonical form from all the candidates using features such as support and confidence, string similarity, and part of speech tags. The model was a constrained system, suggesting that the quality of the proposed model is more important than using additional data and tools. Other approaches were based on conditional random fields (CRF) (Akhtar et al., 2015;Supranovich and Patsepnia, 2015;Akhtar et al., 2015) and recurrent neural networks (RNN) (Min and Mott, 2015;Wagner and Foster, 2015) among others.
Notably, MoNoise (van der Goot and van Noord, 2017) has long been considered state-of-the-art in lexical normalization. MoNoise is a normalization model using spelling correction and word embeddings for candidate generation and a feature-based random forest classifier for candidate ranking. It is a modular normalization system easily reusable and adaptable (van der Goot and van Noord, 2017). The model was at the beginning developed only for English text. Still, then it was later expanded for multi-lingual lexical normalization covering languages such as Dutch, Spanish, Turkish, Slovenian, Croatian and Serbian (van der Goot, 2019b).
The lexical normalization task can also be formulated as a machine translation (MT) task. The noisy user-generated content is the source language, and the canonical form is the target language. Veliz et al. (2019) compare the MT approaches for lexical normalization, focusing on statistical neural translation (SMT) and neural machine translation (NMT) and obtaining better results using the SMT method. Furthermore, the authors show that the SMT approach works better in a low-resource setting than an NMT approach which requires a lot of data.
With the rise in popularity of pre-trained language models for natural language understanding and natural language generation, their ability to perform lexical normalization was also studied. By transforming the task into a token prediction one, Muller et al. (2019) demonstrate that a BERT model can be used as a lexical normalization model in low resource settings.
We propose to make use of the latest transformer models that are capable of multilingual translation in a sequence to sequence manner, namely mBART (Tang et al., 2020). However, we do not perform translation between languages, but instead, we use mBART as a denoising autoencoder, i.e. translating from bad English to good English. This way, we take the whole sentence into consideration when correcting the text. Moreover, this method is more straightforward and can scale to multiple languages without increasing computational demands.

Data and Evaluation
We further describe the dataset for this task and evaluation procedures. MultiLexNorm Dataset The data provided by the organizers includes texts from 12 languages: Croatian, Danish, Dutch, English, German, Italian, Serbian, Slovenian, Spanish, Turkish and codeswitched data for Indonesian-English and Turkish-German, as seen in Table 1. Some examples from the training data are shown in Table 2. For some languages in the dataset, the capitalization (Caps column) is also corrected, and words are split or merged (1-N/N-1 column). The dataset comprises Twitter posts from all languages, but some languages also have texts from additional sources. For example, Danish also has texts from Arto, Denmark's first large-scale social media (Plank et al., 2020) and Dutch texts were also gathered from public Internet forums, and SMS messages (Schuur, 2020). W-NUT Evaluation Methodology The organizers of the W-NUT workshop propose two types of evaluation procedures: intrinsic, word-level and extrinsic, downstream task performance (i.e. dependency parsing). introduced by van der Goot (2019b) is proposed:

As intrinsic evaluation, the Error Reduction Rate
Because accuracy is hard to compare across datasets with different numbers of raw words which have to be normalized, the ERR proposes an evaluation metric that can be used to compare the performance of systems across multiple datasets. It is computed as accuracy normalized for the number of raw words normalized in the gold dataset.
A system that always keeps the raw words has an ERR score of 0.0, while a perfect system will have ERR precisely 1.0. The ERR has a negative value when the system normalizes more words with a wrong form than the correct canonical form.
However, one downside of the ERR is that it fails to distinguish between FP and FN. Thus, in the case of FP, the system may provide a correct normalization, even if the annotators did not normalize the raw word.
Further, two baselines are provided: Leave-As-Is (LAI) -the output is the same as the raw input, the normalization is not performed -and Mostfrequent-Replacement (MFR) -the output is the most frequent replacement from the training data. If the raw word is not found in the training set, no normalization is performed.
As a secondary evaluation, the organizers propose an extrinsic evaluation of the effect of normalization on the task of dependency parsing, previous research showing that lexical normalization improves the performance for this task (van der Goot, 2019a). A dependency parser is trained on
For evaluating on sentiment analysis (SMILE) and offensive language identification (OLID), we trained a simple word-level TF-IDF model together with a linear SVM with balanced weights. For SMILE, we report average macro F 1 score across 5 folds, and for OLID, we report macro F 1 score on the test set. Figure 1: Fine-tuning a mBART model for lexical normalization on all available languages. We use the same model for all languages simultaneously. Lewis et al. (2019) proposed BART in 2019, as a way to pre-train large-scale transformers for sequence-to-sequence tasks. Initially, the authors pre-trained an encoder-decoder transformer only for English, obtaining good results on multiple downstream NLP tasks. Further, mBART (Tang et al., 2020), follows the same procedure, but for multiple languages. The pre-training stage for both BART and mBART is akin to a denoising autoencoder, in which the model receives a noisy (in this case masked) sentence, and it learns to reconstruct it.

Method
While mBART is fine-tuned on multiple language pairs, it is pretrained monolingually, and is capable of acting as an autoencoder for the same language. In our case, we make use of a pretrained mBART on 50 languages 3 from the transformers library (Wolf et al., 2020), and employ a procedure similar to the pre-training stage: a noisy sentence is   Figure 1 showcases our fine-tuning procedure. We tried two different approaches for fine-tuning: Frozen Encoder and Frozen Decoder because, with a fixed encoder, the model suffers from the same OOV-type problems as a typical transformer. However, training with a fixed decoder allows the model to better adapt its representations to each language's noisy version while maintaining its generative properties. For both approaches, we train a single model for all languages. Moreover, we also trained a separate mBART for each language, monolingually.
Training details For all runs, we fine-tune mBART for 50 epochs, using a batch size of 256 and with a cyclical learning rate scheduler (Smith, 2015) that linearly increases the learning rate from 0.00001 to 0.0001 and back across 5 epochs. The workshop organizers provided both the training data and the validation data on most languages. We omit validation on languages where the validation data is missing. The training was performed on an NVIDIA RTX 2070 graphics card. Since the memory requirements of an mBART model are quite high, we employed gradient accumulation to increase the batch size. In addition, we employed early stopping when the validation loss increased for more than 3 epochs. Post-processing Since our model outputs a whole sentence directly, the word-level evaluation requires the noisy input words to be aligned to their normalized counterpart. This phase is essential for sequence-to-sequence text normalization, as bad alignments will reduce the overall word-level performance score, especially in the 1-N/N-1 languages. As such, for the post-processing phase, we aligned input words with their normalized counterparts based on the Levenshtein distance between them. We used a linear sum assignment on the distance matrix to perform matching. Additionally, we matched the capitalization between corrections and left links, hashtags, and user mentions as they are.

Results
We further showcase the results of the pretrained mBART models fine-tuned on the available data: firstly, we kept the transformer encoder fixed and trained only the decoder, and secondly, we kept the decoder fixed and trained the encoder. During this fine-tuning process, we trained a single model for all languages. Further, for the CodaLab submission, we fine-tuned multiple models, one for each language, in the "fixed encoder" regime. Intrinsic Evaluation Table 3 showcases intrinsic, word-level evaluation across languages. Our best model obtained an average ERR across languages of 10.65, corresponding to a separate mBART trained for each language, with the additional postprocessing described in Section 4. In our case, training multilingually did improve performance on some languages (i.e. DE, EN, NL, TR), but over-    all achieved lower scores, especially in the case for HR and SL. For those languages, the model severely diverged, and its output consisted only of repeating the first word in the sentence. As per our intuition, fixing the decoder results in better perfor-mance when compared to fixing the encoder: the model learns to adapt its representations to account for the noisy text.
However, since our method is sentence-based, perfect alignment between words is cumbersome, with many cases of mismatch between punctuation. Moreover, merging or splitting words for normalization is also not taken into account in the post-processing phase. Extrinsic Evaluation For extrinsic evaluation, we showcase the results for our best model in Table 4 on the dependency parsing downstream task from the workshop challenge. Even though our model is not in the top-performing models, the absolute difference in performance is minimal.
Moreover, we also evaluated the effect of lexical normalization on two other tasks -sentiment analysis on the SMILE dataset and offensive language identification on OLID (Table 5). We trained a word-level TF-IDF and a linear SVM with balanced weights for both datasets and reported a macro F 1 score. Our lexical normalization improves results on both these tasks, compared to modelling the raw, unprocessed social media posts. This is because lexical normalization results in a smaller vocabulary for the documents, allowing the SVM model to operate on smaller dimensional data. Moreover, this evaluation procedure is arguably more realistic, as it does not require accurate post-processing to precisely align noisy words with their corrected version and match punctuation. Discussion In Table 6 we showcase some examples for correct, incorrect and questionable text normalizations. The model is able to easily grasp contractions such as u → you and expressive lengthening such as Juhuuuuu → Juhu. However, more complex word abbreviations such as nvm are quite challenging to generate, as the model only outputs the first part (i.e. never). Moreover, code-switched languages are an inherent problem to our approach, as mBART is only trained to receive input from a single language and not code-switched. Interestingly, even though we specified the langauge code for German in the phrase "Why Germany says "nein", the model actually translates the English part into German: Warum Deutschland sagt "nein".
However, as the organizers have pointed out, there are inconsistencies in the training and testing data annotations. In some cases, some words are not normalized (i.e. jajajajaj / w/ / n) even though they were clearly lengthenings or contractions. Despite this, in some of these cases, our model was able to provide correct normalizations.
There also appears to be no correlation between training dataset size and final normalization performance. For example, in the case of Croatian, even though the dataset is the second largest, the performance is lower than for other languages. Thus, the lower performance in some languages may be a cause of the complexity of the language; for English, our model obtained the best results.

Conclusions
In this work, we presented a method to perform lexical normalization by fine-tuning a multilingual machine translation model on pairs of noisy and normalized sentences from various languages. We employed mBART, as it is currently the state-ofthe-art in transformer-based multilingual machine translation, allowing us to fine-tune on all available languages simultaneously. Furthermore, we used mBART as a denoising autoencoder and tuned it in a supervised fashion.
As opposed to current two-stage methods for word candidate generation and ranking, our approach is more straightforward. Moreover, it scales to multiple languages without increasing computational demand (i.e. not increasing vocabulary size, increasing search space and others). Evaluation results show that our method, even though it lacks behind current methods on intrinsic, word-level evaluation, improves performance on downstream tasks.
For future work, we aim to develop our method for better post-processing of the output and increasing augmentation levels -i.e. injecting more noise in the form of spelling mistakes, backwards translations etc. Moreover, since our method is supervised, the quality and quantity of training data play an essential role in the final performance. In this regard, we aim to explore ways to take into account inconsistent annotations. Ana-Maria Bucur, Marcos Zampieri, and Liviu P. Dinu. 2021b. An exploratory analysis of the relation between offensive language and mental health.