Intriguing Properties of Compression on Multilingual Models

Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.


Introduction
Scaling language models benefits multilingual settings, since it is difficult to maintain performance across a growing number of languages at a constant model size, a property also called the "curse of multilinguality" (Conneau and Lample, 2019;Conneau et al., 2020;Artetxe and Schwenk, 2019). However, the extent of growth in language model (LM) size (Radford et al., 2019;Brown et al., 2020;Zhang et al., 2022;Chowdhery et al., 2022) has made deployment to resource-constrained environments much more challenging (Warden and Situnayake, 2019;Samala et al., 2018;Treviso et al., 2022). To benefit from the performance gains conferred by scale, efficiency techniques that reduce model size while maintaining comparable aggregate performance are widely used, such as quantization (Shen et al., 2020), compression (Michel et al., 2019Lagunas et al., 2021) and distillation (Tsai et al., 2019;Sanh et al., 2019;Pu et al., 2021).
While most compression techniques have minimal impact on aggregate performance numbers (Gale et al., 2019;Li et al., 2020;Hou et al., 2020;Chen et al., 2021;Bai et al., 2020;ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can be far more severe (Hooker et al., 2019;Hooker et al., 2020;Ahia et al., 2021). Disparities in resource availability become more apparent at larger scale, both in terms of data and deployment resource availability. This makes compression all the more necessary, but also motivates a thorough consideration of the subsequent impact of compression on generalization.
In this work, we develop an experimental framework to investigate the impact of compression during fine-tuning of pre-trained multilingual models which we apply to Named Entity Recognition (NER) across 40 languages of the WikiAnn benchmark (Pan et al., 2017). We study the impact of compression on groups of languages across multiple dimensions-resourcedness, script, and language family-and evaluate the sensitivity of models to input perturbations along these groupings.
This leads us to discover the following intriguing properties: (1) Lower-performing languages disproportionately suffer under extreme levels of sparsity, as pruning amplifies disparities. However, low-resource languages present an intriguing flipflop moment, where their performance may benefit from medium regimes of sparsity. (2) We find that dense models overfit to typical test cases, achieving a close-to-0 F1 score on slightly perturbed inputs, while compression can recover close to the original test performance. Our results stand in contrast to previous work that find that sparsity erodes robust-ness, suggesting more work is needed to understand the dynamics between compression and robustness.
(3) The choice to prune model embeddings can completely negate the two benefits described in the previous observations, showing the importance of comparing the two cases in future analyses.

Related Work
The "curse-of-multilinguality" creates a tradeoff between number of languages and size of a model (Conneau et al., 2020). However, training smaller models supporting fewer languages may not always be feasible (Abdaoui et al., 2020). Compressing large models has been shown to combat the curse, either by compressing the pretrained model (Tsai et al., 2019;Sanh et al., 2019), or by compressing during finetuning, as in our case. While many studies investigate the impact of pruning on aggregate metrics in monolingual pretrained LMs (Sanh et al., 2020;Goyal et al., 2020;Gordon et al., 2020;Budhraja et al., 2020;Sajjad et al., 2020;Lagunas et al., 2021;Xu et al., 2021;Du et al., 2021a;Ganesh et al., 2021), fewer works focus on multilingual settings (Mukherjee and Hassan Awadallah, 2020;Ansell et al., 2022). Yet, prior analyses find a disparate effect of removing attention heads or model layers on languages and language families distant from the training data in NER (Ma et al., 2021;Budhraja et al., 2021), demonstrating the importance of looking into subpopulations as we do in this study. Concurrent work (Mohammadshahi et al., 2022b) explores compression on multilingual machine translation models, and find that while under-represented languages suffer extreme performance drop, some mediumresource language benefit from compression. However, unlike our proposed framework, they perform this study without additional fine-tuning after compression.
Studies that compare the robustness of compressed and dense models further find that compression may lead to erosion of performance on "challenging" samples and poor generalization (Ahia et al., 2021;Du et al., 2021a;Xu et al., 2021), a finding that we expand on and connect to language resourcedness. The technique we use to study robustness expands on studies that perturb training (Yaseen and Langer, 2021;Dai and Adel, 2020) or evaluation data (Dhole et al., 2021) in NER by introducing perturbations specific to languages, language families, and scripts.

Methodology
Data We conduct our experiments on WikiAnn (Pan et al., 2017), a multilingual NER dataset. WikiAnn was sourced from Wikipedia articles and automatically annotated with LOC (location), PER (person), and ORG (organisation) labels in the IOB2 format (Ramshaw and Marcus, 1995). It is considered a "silver standard" due to its automatic entity labels and noise (Lignos et al., 2022), but with its 176 languages it covers the most languages of any NER dataset. We focus our experiments on the 40 languages from the XTREME benchmark (Hu et al., 2020), with train-test splits defined by Rahimi et al. (2019). These training sets were built with stratified sampling to create a balance across entity types (Lignos et al., 2022), and are thus a subset of the total available data from the original WikiAnn. Table 1 lists language codes in ISO 639-1 and their available training data for fine-tuning.
Perturbations We test the robustness of compressed models by perturbing named entities in the test set. Previous work (Du et al., 2021a) show that sparse pretrained language models are less robust than their dense equivalents when evaluated on adversarial test sets, even when they perform similarly on in-distribution test sets. We adopt a data perturbation technique from Dai and Adel (2020) called entity mention replacement; an entity is randomly swapped with another entity of the same type (example sentences shown in App. D). We first perturb entities within same language for all the languages in our dataset (in-language). Secondly, we propose a new benchmark appropriate for testing the cross-lingual robustness of multilingual models on our downstream task. We perturb entities across different languages that share common linguistic properties. In particular, we group languages by family and script and perturb entities across languages within the same group (in-script, in-family).
Model We use the cased multilingual BERT (mBERT) (Devlin et al., 2019) for all our experiments because it is one of the most widely used and studied multilingual LMs (e.g., Pires et al., 2019;Rönnqvist et al., 2019;Wu and Dredze, 2019;Wang et al., 2020;Chi et al., 2020 ,he,hu,id,it,ja,ko,ms nl,pt,ru,th,tr,ur,vi,zh  trained on Wikipedia data from 104 languages, has approximately 177 million parameters, and a vocabulary size of around 120,000. We finetune mBERT by appending a linear classification layer to the model and updating all its parameters. Full hyperparameters are listed in App. A.1. We evaluate the impact of sparsity in two settings: 1) In the monolingual setting we fine-tune on individual languages. For this setting, we select 10 languages with different available data size, language family and scripts (see Table 1). 2) In the multilingual setting we jointly fine-tune on all languages. We train all models with three random seeds, and evaluate F1 using seqeval (Nakayama, 2018) on the individual languages' evaluation data. We report mean results across runs after computing the micro-average F1 scores across entity classes.
Pruning We induce sparsity by applying Iterative Magnitude Pruning (IMP) (Han et al., 2015(Han et al., , 2016 during fine-tuning. IMP iteratively removes weights that are below a certain threshold until a desired target sparsity is reached. IMP is widely used and competitive with far more compute intenparameters that replicate the prior results.  sive approaches (Gale et al., 2019;Gordon et al., 2020;Du et al., 2021b;Ganesh et al., 2021), while allowing us to sparsify to an exact level. We compare two pruning strategies: 1) partial where we prune all dense layers except for embedding layers, 2) incl. embeddings where we prune all dense weights including embedding layers. Embeddings make up more than half (91M) of the 177M parameters in mBERT, while dense weights make up the rest. Hence, pruning embeddings allows us to significantly reduce the number of mBERT parameters. We consider five sparsity levels: 50%, 70%, 80%, 90%, 95% and 98%, corresponding to the percentage of weights pruned (hyperparameters in App. A.2. Preliminary experiments were conducted with lower sparsity levels (10%-40%) and yielded similar findings to those at moderate sparsity levels (50%-70%), motivating the sparsity intervals chosen. The chosen sparsity levels also align with general best practice in sparsity evaluation as presented in previous works. Moderate to high sparsity levels (50%+) are necessary for efficiency gains in the real-world and are usually studied in literature ( Sparsity (Pruning with Embeddings)

Type of Perturbation
In-Language In-Script In-Family Figure 3: Regular vs Perturbed: We show the aggregated results across all languages after perturbation at different sparsity levels. Without pruning, the model performs poorly, which is overcome by partial pruning, but not pruning with embeddings. The relative performance drop is consistent across all pruning levels above 0.
to benefit more from crosslingual transfer. 2 We find that this finding holds under sparsity -multilingual models achieve higher F1 than monolingual models not only in the dense setting, but across all sparsity levels, as shown in Figure 1. 3 At high sparsity levels, the loss in quality that is generally incurred is considerably lower for multilingual models. This suggests that when high levels of compression are necessary (e.g. for inference efficiency needs), multilingual training should be preferred to monolingual training, as it could help offset some of the erosion in the performance caused by the compression. Thus, we conclude that the benefits of cross-lingual transfer are not inhibited by pruning, and perhaps are even more pronounced at a lower capacity (Dufter and Schütze, 2020) for certain languages.

Impact of pruning across languages
Figure 2 displays the relative differences in F1 score between dense and sparse models across languages, grouped according to fine-tuning size. 4 At moderate sparsity levels (50%-70%), partial pruning surprisingly improves over the dense models, in particular those with less fine-tuning data. The majority of languages (26 out of 40) benefit from moderate pruning and yield slightly higher F1 with pruning than without. All three datasets with only 100 fine-tuning examples (yo, my, jv) benefit. This suggests that moderate pruning may benefit low-resource datasets when introduced during a finetuning regime. However, at high sparsity levels (70%-98%), the findings reverse. Those languages that have a lower frequency of representation in the finetuning dataset incur the highest absolute and relative loss in quality. We can observe the same trend when grouping languages according to their family or script, respectively (see Fig. 4 and 5 in App. B). The groups that start with the lowest average performance under the dense model, also suffer the most under extreme sparsity.
In conclusion, moderate pruning levels should be explored for low-resource languages since they may benefit such languages. This is especially important since models for low-resourced languages are often deployed in resource-constrained environments § (Ahia and Ogueji, 2020;Nekoto et al., 2020;Ahia et al., 2021;Ogueji et al., 2021;Mohammadshahi et al., 2022a). Also, since high sparsity levels reinforce existing disparities (as measured by model performance and data availability) between languages and language groups, it is imperative that practitioners pay attention to possible disparities when sparsifying models. Figure 3 shows the relative performance on the perturbed sets as a fraction of the corresponding unperturbed performance. Across all perturbation types, the dense model performs poorly, indicating that the model may have overfit to typical entities and the semantic context that appear in the training corpora. 5 Surprisingly, partial pruning at any level (2021) find that magnitude pruning improves model robustness to out-of-distribution shifts in machine translation. Despite the contradictions, our work represents an important step in understanding the impact of pruning on robustness, especially since we are one of the firsts to explore it multilingually. Interestingly, our findings are consistent across all perturbation types as their scope increases from languages (in-language) to scripts (in-script) and families (in-family). This suggests that sparsity can be explored as an avenue to improve robustness as has been explored in previous works (Xu et al., 2021;Ahia et al., 2021).

How does pruning impact robustness?
However, pruning the embeddings makes a crucial difference for the perturbed test cases. While pruning the embeddings does not matter for regular test set (see Figure 2), we observe the same severe drop in performance on the perturbed test-set as for the dense model. This suggests that including model embeddings when pruning sharply erodes performance on out-of-distribution rare artefacts, prompting a closer look into what is pruned in the embedding space and the potential impact of sparsifying different parts of a model.

Conclusion
This work investigates the effects of compression on multilingual pre-trained language models during fine-tuning. Our analysis revealed several intriguing properties of pruning that should inform future work in this direction: (1) Pruning dense layers up to ∼ 70% may improve quality for lowfrequency examples in the data and enhance model robustness.
(2) The decision to prune embeddings may have critical impact on model robustness to out-of-distribution performance. (3) While lowperforming languages benefit from moderate pruning, they are disproportionately harmed when pruning more aggressively. Based on these intriguing properties, we also make several recommendations to machine learning practitioners.
performance of dense models on the perturbed test sets.

Limitations
We detail the following potential limitations of our work: Noisy dataset: Lignos et al. (2022) shed light on several quality issues of the WikiAnn dataset that we are treating as a gold standard. Our results might thus not adequately reflect NER performance that can be achieved with cleaner and humanannotated datasets, such as the MasakhaNER (Adelani et al., 2021) or SADiLaR (Eiselen, 2016). Since the perturbations are based on the WikiAnn labels, we might be amplifying the existing label noise for the perturbed test sets and as a result underestimate model quality on clean perturbed examples. We try to combat the randomness by averaging results across three separate runs, but any issues intrinsic to WikiAnn will likely impact all three.
Other Multilingual Models and Downstream tasks: Multilingual pre-trained models such as XLM-R (Conneau et al., 2020) might yield a better performance or show slightly different trends across languages (Adelani et al., 2021). Other downstream tasks, especially generation tasks, might tolerate different levels of sparsity, and also show different crosslingual transfer capabilities (Wu and Dredze, 2019; Hu et al., 2020). However, since fine-grained prior results on the same WikiAnn splits were not available to us, we restricted the analysis to mBERT where we could verify that we can replicate the results reported by XTREME.
Evaluation metrics: We use F1 as the sole evaluation metric and trust it to reflect quality adequately across languages. Human evaluation and the use of qualitative evaluation metrics might reflect the quality for individual languages better.

Unknown factors influencing performance:
The absolute performance for a given language can be influenced by many factors including size, family and script, relatedness to other languages, and the inherent difficulty of the NER task and the evaluation examples, as studied in related works (e.g., Pires et al., 2019;Wu and Dredze, 2020;Shaffer, 2021;Adelani et al., 2021;Muller et al., 2021;Deshpande et al., 2021). As a result, it is impossible to identify the exact cause for all our observations and we have to partially rely on correlational analyses.  Figure 9 shows the correlation between percentage entity overlap and F1 on dense multilingual models.

C Full Results
We present the results for individual languages on both the regular and perturbed test sets obtained via multilingual finetuning in tables 2, 3, 4 and 5 We present the results for individual languages on both the regular and perturbed test sets obtained via monolingual finetuning in tables 6, 7, 8 and 9

D Examples of Perturbed Test Sentences
We present examples of perturbed test sentences in the in-language setting for English (table 10) Table 9: F1 scores for monolingual fine-tuning on the perturbed data for various levels of sparsity with pruning embedding layers.

Example English test sentences
Original Much construction was undertaken during this period , such as the building of Palermo Cathedral . Perturbed Much construction was undertaken during this period , such as the building of Knott 's Soak City .

Original
It is found in Peru . Perturbed It is found in Carbon Cliff , Illinois .

Original
It flows from Ägerisee through Lake Zug into the Reuss . Perturbed It flows from New Orleans through Humboldt County , Nevada into the Crow Agency , Montana .

Original
The album 's lead single " Better Believe It " featuring Young Jeezy and Webbie , was released on July 14 , 2009 . Perturbed The album 's lead single " Better Believe It " featuring W. S. Merwin and Empress Maria Theresa , was released on July 14 , 2009 . ' " ' " jé Fáráò ni Égíptì Ayéijóun . Perturbed ' " ' " j Yousaf Raza Gillani ni Nàìjíríà . Table 11: Example of test sentences for Yoruba language using the entity mention replacement (Dai and Adel, 2020) technique where an entity is randomly swapped with another entity of the same type.