Medicine
Computer Science-2021.gz:143342
Computer Science
3e65f3f0c122cf6bfd9a038e4c2711dbc5286652
A Multilingual Approach to Identify and Classify Exceptional Measures against COVID-19

The COVID-19 pandemic has witnessed the implementations of exceptional measures by governments across the world to counteract its impact. This work presents the initial results of an on-going project, EXCEPTIUS, aiming to automatically identify, classify and com- pare exceptional measures against COVID-19 across 32 countries in Europe. To this goal, we created a corpus of legal documents with sentence-level annotations of eight different classes of exceptional measures that are im- plemented across these countries. We evalu- ated multiple multi-label classifiers on a manu- ally annotated corpus at sentence level. The XLM-RoBERTa model achieves highest per- formance on this multilingual multi-label clas- sification task, with a macro-average F1 score of 59.8%.


Introduction
The increasing availability of digitized and publicly available legal documents has boosted political scientists, legal scholars, lawyers and policy-makers to apply Human Language Technologies (HLT) to discover, analyze, digest, and use automatically extracted information. All of these operations fall under the larger area of study that can be labeled as Legal Artificial Intelligence, or LegalAI (Zhong et al., 2020). Similarly to any domain that wants to apply HLT to process written documents, LegalAI requires the development of both domain-specific languages resources (e.g., annotated corpora or embedding representations (Chalkidis et al., 2019c;Kornilova and Eidelman, 2019;Holzenberger et al., 2020;Luz de Araujo et al., 2020;Samy et al., 2020)) and tools (e.g., language specific natural language processing pipelines (Koeva et al., 2020; or pretrained language models (Chalkidis et al., 2020)). At the same time, LegalAI presents an additional challenge when it comes to the development of multilingual systems. With very few exceptions (e.g., EU-level legislation), in the domain of LegalAI, natural languages are strictly connected to countries, meaning that different legislative systems and practices may be in place. For instance, although French-speaking countries share a common language, their legal traditions widely differ making comparisons across legal systems uneasy. Nevertheless, the adoption of a multilingual approach may prove valuable especially to legal practitioners and scholars as well as political scientists and policy-makers who are increasingly interested in comparing legal systems and examining how legal concepts "travel" across time and spaces. Commonly used data collection and analysis techniques -largely relying on manual or rule-based coding of legal documents -have so far prevented the development of meaningful and systematic analyses allowing the comparison of fine-grained classes.
In this paper we investigate the potential of multilingual pretrained language models in order to facilitate the analysis, exploration, and comparison of legal texts on COVID-19 exceptional measures. Our major contributions can be summarised as follows: • the creation of a new corpus of legislative documents from 21 European countries manually annotated for exceptional measures against COVID-19 (Sections 2 and 3); • the development of a rich taxonomy (eight classes and 83 subclasses) to identify and compare exceptional measures in a consistent way (Sections 4); • the development of a multi-label classifier based on XLM-RoBERTa (Conneau et al., 2019) to identify exceptional measures at sentence level (Section 5).

The Exceptional Measures Against COVID-19 in Europe
The COVID-19 pandemic has led governments around the world to take exceptional measures in order to contain the spread of the virus. Such exceptional decision-making has seen executives challenge the scope and legality of their powers, as well as impose restrictions on democratic processes, the rule of law, fundamental rights and civil liberties. These exceptional measures considerably vary from one country to another, even in cases where some forms of coordination are claimed to be in place, like in the European Union. Countries sharing close political culture and institutions reacted in contrasting ways, as attested by the sharp difference between the Belgian and Dutch responses to the crisis between March and June 2020. While the Dutch government implemented one of the softest approaches in Europe relying on people's compliance with governmental recommendations, Belgium introduced very early a strict lockdown. This diversity is not only practical but also semantic. While many countries relied on a "lockdown" to contain the spread of the virus, restrictions vary in scope while enforcement modalities are unequally coercive (Engler et al., 2021;Egger et al., 2021). This fragmented political response sparked interest from researchers in political science, economics, and law that started to trace exceptional decision-making in times of COVID-19 (Porcher, 2020;Hale et al., 2021). To the best of our knowledge, all current data collection efforts are based on manual or rule-based methods applied to press releases (Hopkins and King, 2007;Grimmer and Stewart, 2013;Wiedemann, 2013;Wettstein, 2014) or on experts survey. Yet, in the specific case of the COVID-19, such methods suffer from three core limitations: Decisions were taken on different legal bases There is a variety of legislative tools that have been put in action across countries to counteract the spreading of the virus. Some governments activated crisis-management instruments and legal frameworks, including, but not limited to, the activation of state of emergency provisions (Bjørnskov and Voigt, 2021) that predate the crisis. Others took decisions in an ad hoc manner on the basis of executive, legal or administrative acts taken not only at the national but also at the subnational level.
Measures evolved quickly At least for the first wave of the COVID-19 (late January -June 2020), measures evolved on a weekly, and sometimes, on a daily basis, requiring researchers to handle a very large amount of constantly evolving textual data. Since the application of close reading methods (i.e., extensive manual annotation) to such a large amount of documents is a daunting task to perform over a reasonable period of time, most of the competing research teams opted for collecting data on broad classes of events (lockdown, border closure, state of emergencies) based on press releases, conferences or experts opinions. The lack of fine grained classes derived from legal texts provides a false impression of homogeneity between various governmental responses, especially in a context of semantic ambiguity about the measures used.
Different countries, multiple languages Being a pandemic, the COVID-19 emergency affected the entire world. This global condition is accompanied by a rich and diverse language composition of any corpus created to investigate and compare the legislative measures of different countries. The intrinsic multi-lingual nature of this corpus has raised additional challenges for coding methods traditionally used in social sciences. Only a few legal texts are translated in English and some national languages are spoken by a fairly limited number of people.
Against such a background, we have initiated a research project, EXCEPTIUS (Exceptional measures in times of COVID-19) 1 to collect and document metrics of exceptionalism in 30 countries of the European Economic Area (EEA), plus UK and Switzerland, starting from late January 2020. EXCEPTIUS intends to address the abovementioned challenges in the analysis of COVID-19 measures in three ways. First, measures are automatically captured from a homogeneous corpus of legal sources uniquely allowing researchers to analyse the diversity of the legal instruments used to contain the COVID-19 pandemic. Press releases or expert surveys commonly used in competing projects only capture such dimensions indirectly and imperfectly. Second, our project defines the most comprehensive taxonomy of exceptional measures in the field of democratic governance, the rule of law and fundamental rights and liberties. The automatic application of such taxonomy to a comprehensive legal corpus allows to conciliate the need to rely on fine-grained categories with the constraints deriving from the analysis of a large and constantly evolving corpus. Last, we adopt a multi-lingual approach to automatically analyse the sources of COVID-19 legislation, limiting the bias associated with the translations of the original texts in English. The reliance on multi-lingual methods adopts a philosophical perspective of Artificial Intelligence (AI) as a problem-solving tool rather than as an adaptive mechanism mimicking human abilities (Winograd, 1997;Auernhammer, 2020;Caselli et al., 2021). The "intelligent" systems developed in this project (see Section 5) do not aim at substituting humans but are designed to account for different development cycles with humans in the loop. The use of automatic methods based on HLT allows us to overcome in a smart and fast way the three challenges previously described.

Corpus Collection
The corpus collection process has been overseen by four political science experts working in partnership with national legal experts. All documents were retrieved from official governmental websites that publish legal acts. The identification of the relevant documents has been done by means of 4 keywords (i.e., "COVID", "COVID-19", "Coronavirus" and "Health emergency"). For each language, the corresponding language specific keywords were used. In this initial phase, we focus on a sample of 19 EEA countries on measures adopted at the national level. To do so, we identify publicly available links to relevant documents 2 plus UK and Switzerland. We could not find corresponding documents for two countries of the EEA (i.e., Bulgaria and Greece). All documents have been collected either by manually downloading them or by automatic scraping. 3 For countries with more than one official language (e.g., Switzerland), legal acts were collected in all available languages. 4 A total of 6,449 documents has been collected and stored in text format so far. Documents form a homogeneous set of existing COVID-19 legislation. Such legislation however includes a variety of texts adopting by political authorities acting at different levels. Beside legal acts -adopted by 2 The EEA countries are: Austria, Belgium, Croatia, Cyprus, Czechia, Denmark, France, Germany, Hungary, Ireland, Italy, Latvia, Lithuania, Netherlands, Poland, Sweden, and Spain.
3 All scripts, corpus, annotated data, and system(s) are available at https://github.com/tommasoc80/ COVID19_emergency_event 4 Belgium is an exception: we could collect only documents in French. national parliaments -the corpus also includes executive acts -adopted by governmental authorities which were granted exceptional powers during the COVID-19 crisis -and administrative acts which mainly specifies how the implementation modalities of measures adopted by parliaments and governments. The distribution of the documents per country varies greatly: from 9 on the federal level in Germany to 969 in Slovenia, with 212.5 being the median. These differences are mainly due to a variability across the countries concerning the institutional levels responsible for taking actions against the spread of COVID-19: for example, the low number of German documents reflects the fact that the Länder were responsible for enacting COVID-related measures. We further process all documents using the SpaCy UDPipe 2 NLP pipeline 5 (Straka and Straková, 2017) using models trained on the Universal Dependencies project (Nivre et al., 2016(Nivre et al., , 2020. Although the UDPipe models may be sub-optimal for processing legal documents, the availability of models and unified representations (i.e., same sets of labels for parts-of-speech tagging and dependency relations) for all the languages of the corpus is an advantage. The SpaCy UDPipe 2 pipeline successfully processed 6,049 documents, providing sentence splitting, tokenization, part-of-speech tagging, and dependency relations. Not processed documents are due to issues in the text format due to the conversion process from pdf or other formats. An full overview of the processed data is reported in Table  A.1 in Appendix A.
The corpus covers 17 languages, 16 belonging to the Indo-european family and one to the Uralic family (i.e., Hungarian). The length of the documents is a dimension of variation among the countries, dependent mostly on different archival practices across the countries. Some countries include only the changes to legal acts, while others combine the original text and the changes. A further peculiarity concerns the levels of detail that accompany the modalities of the measures taken, with some countries being very precise and others addressing the issue in a much broader manner. These dissimilarities can be observed by looking at the average number of sentences per country / per language. Countries such as Switzerland, Latvia, Slovenia, and Czechia appear to have the longest documents # Subclasses Example  The current corpus consists of 18,714,750 tokens. Variation in the number of tokens is quite spread, with Slovenia having more than 4 millions tokens and Norway only 6,037, followed by Germany with 12,011 tokens. Aggregating per language 6 changes the distribution of the data, leaving Norwegian as the least represented language, followed by Lithuanian with 42,761 tokens. In this setting, seven languages have more than 1 million tokens (French, Slovene, Latvian, Greek, English, Dutch, and Span-ish). At this point, two remarks deserve to be made about our corpus. First, although comprehensive, our corpus is relatively small and its limited size may negatively impact the quality of subsequent analyses. We are aware of this limitation and intend to address it in future work. Second, and while the size of the documents varies per country, our corpus includes relatively short documents when compared to other types of legislation. This may be due to the specific nature of the issue at stake as COVID-19 containment measures were taken in an ad hoc, fragmented nature and were often not based on pre-existing crisis-management legislation.

Annotating Exceptional Measures
The identification of the exceptional measures has been conducted by applying a taxonomy of 8 classes. Note that, although the overall project focuses on a large range of subclasses, the size of the corpus and the dispersion of the subclasses in the initial phase of project presented in this paper did not allow to annotate documents at the subclass level.
Defining the taxonomy A multidisciplinary Scientific Board of 8 experts in comparative politics,  crisis-management policies, public health, comparative law and human right law from five EU countries identified the eight classes and 83 subclasses that compose the taxonomy. The classes and their subclasses have been identified by applying both top-down and bottom-up methods. In particular, this was done by reviewing similar annotation initiatives (Cheng et al., 2020;Porcher, 2020;Hale et al., 2021) and by manually analyzing a random sample of 50 documents from the corpus. The sample contains at least one document per country. Table 1 illustrates the eight classes, the associated number of subclasses, and one example. 7 The classes cover key measures and variations in the level of coercion used in their implementation. Gathering data on implementation modalities is crucial to capture differences in policy styles that may be hidden between the use of the same term to refer to COVID-19 policy responses.
Annotating the measures A subset of 281 documents in eight languages has been selected for manual annotation. The annotation of the exceptional measures applies at sentence-level. The sample is based on the French, Polish, Dutch, English, Hungarian, Belgian, Italian, and Norwegian subcorpora. Annotators were allowed to assign as many subclasses as they consider relevant to each sentence, but with a total of eight main classes of exceptional measures. Sentences can potentially entail multiple exceptional classes, making this a multi-label annotation task. The annotation process results in eight binary annotations per sentence, with 0 if the specific class is not identified within the sentence and 1 if it is.The annotation 7 The full list of the subclasses is presented in Appendix C.
has been conducted by three experts in political science working under the supervision of the project's Scientific Board. Since the annotators are not fluent in all languages and due to the impossibility of recruiting expert native speakers, some documents need to be translated 8 into English to be manually annotated. No inter-annotator agreement study has been conducted in this initial phase. We intend to remedy this limitation in the project's next development cycle. However, during the annotation phase, annotators met on a weekly basis to discuss ambiguous cases and the guidelines. Annotators are encouraged to propose new classes or subclasses. For a new (sub)class to be accepted, the measure should have been independently identified by the majority of the annotators. In this phase, no new classes were proposed. Table 2 summarizes the results of the manual annotation. Differences across countries affects the distribution of the exceptional classes. Most sentences are mapped to one class only, making the absence of any measure (i.e., the 0 label) by far the most frequent. This is expected given the peculiar structure and style of legal texts. The variation in the distribution of the exceptional classes affects both the presence of specific classes in some countries and their frequency. For instance, France is the only country where all exceptional classes are present (although with varying frequencies); the Netherlands is the country with the fewest classes (only two, namely E3, indicating restrictions of daily liberties, and E4, regulating closures/lockdown); finally, Hungary is the country with the least amount of mentions of measures,  with only seven annotations. In general, the most frequent classes are E3 and E4, while E6 and E7, involving police and army involvement respectively, are the least frequent. Although partial, the manual annotation already provides an indication of the difference and similarities of how different countries and political systems reacted to COVID-19.

Experiments
We have run a set of experiments to develop a tool to support the work of political scientists and other scholars to analyse the large amount of documents in the corpus. In this paper, we present the first development cycle of this tool that targets the automatic identification of the exceptional measures at class level. We opted for this setting mainly due to the low amount of positive instances and the sparseness of the subclass annotation. In particular, we investigate an array of machine learning algorithms distinguishing between feature-based (Section 5.1) and model-based (Section 5.2). The goal is to identify which approach works best for this task. Following the annotation method, the task is framed as a multi-label sentence classification task. Given the distribution of the manually annotated data per language and country, we opted to experiment directly using a multi-lingual setting. We report in Table 3 the split of the data into training, development, and test that we used for our experiments. Full details of the data distribution per country and per language is presented in Table  A.2 in Appendix B

Feature-based Methods
We adopt a standard method for sentence representation, akin to the Bag of Embeddings (BoE) paradigm (Jin et al., 2016). In particular, we extract features from the input text to embed each sentence and use a shallow learning architecture for classifying it to (possibly more than one) related exceptional class. Representations include: N-grams We extract Term Frequency -Inverse Document Frequency (TF-IDF) features based on both word and character n-grams, with n ∈ {2, ..., 5} for words and {3, 7} for characters. We deal with the sparsity of the resulting features by applying Latent Semantic Analysis (LSA), by decomposing the n-gram feature matrix into its truncated singular value components (Halko et al., 2010). The overall feature extraction -decomposition pipeline transforms each sentence of our input text to a single dense vector.
Word Embeddings We utilize a multilingual version of the pretrained GloVe word vectors (Pennington et al., 2014;Ferreira et al., 2016). Word vectors from each sentence are aggregated using average pooling in order to provide a single vector representation for the sentence.
Feature vectors from both methods are concatenated and used as input to a classifier, which is trained on our supervised corpus using the sum of eight parallel binary cross-entropy losses, one per exceptional class, as to accommodate the potential existence of all classes within the same sentence. We experiment with three classifiers, namely: i) a Support Vector Machine (SVM) with a linear kernel, ii) a Multi-Layered Perceptron (MLP) with a single hidden layer and iii) a bi-directional Gated Recurrent Unit (GRU) neural network encoder (Chung et al., 2014) that contextualizes the input sentence before concatenating with the LSA-based feature vector and passing to the classifier. This latter method operates directly on GloVe embeddings, serving as a trainable alternative to the simple average pooling strategy used in the first two bagging approaches. For training the neural methods we use the AdamW optimizer (Kingma and Ba, 2017; Loshchilov and Hutter, 2017), and select hyperparameters after performing grid-search. Resulting values are: learning rate of 10 −3 , dropout probability 0.25, weight decay of 10 −2 , MLP hidden size of 128, GRU hidden size of 150, 100 LSA compo-nents and an early stop patience of 3 epochs. We perform model selection based on the performance in the development set, using averaged F1-score as the target metric for early stopping and report results on the test set.

Model-based Methods
In this second experiment setting, we initially followed the standard approach of fine-tuning a Transformer-based Language Model (TLM) on the annotated data. After experimenting with a range of multilingual models, we report the results of the best model, namely XLM-RoBERTa (Conneau et al., 2019). We fine-tuned XLM-RoBERTa using the AdamW optimizer with a learning rate of 3 × 10 −5 and weight decay of 10 −2 . Similar to the word-embedding setting, we train using a binarycross entropy per output objective, and report results on the test set after selecting models according to their F1-score performance in the development set.
Besides obtaining state-of-the-art performances, it is known that generic TLMs suffer when applied to domains different from the one ( Given the peculiarity of the task and the domain of the texts, we explore the potential effectiveness of adding an intermediate training step in the performance of the language model in the downstream classification task, aiming at first adapting the pretrained language model to the domain before finetuning. We thus further pre-train XML-RoBERTa using the entire collection of document composing the corpus (i.e, 6,649; Section 3). We replicate the XLM-RoBERTa pretraining process, applying the same random chances for masking and making sure that continuous spans of part-word tokens are mutually masked. We split our dataset into train-dev-test splits using 80 − 10 − 10% of the documents per country (with a minimum of 2 documents for each split) and train with a masked language modeling (MLM) objective. The dev split has been used to select the best further trained model. We use a batch size of 16, and train for a maximum of 36 epochs, where the MLM loss saturates. Once the newly adapted model has been generated, we repeat the fine-tuning and evaluation step that we applied for the generic XLM-RoBERTa model.

Results
Comparative results for the supervised multi-label classification task for all methods are presented in Table 4. We evaluate against several sentence and word-level multi-label metrics and include results from a dummy baseline that always predicts negative existence of exceptional classes for all samples. We followed an evaluation approach similar to Named Entity Recognition (NER), where only the positive classes are evaluated. The high accuracy metrics in the dummy case showcase the under-representation of positive classes, further highlighting the challenge of the task. However, it is apparent that all learning methods greatly aid in modeling the task, with the best performing method being the XLM-RoBERTa language model. Although marginal, the better performance of the further trained model (row XLM(pre@36) in Table 4) suggests the potential effectiveness of this technique.
We further evaluate the best system, XLM(pre@36), for its language adaptation capacity by performing a series of zero-shot experiments. In particular, we fine-tune the XLM-RoBERTa models using the manually annotated training data from all countries/languages except one that is used for testing. With this experiment we want to identify whether our multilingual model is capable of learning cross-lingual concepts that are general enough to successfully detect the measures in new languages. This is also a strategy to check the expected performance of the trained models on the not-annotated documents of the corpus. The results of this experiment (presented in Table 5) highlight the intrinsic challenge of multilingual knowledge transfer across legal domains: even though sharing linguistic information, each country is very much bound to the specifics of their legal system. This is mostly evident in the results for Belgium and France, where even though both datasets are in French and comparable in size (see Table 2), individual scores are higher than 60% only when country specific training material is added. However, we observe that on average the zero-shot performance is on par with the featurebased baselines, hinting towards the benefits of incorporating such a multilingual zero-shot system   in a human-in-the-loop co-annotation scenario, serving as a draft analysis that human experts can iterate over in future development cycles of the system.

Related Work
LegalAI has a longstanding tradition with early works dating back from the 1960s (Kort, 1957;Ulmer, 1963) and has seen the development of a variety of tasks ranging from the development of domain specific ontologies and lexica (Breukers and Hoekstra, 2004;Lame, 2005;Peters et al., 2007;Bonin et al., 2010;Francesconi et al., 2010), to automatic classification of legislative documents (Bartolini et al., 2004;Moens et al., 2007;Gonçalves and Quaresma, 2005;de Maat et al., 2010;Chalkidis et al., 2019b;Soh et al., 2019), automatic summarization (Farzindar and Lapalme, 2004;Galgani et al., 2012;Polsley et al., 2016; Feijo and Moreira, 2019), court judgment predictions (Zhong et al., 2018;Chalkidis et al., 2019a;Medvedeva et al., 2020), legal entities detection and classification (Cardellino et al., 2017;Leitner et al., 2020), and question answering systems (Taniguchi and Kano, 2016;Delfino et al., 2018;Kien et al., 2020). Most of current work is embedded in the paradigm of Deep Learning, using embedding representations, neural networks or large pre-trained language models. The emergence of Deep Learning has been accompanied by a growth in specialized embedding representations for the legal domain. Similarly to other areas of applications of HLT, the legal domain has seen two waves of embedding methods. The first is has seen the application of static methods (e.g., Word2Vec, GloVe, FastText, Doc2Vec, a.o.) based on characters, words, or even documents (Ash and Chen, 2017;Chen and Ash, 2019;Chalkidis and Kampas, 2019;Kayalvizhi et al., 2019;Noguti et al., 2020) and their integration in neural network architectures. The second wave has seen the development of contextualized representations and the generation of domain-specific transformer based language models (Chalkidis et al., 2020). Our work is related to this second wave of embedding representations, in particular, by using a generic multilingual pretrained model such as XLM-RoBERTa. On the contrary, rather than creating domain specific and multilingual language models from scratch, we applied Task Adaptive Pretraining (TAP) on the line of Gururangan et al. (2020b) to the generic XLM-RoBERTa as a strategy to boost early domain adaptation.
Similarly to previous work, we perform an Information Extraction task framed as classification problem at sentence level. Neill et al. (2017) introduces a sentence classification task aiming at extracting different modalities from financial legislative document in English. Modality plays a pivotal role in order to distinguish between what is permitted, prohibited, obliged. They are able to achieve an F1 score of .79 using a BiLSTM model and combining an ensemble of domain specific and generic word embedding representations based on Word2Vec. Chalkidis et al. (2018) improves along different dimensions including a more powerful Bi-LSTM system, a larger dataset, and the use of more fine-grained classes. Other works have applied the same task to sentences in German tenancy law (Waltl et al., 2017), and US and Italian regulations (Kiyavitskaya et al., 2008).
Other works have focused on the extraction of contract elements (Chalkidis et al., 2017) or text classification (Sulea et al., 2017;Wei et al., 2018;Chalkidis et al., 2019b). To the best of our knowledge, we have identified limited previous work on multilingual or cross-lingual applications to the legal domain (Galassi et al., 2020;Chalkidis et al., 2021). In this respect, in our work the multilingual dimension plays a pivotal role in the development of our approach. Given the homogeneity of the topic of the legislative documents taken into account (i.e., COVID-19 legislation), the multilingual dimension has been exploited to account for the limited amount of manually annotated documents in each language and country.

Conclusion and Future Work
In this paper, we have presented a new corpus and a taxonomy to identify exceptional measures implemented to counteract the COVID-19 emergency across 21 countries. A subset of the corpus (281 documents, 4,145 sentences) has been manually annotated. The data have been successfully applied to develop a fist version of a classifier based on a domain-adapted multi-lingual language model (XLM-RoBERTa) to support experts in investigation of the measures and their impact in the society.
Besides the relatively small size of the training data, the final score of the system (F1 score 59.8) indicates promising applicability at the final stage of the project.
In the future, we plan to develop the project in two directions. First, we will extend the corpus to include additional countries/languages and extend the data to the regional/municipal levels. This will allow us to further adapt XLM-RoBERTa. Active learning methods can be adopted to boost the annotation process, leveraging on the fine-tuned models to auto-tag and manually review documents in batches to accelerate the annotation process. Second, we plan to develop a fine-grained version of the classifier to include the taxonomy's subclasses. Finally, we wish to further study the predictions of the classification system in the unsupervised data and identify potential cross-lingual keywords and/or topics that relate to each exceptional class separately.