Universal and Independent: Multilingual Probing Framework for
Exhaustive Model Interpretation and Evaluation
Oleg Serikov‡~, Vi
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation
Linguistic analysis of language models is one of the ways to explain and
describe their reasoning, weaknesses, and limitations. In the probing part of
the model interpretability research, studies concern individual languages as
well as individual linguistic structures. The question arises: are the detected
regularities linguistically coherent, or on the contrary, do they dissonate at
the typological scale? Moreover, the majority of studies address the inherent
set of languages and linguistic structures, leaving the actual typological
diversity knowledge out of scope. In this paper, we present and apply the
GUI-assisted framework allowing us to easily probe a massive number of
languages for all the morphosyntactic features present in the Universal
Dependencies data. We show that reflecting the anglo-centric trend in NLP over
the past years, most of the regularities revealed in the mBERT model are
typical for the western-European languages. Our framework can be integrated
with the existing probing toolboxes, model cards, and leaderboards, allowing
practitioners to use and share their standard probing methods to interpret
multilingual models. Thus we propose a toolkit to systematize the multilingual
flaws in multilingual models, providing a reproducible experimental setup for
104 languages and 80 morphosyntactic features.
https://github.com/AIRI-Institute/Probing_framework
Oleg Serikov Vitaly Protasov Ekaterina Voloshina Viktoria Knyazkova Tatiana Shavrina