arXiv:2207.00560v1  [cs.CL]  1 Jul 2022Is neural language acquisition similar to natural?
A chronological probing study
Ekaterin
Is neural language acquisition similar to natural? A chronological probing study
The probing methodology allows one to obtain a partial representation of
linguistic phenomena stored in the inner layers of the neural network, using
external classifiers and statistical analysis. Pre-trained transformer-based
language models are widely used both for natural language understanding (NLU)
and natural language generation (NLG) tasks making them most commonly used for
downstream applications. However, little analysis was carried out, whether the
models were pre-trained enough or contained knowledge correlated with
linguistic theory. We are presenting the chronological probing study of
transformer English models such as MultiBERT and T5. We sequentially compare
the information about the language learned by the models in the process of
training on corpora. The results show that 1) linguistic information is
acquired in the early stages of training 2) both language models demonstrate
capabilities to capture various features from various levels of language,
including morphology, syntax, and even discourse, while they also can
inconsistently fail on tasks that are perceived as easy. We also introduce the
open-source framework for chronological probing research, compatible with other
transformer-based models.
https://github.com/EkaterinaVoloshina/chronological_probing
Ekaterina Voloshina Oleg Serikov Tatiana Shavrina