To gain linguistic insights into the global and deep sentence representation, we conducted probing taskshttps://github.com/facebookresearch/SentEval/tree/master/data/probing [1]} to evaluate linguistics knowledge embedded in the encoder output and the sentence representation in the variations of the Base model that are trained on En\(\Rightarrow \) De translation task.
The probing tasks are classification problems that focus on simple linguistic properties of sentences. The 10 probing tasks are categories into three groups: (1) Surface information. (2) Syntactic information. (3) Semantic information.
For each task, we trained the classifier on the train set, and validated the classifier on the validation set.
We followed hao-etal-2019-modeling and li-etal-2019-information to set the model configurations.
We also listed the results of lower layer representations (\(L=4, 5\) ) in Transformer-Base to conduct better comparison.
