In this line of work, Liu et al. [21] investigated the linguistic knowledge and transferability of contextual representations by comparing BERT <cite>[2]</cite> with ELMo [15] , and concluded that while the higher levels of LSTM's are more task-specific, this trend does not exhibit in transformer based models.