While the root reason why PLMs are so effective remains an open research topic, a recent study shows that BERT does capture linguistic information, representing the steps of the traditional NLP pipeline (e.g., POS tagging, parsing, NER, semantic roles, then coreference) in an interpretable and localizable way [1]}.
