Although pre-trained language representation models have achieved transformative performance, the pre-training tasks like masked language model and next sentence prediction<cite> (Devlin et al., 2019)</cite> neglect to consider the linguistic knowledge. We argue that such knowledge is important for some NLP tasks, particularly for sentiment analysis.