Compared with the vanilla pretrained models like BERT<cite> (Devlin et al., 2019)</cite> , our model enriches the input sequence with its linguistic knowledge including part-of-speech tags and sentiment polarity labels, and utilizes a modified masked language model to capture the relationship between sentence-level sentiment labels and word-level knowledge in addition to context dependency.