One way to insert sophisticated linguistic information may be through the use of contextualized word embeddings. While other works have explored the use of transformer language models to predict prosodic and stylistic features [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, it has not been fully explored how much the encoded word representations actually imbue high-level knowledge. In other words, do they provide information about the content and context of the message or do they only provide/reinforce low-level linguistic features such as the likelihood of lexical prominence, parts of speech and position in the sentence? For prominence/pitch accent prediction, a fairly high baseline can be achieved using word majority/accent-ratio alone (i.e., if a lexical item is usually prominent in the training set, it is likely to be prominent in the test set) [8]}. Moreover, [9]} found that in a binary pitch accent prediction task, using broad word category distinctions (open/content or closed/function) could achieve 68% accuracy; more fine-grained division of the closed class category brought that number up to 77%. In this work, we probe a language model, in the present case BERT [10]}, by choosing a testing ground that cannot rely on simple heuristics to achieve good results: the prediction of contrastive focus on personal pronouns. Personal pronouns, at least in the corpora typically used to train TTS systems, are majority non-prominent.We conflate the notions of prominence and contrastive focus for personal pronouns; when they are prominent, they typically possess a contrastive meaning.
