Mixed Membership Word Embeddings Corpus Specific Embeddings Without Big Data
Word embeddings provide a nuanced representation of words which can improve the performance of NLP systems by revealing the hidden structural properties of words and their relationships to each other. These models have recently risen in popularity due to the successful performance of scalable algorithms trained in the big data setting. Consequently, word embeddings are commonly trained on very large generic corpora such as Wikipedia, instead of a (typically smaller) corpus of interest, leading to embeddings that are precisely trained but inaccurate relative to the domain of study. I propose a probabilistic model based word embedding method which can recover high dimensional embeddings without big data, due to a sensible parameter sharing scheme. The key insight is to leverage the notion of mixed membership modeling, in which global representations are shared, but individual entities (i.e. dictionary words) are free to use these representations to uniquely differing degrees. Leveraging connections to topic models, I show how to train these models in high dimensions using a combination of state of the art training techniques from the word embedding and topic modeling literatures.