Contextualized word representations such as ELMo and BERT have become the de facto starting point for incorporating pretrained representations for downstream NLP tasks. In these settings, contextual representations have largely made obsolete their static embedding predecessors such as Word2Vec and GloVe. However, static embeddings do have their advantages in that they are straightforward to understand and faster to use. Additionally, embedding analysis methods for static embeddings are far more diverse and mature than those available for their dynamic counterparts. In this work, we introduce simple methods for generating static lookup table embeddings from existing pretrained contextual representations and demonstrate they outperform Word2Vec and GloVe embeddings on a variety of word similarity and word relatedness tasks. In doing so, our results also reveal insights that may be useful for subsequent downstream tasks using our embeddings or the original contextual models. Further, we demonstrate the increased potential for analysis by applying existing approaches for estimating social bias in word embeddings. Our analysis constitutes the most comprehensive study of social bias in contextual word representations (via the proxy of our distilled embeddings) and reveals a number of inconsistencies in current techniques for quantifying social bias in word embeddings. We publicly release our code and distilled word embeddings to support reproducible research and the broader NLP community. Word embeddings (Bengio et al., 2003; Collobert & Weston, 2008; Collobert et al., 2011) have been a hallmark of modern natural language processing (NLP) for several years. Pretrained embeddings in particular have seen widespread use and have experienced parallel and complementary innovations alongside neural networks for NLP. Advances in embedding quality in part have come from integrating additional information such as syntax (Levy & Goldberg, 2014b; Li et al., 2017) , morphology (Cotterell & Schütze, 2015) , subwords (Bojanowski et al., 2017) , subcharacters (Stratos, 2017; Yu et al., 2017) and, most recently, context (Peters et al., 2018; Devlin et al., 2019) . As a consequence of their representational potential, pretrained word representations have seen widespread adoption across almost every task in NLP and reflect one of the greatest successes of both representation learning and transfer learning for NLP (Ruder, 2019b) . The space of pretrained word representations can be partitioned into static vs. dynamic embeddings methods. Static methods such as Word2Vec (Mikolov et al., 2013) , GloVe (Pennington et al., 2014), and FastText (Bojanowski et al., 2017) yield representations that are fixed after training and generally associate a single vector with a given word in the style of a lookup table. While subsequent work addressed the fact that words may have multiple senses and should have different representations for different senses (Pilehvar & Collier, 2016; Lee & Chen, 2017; Pilehvar et al., 2017; Athiwaratkun & Wilson, 2017; Camacho-Collados & Pilehvar, 2018) , fundamentally these methods cannot easily adapt to the inference time context in which they are applied. This contrasts with contextual, or dynamic, methods such as CoVe (McCann et al., 2017) , ELMo (Peters et al., 2018) , and BERT (Devlin et al., 2019) , which produce vector representations for a word conditional on the inference time context in which it appears. Given that dynamic representations are arguably more linguistically valid, more expressive (static embeddings are a special-case of dynamic embeddings that are optimally ineffective at being dynamic), and have yielded significant empirical improvements (Wang et al., 2019b; a; Ruder, 2019a) , it would seem that static embeddings are outdated. Static embeddings, however, have significant advantages over dynamic embeddings with regard to speed, computational resources, and ease of use. These benefits have important implications for time-sensitive systems, resource-constrained settings or environmental concerns (Strubell et al., 2019) , and broader accessibility of NLP technologies 1 . As a consequence of this dichotomy between static and dynamic representations and their disparate benefits, we propose in this work a simple yet effective mechanism for converting from dynamic representations to static representations. We begin by demonstrating that our method when applied to pretrained contextual models (BERT, GPT-2, RoBERTa, XLNet, DistilBERT) yields higher quality static embeddings than Word2Vec and GloVe when evaluated intrinsically on four word similarity and word relatedness datasets. Further, since our procedure does not rely on specific properties of the pretrained contextual model, it can be applied as needed to generate ever-improving static embeddings that will track advances in pretrained contextual word representations. Our approach offers the hope that high-quality embeddings can be maintained in both settings given their unique advantages and appropriateness in different settings. At the same time, we show that by distilling static embeddings from their dynamic counterparts, we can then employ the more comprehensive arsenal of embedding analysis tools that have been developed in the static embedding setting to better understand the original contextual embeddings. As an example, we employ methods for identifying gender, racial, and religious bias (Bolukbasi et al., 2016; Garg et al., 2018; Manzini et al., 2019) to our distilled representations and find that these experiments not only shed light on the properties of our distilled embeddings for downstream use but can also serve as a proxy for understanding existing biases in the original pretrained contextual representations. Our large-scale and exhaustive evaluation of bias further reveals dramatic inconsistencies in existing measures of social bias and highlights sizeable discrepancies in the bias estimates obtained for distilled embeddings drawn from different pretrained models and individual model layers. In this work, we propose simple but effective procedures for converting contextual word representations into static word embeddings. When applied to pretrained models like BERT, we find the resulting embeddings outperform Word2Vec and GloVe substantially under intrinsic evaluation and provide insights into the pretrained model. We further demonstrate the resulting embeddings are more amenable to (existing) embedding analysis methods and report the extent of various social biases (gender, race, religion) across a number of measures. Our large-scale analysis furnishes several findings with respect to social bias encoded in popular pretrained contextual representations via the proxy of our embeddings and has implications towards the reliability of existing protocols for quantifying bias in word embeddings.
A procedure for distilling contextual models into static embeddings; we apply our method to 9 popular models and demonstrate clear gains in representation quality wrt Word2Vec/GloVe and improved analysis potential by thoroughly studying social bias.
{"src": "scitldr-AIC_train", "paper_id": "SJg3T2EFvr"}