Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code
 world knowledge into machine readable knowledge bases tend to be entity-centric,
 we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the contexts in which those entities were mentioned.
 We show that large scale training of neural models allows us to learn extremely high fidelity entity typing information, which we demonstrate with few-shot reconstruction of Wikipedia categories. Our learning approach is powerful enough
 to encode specialized topics such as Giro d’Italia cyclists. A long term goal of artificial intelligence has been the development and population of an entitycentric representation of human knowledge. Efforts have been made to create the knowledge representation with knowledge engineers BID10 or crowdsourcers BID1 . However, these methods have relied heavily on human definitions of their ontologies, which are both limited in scope and brittle in nature. Conversely, due to recent advances in deep learning, we can now learn robust general purpose representations of words BID13 and contextualized phrases BID16 BID6 directly from large textual corpora.Consider the following context in which an entity mention is replaced with the [MASK] symbol:. . . [MASK] , a Russian factory worker, was the first woman in space . . .As readers, we understand that first woman in space is a unique identifier, and we are able to fill in the blank unambiguously. The central hypothesis of this paper is that, by matching entities to the contexts in which they are mentioned, we should be able to build a representation for Valentina Tereshkova that encodes the fact that she was the first woman in space.To do this, we start with BERT BID6 , a powerful pretrained text encoder, to encode contexts-Wikipedia text in which a hyperlinked span has been blanked out-and we train an entity encoder to match the BERT representation of the entity's contexts. We experiment with a lookup table that maps each entity to a fixed length vector, which we call RELIC (Representations of Entities Learned In Context). We hypothesize that the dedicated entity representations in RELIC should be able to capture knowledge that is not present in BERT. To test this, we compare RELIC to two BERT-based entity encoders: one that encodes the entity's canonical name, and one that encodes the first paragraph of the entity's Wikipedia page.Ultimately, we would like our representations to encode all of the salient information about each entity. However, for this initial work, we study our representations' ability to capture Wikipedia categorical information encoded by human experts. We show that given just a few exemplar entities of a Wikipedia category such as Giro d'Italia cyclists, we can use RELIC to recover the remaining entities of that category with good precision. We demonstrated that the RELIC fill-in-the-blank task allows us to learn highly interesting representations of entities with their own latent ontology, which we empirically verify through a few-shot Wikipedia category reconstruction task. We encourage researchers to explore the properties of our entity representations and BERT context encoder, which we will release publicly.
We learn entity representations that can reconstruct Wikipedia categories with just a few exemplars.
{"src": "scitldr-AIC_train", "paper_id": "BJgum4Qgu4"}