Learning knowledge graph embeddings (KGEs) is an efficient approach to knowledge graph completion. Conventional KGEs often suffer from limited knowledge representation, which causes less accuracy especially when training on sparse knowledge graphs. To remedy this, we present Pretrain-KGEs, a training framework for learning better knowledgeable entity and relation embeddings, leveraging the abundant linguistic knowledge from pretrained language models. Specifically, we propose a unified approach in which we first learn entity and relation representations via pretrained language models and use the representations to initialize entity and relation embeddings for training KGE models. Our proposed method is model agnostic in the sense that it can be applied to any variant of KGE models. Experimental results show that our method can consistently improve results and achieve state-of-the-art performance using different KGE models such as TransE and QuatE, across four benchmark KG datasets in link prediction and triplet classification tasks. Knowledge graphs (KGs) constitute an effective access to world knowledge for a wide variety of NLP tasks, such as question-answering, entity linking and information retrieval. A typical KG such as Freebase (Bollacker et al., 2008) and WordNet (Miller, 1995) consists of a set of triplets in the form of (h, r, t) with the head entity h and the tail entity t as nodes and relations r as edges in the graph. A triplet represents the relation between two entities, e.g., (Steve Jobs, founded, Apple Inc.). Despite their effectiveness, KGs in real applications suffer from incompleteness and there have been several attempts for knowledge graph completion among which knowledge graph embedding is one of prominent approaches. Knowledge graph embedding (KGE) models have been designed extensively in recent years (Bordes et al., 2013; Ji et al., 2015; Lin et al., 2015; Sun et al., 2019; Ebisu and Ichise, 2018; Nickel et al., 2011; Kazemi and Poole, 2018; Trouillon et al., 2016; Zhang et al., 2019) . The general methodology of these models is to model entities and relations in vector spaces based on a score function for triplets (h, r, t). The score function measures the plausibility of each candidate triplet (h, r, t) compared to corrupted false triplets (h , r, t) or (h, r, t ). However, traditional KGE models often suffer from limited knowledge representation due to the simply symbolic representation of entities and relations. Some recent works take advantages of both fact triplets and textual description to enrich knowledge representation (Socher et al., 2013a; Xu et al., 2017; Xiao et al., 2017; Xie et al., 2016; , but without exploitation of contextual information of the textual descriptions. Moreover, much of this research effort has been dedicated to developing novel architectures for knowledge representation without applications to KGE models. Unlike many existing works which try to propose new architectures for KGEs or knowledge representation, we focus on model-agnostic pretraining technique for KGE models. We present a unified training framework named as PretrainKGEs which consists of three phases: fine-tuning phase, initializing phase and training phase (see Fig. 1 ). During the fine-tuning phase, we learn better knowledgeable entity and relation representations via pretrained language models using textual descriptions as input sequence. Different from previous works incorporating textual information into knowledge representation, we use pretrained langauge models such as BERT (Devlin et al., 2019) to better understand textual description by making full use of syntactic and semantic information in large- scale corpora on which BERT is pretrained. Thus, we enable to incorporate rich linguistic knowledge learned by BERT into entity and relation representations. Then during the initializing phase, we use knowledgeable entity and relation representations to initialize entity and relation embeddings so that the initialized KGEs inherit the rich knowledge. Finally, during the training phase, we train a KGE model the same way as a traditional KGE model to learn entity and relation embeddings. Extensive experiments using six public KGE models across four benchmark KG datasets show that our proposed training framework can consistently improve results and achieve state-of-the-art performance in link prediction and triplet classification tasks. Our contributions are as follows: • We propose a model-agnostic training framework for learning knowledge graph embeddings by first learning knowledge representation via pretrained language models. • Results on several benchmark datasets show that our method can improve results and achieve state-of-the-art performance over variants of knowledge graph embedding models in link prediction and triplet classification tasks. • Further analysis demonstrates the effects of knowledge incorporation in our method and shows that our Pretrain-KGEs outperforms baselines especially in the case of fewer training triplets, low-frequency and the out-ofknowledge-base (OOKB) entities. 2 Background and Related Work We present Pretrain-KGEs, a simple and efficient pretraining technique for knowledge graph embedding models. Pretrain-KGEs is a general technique that can be applied to any KGE model. It contributes to learn better knowledgeable entity and relation representations from pretrained language models, which are leveraged during the initializing and the training phases for a KGE model to learn entity and relation embeddings. Through extensive experiments, we demonstrate state-of-the-art performances using this effective pretraining technique on various benchmark datasets. Further, we verify the effectiveness of our method by demonstrating promising results in the case of fewer training triplets, infrequent and OOKB entities which are particularly hard to handle due to lack of knowledge representation. We finally analyze the effects of knowledge incorporation by demonstrating the sensitivity of MR and MRR metrics and visualizing the process of knowledge incorporation. A Detailed Implementation A.1 Implementation Our implementations of TransE (Bordes et al., 2013) , DistMult , ComplEx (Trouillon et al., 2016) , RotatE (Sun et al., 2019) , pRotatE (Sun et al., 2019) are based on the framework provided by Sun et al. (2019) 6 . Our implementation of QuatE is based on on the framework provided by Zhang et al. (2019) 7 . In fine-tuning phase, we adopt the following non-linear pointwise function σ(·): x i e i ∈ F (where F can be real number filed R, complex number filed C or quaternion number ring H): where x i ∈ R and e i is the K-dimension hypercomplex-value unit. For instance, when K = 1, F = R; when K = 2, F = C, e 1 = i (the imaginary unit); when K = 4, F = H, e 1,2,3 = i, j, k (the quaternion units). The score functions of baselines are listed in Table 4 .
We propose to learn knowledgeable entity and relation representations from Bert for knowledge graph embeddings.
{"src": "scitldr-AIC_train", "paper_id": "HJlv-Fz-pS"}