Knowledge Graph Embedding (KGE) has attracted more attention in recent years. Most of KGE models learn from time-unaware triples. However, the inclusion of temporal information beside triples would further improve the
 performance of a KGE model. In this regard, we propose LiTSE, a temporal KGE model which incorporates time information
 into entity/relation representations by using linear time series decomposition. Moreover, considering the temporal uncertainty during the evolution of entity/relation representations over time, we map the representations of temporal
 KGs into the space of multi-dimensional Gaussian distributions. The mean of each entity/ relation embedding at a time step shows the current expected position, whereas its covariance (which is stationary over time) represents
 its temporal uncertainty. Experiments show that LiTSE not only achieves the state-of- the-art on link prediction in temporal KGs, but also has the ability to predict the occurrence time of facts with missing time annotations, as well as the existence of future events. To the best of our knowledge, no other model is capable to perform all these tasks. Knowledge Graphs (KGs) are being used for gathering and organizing scattered human knowledge into structured knowledge systems. YAGO (Suchanek et al., 2007) , NELL BID3 , DBpedia BID0 and Freebase BID1 are among existing KGs that have been successfully used in various applications including question answering, assistant systems, information retrieval, etc. In these KGs, knowledge can be represented as RDF triples (s, p ,o) in which s (subject) and o (object) are entities (nodes), and p (predicate) is the relation (edge) between them.KG embedding attempts to learn the representations of entities and relations in high-dimensional latent feature spaces while preserving certain properties of the original graph. Recently, KGE has become a very active research topic due to the wide ranges of downstream applications. Different KGE models have been proposed so far to efficiently learn the representations of KGs and perform KG completion as well as inferencing BID2 BID22 BID23 BID20 BID5 .Most of existing KGE models solely learn from time-unknown facts and ignore the useful temporal information in KGs. In fact , there are many time-aware facts (or events) in some temporal KGs. For instance , (Obama, wasBornIn, Hawaii) happened at August 4, 1961, and (Obama, presidentOf, USA) was true from 2009 to 2017. These temporal KGs, e.g. ICEWS BID9 , YAGO3 BID11 , store such temporal information either explicitly or implicitly. Traditional KGE models such as TransE learn only from time-unknown facts and consequently cannot distinguish relations with similar semantic meaning. For instance, they often confuse relations such as wasBornIn and diedIn when predicting (person,?,location) .To tackle this problem , Temporal KGE models BID4 BID6 BID18 encode time information in their embeddings. Temporal KGE models outperform traditional KGE models on link prediction over temporal KGs. It justifies that incorporation of time information can further improve the performance of a KGE model. Some existing temporal KGE models encode time information in a latent space e.g. representing time as a vector BID4 BID10 . These models cannot capture some prop-erties of time information such as the length of time interval as well as order of two time points. Moreover, some exiting temporal graph embedding models BID18 BID19 consider the changes of entity representations over time as a kind of temporal evolution process, while they ignore the uncertainty during the temporal evolution. We argue that the evolution of entity representations has randomness, because the features of an entity at a certain time are not completely determined by the past information. For example, (Steve Jobs, diedIn , California) happened on 2011-10-05. The semantic characteristics of this entity should have a sudden change at this time point. However, due to the incompleteness of knowledge in KGs, this change can not be predicted only according to its past evolutionary trend. Therefore, the representation of Steve Jobs is supposed to include some random components to handle this uncertainty, e.g. a Gaussian noise component.To address the above problems, we propose a new temporal KGE model based on linear time series decomposition (LiTSE) that captures the evolution process of KG representations. LiTSE fits the evolution process of an entity or relation as a linear function of time with a Gaussian random noise. Inspired by , our approach represents each entity and relation as a multi-dimensional Gaussian distribution at each time step to introduce a random component. The mean of an entity/relation representation at a certain time step indicates its current expected position, which is obtained from its initial representation, its evolutionary direction vector which represents the long-term trend of its evolution and the current time. The covariance which describes the temporal uncertainty during its evolution, is denoted as a constant diagonal matrix for computing efficiency. Our contributions are as follows.• Learning the representations for temporal KGs is a relatively unexplored problem because most of existing KGE models only learn from time-unknown facts. We propose LiTSE, a new KGE model to incorporate the time information into the KG representations.• Different from the previous temporal KGE models which use time encoding to incorporate time information, LiTSE fits the evolution process of KG representations as a linear function of time. This enables us to observe and predict the time information directly from entity/relation representations. In particular, we can predict the occurrence of a fact in a future time, according to the known evolution trends of KG representations learned from the past information.• We specially consider the temporal uncertainty during the evolution process of KG representation. Thus, we model each entity as a Gaussian distribution at each time step and use KL-divergence between two Gaussian distributions to compute the scores of facts for optimization.• Beside performing link prediction in temporal KGs, our models are proved to be capable of estimating the occurrence time of a fact with missing time annotation, and predicting future events.The rest of the paper is organized as follows: Section 2 reviews related works. Our model is introduced in the section 3. The proposed model is evaluated and compared with state-of-the-art models in the section 4. Finally, the paper is concluded in the last section. We introduce LiTSE, a temporal KGE model that incorporates time information into KG representations by using linear time series decomposition. LiTSE fits the temporal evolution of KG representations over time as linear time series, which enables itself to estimate time information of a triple with the missing time annotation and predict the occurrence of a future event. Considering the uncertainty during the temporal evolution of KG representations, LiTSE maps the representations of temporal KGs into the space of multi-dimensional Gaussian distributions. The covariance of an entity/relation representation represents its randomness component. Experimental results demonstrate that our method significantly outperforms the state-of-the-art methods on link prediction and future event prediction. Besides, our method can effectively predict the occurrence time of a fact. Our work establishes a previously unexplored connection between relational processes and time series analysis with a potential to open a new direction of research on reasoning over time. In the future, we will explore to use other time series analysis techniques to model the temporal evolution of KG representations. Along with considering the temporal uncertainty, another benefit of using time series analysis is to enable the embedding model to encode temporal rules. For instance, given two quadruple (s, p, o, t p ) and (s, q, o, t q ), there exists a temporal constraint t p < t q . Since the time information is represented as a numerical variable in a time series model, it is feasible to incorporate such temporal rules into our models. We will investigate the possibility of encoding temporal rules into our proposed models. DISPLAYFORM0 regularize the covariances for each entity and relation with constraint 6. 18. end loop TAB10 shows the statistics of datasets which are anew split for future event prediction, denoted as ICEWS14-F and ICEWS05-15F. As mentioned in Section 4.2, all of the facts in test set occur after the facts in training set and validation set, and the facts of validation set occur after the facts in training set. The time spans of training sets, validation sets and test sets of ICEWS14 and ICEWS05-15 are reported in TAB10 . t e represents the end time of the dataset. For instance, t e of the training set of ICEWS14 is 2014/10/20 and t e of the validation set of ICEWS14 is 2014/11/22, which means the time stamps of quadruples in the validation set of ICEWS14 are between 2014/10/21 and 2014/11/22.
Submitted in EMNLP
{"src": "scitldr-AIC_train", "paper_id": "SJxiIBKG64"}