Universal schema builds a knowledge base ( KB ) of entities and relations by jointly embedding all relation types from input KBs as well as textual patterns observed in raw text . In most previous applications of universal schema , each textual pattern is represented as a single embedding , preventing generalization to unseen patterns . Recent work employs a neural network to capture patterns ' compositional semantics , providing generalization to all possible input text . In response , this paper introduces significant further improvements to the coverage and flexibility of universal schema relation extraction : predictions for entities unseen in training and multilingual transfer learning to domains with no annotation . We evaluate our model through extensive experiments on the English and Spanish TAC KBP benchmark , outperforming the top system from TAC 2013 slot-filling using no handwritten patterns or additional annotation . We also consider a multilingual setting in which English training data entities overlap with the seed KB , but Spanish text does not . Despite having no annotation for Spanish data , we train an accurate predictor , with additional improvements obtained by tying word embeddings across languages . Furthermore , we find that multilingual training improves English relation extraction accuracy . Our approach is thus suited to broad-coverage automated knowledge base construction in a variety of languages and domains . The goal of automatic knowledge base construction ( AKBC ) is to build a structured knowledge base ( KB ) of facts using a noisy corpus of raw text evidence , and perhaps an initial seed KB to be augmented ( Carlson et al . , 2010 ; Suchanek et al . , 2007 ; Bollacker et al . , 2008 ) . AKBC supports downstream reasoning at a high level about extracted entities and their relations , and thus has broad-reaching applications to a variety of domains . One challenge in AKBC is aligning knowledge from a structured KB with a text corpus in order to perform supervised learning through distant supervision . Universal schema ( Riedel et al . , 2013 ) along with its extensions ( Yao et al . , 2013 ; Gardner et al . , 2014 ; Neelakantan et al . , 2015 ; Rocktaschel et al . , 2015 ) , avoids alignment by jointly embedding KB relations , entities , and surface text patterns . This propagates information between KB annotation and corresponding textual evidence . The above applications of universal schema express each text relation as a distinct item to be embedded . This harms its ability to generalize to inputs not precisely seen at training time . Recently , Toutanova et al . ( 2015 ) addressed this issue by embedding text patterns using a deep sentence encoder , which captures the compositional semantics of textual relations and allows for prediction on inputs never seen before . This paper further expands the coverage abilities of universal schema relation extraction by introducing techniques for forming predictions for new entities unseen in training and even for new domains with no associated annotation . In the extreme example of domain adaptation to a completely new language , we may have limited linguistic resources or labeled data such as treebanks , and only rarely a KB with adequate coverage . Our method performs multilingual transfer learning , providing a predictive model for a language with no coverage in an existing KB , by leveraging common representations for shared entities across text corpora . As depicted in Figure 1 , we simply require that one language have an available KB of seed facts . We can further improve our models by tying a small set of word embeddings across languages using only simple knowledge about word-level translations , learning to embed semantically similar textual patterns from different languages into the same latent space . In extensive experiments on the TAC Knowledge Base Population ( KBP ) slot-filling benchmark we outperform the top 2013 system with an F1 score of 40.7 and perform relation extraction in Spanish with no labeled data or direct overlap between the Spanish training corpus and the training KB , demonstrating that our approach is wellsuited for broad-coverage AKBC in low-resource languages and domains . Interestingly , joint training with Spanish improves English accuracy . English Low-resource in KB not in KB Figure 1 : Splitting the entities in a multilingual AKBC training set into parts . We only require that entities in the two corpora overlap . Remarkably , we can train a model for the low-resource language even if entities in the lowresource language do not occur in the KB . By jointly embedding English and Spanish corpora along with a KB , we can train an accurate Spanish relation extraction model using no direct annotation for relations in the Spanish data . This approach has the added benefit of providing significant accuracy improvements for the English model , outperforming the top system on the 2013 TAC KBC slot filling task , without using the hand-coded rules or additional annotations of alternative systems . By using deep sentence encoders , we can perform prediction for arbitrary input text and for entities unseen in training . Sentence encoders also provides opportunities to improve cross-lingual transfer learning by sharing word embeddings across languages . In future work we will apply this model to many more languages and domains besides newswire text . We would also like to avoid the entity detection problem by using a deep architecture to both identify entity mentions and identify relations between them .
Existing approaches to incorporate universal schemas for automatic knowledge base construction has limitation in generalization to unseen inputs from training time.
They propose to combine universal schemas and neural network-based deep encoders to achieve generalization to an unseen language without additional annotations.
The proposed approach outperforms existing methods on benchmarks in English and Spanish while having no hand-coded rules or training data for Spanish.