Modeling Global Semantics for Question Answering over Knowledge Bases

Semantic parsing, as an important approach to question answering over knowledge bases (KBQA), transforms a question into the complete query graph for further generating the correct logical query. Existing semantic parsing approaches mainly focus on relations matching with paying less attention to the underlying internal structure of questions (e.g., the dependencies and relations between all entities in a question) to select the query graph. In this paper, we present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism. Experiments evaluated on benchmarks show that our model outperforms off-the-shelf models.


Introduction
Semantic parsing [BC+13; YC+15] constructs a semantic parsing tree or equivalent query structure (called query graph [LL+18]) that represents the semantics of questions. Semantic parsing based approaches effectively transform questions into logical queries where the reliability of logical querying can ensure the correctness of answering questions. The success of semantics parsing lies in representing the semantics of questions in a syntactic way so that it can better capture the intention of users [HZZ18].
Most existing semantic parsing approaches in question answering over knowledge bases (KBQA) focus on capturing the semantics of questions and query graphs. Some "common" relations occurring in both questions and query graphs are taken as core relations for measuring similarity together with some manual features [YC+15;BD+16]. Another approach to detecting individual relation is presented in [YY+17] to improve the performance of matching questions with query graphs where each relation is represented by integrating its word-level and relation-level representations. [LL+18] extends [YY+17] by improving the representation of questions with answering more complex questions. [SG18] enriches the semantics of all entities via all relations during the learning process by Gated Graph Neural Network [LT+18] (GGNN).
However, the state-of-the-art semantic parsing approaches utilize relational semantics of query graphs with pay little attention to the structure semantics of a question. The structure semantics is an important part of the whole semantics of questions (e.g., Figure 1), especially in complex questions where the complexity of a question often relies on its complicated structure. As a result, existing works only consider relational semantics cannot always perform complex questions better. So it is necessary to pay more attention to the structure semantics of questions together with relational semantics when semantic parsing in KBQA. However, to model multirelational directed graphs with edge features remains an open problem. Therefore, it is not trivial to combine the two semantics of a question. In this paper, based on a RGCN [SK+18], we propose a novel model gRGCN for global semantic parsing of KBQA. In gRGCN, we apply RGCN to extract structure semantics of query graphs by utilizing its capability of learning the better task-specific embeddings of multi-relational graphs. gRGCN focuses on extracting global semantics of both query graphs and relations. The main contributions are summarized as: • We propose a global semantic fusion method for structure semantics of query graphs extracted in RGCN integrated with relational semantics via enhanced finegrained relations (word-level and relation-level representation) learning. • We introduce a hierarchical attention mechanism to represent word-level relations, which can discover latent implicit meaning and remove ambiguity in words. • We introduce a syntactic-sequence-combined representation to encode questions and relation-attention-based RGCN layer to strengthen structure semantics.
In this section, we introduce our gRGCN model for global semantic parsing in KBQA, and the overview framework is shown in Figure 2. Our gRGCN model consists of three modules, namely, question representation encoder, structure semantics extractor, and relational semantics extractor.
• Question Representation Encoder transforms all questions input into their syntactic dependency tree as input of RGCN and then encodes the representations of all questions in RGCN with combining the sequence of words. The module is used to provide the condition of calculating the similarity of a question with its candidate query graphs together with the fused semantics of structure semantics and relational semantics.
• Structure Semantics Extractor embeds query graphs as a whole into a semantic space. The module is used to mainly extract the semantics of the whole structure of query graphs. In this module, we present a relationattention-based RGCN layer for extracting global semantics to be fused with relational semantics extracted in the last module.
• Relational Semantics Extractor embeds relations into a semantic space. The module is used to mainly extract the semantics of relations of query graphs left in the global semantics extractor. In this module, we present a hierarchical relational attention mechanism to reduce the noise of word-level relations via WordNet [M95].

Our Approach
In this section, we introduce our approach in the followings.

Query Graph Generation
We convert a question to query graphs further transformed into SPARQL queries for answering on KB. A query graph G is a directed graph with labeled nodes and typed directed edges. All vertexes of G are divided into three categories: question variable (?q), intermediate variables(v), and KB entities. A ?q-node represents the answer to a question. A vnode is either an unknown entity or an unknown value. An edge represents a relation between two vertexes. In this paper, we improve the candidate graph generation method based on an heuristic algorithm [SK+18] by further considering five kinds of semantic constraints: entity, type, temporal(explicit and inexplicit time), order, and compare.

Question Representation Encoding
This module encodes the syntactic structure of a question via RGCN with combining with the sequence of all words occurring in the question to a final vector as the representations of the question.
Given a question q = (w 1 , . . . , w n ), the syntactic dependency tree (a graph structure) of q, denoted as G q , is of form (V q , E q ) where V q = {w 1 , . . . , w n } (a vertex set) is a set of words and E q ⊆ V q × V q (a edge set) is a set of dependencyrelation over {w 1 , . . . , w n }. Edges are classified into two classes: self-loop and Head to dependent.
Let W glove ∈ R |V |×d be the word embedding matrix where each row vector represents the embedding of a word, d is the dimension of embedding. For each vertex w i ∈ V q , − → h wi is assigned to the corresponding vertex w i as its embedding in V q (i = 1, 2, . . . , n).
Firstly, we initialize the embedding of V q via its all vertexes' embedding and compute the average of word embeddings as − → E avg . Secondly, given the intial embedding of wn }, we apply RGCN encoder to generate its hidden representation. In RGCN, the forward-pass update of each vertex w i in V q is formalized in the following: Here l is a layer, N r wi is the set of all r-neighbors of w i in G q . Note that W 0 and W v are weighted matrixes to be learned.
To access both sequence and syntactic information, we concatenate the word embedding of q and the output of RGCN layers to calculate the final result. The attention weight of the j-th token (i.e., a j ) is calculated based on the following formula: let M is a weight matrix, Finally, we present a fully-connected layer and the ReLU non-linearity to obtain the final representation h q of question q in the following equation:

Structure Semantics Extracting
This module computes the representation of the structure of a query graph. We first formalize query graphs.
Let V e be a set of entites and V r be a set of relations. A query graph G on V e and V r is a quad of (N G , R G , λ, δ) where N G is the set of vertexes, R G is the set of edges, λ : N G → V e assigns each vertex to an entity, and δ : R G → V r assigns each edge to a relation.
Firstly, we initialize hidden states vi for node v i by calculating them in relation-attention-based RGCN layer in the following way: Here l is a layer, N r vi is the set of all r-neighbors of v i in G. Note that W 0 and W v are weighted matrixes to be learned. a r is the attention weight of the edge r to give high weight to relations more relevant to the question in query graphs : Finally, we obtain − → h structure as the structure semantic representation of G by taking the representation of answer node.

Relational Semantics Extracting
To utilize relational semantics both questions and query graphs, we compute the representation of all relations in word-level and relation-level.
Firstly, we define a relational embedding matrix as follows: Given a query graph G = (N G , R G , λ, δ) on V e and V r , we define relation-level relations (e.g. "posistion held") in G, denoted by R whole , as a sequence of vectors as follows: let λ(R G ) = {r 1 , . . . , r n }, R whole 1:n We define word-level relation R word , as a sequence of vectors by applying W glove ∈ R |V |×d to map each word w ij in r i to its word embedding − → w ij as follows: R word occurring in r i (e.g."posistion", "held").
There is some unavoidable noise of fine-grained relations. For instance, for relation type "be subject to," its word-level word may lose the global relation information, as "subject" may have different meanings shown in Figure 4.
To reduce noise, we introduce an external linguistic knowledge WordNet [M95], where a word possibly has multiple senses, and a sense consists of several lemmas.
The vector of sense − → w ijk is defined as follows: The vector of word w ij is defined as follows: Then, we obtain the fine-grained relations embedding of G, denoted by R fine . Based on fine-grained word embedding of a relation r i , we define a vector as follows: We employ max pooling over the R fine to get the final relational semantics − → h relational . Finally, we present a linear combination to fuse structure and relational semantics via ReLU as the global semantics: In this section, we evaluate our approach on the following experiments: an overall contrast experiment with baselines and ablation study experiments of our three different modules.

Experiments Setup
Knowledge Bases We select two representive KBs: • Wikidata: A collaborative KB developed by [VK14] contains more than 40 million entities and 350 million relation instances. We use the full Wikidata dump 1 and host it with Virtuoso engine 2 . • FB2M: The KB collected by [BU+15], as a subset of Freebase, consists of 2 million entities and 10 million triple facts. FB2M is a famous KB for creating many QA datasets particularly simple question datasets.

QA Datasets
We select four popular datasets as follows: • ComplexQuestion (CompQ): The dataset consisting of 2,100 complex questions is developed by [BD+16] to provide questions with structure and expression diversity. To support answering over Wikidata [VK14], we slightly revise it by mapping answers to Wikidata answer IDs instead of Freebase IDs since Freebase was discontinued and is no longer up-to-date, including unavailability of APIs and new dumps [SG18]. • WebQSP-WD (WebQSPS): The dataset collected by [SG18] contains 712 real complex questions (in total 3913). WebQSPS is a corrected version of the common benchmark WebQSP [BC+13] data set for supporting Wikidata. • QALD-7: The small dataset consisting of 42 complex questions and 58 simple questions collected by [UN+17] to be mainly used to test KBQA models. QALD-7 is specially designed for the KBQA task over Wikidata. • SimpleQuestion (SimpQ): The dataset developed by [BU+15], as a popular benchmark of KBQA, contains over 100K questions w.r.t. FB2M. To evaluate the generalization ability of our model, we additionally use SimpQ in our complementary experiment.
Implementation Details In training, we adopt hinge loss which are applied in [LL+18;SG18]. Formally, let q be a question and C be the query graph set of q where C contains all positive graph g + and negative graph g − of q. L = max g∈C (0, (λ − cos( q, g + ) + cos( q, g − ))).
In our experiments, We use S-MART [YC15] as our entity linking tool, GloVe [PSM14] word vectors with dimensions of 100 is employed to initializing word embeddings and Adam optimizer [KB15] is applied to train the model with a batch size of 64 at each epoch. Moreover, we design a multiple-layer RGCN with a fully-connected layer with dropout=0.2 to calculate the representation of questions and the structure of query graphs (three-layer for questions and two layers for graphs). Finally, SpaCy 3 is used to parse syntactic dependency, and Deep Graph Library (DGL 4 ) is used to transfer query graphs into DGL graph objects.

Baselines
We introduce four popular models and one variation of gRGCN as baselines as follows: • STAGG (2015) [YC+15]: The model scores query graphs by some manual features and the representation of the core relation. It's a re-implemented by [SG18]. HZZ18]. Since their codes are not accessible, we will discuss them in the related works.

Overall Results of Our Approach
For evaluating our approach on complex questions, we select Wikidata as KB, CompQ, WebQSPS, and QALD-7 as datasets and precision, recall, and F1-score as metrics.
The experimental results are shown in Table 1, where we take the average results of all questions in a dataset as the final result. Note that our results are different from those original results released in baselines over Freebase.
By Table 1, we show that gRGCN outperforms all datasets and all metrics.

Robustness of Our Approach
Simple questions still contain a little structural information, such as the linear order of the subject entity and the object entity. We complementary evaluate our approach to simple questions to analyze its robustness. This experiment selects FB2M as KB, SimpQ as dataset, and six popular baselines. By the experimental results shown in Table 2, our approach

Ablation Study
In this subsection, we will pay attention to analyze the effectiveness of the proposed question representation and the proposed query graph representation in our model. For the ablation study, we use the F1 score as our metrics and perform experiments on CompQ, WebQSPS, and QALD-7 w.r.t. complex questions. Question Representation To analyze the effectiveness of our proposed question representation encoder, this experiment compares gRGCN to gRGCN with substituting the question encoder in each baseline for our encoder.
The results are shown in Table 3, where the column of QR of Baselines is gRGCN equipping with the corresponding question encoder and gRGCN − denote a variant of gRGCN without concatenating question sequences. We use gRGCN − to analyze the effectiveness of our proposed question encoder further, only considering those information of questions also considered in baselines fairly. (2018) with a 1.3%, 6.9%, and 1.1% improvement on three datasets. • Thirdly, the question sequence is useful to improve the performance question presentation. Specifically, gRGCN betters gRGCN − with a 1.0%, 5.4%, and 1.0% improvement on three datasets. Graph Representation To analyze the effectiveness of our proposed graph representation encoder, we consider two types of relations: fine-grained relation (f-g) and wordnet, two types of structures: attention and non-attention (non-att). Note that gRGCN considers all relations and structures.
The experimental results on the three datasets are shown in Table 4, where there are 18 cases, including 6 cases without structure. • Firstly, the structure of query graphs is verified to improve the performance of graph presentation without considering structure. Specifically, non-attention achieve 24.4%, 18.8%, 10.3% improvement on three datasets, and even when further considering wordnetrelation, it still performs with a 19.4%, 8.3%, 8.4% improvement. • Secondly, the wordnet-relation of query graphs is verified to improve the performance graph presentation without considering the wordnet-relation. Specifically, wordnet-relation performs with a 3.5%, 12.4%, 10.7% improvement on three datasets, and even when further considering attention-structure, it still performs with a 3.5%, 12.3%, 3.3% improvement. • Thirdly, the attention-structure of query graphs is verified to improve the performance graph presentation without considering the attention-structure. Specifically, attention-structure performs with a 24.4%, 18.8%, 10.3% improvement on three datasets, and even when further considering wordnet-relation, it still performs with a 19.4%, 8.3%, 8.4% improvement. Therefore, we can conclude that the structure presented in our graph representation as a global semantics is useful to improve the performance question presentation as well as wordnet-relation.

Error Analysis
We randomly analyze the major causes of errors on 100 questions, which returned incorrect answers.
Semantic ambiguity and complexity (42%): Due to the limitation of semantic graph generation, we can't generate graph space for some question since their complexity and semantic ambiguity.
Correct graphs scored low (16%): This type of error occurred when the model failed to extract the global semantics of questions and graphs. Although the correct graphs existed in the candidate set, we failed to select it.
Entity linking error (9%): This error occurred due to the failure to extract the appropriate entities from the given question. And subsequently, we generate incorrect semantic graphs. We could correct questions by replacing the wrong entities.
Dateset and KB error (33%): This error occurred since the incompatibility between wikidata and Freebase ID in datasets or the defects of datasets. The test datasets contain many open question whose answers may be partially or completely unstored in Wikidata.

Related Works
The most popular approaches proposed for the KBQA task can be roughly classified into two categories: semantic parsing and information retrieval. Recently, neural networks (NN) based approaches have shown great performance on KBQA. Different from [CX+17] and [AY+17] tried to extract questions intention by a large number of templates automatically learned from KB and QA corpora, SP+NN employed neural network to encode the semantics of questions and query graphs, and then select the correct query graphs of given questions in the candidate set for querying later [BD+16; LL+18; HZZ18]. [LL+18] encoded multiple relations into a uniform query graph to capture the fined-grained relations. [SG18] used GGNN to enrich the semantics of all entities with tackling the limited complex questions. Differently, our approach integrates both structure and enhanced relational semantics of query graphs.
[HZZ18] [DH+19] pay more attention to query graph generation and use an SVM rank classifier and a combinational function to rank candidate graphs respectively, which is not an encode-and-compare framework as our model. [CSH18] uses an end-to-end query graph generation process via an RNN model, and it aims to search the best action sequence. However, we are focus on learning global semantics of the query graph. Information Retrieval Information retrieval (IR) based approaches retrieve the candidate entities (as answers) from KB using semantic relation extraction [XR+16; DW+15; WH+19; YR+16; SD+18; LWJ19].Those approaches map answers and questions into the same embedding space, where we could retrieve the answers independent of any grammar, lexicon, or other semantic structure. Our approach belongs to the SP-based method and encoded semantic information of question into a query graph for querying on KB.

Conclusions
In this paper, we propose an RGCN-based model for semantic parsing of KBQA with extracting global semantics to maximize the performance of question learning. Our approach pays more attention to structure semantics of query graphs extracted by RGCN and enhanced relational semantics in KBQA. In this sense, our approach provides a new way to capture the intention of questions better. In this paper, we generalize our model to support complex questions, such as comparing questions. In future work, we are interested in extending our model for more complex practical questions.