When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas.   The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query.   We present a unified framework, based on the relation-aware self-attention mechanism,to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder. On the challenging Spider dataset this framework boosts the exact match accuracy to 53.7%, compared to 47.4% for the previous state-of-the-art model unaugmented with BERT embeddings. In addition, we observe qualitative improvements in the model’s understanding of schema linking and alignment. The ability to effectively query databases with natural language has the potential to unlock the power of large datasets to the vast majority of users who are not proficient in query languages. As such, a large body of research has focused on the task of translating natural language questions into queries that existing database software can execute. The release of large annotated datasets containing questions and the corresponding database SQL queries has catalyzed progress in the field, by enabling the training of supervised learning models for the task. In contrast to prior semantic parsing datasets (Finegan-Dollak et al., 2018) , new tasks such as WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018b) pose the real-life challenge of generalization to unseen database schemas. Every query is conditioned on a multi-table database schema, and the databases do not overlap between the train and test sets. Schema generalization is challenging for three interconnected reasons. First, any text-to-SQL semantic parsing model must encode a given schema into column and table representations suitable for decoding a SQL query that might involve any of the given columns or tables. Second, these representations should encode all the information about the schema, including its column types, foreign key relations, and primary keys used for database joins. Finally, the model must recognize natural language used to refer to database columns and tables, which might differ from the referential language seen in training. The latter challenge is known as schema linking -aligning column/table references in the question to the corresponding schema columns/tables. While the question of schema encoding has been studied in recent literature (Bogin et al., 2019b) , schema linking has been relatively less explored. Consider the example in Figure 1 . It illustrates the challenge of ambiguity in linking: while "model" in the question refers to car_names.model rather than model_list.model, "cars" actually refers to both cars_data and car_names (but not car_makers) for the purpose of table joining. To resolve the column/table references properly, the semantic parser must take into account both the known schema relations (e.g. foreign keys) and the question context. Prior work (Bogin et al., 2019b) addressed the schema representation problem by encoding the directed graph of foreign key relations among the columns with a graph neural network. While effective, this approach has two important shortcomings. First, it does not contextualize schema encoding with the question, thus making it difficult for the model to reason about schema linking after both the column representations and question word representations have been built. Second, it limits information propagation during schema encoding to predefined relations in the schema such as foreign keys. The advent of self-attentional mechanisms in natural language processing (Vaswani et al., 2017) shows that global reasoning is crucial to building effective representations of relational structures. However, we would like any global reasoning to also take into account the aforementioned predefined schema relations. In this work, we present a unified framework, called RAT-SQL, 1 for encoding relational structure in the database schema and a given question. It uses relation-aware self-attention to combine global reasoning over the schema entities and question words with structured reasoning over predefined schema relations. We then apply RAT-SQL to the problems of schema encoding and schema linking. As a result, we obtain 53.7% exact match accuracy on the Spider test set. At the time of writing, this result is the state of the art among models unaugmented with pretrained BERT embeddings. In addition, we experimentally demonstrate that RAT-SQL enables the model to build more accurate internal representations of the question's true alignment with schema columns and tables. Despite the abundance of research in semantic parsing of text to SQL, many contemporary models struggle to learn good representations for a given database schema as well as to properly link column/table references in the question. These problems are related: to encode & use columns/tables from the schema, the model must reason about their role in the context of a given question. In this work, we present a unified framework for addressing the schema encoding and linking challenges. Thanks to relation-aware self-attention, it jointly learns schema and question word representations based on their alignment with each other and predefined schema relations. Empirically, the RAT framework allows us to gain significant state of the art improvement on textto-SQL parsing. Qualitatively, it provides a way to combine predefined hard schema relations and inferred soft self-attended relations in the same encoder architecture. We foresee this joint representation learning being beneficial in many learning tasks beyond text-to-SQL, as long as the input has predefined structure. A THE NEED FOR SCHEMA LINKING One natural question is how often does the decoder fail to select the correct column, even with the schema encoding and linking improvements we have made. To answer this, we conducted an oracle experiment (see Table 3 ). For "oracle sketch", at every grammar nonterminal the decoder is forced to make the correct choice so the final SQL sketch exactly matches that of the correct answer. The rest of the decoding proceeds as if the decoder had made the choice on its own. Similarly, "oracle cols" forces the decoder to output the correct column or table at terminal productions. With both oracles, we see an accuracy of 99.4% which just verifies that our grammar is sufficient to answer nearly every question in the data set. With just "oracle sketch", the accuracy is only 70.9%, which means 73.5% of the questions that RAT-SQL gets wrong and could get right have incorrect column or table selection. Similarly, with just "oracle cols", the accuracy is 67.6%, which means that 82.0% of the questions that RAT-SQL gets wrong have incorrect structure. In other words, most questions have both column and structure wrong, so both problems will continue to be important to work on for the future.
State of the art in complex text-to-SQL parsing by combining hard and soft relational reasoning in schema/question encoding.
{"src": "scitldr-AIC_train", "paper_id": "H1egcgHtvB"}