The problem of building a coherent and non-monotonous conversational agent with proper discourse and coverage is still an area of open research. Current architectures only take care of semantic and contextual information for a given query and fail to completely account for syntactic and external knowledge which are crucial for generating responses in a chit-chat system. To overcome this problem, we propose an end to end multi-stream deep learning architecture which learns unified embeddings for query-response pairs by leveraging contextual information from memory networks and syntactic information by incorporating Graph Convolution Networks (GCN) over their dependency parse. A stream of this network also utilizes transfer learning by pre-training a bidirectional transformer to extract semantic representation for each input sentence and incorporates external knowledge through the neighbourhood of the entities from a Knowledge Base (KB). We benchmark these embeddings on next sentence prediction task and significantly improve upon the existing techniques. Furthermore, we use AMUSED to represent query and responses along with its context to develop a retrieval based conversational agent which has been validated by expert linguists to have comprehensive engagement with humans. With significant advancements in Automatic speech recognition systems (Hinton et al., 2012; Kumar et al., 2018) and the field of natural language processing, conversational agents have become an important part of the current research. It finds its usage in multiple domains ranging from self-driving cars (Chen et al., 2017b) to social robots and virtual assistants (Chen et al., 2017a) . Conversational agents can be broadly classified into two categories: a task oriented chat bot and a chit-chat based system respectively. The former works towards completion of a certain goal and are specifically designed for domain-specific needs such as restaurant reservations (Wen et al., 2017) , movie recommendation (Dhingra et al., 2017) , flight ticket booking systems ) among many others. The latter is more of a personal companion and engages in human-computer interaction for entertainment or emotional companionship. An ideal chit chat system should be able to perform non-monotonous interesting conversation with context and coherence. Current chit chat systems are either generative (Vinyals & Le, 2015) or retrieval based in nature. The generative ones tend to generate natural language sentences as responses and enjoy scalability to multiple domains without much change in the network. Even though easier to train, they suffer from error-prone responses (Zhang et al., 2018b) . IR based methods select the best response from a given set of answers which makes them error-free. But, since the responses come from a specific dataset, they might suffer from distribution bias during the course of conversation. A chit-chat system should capture semantic, syntactic, contextual and external knowledge in a conversation to model human like performance. Recent work by Bordes et al. (2016) proposed a memory network based approach to encode contextual information for a query while performing generation and retrieval later. Such networks can capture long term context but fail to encode relevant syntactic information through their model. Things like anaphora resolution are properly taken care of if we incorporate syntax. Our work improves upon previous architectures by creating enhanced representations of the conversation using multiple streams which includes Graph Convolution networks (Bruna et al., 2014) , Figure 1 : Overview of AMUSED. AMUSED first encodes each sentence by concatenating embeddings (denoted by ⊕) from Bi-LSTM and Syntactic GCN for each token, followed by word attention. The sentence embedding is then concatenated with the knowledge embedding from the Knowledge Module ( Figure 2 ). The query embedding passes through the Memory Module ( Figure 3 ) before being trained using triplet loss. Please see Section 4 for more details. transformers (Vaswani et al., 2017) and memory networks (Bordes et al., 2016) in an end to end setting, where each component captures conversation relevant information from queries, subsequently leading to better responses. Our contribution for this paper can be summarized as follows: • We propose AMUSED, a novel multi stream deep learning model which learns rich unified embeddings for query response pairs using triplet loss as a training metric. • We perform multi-head attention over query-response pairs which has proven to be much more effective than unidirectional or bi-directional attention. • We use Graph Convolutions Networks in a chit-chat setting to incorporate the syntactical information in the dialogue using its dependency parse. • Even with the lack of a concrete metric to judge a conversational agent, our embeddings have shown to perform interesting response retrieval on Persona-Chat dataset. In the paper, we propose AMUSED, a multi-stream architecture which effectively encodes semantic information from the query while properly utilizing external knowledge for improving performance on natural dialogue. It also employs GCN to capture long-range syntactic information and improves context-awareness in dialogue by incorporating memory network. Through our experiments and results using different metrics, we demonstrate that learning these rich representations through smart training (using triplets) would improve the performance of chit-chat systems. The ablation studies show the importance of different components for a better dialogue. Our ideas can easily be extended to various conversational tasks which would benefit from such enhanced representations.
This paper provides a multi -stream end to end approach to learn unified embeddings for query-response pairs in dialogue systems by leveraging contextual, syntactic, semantic and external information together.
{"src": "scitldr-AIC_train", "paper_id": "rJe6t1SFDB"}