The integration of a  Knowledge Base (KB) into a neural dialogue agent is one of the key challenges in Conversational AI. Memory networks has proven to be effective to encode KB information into an external memory to thus generate more fluent and informed responses. Unfortunately, such memory becomes full of latent representations during training, so the most common strategy is to overwrite old memory entries randomly. 

 In this paper, we question this approach and provide experimental evidence showing that conventional memory networks generate many redundant latent vectors resulting in overfitting and the need for larger memories. We introduce memory dropout as an automatic technique that encourages diversity in the latent space by 1) Aging redundant memories to increase their probability of being overwritten during training 2) Sampling new memories that summarize the knowledge acquired by redundant memories. This technique allows us to incorporate  Knowledge Bases to achieve state-of-the-art dialogue generation in the Stanford Multi-Turn Dialogue dataset. Considering the same architecture, its use provides an improvement of +2.2 BLEU points for the automatic  generation of responses and an increase of +8.1% in the recognition of named entities. Given the large amount of dialogue data recorded in human-human or human-chatbot interactions, there is a great need for dialogue systems that infer automatic responses grounded to personal knowledge bases. This approach has the advantage of integrating semantic information that is fundamental to achieve dialogue understanding. We want to leverage the contextual information present in a KB (e.g., a calendar of events) to answer queries like What time is my dentist appointment? . This task is challenging because existing neural dialogue agents often assume that the dialogue history carries the information needed to provide an answer but struggle to interface with the structured data stored in a KB. This assumption prevents to have an end-to-end differentiable model to maintain the kind of contextual conversations that people desire. Memory networks Miller et al. (2016) has proven to be effective to encode KB information into an external memory to generate more fluent and informed responses. However, there is no much work in regularizing the latent representations stored in the external memory. Unlike the conventional dropout technique used to regularize deep neural networks Srivastava et al. (2014) , we propose memory dropout to attain the same goal (i.e., reduction of overfitting) but with different functionality and designed for memory networks Weston et al. (2015) . Given the long-term nature of memory networks, we do not immediately remove redundant memories with some probability as in the original dropout algorithm. Instead, we assign them the current maximum age increasing their probability of being overwritten by more recent latent representations in future training steps. Thus, in contrast to Srivastava et al. (2014) , our memory dropout is a delayed regularization mechanism. The main contributions of our work are the following: • We introduce a new regularization method called memory dropout designed for dealing with overfitting in Memory Augmented Neural Networks. To our best knowledge, ours is the first work on regularizing memory networks. • We build a neural dialogue agent that uses memory dropout to incorporate KB into an external memory for automatic response generation. Our results show that this technique can generate more fluent and accurate responses: an improvement of +2.2 BLUE points and +8.1% Entity F1 score versus not using it in the Stanford Multi-Turn Dialogue dataset. Figure 1: Learning the (h, y) pair transitions the neighborhood of h (represented as an ellipse) to a new state in which a memory h is drawn as the distribution of positive memories. Small circles represent the uncertainty of using a particular memory to model h . In the new memory configuration, we age positive keys (now faded in grey) making them more likely of being overwritten by other training examples. Memory Dropout is a technique for improving memory augmented neural networks by breaking co-adaptating memories built during backpropagation. While conventional dropout works at the level of individual activations, our memory dropout deals with latent representations of the input. These arrays of activations are stored into an external memory module which resembles areas of the human brain that are content-addressable and sensitive to semantic information Wixted et al. (2018) . Central to this technique is the idea that age and uncertainty play important roles to regularize the addressable keys of an external memory module that is persistent across training examples. By doing this, we obtain higher BLEU and Entity F1 scores when training a task-oriented dialogue agent that decodes an answer considering the entries of KB stored in the memory module.
Conventional memory networks generate many redundant latent vectors resulting in overfitting and the need for larger memories. We introduce memory dropout as an automatic technique that encourages diversity in the latent space.
{"src": "scitldr-AIC_train", "paper_id": "SJl7tREFvr"}