We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust  distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train. Recent work in machine learning has examined a variety of novel ways to augment neural networks with fast memory stores. However, the basic problem of how to most efficiently use memory remains an open question. For instance, the slot-based external memory in models like Differentiable Neural Computers (DNCs BID10 ) often collapses reading and writing into single slots, even though the neural network controller can in principle learn more distributed strategies. As as result, information is not shared across memory slots, and additional slots have to be recruited for new inputs, even if they are redundant with existing memories. Similarly, Matching Networks BID25 BID4 and the Neural Episodic Controller BID21 directly store embeddings of data. They therefore require the volume of memory to increase with the number of samples stored. In contrast, the Neural Statistician BID7 summarises a dataset by averaging over their embeddings. The resulting "statistics" are conveniently small, but a large amount of information may be dropped by the averaging process, which is at odds with the desire to have large memories that can capture details of past experience.Historically developed associative memory architectures provide insight into how to design efficient memory structures that store data in overlapping representations. For example, the Hopfield Net BID14 pioneered the idea of storing patterns in low-energy states in a dynamic system. This type of model is robust, but its capacity is limited by the number of recurrent connections, which is in turn constrained by the dimensionality of the input patterns. The Boltzmann Machine BID1 lifts this constraint by introducing latent variables, but at the cost of requiring slow reading and writing mechanisms (i.e. via Gibbs sampling). This issue is resolved by Kanerva's sparse distributed memory model BID15 , which affords fast reads and writes and dissociates capacity from the dimensionality of input by introducing addressing into a distributed memory store whose size is independent of the dimension of the data 1 .In this paper, we present a conditional generative memory model inspired by Kanerva's sparse distributed memory. We generalise Kanerva's original model through learnable addresses and reparametrised latent variables BID23 BID17 BID5 . We solve the challenging problem of learning an effective memory writing operation by exploiting the analytic tractability of our memory model -we derive a Bayesian memory update rule that optimally trades-off preserving old content and storing new content. The resulting hierarchical generative model has a memory dependent prior that quickly adapts to new data, providing top-down knowledge in addition to bottom-up perception from the encoder to form the latent code representing data. As a generative model, our proposal provides a novel way of enriching the often over-simplified priors in VAE-like models BID22 ) through a adaptive memory. As a memory system, our proposal offers an effective way to learn online distributed writing which provides effective compression and storage of complex data. In this paper, we present the Kanerva Machine, a novel memory model that combines slow-learning neural networks and a fast-adapting linear Gaussian model as memory. While our architecture is inspired by Kanerva's seminal model, we have removed the assumption of a uniform data distribution by training a generative model that flexibly learns the observed data distribution. By implementing memory as a generative model, we can retrieve unseen patterns from the memory through sampling. This phenomenon is consistent with the observation of constructive memory neuroscience experiments BID12 .Probabilistic interpretations of Kanerva's model have been developed in previous works: Anderson (1989) explored a conditional probability interpretation of Kanerva's sparse distributed memory, and generalised binary data to discrete data with more than two values. BID0 provides an approximate Bayesian interpretation based on importance sampling. To our knowledge , our model is the first to generalise Kanerva's memory model to continuous, non-uniform data while maintaining an analytic form of Bayesian inference. Moreover, we demonstrate its potential in modern machine learning through integration with deep neural networks.Other models have combined memory mechanisms with neural networks in a generative setting. For example, BID19 used attention to retrieve information from a set of trainable parameters in a memory matrix. Notably, the memory in this model is not updated following learning. As a result, the memory does not quickly adapt to new data as in our model, and so is not suited to the kind of episode-based learning explored here. BID5 used discrete (categorical ) random variables to address an external memory, and train the addressing mechanism, together with the rest of the generative model, though a variational objective. However, the memory in their model is populated by storing images in the form of raw pixels. Although this provides a mechanism for fast adaptation, the cost of storing raw pixels may be overwhelming for large data sets. Our model learns to to store information in a compressed form by taking advantage of statistical regularity in the images via the encoder at the perceptual level, the learned addresses, and Bayes' rule for memory updates.Central to an effective memory model is the efficient updating of memory. While various approaches to learning such updating mechanisms have been examined recently BID10 BID7 BID24 , we designed our model to employ an exact Bayes' update-rule without compromising the flexibility and expressive power of neural networks. The compelling performance of our model and its scalable architecture suggests combining classical statistical models and neural networks may be a promising direction for novel memory models in machine learning.
A generative memory model that combines slow-learning neural networks and a fast-adapting linear Gaussian model as memory.
{"src": "scitldr-AIC_train", "paper_id": "S1HlA-ZAZ"}