Variational Auto Encoders (VAE) are capable of generating realistic images, sounds and video sequences. From practitioners point of view, we are usually interested in solving problems where tasks are learned sequentially, in a way that avoids revisiting all previous data at each stage. We address this problem by introducing a conceptually simple and scalable end-to-end approach of incorporating past knowledge by learning prior directly from the data.   We consider scalable boosting-like approximation for intractable theoretical optimal prior. We provide empirical studies on two commonly used benchmarks, namely  MNIST and Fashion MNIST on disjoint sequential image generation tasks. For each dataset proposed method delivers the best results among comparable approaches,  avoiding catastrophic forgetting in a fully automatic way with a fixed model architecture. Since most of the real-world datasets are unlabeled, unsupervised learning is an essential part of the machine learning field. Generative models allow us to obtain samples from observed empirical distributions of the complicated high-dimensional objects such as images, sounds or texts. This work is mostly devoted to VAEs with the focus on incremental learning setting. It was observed that VAEs ignore dimensions of latent variables and produce blurred reconstructions (Burda et al., 2015; Sønderby et al., 2016) . There are several approaches to address these issues, including amortization gap reduction (Kim et al., 2018) , KL-term annealing (Cremer et al., 2018) and alternative optimization objectives introduction (Rezende and Viola, 2018) . In all cases, it was observed that the choice of the prior distribution is highly important and use of default Gaussian prior overregularizes the encoder. In this work, we address the problem of constructing the optimal prior for VAE. The form of optimal prior (Tomczak and Welling, 2018) was obtained by maximizing a lower bound of the marginal likelihood (ELBO) as the aggregated posterior over the whole training dataset. To construct reasonable approximation, we consider a method of greedy KL-divergence projection. Applying the maximum entropy approach allows us to formulate a feasible optimization problem and avoid overfitting. The greedy manner in which components are added to prior reveals a high potential of the method in an incremental learning setting since we expect prior components to store information about previously learned tasks and overcome catastrophic forgetting (McCloskey and Cohen, 1989; Goodfellow et al., 2013; Nguyen et al., 2017) . From practitioners point of view, it is essential to be able to store one model, capable of solving several tasks arriving sequentially. Hence, we propose the algorithm with one pair of encoderdecoder and update only the prior. We validate our method on the disjoint sequential image generation tasks. We consider MNIST and Fashion-MNIST datasets. In this work, we propose a method for learning a data-driven prior, using a MM algorithm which allows us to reduce the number of components in the prior distribution without the loss of performance. Based on this method, we suggest an efficient algorithm for incremental VAE learning which has single encoder-decoder pair for all the tasks and drastically reduces catastrophic forgetting. To make sure that model keeps encoding and decoding prior components as it did during the training of the corresponding task, we add a regularization term to ELBO. For that purpose at the end of each task we compute reconstructions of the components' mean value p θ j (x|µ i,j ) = p ij (x). All thing considered, objective for the maximization step is the following:
Novel algorithm for Incremental learning of VAE with fixed architecture
{"src": "scitldr-AIC_train", "paper_id": "Sygg5y3NYH"}