Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code. In some cases, autoencoders can "interpolate": By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints. In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefficient from interpolated data. We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting. We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations. One goal of unsupervised learning is to uncover the underlying structure of a dataset without using explicit labels. A common architecture used for this purpose is the autoencoder, which learns to map datapoints to a latent code from which the data can be recovered with minimal information loss. Typically, the latent code is lower dimensional than the data, which indicates that autoencoders can perform some form of dimensionality reduction. For certain architectures, the latent codes have been shown to disentangle important factors of variation in the dataset which makes such models useful for representation learning BID7 BID15 . In the past, they were also used for pre-training other networks by being trained on unlabeled data and then being stacked to initialize a deep network BID1 BID44 . More recently, it was shown that imposing a prior on the latent space allows autoencoders to be used for probabilistic or generative modeling BID18 BID34 BID27 .In some cases, autoencoders have shown the ability to interpolate. Specifically , by mixing codes in latent space and decoding the result, the autoencoder can produce a semantically meaningful combination of the corresponding datapoints. Interpolation has been frequently reported as a qualitative experimental result in studies about autoencoders BID5 BID35 BID30 BID29 BID14 and latent-variable generative models in general BID10 BID33 BID41 . The ability to interpolate can be useful in its own right e.g. for creative applications (Carter & Nielsen, 2017) . However, it also indicates that the autoencoder can "extrapolate" beyond the training data and has learned a latent space with a particular structure. Specifically, if interpolating between two points in latent space produces a smooth semantic warping in data space, this suggests that nearby points in latent space are semantically similar. A visualization of this idea is shown in FIG0 , where a smooth A critic network is fed interpolants and reconstructions and tries to predict the interpolation coefficient α corresponding to its input (with α = 0 for reconstructions). The autoencoder is trained to fool the critic into outputting α = 0 for interpolants. interpolation between a "2" and a "9" suggests that the 2 is surrounded by semantically similar points, i.e. other 2s. This property may suggest that an autoencoder which interpolates well could also provide a good learned representation for downstream tasks because similar points are clustered. If the interpolation is not smooth, there may be "discontinuities" in latent space which could result in the representation being less useful as a learned feature. This connection between interpolation and a "flat" data manifold has been explored in the context of unsupervised representation learning BID3 and regularization BID43 .Given the widespread use of interpolation as a qualitative measure of autoencoder performance, we believe additional investigation into the connection between interpolation and representation learning is warranted. Our goal in this paper is threefold: First , we introduce a regularization strategy with the specific goal of encouraging improved interpolations in autoencoders (section 2); second, we develop a synthetic benchmark where the slippery concept of a "semantically meaningful interpolation" is quantitatively measurable (section 3.1) and evaluate common autoencoders on this task (section 3.2); and third, we confirm the intuition that good interpolation can result in a useful representation by showing that the improved interpolation ability produced by our regularizer elicits improved representation learning performance on downstream tasks (section 4). We also make our codebase available 1 which provides a unified implementation of many common autoencoders including our proposed regularizer. In this paper, we have provided an in-depth perspective on interpolation in autoencoders. We proposed Adversarially Constrained Autoencoder Interpolation (ACAI), which uses a critic to encourage interpolated datapoints to be more realistic. To make interpolation a quantifiable concept, we proposed a synthetic benchmark and showed that ACAI substantially outperformed common autoencoder models. This task also yielded unexpected insights, such as that a VAE which has effectively learned the data distribution might not interpolate. We also studied the effect of improved interpolation on downstream tasks, and showed that ACAI led to improved performance for feature learning and unsupervised clustering. These findings confirm our intuition that improving the interpolation abilities of a baseline autoencoder can also produce a better learned representation for downstream tasks. However, we emphasize that we do not claim that good interpolation always implies a good representation -for example, the AAE produced smooth and realistic interpolations but fared poorly in our representations learning experiments and the denoising autoencoder had low-quality interpolations but provided a useful representation.In future work, we are interested in investigating whether our regularizer improves the performance of autoencoders other than the standard "vanilla" autoencoder we applied it to. In this paper, we primarily focused on image datasets due to the ease of visualizing interpolations, but we are also interested in applying these ideas to non-image datasets. A LINE BENCHMARK
We propose a regularizer that improves interpolation and autoencoders and show that it also improves the learned representation for downstream tasks.
{"src": "scitldr-AIC_train", "paper_id": "S1fQSiCcYm"}