Hierarchical Bayesian methods have the potential to unify many related tasks (e.g. k-shot classification, conditional, and unconditional generation) by framing each as inference within a single generative model. We show that existing approaches for learning such models can fail on expressive generative networks such as PixelCNNs, by describing the global distribution with little reliance on latent variables. To address this, we develop a modification of the Variational Autoencoder in which encoded observations are decoded to new elements from the same class; the result, which we call a Variational Homoencoder (VHE), may be understood as training a hierarchical latent variable model which better utilises latent variables in these cases. Using this framework enables us to train a hierarchical PixelCNN for the Omniglot dataset, outperforming all existing models on test set likelihood. With a single model we achieve both strong one-shot generation and near human-level classification, competitive with state-of-the-art discriminative classifiers. The VHE objective extends naturally to richer dataset structures such as factorial or hierarchical categories, as we illustrate by training models to separate character content from simple variations in drawing style, and to generalise the style of an alphabet to new characters. Learning from few examples is possible only with strong inductive biases. In machine learning such biases can come from hand design, as in the parametrisation of a model, or can be the result of a meta-learning algorithm. Furthermore they may be task-specific, as in discriminative modelling, or may describe the world causally so as to be naturally reused across many tasks.Recent work has approached one-and few-shot learning from all of these perspectives. Siamese Networks BID15 , Matching Networks , Prototypical Networks BID23 and MANNs BID22 ) are all models discriminatively trained for few-shot classification. Such models can achieve state-of-the-art performance at the task they were trained for, but provide no principled means for transferring knowledge to other tasks.Other work such as has developed conditional generative models, which take one or a few observations from a class as input, and return a distribution over new elements p(x|D). These models may be used as classifiers despite not being explicitly trained for this purpose, by comparing conditional likelihoods. They may also be used to generate full sets incrementally as p(X) = i p(x i |x 1 , . . . , x i−1 ), as discussed in Generative Matching Networks BID0 . However, such models are a more natural fit to sequences than sets as they typically lack exchangeability, and furthermore they do not expose any latent representation of shared structure within a set.Finally are hierarchical approaches that model shared structure through latent variables, as p(X) = A VAE treats all datapoints as independent, so only a single random element need be encoded and decoded each step. A Neural Statistician instead feeds a full set of elements X through both encoder and decoder networks, in order to share a latent variable c. In a VHE, we bound the full likelihood p(X) using only random subsamples D and x for encoding/decoding. Optionally, p(x|c) may be defined through a local latent variable z.In this work we propose the Variational Homoencoder (VHE), aiming to combine several advantages of the models described above:1. Like conditional generative approaches, we train on a few-shot generation objective which matches how our model may be used at test time. However, by introducing an encoding cost, we simultaneously optimise a likelihood lower bound for a hierarchical generative model, in which structure shared across elements is made explicit by shared latent variables. 2. Previous work BID6 has learned hierarchical Bayesian models by applying Variational Autoencoders to sets, such as classes of images. However, their approach requires feeding a full set through the model per gradient step FIG1 ), rendering it intractable to train on very large sets. In practice, they avoid computational limits by sampling smaller subset as training data. In a VHE, we instead optimise a likelihood bound for the complete dataset, while constructing this bound by subsampling. This approach can not only improve generalisation, but also departs from previous work by extending to models with richer latent structure, for which the joint likelihood cannot be factorised. 3. As with a VAE, the VHE objective includes both an encoding-and reconstruction-cost.However, by sharing latent variables across a large set of elements, the encoding cost per element is reduced significantly. This facilitates use of powerful autoregressive decoders, which otherwise often suffer from ignoring latent variables BID3 . We demonstrate the significance of this by applying a VHE to the Omniglot dataset. Using a Pixel-CNN decoder (Oord et al., 2016), our generative model is arguably the first with a general purpose architecture to both attain near human-level one-shot classification performance and produce high quality samples in one-shot generation. We introduced the Variational Homoencoder: a deep hierarchical generative model learned by a novel training procedure which resembles few-shot generation. This framework allows latent variables to be shared across a large number of elements in a dataset, encouraging them to be well utilised even alongside highly expressive decoder networks. We demonstrate this by training a hierarchical PixelCNN model on Omniglot dataset, and show that our novel training objective is responsible for the state-of-the-art results it achieves. This model is arguably the first which uses a general purpose architecture to both attain near human-level one-shot classification performance and produce high quality samples in one-shot generation.The VHE framework extends naturally to models with richer latent structure, as we demonstrate with two examples: a hierarchical model which generalises the style of an alphabet to produce new characters, and a factorial model which separates the content and drawing style of coloured character images. In addition to these modelling extensions, our variational bound may also be tightened by learning a subsampling procedure q(D; X), or by introducing an auxiliary inference network r(D; c, X) as discussed in Supplementary Material 6.1. While such modifications were unnecessary for our experiments on Omniglot character classes, we expect that they may yield improvements on other datasets with greater intra-class diversity.6 SUPPLEMENTARY MATERIAL
Technique for learning deep generative models with shared latent variables, applied to Omniglot with a PixelCNN decoder.
{"src": "scitldr-AIC_train", "paper_id": "HJ5AUm-CZ"}