This work presents the Poincaré Wasserstein Autoencoder, a reformulation of
 the recently proposed Wasserstein autoencoder framework on a non-Euclidean
 manifold, the Poincaré ball model of the hyperbolic space H n . By assuming the
 latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure
 on the learned latent space representations. We show that for datasets with latent
 hierarchies, we can recover the structure in a low-dimensional latent space. We
 also demonstrate the model in the visual domain to analyze some of its properties
 and show competitive results on a graph link prediction task. Variational Autoencoders (VAE) (17; 28) are an established class of unsupervised machine learning models, which make use of amortized approximate inference to parametrize the otherwise intractable posterior distribution. They provide an elegant, theoretically sound generative model used in various data domains. Typically, the latent variables are assumed to follow a Gaussian standard prior, a formulation which allows for a closed form evidence lower bound formula and is easy to sample from. However, this constraint on the generative process can be limiting. Real world datasets often possess a notion of structure such as object hierarchies within images or implicit graphs. This notion is often reflected in the interdependence of latent generative factors or multimodality of the latent code distribution. The standard VAE posterior parametrizes a unimodal distribution which does not allow structural assumptions. Attempts at resolving this limitation have been made by either "upgrading" the posterior to be more expressive (27) or imposing structure by using various structured priors (34) , (36) . Furthermore, the explicit treatment of the latent space as a Riemannian manifold has been considered. For instance, the authors of (5) show that the standard VAE framework fails to model data with a latent spherical structure and propose to use a hyperspherical latent space to alleviate this problem. Similarly, we believe that for datasets with a latent tree-like structure, using a hyperbolic latent space, which imbues the latent codes with a notion of hierarchy, is beneficial. There has recently been a number of works which explicitly make use of properties of non-Euclidean geometry in order to perform machine learning tasks. The use of hyperbolic spaces in particular has been shown to yield improved results on datasets which either present a hierarchical tree-like structure such as word ontologies (24) or feature some form of partial ordering (4) . However, most of these approaches have solely considered deterministic hyperbolic embeddings. In this work, we propose the Poincaré Wasserstein Autoencoder (PWA), a Wasserstein autoencoder (33) model which parametrizes a Gaussian distribution in the Poincaré ball model of the hyperbolic space H n . By treating the latent space as a Riemannian manifold with constant negative curvature, we can use the norm ranking property of hyperbolic spaces to impose a notion of hierarchy on the latent space representation, which is better suited for applications where the dataset is hypothesized to possess a latent hierarchy. We demonstrate this aspect on a synthetic dataset and evaluate it using a distortion measure for Euclidean and hyperbolic spaces. We derive a closed form definition of a Gaussian distribution in hyperbolic space H n and sampling procedures for the prior and posterior distributions, which are matched using the Maximum Mean Discrepancy (MMD) objective. We also compare the PWA to the Euclidean VAE visually on an MNIST digit generation task as well quantitatively on a semi-supervised link prediction task. The rest of this paper is structured as follows: we review related work in Section 2, give an overview of the mathematical tools required to work with Riemannian manifolds as well as define the notion of probability distributions on Riemannian manifolds in Section 3. Section 4 describes the model architecture as well as the intuition behind the Wasserstein autoencoder approach. Furthermore, we derive a method to obtain samples from prior and posterior distributions in order to estimate the PWA objective. We present the performed experiments in and discuss the observed results in Section 5 and a summary of our results in Section 6. We have presented an algorithm to perform amortized variational inference on the Poincaré ball model of the hyperbolic space. The underlying geometry of the hyperbolic space allows for an improved performance on tasks which exhibit a partially hierarchical structure. We have discovered certain issues related to the use of the MMD metric in hyperbolic space. Future work will aim to circumvent these issues as well as extend the current results. In particular, we hope to demonstrate the capabilities of our model on more tasks hypothesized to have a latent hyperbolic manifold and explore this technique for mixed curvature settings. A PRIOR REJECTION SAMPLING H (r|0, 1) Result: n samples from prior p(z) while i < n do sampleφ ∼ N (0, I d ); compute direction on the unit sphereφ =φ ||φ|| ; sample u ∼ U(0, 1); get uniform radius samples r i ∈ [0, r max ] via ratio of hyperspheres; where erfc is the complementary error function.
Wasserstein Autoencoder with hyperbolic latent space
{"src": "scitldr-AIC_train", "paper_id": "BJgLpaEtDS"}