---
abstract: 'Generative models for 3D geometric data arise in many important applications in 3D computer vision and graphics. In this paper, we focus on 3D deformable shapes that share a common topological structure, such as human faces and bodies. Morphable Models and their variants, despite their linear formulation, have been widely used for shape representation, while most of the recently proposed non-linear approaches resort to intermediate representations, such as 3D voxel grids or 2D views. In this work, we introduce a novel graph convolutional operator, acting directly on the 3D mesh, that explicitly models the inductive bias of the fixed underlying graph. This is achieved by enforcing consistent local orderings of the vertices of the graph, through the spiral operator, thus breaking the permutation invariance property that is adopted by all the prior work on Graph Neural Networks. Our operator comes by construction with desirable properties (anisotropic, topology-aware, lightweight, easy-to-optimise), and by using it as a building block for traditional deep generative architectures, we demonstrate state-of-the-art results on a variety of 3D shape datasets compared to the linear Morphable Model and other graph convolutional operators.'
author:
- |
    Giorgos Bouritsas [^1] ^1^ Sergiy Bokhnyak ^\*\ 2^ Stylianos Ploumpis^1,3^\
    Michael Bronstein^1,2,4^ Stefanos Zafeiriou^1,3^\
    Imperial College London, UK^1^ Universita Svizzera Italiana, Switzerland^2^ FaceSoft.io^3^ Twitter^4^\
    ^1^[{g.bouritsas18, s.ploumpis, m.bronstein, s.zafeiriou}@imperial.ac.uk]{} ^2^[bokhns@usi.ch]{}
bibliography:
- 'egbib.bib'
title: 'Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation'
---

![image](architecture_figure.png){width="\linewidth"}

Introduction
============

The success of deep learning in computer vision and image analysis, speech recognition, and natural language processing, has driven the recent interest in developing similar models for 3D geometric data. Generalisations of successful architectures such as convolutional neural networks (CNNs) to data with non-Euclidean structure (e.g. manifolds and graphs) is known under the umbrella term [*Geometric deep learning*]{} [@bronstein2017geometric]. In applications dealing with 3D data, the key challenge of geometric deep learning is a meaningful definition of intrinsic operations analogous to convolution and pooling on meshes or point clouds. Among numerous advantages of working directly on mesh or point cloud data is the fact that it is possible to build invariance to shape transformations (both rigid and nonrigid) into the architecture, as a result allowing to use significantly simpler models and much less training data. So far, the main focus of research in the field of geometric deep learning has been on [*analysis*]{} tasks, encompassing shape classification and segmentation [@qi2017pointnet; @qi2017pointnet++], local descriptor learning, correspondence, and retrieval [@monti2017geometric; @boscaini2016learning; @masci2015geodesic].

On the other hand, there has been limited progress in representation learning and generation of geometric data (*shape synthesis*). Obtaining descriptive and compact representations of meshes and point clouds is essential for downstream tasks such as classification and 3D reconstruction, when dealing with limited labelled training data. Additionally, geometric data synthesis is pivotal in applications such as 3D printing, computer graphics and animation, virtual reality, and game design, and can heavily assist graphics designers and speed-up production. Furthermore, given the high cost and time of acquiring quality 3D data, geometric generative models can be used as a cheap alternative for producing training data for geometric ML algorithms.

Most of the previous approaches in this direction rely on intermediate representations of 3D shapes, such as point clouds [@achlioptas2017learning], voxels [@wu2016learning] or mappings to a flat domain [@moschoglou20193dfacegan; @ben2018multi] instead of direct surface representations, such as meshes. Despite the success of such techniques, they either suffer from high computational complexity (voxels) or absence of smoothness of the data representation (point clouds), while usually pre- and post-processing steps are needed in order to obtain the output surface model. Learning directly on the mesh was only recently explored in [@litany2018deformable; @coma; @wang2018pixel2mesh; @kolotouros2019convolutional] for shape completion, non-linear facial morphable model construction and 3D reconstruction from single images, respectively. In this paper, we propose a novel representation learning and generative framework for fixed topology meshes. For this purpose, we formulate an ordering-based graph convolutional operator, contrary to the permutation invariant operators in the literature of Graph Neural Networks. In particular, similarly to image convolutions, for each vertex on the mesh, we enforce an explicit ordering of its neighbours, allowing a “1-1” mapping between the neighbours and the parameters of a learnable local filter. The order is obtained via a spiral scan, as proposed in [@spirals], hence the name of the operator, *Spiral Convolution*. This way we obtain anisotropic filters without sacrificing computational complexity, while simultaneously we explicitly encode the fixed graph connectivity. The operator can potentially be generalised to other domains that accept implicit local orderings, such as arbitrary mesh topologies and point clouds, while it is naturally equivalent to traditional grid convolutions. Via this equivalence, common CNN practices, such as dilated convolutions, can be easily formulated for meshes.

We use spiral convolution as a basic building block for hierarchical intrinsic mesh autoencoders, which we coin *Neural 3D Morphable Models*. We quantitatively evaluate our methods on several popular datasets: human faces with different expressions (COMA [@coma]) and identities (Mein3D [@booth2018large]) and human bodies with shape ad pose variation (DFAUST [@dfaust:CVPR:2017]). Our model achieves state-of-the-art reconstruction results, outperforming the widely used linear 3D Morphable Model [@vetterMM] and the COMA autoencoder [@coma], as well other graph convolutional operators, including the initial formulation of the spiral operator [@spirals]. We also qualitatively assess our framework showing ‘shape arithmetic’ in the latent space of the autoencoder and by synthesising facial identities via a spiral convolution Wasserstein GAN.

Related Work
============

**Generative models for arbitrary shapes:** Perhaps the most common approaches for generating arbitrary shapes are **volumetric CNNs** [@wu20153d; @qi2016volumetric; @maturana2015voxnet] acting on 3D voxels. For example, voxel regression from images [@girdhar2016learning], denoising autoencoders [@sharma2016vconv] and voxel-GANs [@wu2016learning] have been proposed. Among the key drawbacks of volumetric methods are their inherent high computational complexity and that they yield coarse and redundant representations. **Point clouds** are a simple and lightweight alternative to volumetric representation recently gaining popularity. Several methods have been proposed for representation learning of fixed-size point clouds [@achlioptas2017learning] using the PointNet [@qi2017pointnet] architecture. In [@yang2018foldingnet], point clouds of arbitrary size can be synthesised via a 2D grid deformation. Despite their compactness, point clouds are not popular for realistic and high-quality 3D geometry generation due to their lack of an underlying smooth structure. **Image-based** methods have also been proposed, such as multi-view [@arsalan2017synthesizing] and flat domain mappings such as UV maps [@moschoglou20193dfacegan; @ben2018multi], however they are computationally demanding, require pre- and post-processing steps and usually produce undesirable artefacts. It is also worth mentioning the recently introduced **implicit-surface** based approaches [@mescheder2018occupancy; @chen2018learning; @park2019deepsdf], that can yield accurate results, though with the disadvantage of slow inference (dense sampling of the 3D space followed by marching cubes).\
**Morphable models:** In the case of deformable shapes, such as faces, bodies, hands , where a fixed topology can be obtained by establishing dense correspondences with a template, the most popular methods are still statistical models given their simplicity. For **Faces**, the baseline is the PCA-based 3D Morphable Model (3DMM) [@vetterMM]. The Large Scale Face Model (LSFM) [@booth2018large] was proposed for facial identity and made publicly available, [@cao2014facewarehouse; @FLAME:SiggraphAsia2017] were proposed for facial expression, while for the entire head a large scale model was proposed in [@ploumpis2019combining]. For **Body & Hand**, the most well known models are the skinned vertex-based models SMPL [@SMPL:2015] and MANO [@MANO:SIGGRAPHASIA:2017], respectively. SMPL and MANO are non-linear and require (a) joint localisation and (b) solving special optimisation problems in order to project a new shape to the space of the models. In this paper, we take a different approach introducing a new family of differentiable Morphable Models, which can be applied on a variety of objects, with strong (body) and less strong (face) articulations. Our methods have better representational power and also do not require any additional supervision.\
**Geometric Deep Learning** is a set of recent methods trying to generalise neural networks to non-Euclidean domains such as graphs and manifolds [@bronstein2017geometric]. Such methods have achieved promising results in geometry processing and computer graphics [@masci2015geodesic; @boscaini2016learning], computational chemistry [@duvenaud2015convolutional; @gilmer2017neural], and network science [@kipfGCN; @monti2017geometric]. Multiple approaches have been proposed to construct convolution-like operations, including spectral methods [@bruna2013spectral; @defferrard2016convolutional; @kipfGCN; @yi2017syncspeccnn], local charting based [@masci2015geodesic; @boscaini2016learning; @monti2017geometric; @fey2018splinecnn; @spirals] and soft attention [@velivckovic2017graph; @verma2018feastnet]. Finally, graph or mesh coarsening techniques [@defferrard2016convolutional; @ying2018hierarchical] have been proposed, equivalent to image pooling.

Spiral Convolutional Networks
=============================

Spiral Convolution {#spiral_theory}
------------------

For the following discussion, we assume to be given a manifold, discretised as a triangular mesh $\mathcal{M} = (\mathcal{V,E,F})$ where $\mathcal{V}=\{1, \dots, n\}$, $\mathcal{E}$, and $\mathcal{F}$ denote the sets of vertices, edges, and faces respectively. Furthermore, let ${f:V\rightarrow\mathbb{R}}$, a function representing the vertex features.

One of the key challenges in developing convolution-like operators on graphs or manifolds is the lack of a global system of coordinates that can be associated with each point. The first intrinsic mesh convolutional architectures such as GCNN [@masci2015geodesic], ACNN [@boscaini2016learning] or MoNet [@monti2017geometric] overcame this problem by constructing a [*local*]{} system of coordinates $\mathbf{u}(x,y)$ around each vertex $x$ of the mesh, in which a set of local weighting functions $w_1,\hdots, w_L$ is applied to aggregate information from the vertices $y$ of the neighborhood $\mathcal{N}(x)$. This allows to define ‘patch operators’ generalising the sliding window filtering in images: $$\label{soft_attn}
    (f\star g)_x = \sum_{\ell=1}^L g_\ell \sum_{y\in\mathcal{N}(x)} w_\ell(\mathbf{u}(x,y)) f(y)$$ where $\sum_{y\in\mathcal{N}(x)} w_\ell(\mathbf{u}(x,y)) f(y)$ are ‘soft pixels’ ($L$ in total), $f$ are akin to pixel intensity in images, and $g_\ell$ the filter weights. The problem of the absence of a global coordinate system is equivalent to the absence of canonical ordering of the vertices, and the patch-operator based approaches can be also interpreted as attention mechanisms, as in [@velivckovic2017graph] and [@verma2018feastnet]. In particular, the absence of ordering does not allow the construction of a “1-1” mapping between neighbouring features $f(y)$ and and filter weights $g_\ell$, thus a “all-to-all” mapping is performed via learnable *soft-attention* weights $w_\ell(\mathbf{u}(x,y))$. In the Euclidean setting, such operators boil down to the classical convolution, since an ordering can be obtained via the global coordinate system.

Besides the lack of a global coordinate system, another motivation for patch-operator based approaches when working on meshes, is the need for *insensitivity to meshing* of the continuous surface, ideally, each patch operator should be independent of the underlying graph topology.

[0.35]{} ![Spiral ordering on a mesh and an image patch[]{data-label="spiral_fig"}](Spiral_on_Mesh.png "fig:"){width="\linewidth"} \[spiral\_mesh\]

[0.33]{} ![Spiral ordering on a mesh and an image patch[]{data-label="spiral_fig"}](Spiral_on_Grid.png "fig:"){width="\linewidth"} \[spiral\_image\]

However, all the methods falling into this family, come at the cost of high computational complexity and parameter count and can be hard to optimise. Moreover, patch-operator based methods specifically designed for meshes, require hand-crafting and pre-computing the local systems of coordinates. To this end, in this paper we make a crucial observation in order to overcome the disadvantages of the aforementioned approaches: the issues of the absence of a global ordering and insensitivity to graph topology are irrelevant when dealing with [*fixed topology*]{} meshes. In particular, one can locally order the vertices and keep the order fixed. Then, graph convolution can be defined as follows: $$\label{ordering_based}
    (f\star g)_x = \sum_{\ell=1}^L g_\ell f(x_\ell).$$ where $\{x_1, \hdots, x_L\}$ denote the neighbours of vertex $x$ ordered in a fixed way. Here, in analogy with the patch operators, each patch operator is a single neighbouring vertex.

In the Euclidean setting, the order is simply a raster scan of pixels in a patch. On meshes, we opt for a simple and intuitive ordering using spiral trajectories inspired by [@spirals]. Let $x\in \mathcal{V}$ be a mesh vertex, and let $R^d(x)$ be the $d$-[*ring*]{}, an ordered set of vertices whose shortest (graph) path to $x$ is exactly $d$ hops long; $R_j^d(x)$ denotes the $j$th element in the $d$-ring (trivially, $R_1^0(x) = x$). We define the [*spiral patch operator*]{} as the ordered sequence $$S(x) = \{x, R_1^1(x), R_2^1(x), \hdots, R^h_{|R^h|}\},$$ where $h$ denotes the patch radius, similar to the size of the kernel in classical CNNs. Then, [*spiral convolution*]{} is: $$(f * g)_x = \sum_{\ell=1}^{L} g_\ell \, f\big(S_\ell(x)\big).  $$ The uniqueness of the ordering is given by fixing two degrees of freedom: the direction of the rings and the first vertex $R_1^1(x)$. The rest of the vertices of the spiral are ordered inductively. The direction is chosen by moving clockwise or counterclockwise, while the choice of the first vertex, the *reference point*, is based on the underlying geometry of the shape to ensure the robustness of the method. In particular, we fix a reference vertex $x_0$ on a template shape and choose the initial point for each spiral to be in the direction of the shortest geodesic path to $x_0$, . $$R_1^1(x) = \operatorname*{arg\,min}\limits_{y \in R^1(x)}{d_\mathcal{M}(x_0,y)},$$ where $d_{\mathcal{M}}$ is the geodesic distance between two vertices on the mesh $\mathcal{M}$. In order to allow for fixed-sized spirals, we choose a fixed length $L$ as a hyper-parameter and then either truncate or zero-pad each spiral depending on its size.

**Comparison to Lim [@spirals]**: The authors choose the starting point of each spiral at random, for every mesh sample, every vertex, and every epoch during training. This choice prevents us from explicitly encoding the fixed connectivity, since corresponding vertices in different meshes will not undergo the same transformation (as in image convolutions). Moreover, single vertices also undergo different transformations every time a new spiral is sampled. Thus, in order for the network to obtain robustness to different spiral samples, it inevitably has to become invariant to different rotations of the neighbourhoods, thus it has reduced capacity. To this end, we emphasise the need of consistent orderings across different meshes. Moreover, in [@spirals], the authors model the vertices on the spiral via a recurrent network, which has higher computational complexity, is harder to optimise and does not take advantage of the stationary properties of the 3D shape (local statistics are repeated across different patches), which are treated by our spiral kernel with weight sharing.

**Comparison to spectral filters:** Spectral convolutional operators developed in [@chebnet; @kipfGCN] for graphs and used in [@coma] for mesh autoencoders, suffer from the fact that the are inherently [*isotropic*]{}. This is a side-effect when one, under the absence of a canonical ordering, needs to design a permutation-invariant operator with small number of parameters. In particular, spectral filters rely on the [*Laplacian operator*]{}, which performs a weighted averaging of the neighbour vertices : $$(\Delta f)_x = \sum\nolimits_{y: (x,y) \in \mathcal{E}} w_{xy} \big(f(y) - f(x)\big),$$ where $w_{xy}$ denotes an edge weight. A polynomial of degree $r$ with learnable coefficients $\theta_0, \hdots, \theta_r$ is then applied to $\Delta$. Then, the graph convolution amounts to filtering the Laplacian eigenvalues, ${p(\Delta) = \Phi p(\Lambda) \Phi^\top}$. Equivalently: $$(f * g) = p(\Delta) f = \sum_{\ell = 0}^r \theta_\ell \Delta^\ell f,$$ While a necessary evil in general graphs, spectral filters on meshes are rather weak given that they are locally rotationally-invariant. On the other hand, spiral convolutional filters leverage the fact that on a mesh one can canonically order the neighbours. Thus, they are anisotropic by construction and as will be shown in the experimental section \[evaluation\] they are expressive by using just one-hop neighbourhoods, contrary to the large receptive fields used in [@coma]. In Fig \[isotropy\] we visualise the impulse response (centred on a vertex on the forehead) of a selected laplacian polynomial filter from the architecture of [@coma] (left) and from a spiral convolutional filter with $h=1$ (right).

Finally, the equivalence of spiral convolutions to image convolutions allows the use of long-studied practices in the computer vision community. For example, small patches can be used, leading to few parameters and fast computation. Furthermore, dilated convolutions [@yu2015multi] can also be adapted to the spiral operator by simply sub-sampling the spiral. Finally, we argue here that our operator could be applied to other domains, such as point clouds, where an ordering of the data points can be enforced.

Neural 3D Morphable Models
--------------------------

Let $\bm{F}=[\bm{f_0}| \bm{f_1}|..., \bm{f_N}]$, $\bm{f_i} \in \mathbb{R}^{d*m}$ the matrix of all the signals defined on a set of meshes in dense correspondence that are sampled from a distribution $\mathcal{D}$, where $d$ the dimensionality of the signal on the mesh (vertex position, texture etc.) and $m$ the number of vertices. A linear 3D Morphable Model [@vetterMM] represents arbitrary instances $\mathbf{y}\in \mathcal{D}$ as a linear combination of the k largest eigenvectors of the covariance matrix of $\bm{F}$ by making a gaussianity assumption: $$\mathbf{y} \approx \Bar{\bm{f}} + \sum_{i}^k \alpha_i \sqrt{d_i} \mathbf{v_i}$$

where $\Bar{\bm{f}}$ the mean shape, $\mathbf{v_i}$ is the $i$th principal component, $d_i$ the respective eigenvalue and $\alpha_i$ the linear weight coefficient. Given its linear formulation, the representational power of the 3DMM is constrained by the span of the eigenvectors, while its parameters scale linearly w.r.t the number of the eigencomponents used, leading to large parametrisations for meshes of high resolution.

In contrast, in this paper, we use spiral convolutions as a building block to build a fully differentiable non-linear Morphable Model. In essence, a Neural 3D Morphable Model is a deep convolutional mesh autoencoder, that learns hierarchical representations of a shape. An illustration of the architecture can be found in Fig \[fig:architecture\]. Leveraging the connectivity of the graph with spiral convolutional filters, we allow for local processing of each shape, while the hierarchical nature of the model allows learning in multiple scales. This way we manage to learn semantically meaningful representations and considerably reduce the number of parameters. Furthermore, we bypass the need to make assumptions about the distribution of the data.

Similar to traditional convolutional autoencoders, we make use of series of convolutional layers with small receptive fields followed by pooling and unpooling, for the encoder and the decoder respectively, where a decimated or upsampled version of the mesh is obtained each time and the features of the existing vertices are either aggregated or extrapolated. We follow [@coma] for the calculation of the features of the added vertices after upsampling, through interpolation by weighting the nearby vertices with barycentric coordinates. The network is trained by minimising the $L_1$ norm between the input and the predicted output.

Spiral Convolutional GAN
------------------------

In order to improve the synthesis of meshes of high resolution, thus increased detail, we extend our framework with a distribution matching scheme. In particular, we propose a mesh Wasserstein GAN [@pmlr-v70-arjovsky17a] with gradient penalty to enforce the Lipschitz constraint [@gulrajani2017improved], that is trained to minimise the wasserstein divergence between the real distribution of the meshes and the distribution of those produced by the generator network. The generator and discriminator architectures, have the same structure as the decoder and the encoder of the Neural3DMM respectively. Via this framework, we obtain two additional properties that are inherently absent from the autoencoder: high frequency detail and a straightforward way to sample from the latent space.

Evaluation
==========

In this section, we showcase the effectiveness of our proposed method on a variety of shape datasets. We conduct a series of ablation studies in order to compare our operator to other Graph Neural Networks, by using the same autoencoder architecture. Fist, we demonstrate the inherent higher capacity of spiral convolutions compared to ChebNet (spectral). Moreover, we discuss the advantages of our method compared to soft-attention based Graph Neural Networks, such as patch-operator based. Finally, we show the importance of the consistency of the ordering by comparing our method to different variants of the method proposed in [@spirals].

Furthermore, we quantitatively show that our method can yield better representations than the linear 3DMM and COMA, while maintaining a small parameter count and frequently allowing a more compact latent representation. Moreover, we proceed with a qualitative evaluation of our method by generating novel examples through vector space arithmetic. Finally, we assess our intrinsic GAN in terms of its ability to produce high resolution realistic examples.

For all the cases, we choose as signal on the mesh the normalised deformations from the mean shape, for every vertex we subtract its mean position and divide with the standard deviation. In this way, we encourage signal stationarity, thus facilitating optimisation. The code is available at <https://github.com/gbouritsas/neural3DMM>.

Datasets
--------

**COMA.** The facial expression dataset from Ranjan [@coma], consisting of 20K+ 3D scans (5023 vertices) of twelve unique identities performing twelve types of extreme facial expressions. We used the same data split as in [@coma]. **DFAUST.** The dynamic human body shape dataset from Bogo [@dfaust:CVPR:2017], consisting of 40K+ 3D scans (6890 vertices) of ten unique identities performing actions such as leg and arm raises, jumps, etc. We randomly split the data into a test set of 5000, 500 validation, and 34,5K+ train.

**MeIn3D.** The 3D large scale facial identity dataset from Booth [@booth20163d], consisting of more than 10,000 distinct identity scans with 28K vertices which cover a wide range of gender ethnicity and age. For the subsequent experiments, the MeIn3D dataset was randomly split within demographic constraints to ensure gender, ethnic and age diversity, into 9K train and 1K test meshes.

![image](coma_errors.png){width="0.33\linewidth"} ![image](dfaust_errors.png){width="0.33\linewidth"} ![image](mein3d_errors.png){width="0.33\linewidth"}

For the quantitative experiments of sections \[ablation\] and \[3dmm\_comparison\] the evaluation metric used is **generalisation**, which measures the ability of a model to represent novel shapes from the same distribution as it was trained on. More specifically we evaluate the average per sample and per vertex euclidean distance in the 3D space (in millimetres) between corresponding vertices in the input and its reconstruction.

Implementation Details {#implementation}
----------------------

We denote as $SC(h,w)$ a spiral convolution of $h$ hops and $w$ filters, $DS(p)$ and $US(p)$ a downsampling and an upsampling by a factor of $p$, respectively, $FC(d)$ a fully connected layer, $l$ the number of vertices after the last downsampling layer. The simple Neural3DMM for COMA and DFAUST datasets is the following:

$\mathit{Enc:} \, SC(1,16)\rightarrow DS(4)\rightarrow SC(1,16)\rightarrow DS(4)\rightarrow SC(1,16)\rightarrow DS(4)\rightarrow SC(1,32)\rightarrow DS(4)\rightarrow FC(d)$

$\mathit{Dec:} \,FC(l*32)\rightarrow US(4)\rightarrow SC(1,32)\rightarrow US(4)\rightarrow SC(1,16)\rightarrow US(4)\rightarrow SC(1,16)\rightarrow US(4)\rightarrow SC(1,3)$ For Mein3D, due to the high vertex count, we modified the COMA architecture for our simple Neural3DMM by adding an extra convolution and an extra downsampling/upsampling layer in the encoder and the decoder respectively (encoder filter sizes: \[8,16,16,32,32\], decoder: mirror of the encoder). The larger Neural3DMM follows the above architecture, but with an increased parameter space. For COMA, the convolutional filters of the encoder had sizes \[64,64,64,128\] and for Mein3D the sizes were \[8,16,32,64,128\], while the decoder is a mirror of the encoder. For DFAUST, the sizes were \[16,32,64,128\] and \[128,64,32,32,16\] and dilated convolutions with $h=2$ hops and dilation ratio $r=2$ were used for the first and the last two layers of the encoder and the decoder respectively. We observed that by adding an additional convolution at the very end (of size equal to the size of the input feature space), training was accelerated. All of our activation functions were ELUs [@ELU]. Our learning rate was $10^{-3}$ with a decay of $0.99$ after each epoch, and our weight decay was $5\times 10^{-5}$. All models were trained for 300 epochs.

![Spiral vs ChebNet (spectral) filters[]{data-label="fig:spectral_spiral_comparison"}](coma_isotropic_anisotropic.png "fig:"){width="0.6\linewidth"} \[fig:coma\_tropic\_comparison\]

![Spiral vs ChebNet (spectral) filters[]{data-label="fig:spectral_spiral_comparison"}](dfaust_isotropic_anisotropic.png "fig:"){width="0.6\linewidth"} \[fig:dfaust\_tropic\_comparison\]

Ablation Studies {#ablation}
----------------

### Isotropic vs Anisotropic Convolutions {#iso_comaparison}

For the purposes of this experiment we used the architecture deployed by the authors of [@coma]. The number of parameters in our case is slightly larger due to the fact that the immediate neighbours, that affect the size of the spiral, range from 7 to 10, while the polynomials used in [@coma] go up to the 6th power of the Laplacian. For both datasets, as clearly illustrated in Fig \[fig:spectral\_spiral\_comparison\], spiral convolution-based autoencoders consistently outperform the spectral ones for every latent dimension, in accordance with the analysis made in section \[spiral\_theory\]. Additionally, increasing the latent dimensions, our model’s performance increases at a higher rate than its counterpart. Notice that the number of parameters scales the same way as the latent size grows, but the spiral model makes better use of the added parameters especially looking at dimensions 16, 32, 64, and 128. Especially on the COMA dataset, the spectral model seems to be flattening between 64 and 128 while the spiral is still noticeably decreasing.

### Spiral vs Attention based Convolutions {#attention_comaparison}

In this experiment we compare our method with certain state-of-the-art soft-attention based Graph Neural Networks: **MoNet**: the patch-operator based model of [@monti2017geometric], where the attention weights are the learnable parameters of gaussian kernels defined on a pseudo-coordinate space[^2], **FeastNet** [@verma2018feastnet] and **Graph Attention** [@velivckovic2017graph], where the attention weights are learnable functions of the input features. In table \[soft\_attn\], we provide results on COMA dataset, using the simple Neural3DMM architecture with latent size 16. We choose the number of attention heads (gaussian kernels in [@monti2017geometric]) to be either 9 (equal to the size of the spiral in our method, for a fair comparison) or 25 (as in [@monti2017geometric], to showcase the effect of over-parametrisation). When it comes to similar number of parameters our method manages to outperform its counterparts, while compared to over-parametrised soft attention networks it either outperforms them, or achieves slightly worse performance. This shows that the spiral operator can make more efficient use of the available learnable parameters, thus being a lightweight alternative to attention-based methods without sacrificing performance. Also, its formulation allows for fast computation; in table \[soft\_attn\] we measure per mesh inference time in ms (on a GeForce RTX 2080 Ti GPU).

                                     Ours                       
  --------- ------- ------- ------- ------- ------- ----------- ------
  kernels      9      25       9      25       9        25        -
  error      0,762   0,732   0,750   0,623   0,708   **0,583**   ****
  params             101K             98K               95K      ****
  time       12,77   15,37   9,04    9,66    10,55     10,96     ****

  : Spirals vs soft-attention operators[]{data-label="spiral_vs_attn"}

### Comparison to Lim et al. [@spirals] {#spiral_comaparison}

In order to showcase how the operator behaves when the ordering is not consistent, we perform experiments under four scenarios: the original formulation of [@spirals], where each spiral is randomly oriented for every mesh and every epoch (*rand mesh & epoch)*; choosing the same orientation across all the meshes randomly at every epoch (*rand epoch*); choosing different orientations for every mesh, but keeping them fixed across epochs (*rand mesh*); and fixed ordering (Ours). We compare the LSTM-based approach of [@spirals] and our linear projection formulation (Eq ). The experimental setting and architecture is the same as in the previous section. The proposed approach achieves over 28% improved performance compared to [@spirals], which substantiates the benefits of passing corresponding points through the same transformations.

  operation     rand mesh & epoch   rand mesh   rand epoch    fixed ordering
  ------------ ------------------- ----------- ------------ ------------------
  LSTM          0.888 [@spirals]      0.880       0,996           0.792
  lin. proj.          0.829           0.825       0.951      **0.635 (Ours)**

  : Importance of the ordering consistency

\[text\]

Neural 3D Morphable models {#3dmm_comparison}
--------------------------

![Colour coding of the per vertex euclidean error of the reconstructions produced by PCA (2nd), COMA (3rd), and our Neural3DMM (bottom). Top row is ground truth.[]{data-label="fig:dfaust_color_coding"}](dfaust_color_coding.jpg){width="0.6\linewidth"}

### Quantitative results

In this section, we compare the following methods for different dimensions of the latent space: **PCA**, the 3D Morphable Model [@vetterMM], **COMA**, the ChebNet-based Mesh Autoencoder, **Neural3DMM (small)**, ours spiral convolution autoencoder with the same architecture as in COMA, **Neural3DMM (ours)**, our proposed Neural3DMM framework, where we enhanced our model with a larger parameter space (see Sec. \[implementation\]). The latent sizes were chosen based on the variance explained by PCA (explained variance of roughly 85%, 95% and 99% of the total variance).

As can be seen from the graphs in Fig \[quantitative\], our Neural3DMM achieves smaller generalisation errors in every case it was tested on. For the COMA and DFAUST datasets all hierarchical intrinsic architectures outperform PCA for small latent sizes. That should probably be attributed to the fact that the localised filters used allow for effective reconstruction of smaller patches of the shape, such as arms and legs (for the DFAUST case), whilst PCA attempts a more global reconstruction, thus its error is distributed equally across the entire shape. This is well shown in Fig \[fig:dfaust\_color\_coding\], where we compare exemplar reconstructions of samples from the test set (latent size 16). It is clearly visible that PCA prioritises body shape over pose resulting to body parts in the wrong locations (for example see the right leg of the woman on the leftmost column). On the contrary COMA places the vertices in approximately correct locations, but struggles to recover the fine details of the shape leading to various artefacts and deformities; our model on the other hand seemingly balances these two difficult tasks resulting in quality reconstructions that preserve pose and shape.

Comparing to [@coma], it is again apparent here that our spiral-based autoencoder has increased capacity, which together with the increased parameter space, makes our larger Neural3DMM outperform the other methods by a considerably large margin in terms of both generalisation and compression. Despite the fact that for higher dimensions, PCA can explain more than 99% of the total variance, thus making it a tough-to-beat baseline, our larger model still manages to outperform it. The main advantage here is the substantially smaller number of parameters of which we make use. This is clearly seen in the comparison for the MeIn3D dataset, where the large vertex count makes non-local methods as PCA impractical. It is necessary to mention here, that larger latent space sizes are not necessarily desirable for an autoencoder because they might lead to less semantically meaningful and discriminative representation for downstream tasks.

### Qualitative results

Here, we assess the representational power of our models by the common practice of testing their ability to perform linear algebra in their latent spaces.

**Interpolation** Fig \[fig:interpolation\]: We choose two sufficiently different samples $\mathbf{x_1}$ and $\mathbf{x_2}$ from our test set, encode them in their latent representations $\mathbf{z_1}$ and $\mathbf{z_2}$ and then produce intermediate encodings by sampling the line that connects them $\mathbf{z} = a\mathbf{z_1}+(1-a)\mathbf{z_2}$, where $a\in(0,1)$.

**Extrapolation** Fig \[fig:extrapolation\]: Similarly, we decode latent representations that reside on the line defined by $\mathbf{z_1}$ and $\mathbf{z_2}$, but outside the respective line segment, $\mathbf{z} = a*\mathbf{z_1}+(1-a)*\mathbf{z_2}$, where $a\in(-\infty,0)\cup(1,+\infty)$. We choose $\mathbf{z_1}$ to be our neutral expression for COMA and neutral pose for DFAUST, in order to showcase the exaggeration of a specific characteristic on the shape.

![Extrapolation. Left: neutral expression/pose[]{data-label="fig:extrapolation"}](extrapolation_final.png){width="0.6\linewidth"}

**Shape Analogies** Fig \[fig:analogies\]: We choose three meshes $A$, $B$, $C$, and construct a $D$ such that it satisfies $A$:$B$::$C$:$D$ using linear algebra in the latent space as in [@mikolov]: $e(B)-e(A)=e(D)-e(C)$ ($e(*)$ the encoding), where we then solve for $e(D)$ and decode it. This way we transfer a specific characteristic using meshes from our dataset.

![Analogies in MeIn3D and DFAUST[]{data-label="fig:analogies"}](analogies.png){width="0.6\linewidth"}

GAN evaluation
--------------

In figure \[fig:gan\_samples\], we sampled several faces from the latent distribution of the trained generator. Notice that they are realistically looking and, following the statistics of the dataset, span a large proportion of the real distribution of the human faces, in terms of ethnicity, gender and age. Compared to the most popular approach for synthesizing faces, the 3DMM, our model learns to produce fine details on the facial structure, making them hard to distinguish from real 3D scans, whereas the 3DMM, although it produces smooth surfaces, frequently makes it easy to tell the difference between real and artificially produced samples. We direct the reader to the supplementary material to compare with samples drawn from the 3DMM’s latent space.

![Generated identities from our intrinsic 3D GAN[]{data-label="fig:gan_samples"}](gan_tile.png){width="0.7\columnwidth"}

Conclusion
==========

In this paper we introduced a representation learning and generative framework for fixed topology 3D deformable shapes, by using a mesh convolutional operator, spiral convolutions, that efficiently encodes the inductive bias of the fixed topology. We showcased the inherent representational power of the operator, as well as its reduced computational complexity, compared to prior work on graph convolutional operators and show that our mesh autoencoder achieves state-of-the-art results in mesh reconstruction. Finally, we present the generation capabilities of our models through vector space arithmetic, as well as by synthesising novel facial identities. Regarding future work, we plan to extend our framework to general graphs and 3D shapes of arbitrary topology, as well as to other domains that have capacity for an implicit ordering of their primitives, such as point clouds.

Acknowledgements
================

This research was partially supported by ERC Consolidator Grant No. 724228 (LEMAN), Google Research Faculty awards, and the Royal Society Wolfson Research Merit award. G. Bouritsas was funded by the Imperial College London, Department of Computing, PhD scholarship. Dr. Zafeiriou acknowledges support from a Google Faculty award and EPSRC fellowship Deform (EP/S010203/1). S. Ploumpis was suppored by EPSRC Project (EP/N007743/1) FACER2VM.

[^1]: Equal Contribution

[^2]: here we display the best obtained results when choosing the pseudo-coordinates to be local cartesian.
