Convolutional neural networks (CNNs) have achieved state of the art performance on recognizing and representing audio, images, videos and 3D volumes; that is, domains where the input can be characterized by a regular graph structure. 
 However, generalizing CNNs to irregular domains like 3D meshes is challenging. Additionally, training data for 3D meshes is often limited. In this work, we generalize convolutional autoencoders to mesh surfaces. We perform spectral decomposition of meshes and apply convolutions directly in frequency space. In addition, we use max pooling and introduce upsampling within the network to represent meshes in a low dimensional space. We construct a complex dataset of 20,466 high resolution meshes with extreme facial expressions and encode it using our Convolutional Mesh Autoencoder. Despite limited training data, our method outperforms state-of-the-art PCA models of faces with 50% lower error,  while using 75% fewer parameters. Convolutional neural networks BID27 have achieved state of the art performance in a large number of problems in computer vision BID26 BID22 , natural language processing BID32 and speech processing BID20 . In recent years, CNNs have also emerged as rich models for generating both images BID18 and audio . These successes may be attributed to the multi-scale hierarchical structure of CNNs that allows them to learn translational-invariant localized features. Since the learned filters are shared across the global domain, the number of filter parameters is independent of the domain size. We refer the reader to BID19 for a comprehensive overview of deep learning methods and the recent developments in the field.Despite the recent success, CNNs have mostly been successful in Euclidean domains with gridbased structured data. In particular, most applications of CNNs deal with regular data structures such as images, videos, text and audio, while the generalization of CNNs to irregular structures like graphs and meshes is not trivial. Extending CNNs to graph structures and meshes has only recently drawn significant attention BID8 BID14 . Following the work of BID14 on generalizing the CNNs on graphs using fast Chebyshev filters, we introduce a convolutional mesh autoencoder architecture for realistically representing high-dimensional meshes of 3D human faces and heads.The human face is highly variable in shape as it is affected by many factors such as age, gender, ethnicity etc. The face also deforms significantly with expressions. The existing state of the art 3D face representations mostly use linear transformations BID39 BID29 BID40 or higher-order tensor generalizations BID43 BID9 . While these linear models achieve state of the art results in terms of realistic appearance and Euclidean reconstruction error, we show that CNNs can perform much better at capturing highly non-linear extreme facial expressions with many fewer model parameters.One challenge of training CNNs on 3D facial data is the limited size of current datasets. Here we demonstrate that, since these networks have fewer parameters than traditional linear models, they can be effectively learned with limited data. This reduction in parameters is attributed to the locally invariant convolutional filters that can be shared on the surface of the mesh. Recent work has exploited thousands of 3D scans and 4D scan sequences for learning detailed models of 3D faces BID13 BID46 BID37 BID11 . The availability of this data enables us to a learn rich non-linear representation of 3D face meshes that can not be captured easily by existing linear models.In summary, our work introduces a convolutional mesh autoencoder suitable for 3D mesh processing. Our main contributions are:• We introduce a mesh convolutional autoencoder consisting of mesh downsampling and mesh upsampling layers with fast localized convolutional filters defined on the mesh surface.• We use the mesh autoencoder to accurately represent 3D faces in a low-dimensional latent space performing 50% better than a PCA model that is used in state of the art methods BID39 for face representation.• Our autoencoder uses up to 75% fewer parameters than linear PCA models, while being more accurate on the reconstruction error.• We provide 20,466 frames of highly detailed and complex 3D meshes from 12 different subjects for a range of extreme facial expressions along with our code for research purposes. Our data and code is located at http://withheld.for.review.This work takes a step towards the application of CNNs to problems in graphics involving 3D meshes. Key aspects of such problems are the limited availability of training data and the need for realism. Our work addresses these issues and provides a new tool for 3D mesh modeling. While our convolutional Mesh Autoencoder leads to a representation that generalizes better for unseen 3D faces than PCA with much fewer parameters, our model has several limitations. Our network is restricted to learning face representation for a fixed topology, i.e., all our data samples needs to have the same adjacency matrix, A. The mesh sampling layers are also based on this fixed adjacency matrix A, which defines only the edge connections. The adjacency matrix does not take in to account the vertex positions thus affecting the performance of our sampling operations. In future, we would like to incorporate this information into our learning framework. Mesh Autoencoder PCA FLAME BID29 Table 5 : Quantitative evaluation of Extrapolation experiment. The training set consists of the rest of the expressions. Mean error is of the form [µ ± σ] with mean Euclidean distance µ and standard deviation σ. The median error and number of frames in each expression sequnece is also shown. All errors are in millimeters (mm).The amount of data for high resolution faces is very limited. We believe that generating more of such data with high variability between faces would improve the performance of Mesh Autoencoders for 3D face representations. The data scarcity also limits our ability to learn models that can be trained for superior performance at higher dimensional latent space. The data scarcity also produces noise in some reconstructions. We have introduced a generalization of convolutional autoencoders to mesh surfaces with mesh downsampling and upsampling layers combined with fast localized convolutional filters in spectral space. The locally invariant filters that are shared across the surface of the mesh significantly reduce the number of filter parameters in the network. While the autoencoder is applicable to any class of mesh objects, we evaluated its quality on a dataset of realistic extreme facial expressions. Table 6 : Comparison of FLAME and FLAME++. FLAME++ is obtained by replacing expression model of FLAME with our mesh autoencoder. All errors are in millimeters (mm).convolutional filters capture a lot of surface details that are generally missed in linear models like PCA while using 75% fewer parameters. Our Mesh Autoencoder outperforms the linear PCA model by 50% on interpolation experiments and generalizes better on completely unseen facial expressions.Face models are used in a large number of applications in computer animations, visual avatars and interactions. In recent years, a lot of focus has been given to capturing highly detailed static and dynamic facial expressions. This work introduces a direction in modeling these high dimensional face meshes that can be useful in a range of computer graphics applications.
Convolutional autoencoders generalized to mesh surfaces for encoding and reconstructing extreme 3D facial expressions.
{"src": "scitldr-AIC_train", "paper_id": "SkgYEQ9h4m"}