3D shape reconstruction from a single image is a highly ill-posed problem. Modern deep learning based systems try to solve this problem by learning an end-to-end mapping from image to shape via a deep network. In this paper, we aim to solve this problem via an online optimization framework inspired by traditional methods. Our framework employs a deep autoencoder to learn a set of latent codes of 3D object shapes, which are fitted by a probabilistic shape prior using Gaussian Mixture Model (GMM). At inference, the shape and pose are jointly optimized guided by both image cues and deep shape prior without relying on an initialization from any trained deep nets. Surprisingly, our method achieves comparable performance to state-of-the-art methods even without training an end-to-end network, which shows a promising step in this direction.
Following the best practices in deep learning regime, we choose to use point cloud as shape representation, decouple shape and pose by reconstructing 3D shapes in a canonical pose and estimating object pose separately, and a two-step training schedule for shape reconstruction network used TL-embedding network @cite_34 , where a latent space for 3D shapes is learned explicitly via an auto-encoder and then an image regressor that maps an RGB image to the learned latent space is trained. On top of these deep learning components, we use the predicted silhouette and learned shape prior to transform the one-shot prediction from deep networks to an optimization at inference time.
@cite_34
2964137676
What is a good vector representation of an object? We believe that it should be generative in 3D, in the sense that it can produce new 3D objects; as well as be predictable from 2D, in the sense that it can be perceived from 2D images. We propose a novel architecture, called the TL-embedding network, to learn an embedding space with these properties. The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable. This enables tackling a number of tasks including voxel prediction from 2D images and 3D model retrieval. Extensive experimental analysis demonstrates the usefulness and versatility of this embedding.