We present a convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object. Concretely, the network can predict an RGB image and a depth map of the object as seen from an arbitrary view. Several of these depth maps fused together give a full point cloud of the object. The point cloud can in turn be transformed into a surface mesh. The network is trained on renderings of synthetic 3D models of cars and chairs. It successfully deals with objects on cluttered background and generates reasonable predictions for real images of cars.
The multi-view perceptron @cite_23 is a network that takes a face image and a random vector as input and generates a random view of this face together with the corresponding viewpoint. In contrast, our model can generate directly the desired view without the need for random sampling. @cite_29 trained a variant of a variational autoencoder with factored hidden representations, where certain dimensions are constrained to correspond to specific factors of variations in the input data, such as viewpoint and lighting. This method is conceptually interesting and it allows to generate previously unseen views of objects, but the quality of predictions made by our network is significantly better, as we show in the experimental section.
@cite_29 @cite_23
2953255770 2131024102
We present a convolutional network capable of generating images of a previously unseen object from arbitrary viewpoints given a single image of this object. The input to the network is a single image and the desired new viewpoint; the output is a view of the object from this desired viewpoint. The network is trained on renderings of synthetic 3D models. It learns an implicit 3D representation of the object class, which allows it to transfer shape knowledge from training instances to a new object instance. Beside the color image, the network can also generate the depth map of an object from arbitrary viewpoints. This allows us to predict 3D point clouds from a single image, which can be fused into a surface mesh. We experimented with cars and chairs. Even though the network is trained on artificial data, it generalizes well to objects in natural images without any modifications. Various factors, such as identity, view, and illumination, are coupled in face images. Disentangling the identity and view representations is a major challenge in face recognition. Existing face recognition systems either use handcrafted features or learn features discriminatively to improve recognition accuracy. This is different from the behavior of primate brain. Recent studies [5, 19] discovered that primate brain has a face-processing network, where view and identity are processed by different neurons. Taking into account this instinct, this paper proposes a novel deep neural net, named multi-view perceptron (MVP), which can untangle the identity and view features, and in the meanwhile infer a full spectrum of multi-view images, given a single 2D face image. The identity features of MVP achieve superior performance on the MultiPIE dataset. MVP is also capable to interpolate and predict images under viewpoints that are unobserved in the training data.