We present a convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object. Concretely, the network can predict an RGB image and a depth map of the object as seen from an arbitrary view. Several of these depth maps fused together give a full point cloud of the object. The point cloud can in turn be transformed into a surface mesh. The network is trained on renderings of synthetic 3D models of cars and chairs. It successfully deals with objects on cluttered background and generates reasonable predictions for real images of cars.
@cite_27 proposed an approach for aligning 3D models of objects with images of these objects. The method makes use of discriminative part detectors and works on complicated real scenes. On the downside, this is a nearest-neighbor kind of method: it selects the best fitting 3D models from a fixed set of models. This limits the generalization capability of the method and makes it proportionally slower if the model collection grows in size.
@cite_27
2154880882
We present a new algorithm 3DNN (3D Nearest-Neighbor), which is capable of matching an image with 3D data, independently of the viewpoint from which the image was captured. By leveraging rich annotations associated with each image, our algorithm can automatically produce precise and detailed 3D models of a scene from a single image. Moreover, we can transfer information across images to accurately label and segment objects in a scene. The true benefit of 3DNN compared to a traditional 2D nearest-neighbor approach is that by generalizing across viewpoints, we free ourselves from the need to have training examples captured from all possible viewpoints. Thus, we are able to achieve comparable results using orders of magnitude less data, and recognize objects from never-before-seen viewpoints. In this work, we describe the 3DNN algorithm and rigorously evaluate its performance for the tasks of geometry estimation and object detection segmentation. By decoupling the viewpoint and the geometry of an image, we develop a scene matching approach which is truly 100 viewpoint invariant, yielding state-of-the-art performance on challenging data.