This paper describes a new approach for training generative adversarial networks (GAN) to understand the detailed 3D shape of objects. While GANs have been used in this domain previously, they are notoriously hard to train, especially for the complex joint data distribution over 3D objects of many categories and orientations. Our method extends previous work by employing the Wasserstein distance normalized with gradient penalization as a training objective. This enables improved generation from the joint object shape distribution. Our system can also reconstruct 3D shape from 2D images and perform shape completion from occluded 2.5D range scans. We achieve notable quantitative improvements in comparison to existing baselines
The pioneering work of Roberts @cite_19 and Marr @cite_18 exemplify early shape models composed of 3D geometric primitives with recognizable 2D properties. Similarly intuitive properties are still in common use, such as symmetric parts @cite_21 , skeletons @cite_0 and CAD wire-frames @cite_22 .
@cite_18 @cite_22 @cite_21 @cite_0 @cite_19
2136393262 1971254313 2107198582 2116735535 2016316738 1491968272 2114111978 2071657752 2070568264 2963369474 2612843093 1901654600 2088711760 2742468459 2604493845 2891377836 2113877705 2622333333 2557269700 2949896890
In this paper, we present a two-level generative model for representing the images and surface depth maps of drapery and clothes. The upper level consists of a number of folds which will generate the high contrast (ridge) areas with a dictionary of shading primitives (for 2D images) and fold primitives (for 3D depth maps). These primitives are represented in parametric forms and are learned in a supervised learning phase using 3D surfaces of clothes acquired through photometric stereo. The lower level consists of the remaining flat areas which fill between the folds with a smoothness prior (Markov random field). We show that the classical ill-posed problem-shape from shading (SFS) can be much improved by this two-level model for its reduced dimensionality and incorporation of middle-level visual knowledge, i.e., the dictionary of primitives. Given an input image, we first infer the folds and compute a sketch graph using a sketch pursuit algorithm as in the primal sketch (, 2003). The 3D folds are estimated by parameter fitting using the fold dictionary and they form the "skeleton" of the drapery cloth surfaces. Then, the lower level is computed by conventional SFS method using the fold areas as boundary conditions. The two levels interact at the final stage by optimizing a joint Bayesian posterior probability on the depth map. We show a number of experiments which demonstrate more robust results in comparison with state-of-the-art work. In a broader scope, our representation can be viewed as a two-level inhomogeneous MRF model which is applicable to general shape-from-X problems. Our study is an attempt to revisit Marr's idea (Marr and Freeman, 1982) of computing the 2frac12D sketch from primal sketch. In a companion paper (Barbu and Zhu, 2005), we study shape from stereo based on a similar two-level generative sketch representation. A novel self-adaptive geometric primitive for functional geometric shape synthesis is presented. This novel geometric primitive, for CAD use, is specifically designed to reproduce geometric shapes with functional requirements, such as the aerodynamic and hydrodynamic ones, once the functional parameters are furnished. It produces a typical CAD representation of a functional profile: a set of Bezier curves. The proposed primitive follows a generate-and-test approach and takes advantage of the use of a properly designed artificial neural network (BNN). It combines the properties of a geometric primitive and the capability to manage the engineering knowledge in a specific field of application. The proposed evolutionary primitive is applied to a real engineering application: the automatic synthesis of airfoils. Some examples are simulated in order to test the effectiveness of the proposed method. The results obtained by an original prototypal software are presented and critically discussed. We propose models of 3D shape which may be viewed as deformable bodies composed of simulated elastic material. In contrast to traditional, purely geometric models of shape, deformable models are active—their shapes change in response to externally applied forces. We develop a deformable model for 3D shape which has a preference for axial symmetry. Symmetry is represented even though the model does not belong to a parametric shape family such as (generalized) cylinders. Rather, a symmetry-seeking property is designed into internal forces that constrain the deformations of the model. We develop a framework for 3D object reconstruction based on symmetry-seeking models. Instances of these models are formed from monocular image data through the action of external forces derived from the data. The forces proposed in this paper deform the model in space so that the shape of its projection into the image plane is consistent with the 2D silhouette of an object of interest. The effectiveness of our approach is demonstrated using natural images. We present an image-based approach to infer 3D structure parameters using a probabilistic "shape+structure" model. The 3D shape of an object class is represented by sets of contours from silhouette views simultaneously observed from multiple calibrated cameras, while structural features of interest on the object are denoted by a number of 3D locations. A prior density over the multiview shape and corresponding structure is constructed with a mixture of probabilistic principal components analyzers. Given a novel set of contours, we infer the unknown structure parameters from the new shape's Bayesian reconstruction. Model matching and parameter inference are done entirely in the image domain and require no explicit 3D construction. Our shape model enables accurate estimation of structure despite segmentation errors or missing views in the input silhouettes, and it works even with only a single input view. Using a training set of thousands of pedestrian images generated from a synthetic model, we can accurately infer the 3D locations of 19 joints on the body based on observed silhouette contours from real images. A large amount of 2D drawings have been produced in engineering fields. To reuse and share the available drawings efficiently, we propose two methods in this paper, namely 2.5D spherical harmonics transformation and 2D shape histogram, to retrieve 2D drawings by measuring their shape similarity. The first approach represents a drawing as a spherical function by transforming it from a 2D space into a 3D space. Then a fast spherical harmonics transformation is employed to get a rotation invariant descriptor. The second statistics-based approach represents the shape of a 2D drawing using a distance distribution between two randomly sampled points. To allow users to interactively emphasize certain local shapes that they are interested in, we have adopted a flexible sampling strategy by specifying a bias sampling density upon these local shapes. The two proposed methods have many valuable properties, including transform invariance, efficiency, and robustness. In addition, their insensitivity to noise allows for the user's causal input, thus supporting a freehand sketch-based retrieval user interface. Experiments show that a better performance can be achieved by combining them together using weights. This paper explores a method, first proposed by Wei and Chai [1], for estimating 3D human pose from several frames of uncalibrated 2D point correspondences containing projected body joint locations. In their work Wei and Chai boldly claimed that, through the introduction of rigid constraints to the torso and hip, camera scales, bone lengths and absolute depths could be estimated from a finite number of frames (i.e. ≥ 5). In this paper we show this claim to be false, demonstrating in principle one can never estimate these parameters in a finite number of frames. Further, we demonstrate their approach is only valid for rigid sub-structures of the body (e.g. torso). Based on this analysis we propose a novel approach using deterministic structure from motion based on assumptions of rigidity in the body's torso. Our approach provides notably more accurate estimates and is substantially faster than Wei and Chai's approach, and unlike the original, can be solved as a deterministic least-squares problem. Geometric 3D reasoning at the level of objects has received renewed attention recently in the context of visual scene understanding. The level of geometric detail, however, is typically limited to qualitative representations or coarse boxes. This is linked to the fact that today's object class detectors are tuned toward robust 2D matching rather than accurate 3D geometry, encouraged by bounding-box-based benchmarks such as Pascal VOC. In this paper, we revisit ideas from the early days of computer vision, namely, detailed, 3D geometric object class representations for recognition. These representations can recover geometrically far more accurate object hypotheses than just bounding boxes, including continuous estimates of object pose and 3D wireframes with relative 3D positions of object parts. In combination with robust techniques for shape description and inference, we outperform state-of-the-art results in monocular 3D pose estimation. In a series of experiments, we analyze our approach in detail and demonstrate novel applications enabled by such an object class representation, such as fine-grained categorization of cars and bicycles, according to their 3D geometry, and ultrawide baseline matching. Skeletons provide a synthetic and thin representation of objects. Therefore, they are useful for shape description. Recent papers have proposed to approximate the skeleton of continuous shapes using the Voronoi graph of boundary points. An original formulation is presented here, using the notion of polyballs (we call polyball any finite union of balls). A preliminary work shows that their skeletons consist of simple components (line segments in 2D and polygons in 3D). An efficient method for simplifying 3D continuous skeletons is also presented. The originality of our approach consists in simplifying the shape without modifying its topology and in including these modifications on the skeleton. Depending on the desired result, we propose two strategies which lead to either surfacical skeletons or wireframe skeletons. Two angular criteria are proposed that allow us to build a size-invariant hierarchy of simplified skeletons. We present a new approach to the problem of modeling smoothly deformable shapes with convex polyhedral bounds. Our hyperquadric modeling primitives, which include superquadrics as a special case, can be viewed as hyperplanar slices of deformed hyperspheres. As the original hypersphere is deformed to its bounding hypercube, the slices undergo corresponding smooth deformations to convex polytopes. The possible shape classes include arbitrary convex polygons and polyhedra, as well as taperings and distortions that are not naturally included within the conventional superquadric framework. By generalizing Blinn's “blobby” approach to modeling complex objects, we construct single equations for nonconvex, composite shapes starting with our basic convex primitives. Hyperquadrics are of potential interest for the generation of synthetic images, for automated image interpretation and for psychological models of geometric shape representation, manipulation, and perception. The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data. Inspired by the nature of human perception of 3D shapes as a collection of simple parts, we explore such an abstract shape representation based on primitives. Given a single depth image of an object, we present 3DPRNN, a generative recurrent neural network that synthesizes multiple plausible shapes composed of a set of primitives. Our generative model encodes symmetry characteristics of common man-made objects, preserves long-range structural coherence, and describes objects of varying complexity with a compact representation. We also propose a method based on Gaussian Fields to generate a large scale dataset of primitive-based shape representations to train our network. We evaluate our approach on a wide range of examples and show that it outperforms nearest-neighbor based shape retrieval methods and is on-par with voxelbased generative models while using a significantly reduced parameter space. We introduce a novel neural network architecture for encoding and synthesis of 3D shapes, particularly their structures. Our key insight is that 3D shapes are effectively characterized by their hierarchical organization of parts, which reflects fundamental intra-shape relationships such as adjacency and symmetry. We develop a recursive neural net (RvNN) based autoencoder to map a flat, unlabeled, arbitrary part layout to a compact code. The code effectively captures hierarchical structures of man-made 3D objects of varying structural complexities despite being fixed-dimensional: an associated decoder maps a code back to a full hierarchy. The learned bidirectional mapping is further tuned using an adversarial setup to yield a generative model of plausible structures, from which novel structures can be sampled. Finally, our structure synthesis framework is augmented by a second trained module that produces fine-grained part geometry, conditioned on global and local structural context, leading to a full generative pipeline for 3D shapes. We demonstrate that without supervision, our network learns meaningful structural hierarchies adhering to perceptual grouping principles, produces compact codes which enable applications such as shape classification and partial matching, and supports shape synthesis and interpolation with significant variations in topology and geometry. We propose a new 3D model of the human body that is both realistic and part-based. The body is represented by a graphical model in which nodes of the graph correspond to body parts that can independently translate and rotate in 3D and deform to represent different body shapes and to capture pose-dependent shape variations. Pairwise potentials define a “stitching cost” for pulling the limbs apart, giving rise to the stitched puppet (SP) model. Unlike existing realistic 3D body models, the distributed representation facilitates inference by allowing the model to more effectively explore the space of poses, much like existing 2D pictorial structures models. We infer pose and body shape using a form of particle-based max-product belief propagation. This gives SP the realism of recent 3D body models with the computational advantages of part-based models. We apply SP to two challenging problems involving estimating human shape and pose from 3D data. The first is the FAUST mesh alignment challenge, where ours is the first method to successfully align all 3D meshes with no pose prior. The second involves estimating pose and shape from crude visual hull representations of complex body movements. This paper proposes a novel part-based representation for modeling object categories. Our representation combines the effectiveness of deformable part-based models with the richness of geometric representation by defining parts based on consistent underlying 3D geometry. Our key hypothesis is that while the appearance and the arrangement of parts might vary across the instances of object categories, the constituent parts will still have consistent underlying 3D geometry. We propose to learn this geometry-driven deformable part-based model (gDPM) from a set of labeled RGBD images. We also demonstrate how the geometric representation of gDPM can help us leverage depth data during training and constrain the latent model learning problem. But most importantly, a joint geometric and appearance based representation not only allows us to achieve state-of-the-art results on object detection but also allows us to tackle the grand challenge of understanding 3D objects from 2D images. The success of various applications including robotics, digital content creation, and visualization demand a structured and abstract representation of the 3D world from limited sensor data. Inspired by the nature of human perception of 3D shapes as a collection of simple parts, we explore such an abstract shape representation based on primitives. Given a single depth image of an object, we present 3D-PRNN, a generative recurrent neural network that synthesizes multiple plausible shapes composed of a set of primitives. Our generative model encodes symmetry characteristics of common man-made objects, preserves long-range structural coherence, and describes objects of varying complexity with a compact representation. We also propose a method based on Gaussian Fields to generate a large scale dataset of primitive-based shape representations to train our network. We evaluate our approach on a wide range of examples and show that it outperforms nearest-neighbor based shape retrieval methods and is on-par with voxel-based generative models while using a significantly reduced parameter space. 3D shape models are naturally parameterized using vertices and faces, i.e., composed of polygons forming a surface. However, current 3D learning paradigms for predictive and generative tasks using convolutional neural networks focus on a voxelized representation of the object. Lifting convolution operators from the traditional 2D to 3D results in high computational overhead with little additional benefit as most of the geometry information is contained on the surface boundary. Here we study the problem of directly generating the 3D shape surface of rigid and non-rigid shapes using deep convolutional neural networks. We develop a procedure to create consistent &#x2018;geometry images&#x2019; representing the shape surface of a category of 3D objects. We then use this consistent representation for category-specific shape surface generation from a parametric representation or an image by developing novel extensions of deep residual networks for the task of geometry image generation. Our experiments indicate that our network learns a meaningful representation of shape surfaces allowing it to interpolate between shape orientations and poses, invent new shape surfaces and reconstruct 3D shape surfaces from previously unseen images. Our code is available at https: github.com sinhayan surfnet. We present a feed-forward, multitask, end-to-end trainable system for the integrated 2d localization, as well as 3d pose and shape estimation, of multiple people in monocular images. The challenge is the formal modeling of the problem that intrinsically requires discrete and continuous computation (e.g. grouping people vs. predicting 3d pose). The model identifies human body structures (joints and limbs) in images, groups them based on 2d and 3d information fused using learned scoring functions, and optimally aggregates such responses into partial or complete 3d human skeleton hypotheses under kinematic tree constraints, but without knowing in advance the number of people in the scene and their visibility relations. We design a single multi-task deep neural network with differentiable stages where the person grouping problem is formulated as an integer program based on learned body part scores parameterized by both 2d and 3d information. This avoids suboptimality resulting from separate 2d and 3d reasoning, with grouping performed based on the combined information. The calculation can be formally described as a linear binary integer program with globally optimal solution. The final predictive stage of 3d pose and shape is based on a learned attention process where information from different human body parts is optimally fused. State-of-the-art results are obtained in large scale datasets like Human3.6M and Panoptic. A physically-based approach is presented to fitting complex 3D shapes using a novel class of dynamic models. These models can deform both locally and globally. The authors formulate deformable superquadrics which incorporate the global shape parameters of a conventional superellipsoid with the local degrees of freedom of a spline. The local global representational power of a deformable superquadric simultaneously satisfies the conflicting requirements of shape reconstruction and shape recognition. The model's six global deformational degrees of freedom capture gross shape features from visual data and provide salient part descriptors for efficient indexing into a database of stored models. Model fitting experiments involving 2D monocular image data and 3D range data are reported. > Abstract Skeletons are simplified shape representations with many applications involving image processing, analysis, and visualization. A fundamental problem in shape skeletonization is the sensitivity of the skeletons to small perturbations of the input shape, which leads to the appearance of spurious branches. By assigning an importance metric to every skeleton point, one encodes the scale of the shape details. Subsequently, simplified skeletons can be obtained by simply thresholding at continuous values of that metric. Such a multiscale regularization procedure should ideally produce one-spel-wide skeletons in all scales. In this chapter, we present a new method based on the image foresting transform framework, which achieves this result for medial axes of 2D shapes and for surface skeletons of 3D shapes. Our approach relies on simple and efficient algorithms, faster than several methods based on the same importance metric, simpler than others, far less sensitive to numerical noise than a recent one, and attains similar quality of centeredness, smoothness, thinness, and ease to simplify the skeleton. Such a conclusion is substantiated with a comparative analysis on a wide set of 2D and 3D real-word shapes against its multiscale counterparts. We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives. In addition to generating simple and geometrically interpretable explanations of 3D objects, our framework also allows us to automatically discover and exploit consistent structure in the data. We demonstrate that using our method allows predicting shape representations which can be leveraged for obtaining a consistent parsing across the instances of a shape collection and constructing an interpretable shape similarity measure. We also examine applications for image-based prediction as well as shape manipulation. We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives. In addition to generating simple and geometrically interpretable explanations of 3D objects, our framework also allows us to automatically discover and exploit consistent structure in the data. We demonstrate that using our method allows predicting shape representations which can be leveraged for obtaining a consistent parsing across the instances of a shape collection and constructing an interpretable shape similarity measure. We also examine applications for image-based prediction as well as shape manipulation.