This paper proposes the idea of using a generative adversarial network (GAN) to assist a novice user in designing real-world shapes with a simple interface. The user edits a voxel grid with a painting interface (like Minecraft). Yet, at any time, he she can execute a SNAP command, which projects the current voxel grid onto a latent shape manifold with a learned projection operator and then generates a similar, but more realistic, shape using a learned generator network. Then the user can edit the resulting shape and snap again until he she is satisfied with the result. The main advantage of this approach is that the projection and generation operators assist novice users to create 3D models characteristic of a background distribution of object shapes, but without having to specify all the details. The core new research idea is to use a GAN to support this application. 3D GANs have previously been used for shape generation, interpolation, and completion, but never for interactive modeling. The new challenge for this application is to learn a projection operator that takes an arbitrary 3D voxel model and produces a latent vector on the shape manifold from which a similar and realistic shape can be generated. We develop algorithms for this and other steps of the SNAP processing pipeline and integrate them into a simple modeling tool. Experiments with these algorithms and tool suggest that GANs provide a promising approach to computer-assisted interactive modeling.
* 1mm Interactive 3D Modeling for Novices: Most interactive modeling tools are designed for experts (e.g., Maya @cite_30 ) and are too difficult to use for casual, novice users. To address this issue, several researchers have proposed simpler interaction techniques for specifying 3D shapes, including ones based on sketching curves @cite_2 , making gestures @cite_28 , or sculpting volumes @cite_7 . However, these interfaces are limited to creating simple objects, since every shape feature of the output must be specified explicitly by the user.
@cite_30 @cite_28 @cite_7 @cite_2
2218614209 1530957309 2030682703 1992207512
Interactively articulating virtual 3D characters lies at the heart of computer animation and geometric modeling. Expressive articulation requires control over many degrees of freedom: most often the joint angles of an internal skeleton. We introduce a physical input device assembled on the fly to control any character's skeleton directly. With traditional mouse and keyboard input, animators must rely on indirect methods such as inverse kinematics or decompose complex and integrated motions into smaller sequential manipulations---for example, iteratively positioning each bone of a skeleton hierarchy. While direct manipulation mouse and touch interfaces are successful in 2D [Shneiderman 1997], 3D interactions with 2D input are ill-posed and thus more challenging. Successful commercial products with 2D interfaces, e.g. Autodesk's Maya, have notoriously steep learning curves and require interface-specific training. While many sophisticated 3D rendering methods become available to produce realistic output, 3D data input still is a tedious and time-consuming task. The paper proposes a method for creating 3D objects using hand gestures. Two types of 3D shape forming tasks are considered. One is to create a completely new object from scratch and the other is to make it by changing an existing one. In the first case, a 3D modeling method is designed and implemented to provide functions like creation, blending and deformation of multiple 3D shapes easily controlled by intuitive bimanual gestures. The implicit superquadric functions are applied to building a gesture-based 3D object modeler with extensions of blending and deformations. In the second case, a 3D data management function is proposed to share and reuse existing shapes created by using the 3D modeler mentioned above. All 3D data are kept small in size and retrieved by picturing partial shapes with the user's hands. As a consequence, the proposed method allows users to consistently deal with all object modeling tasks including creation, storage and reuse of 3D objects through natural and intuitive hand gestures. Some 3D objects and a miniature solid model are shown to present the expressive power and usefulness of the proposed method. We introduce an interactive technique for manipulating simple 3D shapes based on extracting them from a single photograph. Such extraction requires understanding of the components of the shape, their projections, and relations. These simple cognitive tasks for humans are particularly difficult for automatic algorithms. Thus, our approach combines the cognitive abilities of humans with the computational accuracy of the machine to solve this problem. Our technique provides the user the means to quickly create editable 3D parts---human assistance implicitly segments a complex object into its components, and positions them in space. In our interface, three strokes are used to generate a 3D component that snaps to the shape's outline in the photograph, where each stroke defines one dimension of the component. The computer reshapes the component to fit the image of the object in the photograph as well as to satisfy various inferred geometric constraints imposed by its global 3D structure. We show that with this intelligent interactive modeling tool, the daunting task of object extraction is made simple. Once the 3D object has been extracted, it can be quickly edited and placed back into photos or 3D scenes, permitting object-driven photo editing tasks which are impossible to perform in image-space. We show several examples and present a user study illustrating the usefulness of our technique. This paper introduces a new type of interface for 3D drawings that improves the usability of gestural interfaces and augments typical command-based modeling systems. In our suggestive interface, the user gives hints about a desired operation to the system by highlighting related geometric components in the scene. The system then infers possible operations based on the hints and presents the results of these operations as small thumbnails. The user completes the editing operation simply by clicking on the desired thumbnail. The hinting mechanism lets the user specify geometric relations among graphical components in the scene, and the multiple thumbnail suggestions make it possible to define many operations with relatively few distinct hint patterns. The suggestive interface system is implemented as a set of suggestion engines working in parallel, and is easily extended by adding customized engines. Our prototype 3D drawing system, Chateau, shows that a suggestive interface can effectively support construction of various 3D drawings.