We introduce Code Park, a novel tool for visualizing codebases in a 3D game-like environment. Code Park aims to improve a programmer's understanding of an existing codebase in a manner that is both engaging and intuitive, appealing to novice users such as students. It achieves these goals by laying out the codebase in a 3D park-like environment. Each class in the codebase is represented as a 3D room-like structure. Constituent parts of the class (variable, member functions, etc.) are laid out on the walls, resembling a syntax-aware "wallpaper". The users can interact with the codebase using an overview, and a first-person viewer mode. We conducted two user studies to evaluate Code Park's usability and suitability for organizing an existing project. Our results indicate that Code Park is easy to get familiar with and significantly helps in code understanding compared to a traditional IDE. Further, the users unanimously believed that Code Park was a fun tool to work with.
Remembering code structure will result in faster development so it is an essential part of being a programmer. Specifically, 3D environments tap into the spatial memory of the user and help with memorizing the position of objects @cite_2 . These objects could be classes or methods. There are also studies which provide evidence that spatial aptitude is a strong predictor of performance with computer-based user interfaces. For instance, Cockburn and McKenzie @cite_4 have shown that 3D interfaces that leverage the human's spatial memory result in better performance even though some of their subjects believed that 3D interfaces are less efficient. Robertson @cite_7 have also shown that spatial memory does in fact play a role in 3D virtual environments. A number of researchers have attempted to solve the problem of understanding code structure. Graham @cite_40 suggested a solar system metaphor, in which each planet represented a Java class and the orbits showed various inheritance levels. Balzer @cite_21 presented the static structure and the relation of object-oriented programs using 3D blocks in a 2D landscape model.
@cite_4 @cite_7 @cite_21 @cite_40 @cite_2
2130968124 2150804029 1532682605 1970006557
Effective management of documents on computers has been a central user interface problem for many years. One common approach involves using 2D spatial layouts of icons representing the documents, particularly for information workspace tasks. This approach takes advantage of human 2D spatial cognition. More recently, several 3D spatial layouts have engaged 3D spatial cognition capabilities. Some have attempted to use spatial memory in 3D virtual environments. However, there has been no proof to date that spatial memory works the same way in 3D virtual environments as it does in the real world. We describe a new technique for document management called the Data Mountain, which allows users to place documents at arbitrary positions on an inclined plane in a 3D desktop virtual environment using a simple 2D interaction technique. We discuss how the design evolved in response to user feedback. We also describe a user study that shows that the Data Mountain does take advantage of spatial memory. Our study shows that the Data Mountain has statistically reliable advantages over the Microsoft Internet Explorer Favorites mechanism for managing documents of interest in an information workspace. User interfaces can improve task performance by exploiting the powerful human capabilities for spatial cognition. This opportunity has been demonstrated by many prior experiments. It is tempting to believe that providing greater spatial flexibility-by moving from flat 2D to 3D user interfaces-will further enhance user performance. This paper describes an experiment that investigates the effectiveness of spatial memory in real-world physical models and in equivalent computer-based virtual systems. The different models vary the user's freedom to use depth and perspective in spatial arrangements of images representing web pages. Results show that the subjects' performance deteriorated in both the physical and virtual systems as their freedom to locate items in the third dimension increased. Subjective measures reinforce the performance measures, indicating that users found interfaces with higher dimensions more 'cluttered' and less efficient 3D interfaces use motion sensing, physical input, and spatial interaction techniques to effectively control highly dynamic virtual content. Now, with the advent of the Nintendo Wii, Sony Move, and Microsoft Kinect, game developers and researchers must create compelling interface techniques and game-play mechanics that make use of these technologies. At the same time, it is becoming increasingly clear that emerging game technologies are not just going to change the way we play games, they are also going to change the way we make and view art, design new products, analyze scientific datasets, and more. This introduction to 3D spatial interfaces demystifies the workings of modern videogame motion controllers and provides an overview of how it is used to create 3D interfaces for tasks such as 2D and 3D navigation, object selection and manipulation, and gesture-based application control. Topics include the strengths and limitations of various motion-controller sensing technologies in today's peripherals, useful techniques for working with these devices, and current and future applications of these technologies to areas beyond games. The course presents valuable information on how to utilize existing 3D user-interface techniques with emerging technologies, how to develop interface techniques, and how to learn from the successes and failures of spatial interfaces created for a variety of application domains. A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (, 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training. Finding one's way around an environment and remembering the events that occur within it are crucial cognitive abilities that have been linked to the hippocampus and medial temporal lobes. Our review of neuropsychological, behavioral, and neuroimaging studies of human hippocampal involvement in spatial memory concentrates on three important concepts in this field: spatial frameworks, dimensionality, and orientation and self-motion. We also compare variation in hippocampal structure and function across and within species. We discuss how its spatial role relates to its accepted role in episodic memory. Five related studies use virtual reality to examine these two types of memory in ecologically valid situations. While processing of spatial scenes involves the parahippocampus, the right hippocampus appears particularly involved in memory for locations within an environment, with the left hippocampus more involved in context-dependent episodic or autobiographical memory. Modeling instance-level context and object-object relationships is extremely challenging. It requires reasoning about bounding boxes of different classes, locations . Above all, instance-level spatial reasoning inherently requires modeling conditional distributions on previous detections. Unfortunately, our current object detection systems do not have any memory to remember what to condition on! The state-of-the-art object detectors still detect all object in parallel followed by non-maximal suppression (NMS). While memory has been used for tasks such as captioning, they mostly use image-level memory cells without capturing the spatial layout. On the other hand, modeling object-object relationships requires spatial reasoning -- not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns. This paper presents a conceptually simple yet powerful solution -- Spatial Memory Network (SMN), to model the instance-level context efficiently and effectively. Our spatial memory essentially assembles object instances back into a pseudo "image" representation that is easy to be fed into another ConvNet for object-object context reasoning. This leads to a new sequential reasoning architecture where image and memory are processed in parallel to obtain detections which update the memory again. We show our SMN direction is promising as it provides 2.2 improvement over baseline Faster RCNN on the COCO dataset so far. We introduce an interactive technique for manipulating simple 3D shapes based on extracting them from a single photograph. Such extraction requires understanding of the components of the shape, their projections, and relations. These simple cognitive tasks for humans are particularly difficult for automatic algorithms. Thus, our approach combines the cognitive abilities of humans with the computational accuracy of the machine to solve this problem. Our technique provides the user the means to quickly create editable 3D parts---human assistance implicitly segments a complex object into its components, and positions them in space. In our interface, three strokes are used to generate a 3D component that snaps to the shape's outline in the photograph, where each stroke defines one dimension of the component. The computer reshapes the component to fit the image of the object in the photograph as well as to satisfy various inferred geometric constraints imposed by its global 3D structure. We show that with this intelligent interactive modeling tool, the daunting task of object extraction is made simple. Once the 3D object has been extracted, it can be quickly edited and placed back into photos or 3D scenes, permitting object-driven photo editing tasks which are impossible to perform in image-space. We show several examples and present a user study illustrating the usefulness of our technique. Triggering shortcuts or actions on a mobile device often requires a long sequence of key presses. Because the functions of buttons are highly dependent on the current application's context, users are required to look at the display during interaction, even in many mobile situations when eyes-free interactions may be preferable. We present Virtual Shelves, a technique to trigger programmable shortcuts that leverages the user's spatial awareness and kinesthetic memory. With Virtual Shelves, the user triggers shortcuts by orienting a spatially-aware mobile device within the circular hemisphere in front of her. This space is segmented into definable and selectable regions along the phi and theta planes. We show that users can accurately point to 7 regions on the theta and 4 regions on the phi plane using only their kinesthetic memory. Building upon these results, we then evaluate a proof-of-concept prototype of the Virtual Shelves using a Nokia N93. The results show that Virtual Shelves is faster than the N93's native interface for common mobile phone tasks. Inheritance is a crucial part of object-oriented programming, but its use in practice, and the resulting large-scale inheritance structures in programs, remain poorly understood. Previous studies of inheritance have been relatively small and have generally not considered issues such as Java's distinction between classes and interfaces, nor have they considered the use of external libraries. In this paper we present the first substantial empirical study of the large-scale use of inheritance in a contemporary OO programming language. We present a suite of structured metrics for quantifying inheritance in Java programs. We present the results of performing a corpus analysis using those metrics to over 90 applications consisting of over 100,000 separate classes and interfaces. Our analysis finds higher use of inheritance than anticipated, variation in the use of inheritance between interfaces and classes, and differences between inheritance within application types compared with inheritance from external libraries. An increase number of gesture sensors and AR(augmented reality) displays are emerging these years. There are more and more study working on spatial interaction. In this work, we focus on evaluating the feasibility of hand-based interaction designed for AR glasses to do basic manipulation on 3D virtual objects in AR applications with current interaction technology. We design a user study with 10 volunteers to discuss whether and how to use natural user interfaces like gestures, with the help of 3D menus, to finish the tasks of manipulating AR scenes. The interface including 3D menus presents the advantages of usability and merits of intuition; However, it also exposes some issues like ergonomic discomfort and limitation like imprecise input. So our results indicate the kind of interactive device and technology are valuable in natural interaction without extra knowledge, though they are limited in serious applications now. Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating @math convolutions with @math convolutional filters on spatial domain (equivalent to 2D CNN) plus @math convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3 and 1.8 , respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques. Foreword. Preface. I. FOUNDATIONS OF 3D USER INTERFACES . 1. Introduction to 3D User Interfaces. What Are 3D User Interfaces? Why 3D User Interfaces? Terminology. Application Areas. Conclusion. 2. 3D User Interfaces: History and Roadmap. History of 3D UIs. Roadmap to 3D UIs. Scope of This Book. Conclusion. II. Hardware Technologies for 3D User Interfaces. 3. 3D User Interface Output Hardware. Introduction. Visual Displays. Auditory Displays. Haptic Displays. Design Guidelines: Choosing Output Devices for 3D User Interfaces. Conclusion. 4. 3D User Interface Input Hardware. Introduction. Desktop Input Devices. Tracking Devices. 3D Mice. Special-Purpose Input Devices. Direct Human Input. Home-Brewed Input Devices. Choosing Input Devices for 3D Interfaces. III. 3D INTERACTION TECHNIQUES. 5. Selection and Manipulation. Introduction. 3D Manipulation Tasks. Manipulation Techniques and Input Devices. Interaction Techniques for 3D Manipulation. Design Guidelines. 6. Travel. Introduction. 3D Travel Tasks. Travel Techniques. Design Guidelines. 7. Wayfinding. Introduction. Theoretical Foundations. User-Centered Wayfinding Support. Environment-Centered Wayfinding Support. Evaluating Wayfinding Aids. Design Guidelines. Conclusion. 8. System Control. Introduction. Classification. Graphical Menus. Voice Commands. Gestural Commands. Tools. Multimodal System Control Techniques. Design Guidelines. Case Study: Mixing System Control Methods. 8.10. Conclusion. 9. Symbolic Input. Introduction. Symbolic Input Tasks. Symbolic Input Techniques. Design Guidelines. Beyond Text and Number Entry. IV. DESIGNING AND DEVELOPING 3D USER INTERFACES. 10. Strategies for Designing and Developing 3D User Interfaces. Introduction. Designing for Humans. Inventing 3D User Interfaces. Design Guidelines. 11. Evaluation of 3D User Interfaces. Introduction. Background. Evaluation Metrics for 3D Interfaces. Distinctive Characteristics of 3D Interface Evaluation. Classification of 3D Evaluation Methods. Two Multimethod Approaches. Guidelines for 3D Interface Evaluation. V. THE FUTURE OF 3D USER INTERFACES. 12. Beyond Virtual: 3D User Interfaces for the Real World. Introduction. AR Interfaces as 3D Data Browsers. 3D Augmented Reality Interfaces. Augmented Surfaces and Tangible Interfaces. Tangible AR Interfaces. Agents in AR. Transitional AR-VR Interfaces. Conclusion. 13. The Future of 3D User Interfaces. Questions about 3D UI Technology. Questions about 3D Interaction Techniques. Questions about 3D UI Design and Development. Questions about 3D UI Evaluation. Million-Dollar Questions. Appendix A: Quick Reference Guide to 3D User Interface Mathematics. Scalars. Vectors. Points. Matrices. Quaternions. Bibliography. Index. This paper reports empirical results from a study into the useof 2D widgets in 3D immersive virtual environments. Severalresearchers have proposed the use of 2D interaction techniques in3D environments, however little empirical work has been done totest the usability of such approaches. We present the results oftwo experiments conducted on low-level 2D manipulation tasks withinan immersive virtual environment. We empirically show that theaddition of passive-haptic feedback for use in precise UImanipulation tasks can significantly increase user performance.Furthermore, users prefer interfaces that provide a physicalsurface, and that allow them to work with interface widgets in thesame visual field of view as the objects they are modifying. Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating 3 x 3 x 3 convolutions with 1 × 3 × 3 convolutional filters on spatial domain (equivalent to 2D CNN) plus 3 × 1 × 1 convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3 and 1.8 , respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques. Navigating through large-scale virtual environments such as simulations of the astrophysical Universe is difficult. The huge spatial range of astronomical models and the dominance of empty space make it hard for users to travel across cosmological scales effectively, and the problem of wayfinding further impedes the user's ability to acquire reliable spatial knowledge of astronomical contexts. We introduce a new technique called the scalable world-in-miniature (WIM) map as a unifying interface to facilitate travel and wayfinding in a virtual environment spanning gigantic spatial scales: power-law spatial seating enables rapid and accurate transitions among widely separated regions; logarithmically mapped miniature spaces offer a global overview mode when the full context is too large; 3D landmarks represented in the WIM are enhanced by scale, positional, and directional cues to augment spatial context awareness; a series of navigation models are incorporated into the scalable WIM to improve the performance of travel tasks posed by the unique characteristics of virtual cosmic exploration. The scalable WIM user interface supports an improved physical navigation experience and assists pragmatic cognitive understanding of a visualization context that incorporates the features of large-scale astronomy Preface 1. Introduction 2. Some context for Context Analysis: a view of the origins of structural studies of face-to-face interaction 3. Some functions of gaze direction in two-person conversation 4. Movement co-ordination in social interaction 5. Some functions of the face in a kissing round 6. A description of some human greetings 7. Spatial organisation in social encounters: the F-formation system 8. Behavioural foundations for the process of frame-attunement in face-to-face interaction List of films cited References Index. The photorealistic modeling of large-scale objects, such as urban scenes, requires the combination of range sensing technology and digital photography. In this paper, we attack the key problem of camera pose estimation, in an automatic and efficient way. First, the camera orientation is recovered by matching vanishing points (extracted from 2D images) with 3D directions (derived from a 3D range model). Then, a hypothesis-and-test algorithm computes the camera positions with respect to the 3D range model by matching corresponding 2D and 3D linear features. The camera positions are further optimized by minimizing a line-to-line distance. The advantage of our method over earlier work has to do with the fact we do not need to rely on extracted planar facades, or other higher-order features; we are utilizing low- level linear features. That makes this method more general, robust, and efficient. Our method can also be enhanced by the incorporation of traditional structure-from-motion algorithms. We have also developed a user-interface for allowing users to accurately texture-map 2D images onto 3D range models at interactive rates. We have tested our system in a large variety of urban scenes. Virtual environments are synthetic 3D worlds typically viewed from a first-person point of view with many potential applications within areas such as visualisation, entertainment and training simulators. To effectively develop and utilise virtual environments, user-centric evaluations are commonly performed. Anecdotal evidence suggests that factors such as prior experience with computer games may affect the results of such evaluations. This paper examines the effects of previous computer gaming experience, user perceived gaming ability and actual gaming performance on navigation tasks in a virtual environment. Two computer games and a virtual environment were developed to elicit performance metrics for use within a user study. Results indicated that perceived gaming skill and progress in a First-Person-Shooter (FPS) game were the most consistent metrics showing significant correlations with performance in time-based navigation tasks. There was also strong evidence that these relations were significantly intensified by the inclusion of participants who play FPS games. In addition, it was found that increased gaming experience decreased spatial perception performance. This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are "aware" of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions including direct touch, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities. While class-based object-oriented programming languages provide a flexible mechanism for re-using and managing related pieces of code, they typically lack linguistic facilities for specifying a uniform extension of many classes with one set of fields and methods. As a result, programmers are unable to express certain abstractions over classes.In this paper we develop a model of class-to-class functions that we refer to as mixins. A mixin function maps a class to an extended class by adding or overriding fields and methods. Programming with mixins is similar to programming with single inheritance classes, but mixins more directly encourage programming to interfaces.The paper develops these ideas within the context of Java. The results are 1. an intuitive model of an essential Java subset; 2. an extension that explains and models mixins; and 3. type soundness theorems for these languages.