In this paper, we propose a new interface for virtual reality headset: a touchpad in front of the headset. To demonstrate the feasibility of the front touch interface, we built a prototype device, explored VR UI design space expansion, and performed various user studies. We started with preliminary tests to see how intuitively and accurately people can interact with the front touchpad. Then, we further experimented various user interfaces such as a binary selection, a typical menu layout, and a keyboard. Two-Finger and Drag-n-Tap were also explored to find the appropriate selection technique. As a low-cost, light-weight, and in low power budget technology, a touch sensor can make an ideal interface for mobile headset. Also, front touch area can be large enough to allow wide range of interaction types such as multi-finger interactions. With this novel front touch interface, we paved a way to new virtual reality interaction methods.
Bowman @cite_21 characterized the four universal interaction tasks of virtual environment systems: selection, navigation, manipulation, and system control. Although simple selection would be adequate for static menu layouts, delicate manipulation is needed for complex UI where buttons are smaller and packed.
@cite_21
2126889951
We present a mobile manipulation platform operated by a motor-impaired person using input from a head-tracker, single-button mouse. The platform is used to perform varied and unscripted manipulation tasks in a real home, combining navigation, perception and manipulation. The operator can make use of a wide range of interaction methods and tools, from direct tele-operation of the gripper or mobile base to autonomous sub-modules performing collision-free base navigation or arm motion planning. We describe the complete set of tools that enable the execution of complex tasks, and share the lessons learned from testing them in a real user's home. In the context of grasping, we show how the use of autonomous sub-modules improves performance in complex, cluttered environments, and compare the results to those obtained by novice, able-bodied users operating the same system. Foreword. Preface. I. FOUNDATIONS OF 3D USER INTERFACES . 1. Introduction to 3D User Interfaces. What Are 3D User Interfaces? Why 3D User Interfaces? Terminology. Application Areas. Conclusion. 2. 3D User Interfaces: History and Roadmap. History of 3D UIs. Roadmap to 3D UIs. Scope of This Book. Conclusion. II. Hardware Technologies for 3D User Interfaces. 3. 3D User Interface Output Hardware. Introduction. Visual Displays. Auditory Displays. Haptic Displays. Design Guidelines: Choosing Output Devices for 3D User Interfaces. Conclusion. 4. 3D User Interface Input Hardware. Introduction. Desktop Input Devices. Tracking Devices. 3D Mice. Special-Purpose Input Devices. Direct Human Input. Home-Brewed Input Devices. Choosing Input Devices for 3D Interfaces. III. 3D INTERACTION TECHNIQUES. 5. Selection and Manipulation. Introduction. 3D Manipulation Tasks. Manipulation Techniques and Input Devices. Interaction Techniques for 3D Manipulation. Design Guidelines. 6. Travel. Introduction. 3D Travel Tasks. Travel Techniques. Design Guidelines. 7. Wayfinding. Introduction. Theoretical Foundations. User-Centered Wayfinding Support. Environment-Centered Wayfinding Support. Evaluating Wayfinding Aids. Design Guidelines. Conclusion. 8. System Control. Introduction. Classification. Graphical Menus. Voice Commands. Gestural Commands. Tools. Multimodal System Control Techniques. Design Guidelines. Case Study: Mixing System Control Methods. 8.10. Conclusion. 9. Symbolic Input. Introduction. Symbolic Input Tasks. Symbolic Input Techniques. Design Guidelines. Beyond Text and Number Entry. IV. DESIGNING AND DEVELOPING 3D USER INTERFACES. 10. Strategies for Designing and Developing 3D User Interfaces. Introduction. Designing for Humans. Inventing 3D User Interfaces. Design Guidelines. 11. Evaluation of 3D User Interfaces. Introduction. Background. Evaluation Metrics for 3D Interfaces. Distinctive Characteristics of 3D Interface Evaluation. Classification of 3D Evaluation Methods. Two Multimethod Approaches. Guidelines for 3D Interface Evaluation. V. THE FUTURE OF 3D USER INTERFACES. 12. Beyond Virtual: 3D User Interfaces for the Real World. Introduction. AR Interfaces as 3D Data Browsers. 3D Augmented Reality Interfaces. Augmented Surfaces and Tangible Interfaces. Tangible AR Interfaces. Agents in AR. Transitional AR-VR Interfaces. Conclusion. 13. The Future of 3D User Interfaces. Questions about 3D UI Technology. Questions about 3D Interaction Techniques. Questions about 3D UI Design and Development. Questions about 3D UI Evaluation. Million-Dollar Questions. Appendix A: Quick Reference Guide to 3D User Interface Mathematics. Scalars. Vectors. Points. Matrices. Quaternions. Bibliography. Index. A controlled experiment was performed to evaluate the usability of freehand gestural target selection with different 3D marking menu layouts and target directions. We found that a rectangular layout was faster than an octagonal layout, with no significant increase in errors, and that our right-handed participants preferred to select to the right and forwards in a 3D marking menu. We propose an improved design for 3D marking menus based on our findings. The experimental results also suggest that designers should consider carefully whether or not findings from similar interaction techniques using hands-on devices can be carried over to the design of freehand gestural interaction. This paper presents a set of interaction techniques for hands-free multi-scale navigation through virtual environments. We believe that hands-free navigation, unlike the majority of navigation techniques based on hand motions, has the greatest potential for maximizing the interactivity of virtual environments since navigation modes are offloaded from modal hand gestures to more direct motions of the feet and torso. Not only are the users’ hands freed to perform tasks such as modeling, notetaking and object manipulation, but we also believe that foot and torso movements may inherently be more natural for some navigation tasks. The particular interactions that we developed include a leaning technique for moving small and medium distances, a foot-gesture controlled Step WIM that acts as a floor map for moving larger distances, and a viewing technique that enables a user to view a full 360 degrees in only a three-walled semi-immersive environment by subtly amplifying the mapping between their torso rotation and the virtual world. We formatively designed and evaluated our techniques in existing projects related to archaeological reconstructions, free-form modeling, and interior design. In each case, our informal observations have indicated that motions such as walking and leaning are both appropriate for navigation and are effective in cognitively simplifying complex virtual environment interactions since functionality is more evenly distributed across the body. Presents a novel class of virtual reality input devices that combine pop-through buttons with 6-DOF trackers. Compared to similar devices that use conventional buttons, pop-through devices double the number of potential discrete interaction modes, since each button has two activation states corresponding to light and firm pressure. This additional state per button provides a foundation to address a range of shortcomings with conventional virtual environment (VE) input devices that includes reducing the physical dexterity required to perform interactions, reducing the cognitive complexity of some compound tasks and enabling the design of less obtrusive devices without sacrificing expressive power. Specifically, we present two novel input devices: the FingerSleeve was designed to be minimally obtrusive physically, whereas the TriggerGun was designed to be physically similar to, yet more functional than a conventional hand-held trigger device. Further, we present a set of novel navigation and interaction techniques that leverage the capabilities of our pop-through button devices to improve interaction quality and provide insight into harnessing the potential of pop-through buttons for other tasks. Finally, we discuss how we incorporated one of our devices into a real application. This paper addresses problems on the structural design of large-scale control systems. An efficient and unified framework is proposed to select the minimum number of manipulated measured variables to achieve structural controllability observability of the system, and to select the minimum number of feedback interconnections between measured and manipulated variables such that the closed-loop system has no structural fixed modes. Global solutions are computed using polynomial complexity algorithms in the number of the state variables of the system. Finally, graph-theoretic characterizations are proposed, which allow a characterization of all possible solutions. Unmanned aerial vehicle (UAV) control stations feature multiple menu pages with systems accessed by keyboard presses. Use of speech-based input may enable operators to navigate through menus and select options more quickly. This experiment examined the utility of conventional manual input versus speech input for tasks performed by operators of a UAV control station simulator at two levels of mission difficulty. Pilots performed a continuous flight navigation control task while completing eight different data entry task types with each input modality. Results showed that speech input was significantly better than manual input in terms of task completion time, task accuracy, flight navigation measures, and pilot ratings. Across tasks, data entry time was reduced by approximately 40 with speech input. Additional research is warranted to confirm that this head-up, hands-free control is still beneficial in operational UAV control station auditory environments and does not conflict with intercom operations and... A simple, fast, and approximate voxel-based approach to 6-DOF haptic rendering is presented. It can reliably sustain a 1000 Hz haptic refresh rate without resorting to asynchronous physics and haptic rendering loops. It enables the manipulation of a modestly complex rigid object within an arbitrarily complex environment of static rigid objects. It renders a short-range force field surrounding the static objects, which repels the manipulated object and strives to maintain a voxel-scale minimum separation distance that is known to preclude exact surface interpenetration. Force discontinuities arising from the use of a simple penalty force model are mitigated by a dynamic simulation based on virtual coupling. A generalization of octree improves voxel memory efficiency. In a preliminary implementation, a commercially available 6-DOF haptic prototype device is driven at a constant 1000 Hz haptic refresh rate from one dedicated haptic processor, with a separate processor for graphics. This system yields stable and convincing force feedback for a wide range of user controlled motion inside a large, complex virtual environment, with very few surface interpenetration events. This level of performance appears suited to applications such as certain maintenance and assembly task simulations that can tolerate voxel-scale minimum separation distances. An increase number of gesture sensors and AR(augmented reality) displays are emerging these years. There are more and more study working on spatial interaction. In this work, we focus on evaluating the feasibility of hand-based interaction designed for AR glasses to do basic manipulation on 3D virtual objects in AR applications with current interaction technology. We design a user study with 10 volunteers to discuss whether and how to use natural user interfaces like gestures, with the help of 3D menus, to finish the tasks of manipulating AR scenes. The interface including 3D menus presents the advantages of usability and merits of intuition; However, it also exposes some issues like ergonomic discomfort and limitation like imprecise input. So our results indicate the kind of interactive device and technology are valuable in natural interaction without extra knowledge, though they are limited in serious applications now. Many tasks in human environments require performing a sequence of complex navigation and manipulation tasks. In unstructured human environments, the locations and conguration of the objects can change in unpredictable ways. This requires a high-level planning strategy that is robust and exible in an uncertain environment. We propose a novel dynamic planning strategy, which can be trained from a set of example activities. High level activities are expressed as a sequence of primitive actions or controllers (with appropriate parameters). Our planning model synthesizes a universal strategy, where the a suitable next action is selected based on the current state of the environment. By expressing the environment using sets of attributes, the approach generalizes well to unseen scenarios. By unfolding our planning strategy into a Markov Random Field approximation, we can eectively train parameters using a maximum margin learning strategy. We provide a detailed empirical validation of our overall framework demonstrating successful plan strategies for a variety of tasks. The theoretical ability of modular robots to reconfigure in response to complex tasks in a priori unknown environments has frequently been cited as an advantage and remains a major motivator for work in the field. We present a modular robot system capable of autonomously completing high-level tasks by reactively reconfiguring to meet the needs of a perceived, a priori unknown environment. The system integrates perception, high-level planning, and modular hardware and is validated in three hardware demonstrations. Given a high-level task specification, a modular robot autonomously explores an unknown environment, decides when and how to reconfigure, and manipulates objects to complete its task. The system architecture balances distributed mechanical elements with centralized perception, planning, and control. By providing an example of how a modular robot system can be designed to leverage reactive reconfigurability in unknown environments, we have begun to lay the groundwork for modular self-reconfigurable robots to address tasks in the real world. Autonomous manipulation in unstructured environments will enable a large variety of exciting and important applications. Despite its promise, autonomous manipulation remains largely unsolved. Even the most rudimentary manipulation task--such as removing objects from a pile--remains challenging for robots. We identify three major challenges that must be addressed to enable autonomous manipulation: object segmentation, action selection, and motion generation. These challenges become more pronounced when unknown man-made or natural objects are cluttered together in a pile. We present a system capable of manipulating unknown objects in such an environment. Our robot is tasked with clearing a table by removing objects from a pile and placing them into a bin. To that end, we address the three aforementioned challenges. Our robot perceives the environment with an RGB-D sensor, segmenting the pile into object hypotheses using non-parametric surface models. Our system then computes the affordances of each object, and selects the best affordance and its associated action to execute. Finally, our robot instantiates the proper compliant motion primitive to safely execute the desired action. For efficient and reliable action selection, we developed a framework for supervised learning of manipulation expertise. To verify the performance of our system, we conducted dozens of trials and report on several hours of experiments involving more than 1,500 interactions. The results show that our learning-based approach for pile manipulation outperforms a common sense heuristic as well as a random strategy, and is on par with human action selection. This paper describes and evaluates the design of four virtual controllers for use in rotating three-dimensional objects using the mouse. Three of four of these controllers are "new" in that they extend traditional direct manipulation techniques to a 3-D environment. User performance is compared during simple and complex rotation tasks. The results indicate faster performance for complex rotations using the new continuous axes controllers compared to more traditional slider approaches. No significant differences in accuracy for complex rotations were found across the virtual controllers.A second study compared the best of these four virtual controllers (the Virtual Sphere) to a control device by Evans, Tanner and Wein. No significant differences either in time to complete rotation task or accuracy of performance were found. All but one subject indicated they preferred the Virtual Sphere because it seemed more "natural". Manipulation in immersive virtual environments is difficult partly because users must do without the haptic contact with real objects they rely on in the real world to orient themselves and their manipulanda. To compensate for this lack, we propose exploiting the one real object every user has in a virtual environment, his body. We present a unified framework for virtual-environment interaction based on proprioception, a person's sense of the position and orientation of his body and limbs. We describe three forms of body-relative interaction: • Direct manipulation—ways to use body sense to help control manipulation • Physical mnemonics—ways to store recall information relative to the body • Gestural actions—ways to use body-relative actions to issue commands Automatic scaling is a way to bring objects instantly within reach so that users can manipulate them using proprioceptive cues. Several novel virtual interaction techniques based upon automatic scaling and our proposed framework of proprioception allow a user to interact with a virtual world intuitively, efficiently, precisely, and lazily. We report the results of both informal user trials and formal user studies of the usability of the body-relative interaction techniques presented. CR Developing robots capable of fine manipulation skills is of major importance in order to build truly assistive robots. These robots need to be compliant in their actuation and control in order to operate safely in human environments. Manipulation tasks imply complex contact interactions with the external world, and involve reasoning about the forces and torques to be applied. Planning under contact conditions is usually impractical due to computational complexity, and a lack of precise dynamics models of the environment. We present an approach to acquiring manipulation skills on compliant robots through reinforcement learning. The initial position control policy for manipulation is initialized through kinesthetic demonstration. We augment this policy with a force torque profile to be controlled in combination with the position trajectories. We use the Policy Improvement with Path Integrals (PI2) algorithm to learn these force torque profiles by optimizing a cost function that measures task success. We demonstrate our approach on the Barrett WAM robot arm equipped with a 6-DOF force torque sensor on two different manipulation tasks: opening a door with a lever door handle, and picking up a pen off the table. We show that the learnt force control policies allow successful, robust execution of the tasks. The theoretical ability of modular robots to reconfigure in response to complex tasks in a priori unknown environments has frequently been cited as an advantage, but has never been experimentally demonstrated. For the first time, we present a system that integrates perception, high-level mission planning, and modular robot hardware, allowing a modular robot to autonomously reconfigure in response to an a priori unknown environment in order to complete high-level tasks. Three hardware experiments validate the system, and demonstrate a modular robot autonomously exploring, reconfiguring, and manipulating objects to complete high-level tasks in unknown environments. We present system architecture, software and hardware in a general framework that enables modular robots to solve tasks in unknown environments using autonomous, reactive reconfiguration. The physical robot is composed of modules that support multiple robot configurations. An onboard 3D sensor provides information about the environment and informs exploration, reconfiguration decision making and feedback control. A centralized high-level mission planner uses information from the environment and the user-specified task description to autonomously compose low-level controllers to perform locomotion, reconfiguration, and other behaviors. A novel, centralized self-reconfiguration method is used to change robot configurations as needed. A standard approach for visualizing multivariate networks is to use one or more multidimensional views (for example, scatterplots) for selecting nodes by various metrics, possibly coordinated with a node-link view of the network. In this paper, we present three novel approaches for achieving a tighter integration of these views through hybrid techniques for multidimensional visualization, graph selection and layout. First, we present the FlowVizMenu, a radial menu containing a scatterplot that can be popped up transiently and manipulated with rapid, fluid gestures to select and modify the axes of its scatterplot. Second, the FlowVizMenu can be used to steer an attribute-driven layout of the network, causing certain nodes of a node-link diagram to move toward their corresponding positions in a scatterplot while others can be positioned manually or by force-directed layout. Third, we describe a novel hybrid approach that combines a scatterplot matrix (SPLOM) and parallel coordinates called the Parallel Scatterplot Matrix (P-SPLOM), which can be used to visualize and select features within the network. We also describe a novel arrangement of scatterplots called the Scatterplot Staircase (SPLOS) that requires less space than a traditional scatterplot matrix. Initial user feedback is reported. We consider the problem of manipulating elections by cloning candidates. In our model, a manipulator can replace each candidate c by several clones, i.e., new candidates that are so similar to c that each voter simply replaces c in his vote with a block of these new candidates, ranked consecutively. The outcome of the resulting election may then depend on the number of clones as well as on how each voter orders the clones within the block. We formalize what it means for a cloning manipulation to be successful (which turns out to be a surprisingly delicate issue), and, for a number of common voting rules, characterize the preference profiles for which a successful cloning manipulation exists. We also consider the model where there is a cost associated with producing each clone, and study the complexity of finding a minimum-cost cloning manipulation. Finally, we compare cloning with two related problems: the problem of control by adding candidates and the problem of possible (co)winners when new alternatives can join. In a typical Human-Robot Interaction (HRI) scenario, the robot needs to perform various tasks for the human, hence should take into account human oriented constraints. In this context it is not sufficient that the robot selects grasp and placement of the object from the stability point of view only. Motivated from human behavioral psychology, in this paper we emphasize on the mutually depended nature of grasp and placement selections, which is further constrained by the task, the environment and the human's perspective. We will explore essential human oriented constraints on grasp and placement selections and present a framework to incorporate them in synthesizing key configurations of planning basic interactive manipulation tasks. Today's autonomous robots accomplish only selected tasks in constrained environments. But if robots are to master everyday manipulation, they need general problem-solving capabilities as humans have. We have developed commonsense reasoning methods to make qualitative spatio-temporal inferences about manipulation tasks performed by humans and robots. Hence, robots are enabled to reason about a wider spectrum of tasks, which will considerably deepen their understanding of everyday manipulation.