We present and evaluate an approach for human-in-the-loop specification of shape reconstruction with annotations for basic robot-object interactions. Our method is based on the idea of model annotation: the addition of simple cues to an underlying object model to specify shape and delineate a simple task. The goal is to explore reducing the complexity of CAD-like interfaces so that novice users can quickly recover an object's shape and describe a manipulation task that is then carried out by a robot. The object modeling and interaction annotation capabilities are tested with a user study and compared against results obtained using existing approaches. The approach has been analyzed using a variety of shape comparison, grasping, and manipulation metrics, and tested with the PR2 robot platform, where it was shown to be successful.
Remote control may be accomplished through the use of a wide variety of interfaces @cite_91 . These have changed considerably with the advent of Human-Centered Design (HCD) and have evolved from joystick-and-button black boxes to force-feedback devices @cite_21 , smart gloves @cite_33 , body-suits @cite_132 , verbal instructions @cite_9 , and vision-based analysis of natural body motions @cite_36 @cite_44 @cite_126 @cite_2 . While attractive, an immediate complication in using these methods is obtaining and learning to use the required equipment. In addition, mixed results from immersive teleoperation have been reported, resulting in efforts to reduce the cognitive load of the operators @cite_38 . In @cite_116 a clear trade-off was detected: greater control and situational awareness came at the cost of possible cognitive overload and impaired performance.
@cite_38 @cite_91 @cite_33 @cite_132 @cite_36 @cite_9 @cite_21 @cite_44 @cite_116 @cite_126 @cite_2
2017919391 2588104504 2137940226 2037986392 2168512214 2592582799 2476739747 2725320964 1556798239 2785929605 2170674250
Remote control robots are being found in an increasing number of application domains, including search and rescue, exploration, and reconnaissance. There is a large body of HRI research that investigates interface design for remote navigation, control, and sensor monitoring, while aiming for interface enhancements that benefit the remote operator such as improving ease of use, reducing operator mental load, and maximizing awareness of a robot's state and remote environment. Even though many remote control robots have multi-degree-of-freedom robotic manipulator arms for interacting with the environment, there is only limited research into easy-to-use remote control interfaces for such manipulators, and many commercial robotic products are still using simplistic interface technologies such as keypads or gamepads with arbitrary mappings to arm morphology. In this paper, we present an original interface for the remote control of a multi-degree of freedom robotic arm. We conducted a controlled experiment to compare our interface to an existing commercial keypad interface and detail our results that indicate our interface was easier to use, required less cognitive task load, and enabled people to complete tasks more quickly. In this paper, we present an original interface for the remote control of a multi-degree of freedom robotic arm. We conducted a controlled experiment to compare our interface to an existing commercial keypad interface and detail our results that indicate our interface was easier to use, required less cognitive task load, and enabled people to complete tasks more quickly. Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain---Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operators capabilities and their perception of control authority. Additionally, a custom servo controller design allow for safe interactions of the robotic arm with the environment. We present experimental results demonstrating significant performance improvement using our shared-control assistance framework on adapted rehabilitation benchmarks with two subjects at various timepoints relative to their implantation with intracortical BCIs. Our results indicate that shared assistance mitigates perceived user difficulty in using a seven-degree of freedom robotic arm as a prosthetic and enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with objects previously unknown to the system in densely cluttered environments. Direct use of the hand as an input device is an attractive method for providing natural human-computer interaction (HCI). Currently, the only technology that satisfies the advanced requirements of hand-based input for HCI is glove-based sensing. This technology, however, has several drawbacks including that it hinders the ease and naturalness with which the user can interact with the computer-controlled environment, and it requires long calibration and setup procedures. Computer vision (CV) has the potential to provide more natural, non-contact solutions. As a result, there have been considerable research efforts to use the hand as an input device for HCI. In particular, two types of research directions have emerged. One is based on gesture classification and aims to extract high-level abstract information corresponding to motion patterns or postures of the hand. The second is based on pose estimation systems and aims to capture the real 3D motion of the hand. This paper presents a literature review on the latter research direction, which is a very challenging problem in the context of HCI. Abstract Internet telerobotics has emerged in recent decade with direct control and supervisory control as the main teleoperation paradigms. Both paradigms, however, are difficult to use on applications operating in the unknown and dynamic real world, while they do not provide adequate feeling of interaction or a human-friendly control interface to human operator. This paper proposes a novel interactive control (i.e., active supervisory control) paradigm: telecommanding, which is used for Internet-based wheeled robot teleoperation. Telecommanding involves two parts: basic telecommanding using joystick commands, and advanced telecommanding using linguistic commands. Each joystick or linguistic command is designed to perform an independent task and is defined with multiple events (non-time action references), and the corresponding response functions. This event-driven mechanism enables the robot to deliberately respond to expected events while to reactively respond to unexpected events. Assisted by up-to-date media streaming technologies, telecommanding can help a novice operator to easily control an Internet robot navigating in an unknown and dynamic real world. Experiments, including an Internet-based teleoperation test over 1500 km from Beijing to Hong Kong, demonstrate the promising performance. The ability to control a prosthetic device directly from the neocortex has been demonstrated in rats, monkeys and humans. Here we investigate whether neural control can be accomplished in situations where (1) subjects have not received prior motor training to control the device (na¨ user) and (2) the neural encoding of movement parameters in the cortex is unknown to the prosthetic device (nacontroller). By adopting a decoding strategy that identifies and focuses on units whose firing rate properties are best suited for control, we show that na¨ subjects mutually adapt to learn control of a neural prosthetic system. Six untrained Long-Evans rats, implanted with silicon micro-electrodes in the motor cortex, learned cortical control of an auditory device without prior motor characterization of the recorded neural ensemble. Single- and multi-unit activities were decoded using a Kalman filter to represent an audio 'cursor' (90 ms tone pips ranging from 250 Hz to 16 kHz) which subjects controlled to match a given target frequency. After each trial, a novel adaptive algorithm trained the decoding filter based on correlations of the firing patterns with expected cursor movement. Each behavioral session consisted of 100 trials and began with randomized decoding weights. Within 7 ± 1.4 (mean ± SD) sessions, all subjects were able to significantly score above chance ( P< 0.05, randomization method) in a fixed target paradigm. Training lasted 24 sessions in which both the behavioral performance and signal to noise ratio of the peri-event histograms increased significantly ( P< 0.01, ANOVA). Two rats continued training on a more complex task using a bilateral, two-target control paradigm. Both subjects were able to significantly discriminate the target tones ( P< 0.05, Z-test), while one subject demonstrated control above chance ( P< 0.05, Z-test) after 12 sessions and continued improvement with many sessions achieving over 90 correct targets. Dynamic analysis of binary trial responses indicated that early learning for this subject occurred during session 6. This study demonstrates that subjects can learn to generate neural control signals that are well suited for use with external devices without prior experience or training. Robust remote teleoperation of high-DOF manipulators is of critical importance across a wide range of robotics applications. Contemporary robot manipulation interfaces primarily utilize a free-positioning pose specification approach to independently control each axis of translation and orientation in free space. In this work, we present two novel interfaces, constrained positioning and point-and-click, which incorporate scene information, including points-of-interest and local surface geometry, into the grasp specification process. We also present results of a user study evaluation comparing the effects of increased use of scene information in grasp pose specification algorithms for general object manipulation. The results of our study show that constrained positioning and point-and-click significantly outperform the widely used free positioning approach by significantly reducing the number of grasping errors and the number of user interactions required to specify poses. Furthermore, the point-and-click interface significantly increased the number of tasks users were able to complete. The efficient control of our body and successful interaction with the environment are possible through the integration of multisensory information. Brain–computer interface (BCI) may allow people with sensorimotor disorders to actively interact in the world. In this study, visual information was paired with auditory feedback to improve the BCI control of a humanoid surrogate. Healthy and spinal cord injured (SCI) people were asked to embody a humanoid robot and complete a pick-and-place task by means of a visual evoked potentials BCI system. Participants observed the remote environment from the robot’s perspective through a head mounted display. Human-footsteps and computer-beep sounds were used as synchronous asynchronous auditory feedback. Healthy participants achieved better placing accuracy when listening to human footstep sounds relative to a computer-generated sound. SCI people demonstrated more difficulty in steering the robot during asynchronous auditory feedback conditions. Importantly, subjective reports highlighted that the BCI mask overlaying the display did not limit the observation of the scenario and the feeling of being in control of the robot. Overall, the data seem to suggest that sensorimotor-related information may improve the control of external devices. Further studies are required to understand how the contribution of residual sensory channels could improve the reliability of BCI systems. End-to-end control for robot manipulation and grasping is emerging as an attractive alternative to traditional pipelined approaches. However, end-to-end methods tend to either be slow to train, exhibit little or no generalisability, or lack the ability to accomplish long-horizon or multi-stage tasks. In this paper, we show how two simple techniques can lead to end-to-end (image to velocity) execution of a multi-stage task, which is analogous to a simple tidying routine, without having seen a single real image. This involves locating, reaching for, and grasping a cube, then locating a basket and dropping the cube inside. To achieve this, robot trajectories are computed in a simulator, to collect a series of control velocities which accomplish the task. Then, a CNN is trained to map observed images to velocities, using domain randomisation to enable generalisation to real world images. Results show that we are able to successfully accomplish the task in the real world with the ability to generalise to novel environments, including those with dynamic lighting conditions, distractor objects, and moving objects, including the basket itself. We believe our approach to be simple, highly scalable, and capable of learning long-horizon tasks that have until now not been shown with the state-of-the-art in end-to-end robot control. This dissertation presents a hierarchical controller which can learn to perform complex motor skills. Humans routinely coordinate many degrees of freedom smoothly and effortlessly to achieve complex goals. Moreover, we are good at learning new patterns of coordination to produce new skills. Robots and artificial systems, on the other hand, typically have difficulty with the kinds of behaviors that come most naturally to us. Skills such as running, skiing, playing basketball, or diving involve complex nonlinear dynamics, many degrees of freedom, and behavioral goals that can be difficult to specify mathematically; goals such as “ski down the mountain without falling down” or “shoot a layup” must be translated from linguistic requirements into dynamic system constraints. The focus in this dissertation will be on the skill of platform diving, in which the diver's goal is to execute a certain dive and enter the water in a fully-extended, vertical position. Controlling a simulated diver is a difficult problem for standard control and planning algorithms; conservation of angular momentum gives the system dynamics a nonholonomic constraint with nonlinear drift. In this dissertation, ideas from the fields of biological motor control and learning are combined with new learning algorithms in the design of a hierarchical controller which learns to dive. At the lower level of the control hierarchy, each degree of freedom in the diver's joints is assigned a controller based on biological pattern generators for fast, single-joint movements. These controllers contain neural networks, which are trained on data generated by simulation. The higher level of the control hierarchy incorporates ideas from human skill learning: to achieve a desired behavior pattern, a human learning a new skill uses information from instructors and from watching other performers to build a mental model of the task requirements, and then practices to refine the parameters of this behavioral model. In the high-level controller, each dive is represented as a sequence of multi-joint synergies. The controller learns initial estimates of the timing of these synergies from observational data and then refines these estimates through Q-learning with repeated simulations. Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally prove that it is possible for teleoperation subspaces to effectively and intuitively enable teleoperation. In experiments, novice users completed pick and place tasks significantly faster using teleoperation subspace mapping than they did using state of the art teleoperation methods. Shared compliant control has been incorporated into an advanced six-degree-of-freedom force-reflecting telemanipulation system. With this system the authors have investigated the effect of time delay on human telemanipulation task performance. Time delays of between 2 and 4096 ms were introduced between master and slave arms, and high-precision peg-in-hole tasks were performed by six test operators with two modes of control: kinesthetic force feedback (KFF) and shared compliant control (SCC). Task performance was quantified in terms of the completion time (CT) and the sum of square forces (SOSF). The experimental results demonstrate the superiority of SCC over KFF for time-delayed telemanipulation. SCC has significantly lower rates of increase than KFF in both CT and SOSF with time delay. Only SCC enabled task performance at delays above 1 s. Constant force maintenance tasks were also performed to investigate the effect of time delay on the stability of force reflection. SCC also has beneficial effects on telemanipulation without time delay. >