We present and evaluate an approach for human-in-the-loop specification of shape reconstruction with annotations for basic robot-object interactions. Our method is based on the idea of model annotation: the addition of simple cues to an underlying object model to specify shape and delineate a simple task. The goal is to explore reducing the complexity of CAD-like interfaces so that novice users can quickly recover an object's shape and describe a manipulation task that is then carried out by a robot. The object modeling and interaction annotation capabilities are tested with a user study and compared against results obtained using existing approaches. The approach has been analyzed using a variety of shape comparison, grasping, and manipulation metrics, and tested with the PR2 robot platform, where it was shown to be successful.
A third approach is , where an operator indicates appropriate motions by physically guiding the robot while it records the event @cite_125 . Alternatively, the robot may be guided virtually, aided with a variety of feedback options @cite_102 @cite_10 .
@cite_10 @cite_102 @cite_125
1983885373 2344260694 1992461594 2072084471 757786120 2171066085 2766209129 2106338164 2161395589 2403419699 638212880 2166974693 1581800915 1991667713 2158662146 2056600097 2281874712 2001709653 1496308346 2330082268
The approach of inferring user's intended task and optimizing low-level robot motions has promise for making robot teleoperation interfaces more intuitive and responsive. But most existing methods assume a finite set of candidate tasks, which limits a robot's functionality. This paper proposes the notion of freeform tasks that encode an infinite number of possible goals (e.g., desired target positions) within a finite set of types (e.g., reach, orient, pick up). It also presents two technical contributions to help make freeform UIs possible. First, an intent predictor estimates the user's desired task, and accepts freeform tasks that include both discrete types and continuous parameters. Second, a cooperative motion planner continuously updates the robot's trajectories to achieve the inferred tasks by repeatedly solving optimal control problems. The planner is designed to respond interactively to changes in the indicated task, avoid collisions in cluttered environments, handle time-varying objective functions, and achieve high-quality motions using a hybrid of numerical and sampling-based techniques. The system is applied to the problem of controlling a 6D robot manipulator using 2D mouse input in the context of two tasks: static target reaching and dynamic trajectory tracking. Simulations suggest that it enables the robot to reach intended targets faster and to track intended trajectories more closely than comparable techniques. In this paper, we address the problem of simultaneously achieving low trajectory tracking errors and compliant control without using explicit mathematical models of task dynamics. To achieve this goal, we propose a new movement representation called compliant movement primitives (CMPs), which encodes position trajectory and associated torque profiles and can be learned from a single user demonstration. With the proposed control framework, the robot can remain compliant and consequently safe for humans sharing its workspace, even if high trajectory tracking accuracy is required. We developed a statistical learning approach that can use a database of existing CMPs and compute new ones, adapted for novel task variations. The proposed approach was evaluated on a Kuka LWR-4 robot performing 1) a discrete pick-and-place task with objects of varying weight and 2) a periodic handle turning operation. The evaluation of the discrete task showed a 15-fold decrease of the tracking error while exhibiting compliant behavior compared to the standard feedback control approach. It also indicated no significant rise in the tracking error while using generalized primitives computed by the statistical learning method. With respect to unforeseen collisions, the proposed approach resulted in a 75 drop of contact forces compared to standard feedback control. The periodic task demonstrated on-line use of the proposed approach to accomplish a task of handle turning. In this paper we present a framework that allows a human and a robot to perform simultaneous manipulation tasks safely in close proximity. The proposed framework is based on early prediction of the human's motion. The prediction system, which builds on previous work in the area of gesture recognition, generates a prediction of human workspace occupancy by computing the swept volume of learned human motion trajectories. The motion planner then plans robot trajectories that minimize a penetration cost in the human workspace occupancy while interleaving planning and execution. Multiple plans are computed in parallel, one for each robot task available at the current time, and the trajectory with the least cost is selected for execution. We test our framework in simulation using recorded human motions and a simulated PR2 robot. Our results show that our framework enables the robot to avoid the human while still accomplishing the robot's task, even in cases where the initial prediction of the human's motion is incorrect. We also show that taking into account the predicted human workspace occupancy in the robot's motion planner leads to safer and more efficient interactions between the user and the robot than only considering the human's current configuration. For robots to work effectively with humans, they must learn and recognize activities that humans perform. We enable a robot to learn a library of activities from user demonstrations and use it to recognize an action performed by an operator in real time. Our contributions are threefold: (1) a novel probabilistic flow tube representation that can intuitively capture a wide range of motions and can be used to support compliant execution; (2) a method to identify the relevant features of a motion, and ensure that the learned representation preserves these features in new and unforeseen situations; (3) a fast incremental algorithm for recognizing user-performed motions using this representation. Our approach provides several capabilities beyond those of existing algorithms. First, we leverage temporal information to model motions that may exhibit non-Markovian characteristics. Second, our approach can identify parameters of a motion not explicitly specified by the user. Third, we model hybrid continuous and discrete motions in a unified representation that avoids abstracting out the continuous details of the data. Experimental results show a 49 improvement over prior art in recognition rate for varying environments, and a 24 improvement for a static environment, while maintaining average computing times for incremental recognition of less than half of human reaction time. We also demonstrate motion learning and recognition capabilities on real-world robot platforms. We propose a planning method for grasping in cluttered environments, where the robot can make simultaneous contact with multiple objects, manipulating them in a deliberate and controlled fashion. This enables the robot to reach for and grasp the target while simultaneously contacting and moving aside obstacles in order to clear a desired path. We use a physicsbased analysis of pushing to compute the motion of each object in the scene in response to a set of possible robot motions. In order to make the problem computationally tractable, we enable multiple simultaneous robot-object interactions, which we pre-compute and cache, but avoid object-object interactions. Tests on large sets of simulated scenes show that our planner produces more successful grasps in more complex scenes than versions that avoid any interaction with surrounding clutter. Validation on a real robot shows that our grasp evaluation method accurately predicts the outcome of a grasp, and that our approach, in conjunction with state-of-the-art object recognition tools, is applicable in reallife scenes that are highly cluttered and constrained. Using visual feedback to control the movement of the end-effector is a common approach for robust execution of robot movements in real-world scenarios. Over the years several visual servoing algorithms have been developed and implemented for various types of robot hardware. In this paper, we present a hybrid approach which combines visual estimations with kinematically determined orientations to control the movement of a humanoid arm. The approach has been evaluated with the humanoid robot ARMAR III using the stereo system of the active head for perception as well as the torso and arms equipped with five finger hands for actuation. We show how a robust visual perception is used to control complex robots without any hand-eye calibration. Furthermore, the robustness of the system is improved by estimating the hand position in case of failed visual hand tracking due to lightning artifacts or occlusions. The proposed control scheme is based on the fusion of the sensor channels for visual perception, force measurement and motor encoder data. The combination of these different data sources results in a reactive, visually guided control that allows the robot ARMAR-III to execute grasping tasks in a real-world scenario. Abstract Time-optimal motion planning is one of the most essential problems in robotic theories. The presented direct transcription approach uses convex optimisation method to obtain a minimum-time control for robot manipulators. One subset of motion planning is the waiter motion problem that consists of moving a tablet from one place to another as fast as possible, avoiding the slip of the object that is placed upon it. In this paper the direct transcription method is presented in order to solve the waiter problem. This path tracking problem is also approximated by sequential two dimensional second-order cone programs (SOCP) using nonlinear change of variables. The computational benefit of the sequential method compared to the direct transcription approach is presented. The waiter motion problem is validated by real-life experimental results with a 6-DoF robotic arm. In this paper, we present an approach that allows a robot to observe, generalize, and reproduce tasks observed from multiple demonstrations. Motion capture data is recorded in which a human instructor manipulates a set of objects. In our approach, we learn relations between body parts of the demonstrator and objects in the scene. These relations result in a generalized task description. The problem of learning and reproducing human actions is formulated using a dynamic Bayesian network (DBN). The posteriors corresponding to the nodes of the DBN are estimated by observing objects in the scene and body parts of the demonstrator. To reproduce a task, we seek for the maximum-likelihood action sequence according to the DBN. We additionally show how further constraints can be incorporated online, for example, to robustly deal with unforeseen obstacles. Experiments carried out with a real 6-DoF robotic manipulator as well as in simulation show that our approach enables a robot to reproduce a task carried out by a human demonstrator. Our approach yields a high degree of generalization illustrated by performing a pick-and-place and a whiteboard cleaning task. We provide a general approach for learning robotic motor skills from human demonstration. To represent an observed movement, a non-linear differential equation is learned such that it reproduces this movement. Based on this representation, we build a library of movements by labeling each recorded movement according to task and context (e.g., grasping, placing, and releasing). Our differential equation is formulated such that generalization can be achieved simply by adapting a start and a goal parameter in the equation to the desired position values of a movement. For object manipulation, we present how our framework extends to the control of gripper orientation and finger position. The feasibility of our approach is demonstrated in simulation as well as on the Sarcos dextrous robot arm. The robot learned a pick-and-place operation and a water-serving task and could generalize these tasks to novel situations. To facilitate interaction with people, robots must not only recognize current actions, but also infer a person's intentions and future behavior. Recent advances in depth camera technology have significantly improved human motion tracking. However, the inherent high dimensionality of interacting with the physical world makes efficiently forecasting human intention and future behavior a challenging task. Predictive methods that estimate uncertainty are therefore critical for supporting appropriate robotic responses to the many ambiguities posed within the human-robot interaction setting. We address these two challenges, high dimensionality and uncertainty, by employing predictive inverse optimal control methods to estimate a probabilistic model of human motion trajectories. Our inverse optimal control formulation estimates quadratic cost functions that best rationalize observed trajectories framed as solutions to linear-quadratic regularization problems. The formulation calibrates its uncertainty from observed motion trajectories, and is efficient in high-dimensional state spaces with linear dynamics. We demonstrate its effectiveness on a task of anticipating the future trajectories, target locations and activity intentions of hand motions. Directing robot attention to recognise activities and to anticipate events like goal-directed actions is a crucial skill for human-robot interaction. Unfortunately, issues like intrinsic time constraints, the spatially distributed nature of the entailed information sources, and the existence of a multitude of unobservable states affecting the system, like latent intentions, have long rendered achievement of such skills a rather elusive goal. The problem tests the limits of current attention control systems. It requires an integrated solution for tracking, exploration and recognition, which traditionally have been seen as separate problems in active vision. We propose a probabilistic generative framework based on information gain maximisation and a mixture of Kalman Filters that uses predictions in both recognition and attention-control. This framework can efficiently use the observations of one element in a dynamic environment to provide information on other elements, and consequently enables guided exploration. Interestingly, the sensors control policy, directly derived from first principles, represents the intuitive trade-off between finding the most discriminative clues and maintaining overall awareness. Experiments on a simulated humanoid robot observing a human executing goal-oriented actions demonstrated improvement on recognition time and precision over baseline systems. A gesture-based interaction framework is presented for controlling mobile robots. This natural interaction paradigm has few physical requirements, and thus can be deployed in many restrictive and challenging environments. We present an implementation of this scheme in the control of an underwater robot by an on-site human operator. The operator performs discrete gestures using engineered visual targets, which are interpreted by the robot as parametrized actionable commands. By combining the symbolic alphabets resulting from several visual cues, a large vocabulary of statements can be produced. An iterative closest point algorithm is used to detect these observed motions, by comparing them with an established database of gestures. Finally, we present quantitative data collected from human participants indicating accuracy and performance of our proposed scheme. Interest in human-robot coexistence, in which humans and robots share a common work volume, is increasing in manufacturing environments. Efficient work coordination requires both awareness of the human pose and a plan of action for both human and robot agents in order to compute robot motion trajectories that synchronize naturally with human motion. In this paper, we present a data-driven approach that synthesizes anticipatory knowledge of both human motions and subsequent action steps in order to predict in real-time the intended target of a human performing a reaching motion. Motion-level anticipatory models are constructed using multiple demonstrations of human reaching motions. We produce a library of motions from human demonstrations, based on a statistical representation of the degrees of freedom of the human arm, using time series analysis, wherein each time step is encoded as a multivariate Gaussian distribution. We demonstrate the benefits of this approach through offline statistical analysis of human motion data. The results indicate a considerable improvement over prior techniques in early prediction, achieving 70 or higher correct classification on average for the first third of the trajectory (< 500msec). We also indicate proof-of-concept through the demonstration of a human-robot cooperative manipulation task performed with a PR2 robot. Finally, we analyze the quality of task-level anticipatory knowledge required to improve prediction performance early in the human motion trajectory. We consider an imitation learning approach to model robot point-to-point (also known as discrete or reaching) movements with a set of autonomous Dynamical Systems (DS). Each DS model codes a behavior (such as reaching for a cup and swinging a golf club) at the kinematic level. An estimate of these DS models are usually obtained from a set of demonstrations of the task. When modeling robot discrete motions with DS, ensuring stability of the learned DS is a key requirement to provide a useful policy. In this paper we propose an imitation learning approach that exploits the power of Control Lyapunov Function (CLF) control scheme to ensure global asymptotic stability of nonlinear DS. Given a set of demonstrations of a task, our approach proceeds in three steps: (1) Learning a valid Lyapunov function from the demonstrations by solving a constrained optimization problem, (2) Using one of the-state-of-the-art regression techniques to model an (unstable) estimate of the motion from the demonstrations, and (3) Using (1) to ensure stability of (2) during the task execution via solving a constrained convex optimization problem. The proposed approach allows learning a larger set of robot motions compared to existing methods that are based on quadratic Lyapunov functions. Additionally, by using the CLF formalism, the problem of ensuring stability of DS motions becomes independent from the choice of regression method. Hence it allows the user to adopt the most appropriate technique based on the requirements of the task at hand without compromising stability. We evaluate our approach both in simulation and on the 7 degrees of freedom Barrett WAM arm. Proposing a new parameterization to model complex Lyapunov functions.Estimating task-oriented Lyapunov functions from demonstrations.Ensuring stability of nonlinear autonomous dynamical systems.Applicability to any smooth regression method. The demand for flexible and re-programmable robots has increased the need for programming by demonstration systems. In this paper, grasp recognition is considered in a programming by demonstration framework. Three methods for grasp recognition are presented and evaluated. The first method uses Hidden Markov Models to model the hand posture sequence during the grasp sequence, while the second method relies on the hand trajectory and hand rotation. The third method is a hybrid method, in which both the first two methods are active in parallel. The particular contribution is that all methods rely on the grasp sequence and not just the final posture of the hand. This facilitates grasp recognition before the grasp is completed. Also, by analyzing the entire sequence and not just the final grasp, the decision is based on more information and increased robustness of the overall system is achieved. The experimental results show that both arm trajectory and final hand posture provide important information for grasp classification. By combining them, the recognition rate of the overall system is increased. This work extends the framework of bilateral shared control of mobile robots with the aim of increasing the robot autonomy and decreasing the operator commitment. We consider persistent autonomous behaviors where a cyclic motion must be executed by the robot. The human operator is in charge of modifying online some geometric properties of the desired path. This is then autonomously processed by the robot in order to produce an actual path guaranteeing: i) tracking feasibility, ii) collision avoidance with obstacles, iii) closeness to the desired path set by the human operator, and iv) proximity to some points of interest. A force feedback is implemented to inform the human operator of the global deformation of the path rather than using the classical mismatch between desired and executed motion commands. Physically-based simulations, with human hardware-in-the-loop and a quadrotor UAV as robotic platform, demonstrate the feasibility of the method. For robotic systems to realise lifelong autonomy they must be able to navigate accurately in changing environments. In this thesis we describe, implement and validate a new approach to the problem of long-term navigation. To begin, we present our stereo visual odometry system which provides highly accurate pose estimation. Our approach combines several techniques found in existing implementations and a recently published image descriptor that simplifies the solution architecture. The performance and versatility of our system is demonstrated through testing on multiple datasets. Equipped with our visual odometry system, we describe a new approach to the problem of lifelong navigation. We learn a model whose complexity varies naturally in accordance with the variation of scene appearance. As the robot repeatedly traverses its workspace, it accumulates distinct visual experiences that, in concert, implicitly represent the scene variation - each experience captures a visual mode. When operating in a previously visited area, we continually try to localise in these previous experiences while simultaneously running the visual odometry. Failure to localise in a sufficient number of prior experiences indicates an insufficient model of the workspace and instigates the laying down of the live image sequence as a new distinct experience. In this way, over time we can capture the typical temporally varying appearance of an environment and the number of experiences required tends to a constant. Although we focus on vision as a primary sensor, the ideas we present here are equally applicable to other sensor modalities. We demonstrate our approach working on a road vehicle operating over a three month period at different times of day, in different weather and lighting conditions. Gestures are characterized by intermediary or final landmarks (real or virtual) in task space or joint space that can change during the course of the motion, and that are described by varying accuracy and correlation constraints. Generalizing these trajectories in robot learning by imitation is challenging, because of the small number of demonstrations provided by the user. We present an approach to statistically encode movements in a task-parameterized mixture model, and derive an expectation-maximization (EM) algorithm to train it. The model automatically extracts the relevance of candidate coordinate systems during the task, and exploits this information during reproduction to adapt the movement in real-time to changing position and orientation of landmarks or objects. The approach is tested with a robotic arm learning to roll out a pizza dough. It is compared to three categories of task-parameterized models: 1) Gaussian process regression (GPR) with a trajectory models database; 2) Multi-streams approach with models trained in several frames of reference; and 3) Parametric Gaussian mixture model (PGMM) modulating the Gaussian centers with the task parameters. We show that the extrapolation capability of the proposed approach outperforms existing methods, by extracting the local structures of the task instead of relying on interpolation principles. This paper proposes an optimization method for dual-arm robots, inspired by a study on human asymmetric bimanual action called Guiard's principle which states that when humans perform asymmetric bimanual tasks, the right hand (as the lateral preference) performs a fine motion (and force) resolution, while the left hand performs a coarse resolution. To effectively transfer the human bimanual-task knowledge to dual-arm robots, we proposed a cost function, based on task-compatibility index, which is used to set the desired motion and force resolution for each end-effector of the dual-arm according to its role in the bimanual action. Thus the right-arm posture is optimized such that the tool attached to its end-effector can exert fine motion and force. And the left-arm posture is optimized to assume a strong and dynamic structural support for the right arm action. We experimentally compare the proposed cost function against previous methods for dual-arm robots using two six degrees-of-freedom torque-controlled manipulators. The control performance, when mimicking Guiard's principle shows considerably better results. In this paper we address the question of how to provide feedback information to guide users of a handheld robotic device when performing a spa- tial exploration task. We consider various feedback methods for communicating a five degree of freedom target pose to a user including a stereoscopic VR display, a monocular see-through AR display and a 2D screen as well as robot arm gesturing. The spatial exploration task with the handheld robot arm was compared against a baseline of a passive handheld wand. We compared the performance of each of the methods with 21 volunteers using a repeated measures ANOVA experimental design, and recorded users' opinions via a NASA Task Load Index Survey. The robot assisted reaching feedback methods significantly outperform manual reaching with the wand for all three visual feedback methods. However, there is little difference between each of the three visual feedback methods when using the robot. The completion time of the task varies with changing difficulty when using the wand but remains stable when assisted by the robot. These results convey useful information for the design of cooperative handheld robots.