This paper focuses on semantic task planning, i.e., predicting a sequence of actions toward accomplishing a specific task under a certain scene, which is a new problem in computer vision research. The primary challenges are how to model task-specific knowledge and how to integrate this knowledge into the learning procedure. In this work, we propose training a recurrent long short-term memory (LSTM) network to address this problem, i.e., taking a scene image (including pre-located objects) and the specified task as input and recurrently predicting action sequences. However, training such a network generally requires large numbers of annotated samples to cover the semantic space (e.g., diverse action decomposition and ordering). To overcome this issue, we introduce a knowledge and-or graph (AOG) for task description, which hierarchically represents a task as atomic actions. With this AOG representation, we can produce many valid samples (i.e., action sequences according to common sense) by training another auxiliary LSTM network with a small set of annotated samples. Furthermore, these generated samples (i.e., task-oriented action sequences) effectively facilitate training of the model for semantic task planning. In our experiments, we create a new dataset that contains diverse daily tasks and extensively evaluate the effectiveness of our approach.
Some other works also manually defined the controller sequences for completing certain tasks, including baking cookies @cite_55 , making pancakes @cite_31 , and folding laundry @cite_60 . In these works, the controller sequences were selected from several predefined sequences or retrieved from a website. For example, in the work @cite_31 , the robot retrieved instructions for making pancakes from the Internet to generate the controller sequence. Although they often achieve correct sequences in controlled environments, these methods cannot scale up to a large number of tasks in unconstrained household environments because each task requires defining complicated rules for the controller sequence to adapt to various environments.
@cite_55 @cite_31 @cite_60
2156047221 2344260694 2803633138 2132714442
In this paper we report on a recent public experiment that shows two robots making pancakes using web instructions. In the experiment, the robots retrieve instructions for making pancakes from the World Wide Web and generate robot action plans from the instructions. This task is jointly performed by two autonomous robots: The first robot opens and closes cupboards and drawers, takes a pancake mix from the refrigerator, and hands it to the robot B. The second robot cooks and flips the pancakes, and then delivers them back to the first robot. While the robot plans in the scenario are all percept-guided, they are also limited in different ways and rely on manually implemented sub-plans for parts of the task. We will thus discuss the potential of the underlying technologies as well as the research challenges raised by the experiment. In this paper, we address the problem of simultaneously achieving low trajectory tracking errors and compliant control without using explicit mathematical models of task dynamics. To achieve this goal, we propose a new movement representation called compliant movement primitives (CMPs), which encodes position trajectory and associated torque profiles and can be learned from a single user demonstration. With the proposed control framework, the robot can remain compliant and consequently safe for humans sharing its workspace, even if high trajectory tracking accuracy is required. We developed a statistical learning approach that can use a database of existing CMPs and compute new ones, adapted for novel task variations. The proposed approach was evaluated on a Kuka LWR-4 robot performing 1) a discrete pick-and-place task with objects of varying weight and 2) a periodic handle turning operation. The evaluation of the discrete task showed a 15-fold decrease of the tracking error while exhibiting compliant behavior compared to the standard feedback control approach. It also indicated no significant rise in the tracking error while using generalized primitives computed by the statistical learning method. With respect to unforeseen collisions, the proposed approach resulted in a 75 drop of contact forces compared to standard feedback control. The periodic task demonstrated on-line use of the proposed approach to accomplish a task of handle turning. In robotics, knowing the object states and recognizing the desired states are very important. Objects at different states would require different grasping. To achieve different states, different manipulations would be required, as well as different grasping. To analyze the objects at different states, a dataset of cooking objects was created. Cooking consists of various cutting techniques needed for different dishes (e.g. diced, julienne etc.). Identifying each of this state of cooking objects by the human can be difficult sometimes too. In this paper, we have analyzed seven different cooking object states by tuning a convolutional neural network (CNN). For this task, images were downloaded and annotated by students and they are divided into training and a completely different test set. By tuning the vgg-16 CNN 77 accuracy was obtained. The work presented in this paper focuses on classification between various object states rather than task recognition or recipe prediction. This framework can be easily adapted in any other object state classification activity. In this thesis we present a technique for the composition of robot control laws in dynamical environments. We propose a challenging robotic task, called Dynamical Pick and Place, in which a robot equipped with merely a soft paddle must capture and contain a ball, safely negotiate it past obstacles, and bring it to rest at a desired location. We develop a composition technique for local controllers that provides a formal guarantee of the stability of the switching behavior required in this task, and provide descriptive statistics of a working implementation. Our robotic system displays unusually dexterous behavior in the face of significant system noise, and recovers gracefully from large unexpected perturbations caused by the experimenters. Our approach to controller composition makes use of the funnel as a metaphor for asymptotic stability, is motivated by the pre-image backchaining techniques developed by Lozano-Perez, Mason and Taylor, and extends their ideas from quasi-static environments to systems with full dynamics. We introduce the concepts of "dynamical obstacle avoidance" and "dynamical safety" for systems with only intermittent control of their environment, and show that it is important not only that the system avoid obstacles directly, but also that the system will never reach an obstacle before getting another chance to effect control. The Dynamical Pick and Place problem addressed by this thesis is a difficult control problem, but an easy planning problem. The system we develop provides a way to engage more powerful AI planning tools without sacrificing access to the stability arguments of dynamical systems theory.