Guided policy search is a method for reinforcement learning that trains a general policy for accomplishing a given task by guiding the learning of the policy with multiple guiding distributions. Guided policy search relies on learning an underlying dynamical model of the environment and then, at each iteration of the algorithm, using that model to gradually improve the policy. This model, though, often makes the assumption that the environment dynamics are markovian, e.g., depend only on the current state and control signal. In this paper we apply guided policy search to a problem with non-markovian dynamics. Specifically, we apply it to the problem of pouring a precise amount of liquid from a cup into a bowl, where many of the sensor measurements experience non-trivial amounts of delay. We show that, with relatively simple state augmentation, guided policy search can be extended to non-markovian dynamical systems, where the non-markovianess is caused by delayed sensor readings.
Many various aspects of the task of robotic pouring have been investigated by prior work. Some studies have focused on utilizing specialized hardware and algorithms to achieve very precise pouring results @cite_2 , while others have focused on learning the broad motions of pouring through human demonstrations @cite_25 @cite_23 . Okada @cite_18 used a motion planner to manipulate a pouring object, Yamaguchi and Atkeson @cite_11 used differential dynamic programming to pour in a simulator, and Tamosiunaite used dynamic movement primitives to learn the goal and shape of a pouring trajectory, but all of these were designed to pour the entire contents of the container out, rather than a precise amount. To the authors' knowledge, the only study to attempt to combine learning and precise pouring was done by Rozo @cite_20 , who used human demonstrations to learn to pour 100 ml from a bottle into a cup.
@cite_18 @cite_23 @cite_2 @cite_25 @cite_20 @cite_11
2081857580 1844329104 2003612239 2909112693
Robot learning from demonstration faces new challenges when applied to tasks in which forces play a key role. Pouring liquid from a bottle into a glass is one such task, where not just a motion with a certain force profile needs to be learned, but the motion is subtly conditioned by the amount of liquid in the bottle. In this paper, the pouring skill is taught to a robot as follows. In a training phase, the human teleoperates the robot using a haptic device, and data from the demonstrations are statistically encoded by a parametric hidden Markov model, which compactly encapsulates the relation between the task parameter (dependent on the bottle weight) and the force-torque traces. Gaussian mixture regression is then used at the reproduction stage for retrieving the suitable robot actions based on the force perceptions. Computational and experimental results show that the robot is able to learn to pour drinks using the proposed framework, outperforming other approaches such as the classical hidden Markov models in that it requires less training, yields more compact encodings and shows better generalization capabilities. We explore how to represent, plan and learn robot pouring. This is a case study of a complex task that has many variations and involves manipulating non-rigid materials such as liquids and granular substances. Variations of pouring we consider are the type of pouring (such as pouring into a glass or spreading a sauce on an object), material, container shapes, initial poses of containers and target amounts. The robot learns to select appropriate behaviors from a library of skills, such as tipping, shaking and tapping, to pour a range of materials from a variety of containers. The robot also learns to select behavioral parameters. Planning methods are used to adapt skills for some variations such as initial poses of containers. We show using simulation and experiments on a PR2 robot that our pouring behavior model is able to plan and learn to handle a wide variety of pouring tasks. This case study is a step towards enabling humanoid robots to perform tasks of daily living. This paper focuses on improving performance with practice for tasks that are difficult to model or plan, such as pouring (manipulating a liquid or granular material such as sugar). We are also interested in tasks that involve the possible use of many skills, such as pouring by tipping, shaking, and tapping. Although our ultimate goal is to learn and optimize skills automatically from demonstration and practice, in this paper, we explore manually obtaining skills from human demonstration, and automatically selecting skills and optimizing continuous parameters for these skills. Behaviors such as pouring, shaking, and tapping are modeled with finite state machines. We unify the pouring and the two shaking skills as a general pouring model. The constructed models are verified by implementing them on a PR2 robot. The robot experiments demonstrate that our approach is able to appropriately generalize knowledge about different pouring skills and optimize behavior parameters. Pouring is a fundamental skill for robots in both domestic and industrial environments. Ideally, a robot should be able to pour with high accuracy to specific, pre-defined heights and without spilling. However, due to the complex dynamics of liquids, it is difficult to learn how to pour to achieve these goals. In this paper we present an approach to learn a policy for pouring using Deep Deterministic Policy Gradients (DDPG). We remove the need for collecting training experiences on a real robot, by using a state-of-the-art liquid simulator, which allows for learning the liquid dynamics. We show through our experiments, performed with a PR2 robot, that it is possible to successfully transfer the learned policy to a real robot and even apply it to different liquids.