We present a framework for learning to plan hierarchically in domains with unknown dynamics. We enhance planning performance by exploiting problem structure in several ways: First, we simplify the search over plans by leveraging knowledge of skill objectives; second, shorter plans are generated by enforcing aggressively hierarchical planning; and third, we learn transition dynamics with sparse local models for better generalization. Our framework decomposes transition dynamics into skill effects and success conditions, which allows fast planning by reasoning on effects, while learning conditions from interactions with the world. We propose a simple method for learning new abstract skills, using successful trajectories stemming from completing the goals of a curriculum. Learned skills are then refined to leverage other abstract skills and enhance subsequent planning. We show that both conditions and abstract skills can be learned simultaneously while planning, even in stochastic domains. Our method is validated in experiments of increasing complexity, with up to @math states, showing superior planning to classic non-hierarchical planners or reinforcement learning methods. Applicability to real-world problems is demonstrated in a simulation-to-real transfer experiment on a robotic manipulator.
Planning with policies or skills instead of actions was investigated in @cite_30 , where skills are black-box controllers. The problem is extended to the case of parametrised skills by @cite_9 , which improves the range of available robot behaviours at the expense of requiring planning over both skill and parameters. Lastly, actions are replaced with predefined in @cite_21 , yielding advantages over planning with action directly.
@cite_30 @cite_9 @cite_21
2909064322 2007813200 1579750597 2133067819
A critical capability required for generally intelligent robot behavior is the ability to sequence motor skills to reach a goal. This requires a (typically abstract) representation that supports goal-directed planning, which raises the question of how to construct such a representation. Previous work has addressed this question in the context of simple black-box motor skills, which are insufficiently flexible to support the wide range of behavior required of intelligent robots. We now extend that work to include parametrized motor skills, where a robot must both select an action to execute and also decide how to parametrize it. We show how to construct a representation suitable for planning with parametrized motor skills, and specify conditions which are sufficient to separate the selection of motor skills from the parametrization of those skills. Our method results in a simple discrete abstract representation for planning followed by a parameter selection process that operates on a fixed plan. We first demonstrate learning this representation in a virtual domain based on Angry Birds and then learn an abstract symbolic representation for a robot manipulation task. We demonstrate a sample-efficient method for constructing reusable parameterized skills that can solve families of related motor tasks. Our method uses learned policies to analyze the policy space topology and learn a set of regression models which, given a novel task, appropriately parameterizes an underlying low-level controller. By identifying the disjoint charts that compose the policy manifold, the method can separately model the qualitatively different sub-skills required for solving distinct classes of tasks. Such sub-skills are useful because they can be treated as new discrete, specialized actions by higher-level planning processes. We also propose a method for reusing seemingly unsuccessful policies as additional, valid training samples for synthesizing the skill, thus accelerating learning. We evaluate our method on a humanoid iCub robot tasked with learning to accurately throw plastic balls at parameterized target locations. From an automated planning perspective the problem of practical mobile robot control in realistic environments poses many important and contrary challenges. On the one hand, the planning process must be lightweight, robust, and timely. Over the lifetime of the robot it must always respond quickly with new plans that accommodate exogenous events, changing objectives, and the underlying unpredictability of the environment. On the other hand, in order to promote efficient behaviours the planning process must perform computationally expensive reasoning about contingencies and possible revisions of subjective beliefs according to quantitatively modelled uncertainty in acting and sensing. Towards addressing these challenges, we develop a continual planning approach that switches between using a fast satisficing "classical" planner, to decide on the overall strategy, and decision-theoretic planning to solve small abstract subproblems where deeper consideration of the sensing model is both practical, and can significantly impact overall performance. We evaluate our approach in large problems from a realistic robot exploration domain. Describes experiments with a probabilistic roadmap planner (PRM) on a spot-welding station with 2 to 6 robot manipulators combining 12 to 36 degrees of freedom. When performing centralized planning, the planner has proven to be reliable and fast. When performing decoupled planning, it was not significantly faster, but it was much less reliable, failing to find a solution 30 to 75 of the times in 6-robot examples. This is an important result as it invalidates the assumption that the loss of completeness in performing decoupled planning is not very substantial in practice and indicates that centralized planning is a more desirable approach-at least in applications like spot-welding, which requires rather tight robot coordination.