Planning with preferences has been employed extensively to quickly generate high-quality plans. However, it may be difficult for the human expert to supply this information without knowledge of the reasoning employed by the planner and the distribution of planning problems. We consider the problem of actively eliciting preferences from a human expert during the planning process. Specifically, we study this problem in the context of the Hierarchical Task Network (HTN) planning framework as it allows easy interaction with the human. Our experimental results on several diverse planning domains show that the preferences gathered using the proposed approach improve the quality and speed of the planner, while reducing the burden on the human expert.
There has been a surge in the interest for using human experts to improve decision-making @cite_0 @cite_6 @cite_19 @cite_14 @cite_24 . While distinct methods differ in the form of knowledge that can be specified, preference elicitation has been explored inside automated planning @cite_27 @cite_4 @cite_7 @cite_21 @cite_0 , in reinforcement learning (RL) @cite_8 @cite_3 @cite_29 @cite_9 and inverse reinforcement learning (IRL) @cite_19 @cite_24 .
@cite_14 @cite_4 @cite_7 @cite_8 @cite_29 @cite_21 @cite_9 @cite_6 @cite_3 @cite_0 @cite_19 @cite_24 @cite_27
158183001 160902222 2169562625 2153353285 64129406 2129975315 2070301851 1910090669 2059517962 2399750036 2186505677
Imitation Learning (IL) is a popular approach for teaching behavior policies to agents by demonstrating the desired target policy. While the approach has lead to many successes, IL often requires a large set of demonstrations to achieve robust learning, which can be expensive for the teacher. In this paper, we consider a novel approach to improve the learning efficiency of IL by providing a shaping reward function in addition to the usual demonstrations. Shaping rewards are numeric functions of states (and possibly actions) that are generally easily specified, and capture general principles of desired behavior, without necessarily completely specifying the behavior. Shaping rewards have been used extensively in reinforcement learning, but have been seldom considered for IL, though they are often easy to specify. Our main contribution is to propose an IL approach that learns from both shaping rewards and demonstrations. We demonstrate the effectiveness of the approach across several IL problems, even when the shaping reward is not fully consistent with the demonstrations. Recent new planning paradigms, such as Graphplan and Satplan, have been shown to outperform more traditional domain-independent planners. An interesting aspect of these planners is that they do not incorporate domain specific control knowledge, but instead rely on efficient graph-based or propositional representations and advanced search techniques. An alternative approach has been proposed in the TLPlan system. TLPlan is an example of a powerful planner incorporating declarative control specified in temporal logic formulas. We show how these control rules can be parsed into Satplan. Our empirical results show up to an order of magnitude speed up. We also provide a detailed comparison with TLPlan, and show how the search strategies in TLPlan lead to efficient plans in terms of the number of actions but with little or no parallelism. The Satplan and Graphplan formalisms on the other hand do find highly parallel plans, but are less effective in sequential domains. Our results enhance our understanding of the various tradeoffs in planning technology, and extend earlier work on control knowledge in the Satplan framework by (1997) and Kautz and Selman (1998). Learning from reinforcements is a promising approach for creating intelligent agents. However, reinforcement learning usually requires a large number of training episodes. We present an approach that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach, the advice-giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple programming language. Based on techniques from knowledge-based neural networks, these programs axe inserted directly into the agent's utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that shows our approach leads to statistically-significant gains in expected reward. Importantly, the advice improves the expected reward regardless of the stage of training at which it is given. We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice. In classical planning, the planner is given a concrete goal; it returns a plan for it or a failure message. In the latter case, the user can either quit or modify the goal. For many applications, it is more convenient to let the user provide a more elaborate specification consisting of constraints and preferences over possible goal states. Then, let the system discover a plan for the most desirable among the feasible goal states. To materialize such an approach we require a formalism for specifying preferences and constraints over goals and an algorithm for solving the resulting constrained optimization problem. In this work we motivate the need for planning with preferences and constraints, suggest a rich, yet intuitive formalism for representing goal preferences in the context of a deterministic action model, discuss some of its properties, propose an efficient algorithm for planning with preferences and constraints based on this formalism, and provide extensive experimental analysis in an interesting new domain of configuration planning. In this paper we address the problem of generating preferred plans by combining the procedural control knowledge specified by Hierarchical Task Networks (HTNs) with rich user preferences. To this end, we extend the popular Planning Domain Definition Language, PDDL3, to support specification of simple and temporally extended preferences over HTN constructs. To compute preferred HTN plans, we propose a branch-and-bound algorithm, together with a set of heuristics that, leveraging HTN structure, measure progress towards satisfaction of preferences. Our preference-based planner, HTNPLAN-P, is implemented as an extension of the SHOP2 planner. We compared our planner with SGPlan5 and HPLAN-P- the top performers in the 2006 International Planning Competition preference tracks. HTNPLAN-P generated plans that in all but a few cases equalled or exceeded the quality of plans returned by HPLAN-P and SGPlan5. While our implementation builds on SHOP2, the language and techniques proposed here are relevant to a broad range of HTN planners. Learning from reinforcements is a promising approach for creating intelligent agents. However, reinforcement learning usually requires a large number of training episodes. We present and evaluate a design that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach, the advice-giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple imperative programming language. Based on techniques from knowledge-based neural networks, we insert these programs directly into the agent’s utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that investigates several aspects of our approach and shows that, given good advice, a learner can achieve statistically significant gains in expected reward. A second experiment shows that advice improves the expected reward regardless of the stage of training at which it is given, while another study demonstrates that subsequent advice can result in further gains in reward. Finally, we present experimental results that indicate our method is more powerful than a naive technique for making use of advice. In this paper, we address the problem of generating preferred plans by combining the procedural control knowledge specified by Hierarchical Task Networks (HTNs) with rich qualitative user preferences. The outcome of our work is a language for specifying user preferences, tailored to HTN planning, together with a provably optimal preference-based planner, HTNPLAN, that is implemented as an extension of SHOP2. To compute preferred plans, we propose an approach based on forward-chaining heuristic search. Our heuristic uses an admissible evaluation function measuring the satisfaction of preferences over partial plans. Our empirical evaluation demonstrates the effectiveness of our HTNPLAN heuristics. We prove our approach sound and optimal with respect to the plans it generates by appealing to a situation calculus semantics of our preference language and of HTN planning. While our implementation builds on SHOP2, the language and techniques proposed here are relevant to a broad range of HTN planners. Inverse Reinforcement Learning (IRL) is an approach for domain-reward discovery from demonstration, where an agent mines the reward function of a Markov decision process by observing an expert acting in the domain. In the standard setting, it is assumed that the expert acts (nearly) optimally, and a large number of trajectories, i.e., training examples are available for reward discovery (and consequently, learning domain behavior). These are not practical assumptions: trajectories are often noisy, and there can be a paucity of examples. Our novel approach incorporates advice-giving into the IRL framework to address these issues. Inspired by preference elicitation, a domain expert provides advice on states and actions (features) by stating preferences over them. We evaluate our approach on several domains and show that with small amounts of targeted preference advice, learning is possible from noisy demonstrations, and requires far fewer trajectories compared to simply learning from trajectories alone. Intelligent systems that interact with humans typically require demonstrations and or advice from the expert for optimal decision making. While the active learning formalism allows for these systems to incrementally acquire demonstrations from the human expert, most learning systems require all the advice about the domain in advance. We consider the problem of actively soliciting human advice in an inverse reinforcement learning setting where the utilities are learned from demonstrations. Our hypothesis is that such solicitation of advice reduces the burden on the human to provide advice about every scenario in advance. The Advisable Planner project seeks to make AI planning technology more accessible and easier to use through the metaphor of advice-taking. In particular, we are developing an advisable planning system that marries an advice-taking interface to state-of-the-axt planning technology. The advice-taking module will accept user specifications of characteristics for both the desired solution and the problem-solving process to be employed for a given task, while the planning component will employ those directives to guide plan construction. This design will enable users to interact with planning systems at high levels of abstraction to influence the plan generation process in terms that are meaningful to them. This paper describes the overall objectives of the project and outlines the technical approach to be employed.