The soundness and optimality of a plan depends on the correctness of the domain model. In real-world applications, specifying complete domain models is difficult as the interactions between the agent and its environment can be quite complex. We propose a framework to learn a PPDDL representation of the model incrementally over multiple planning problems using only experiences from the current planning problem, which suits non-stationary environments. We introduce the novel concept of reliability as an intrinsic motivation for reinforcement learning, and as a means of learning from failure to prevent repeated instances of similar failures. Our motivation is to improve both learning efficiency and goal-directedness. We evaluate our work with experimental results for three planning domains. Planning requires as input a model which describes the dynamics of a domain. While domain models are normally hand-coded by human experts, complex dynamics typical of real-world applications can be difficult to capture in this way. This is known as the knowledge engineering problem BID3 . One solution is to learn the model from data which is then used to synthesize a plan or policy. In this work, we are interested in applications where the training data has to be acquired by acting or executing an action. However, training data acquired in a planning problem could be insufficient to infer a complete model. While this is mitigated by including past training data from previous planning problems, this would be ill-suited for nonstationary domains where distributions of stochastic dynamics shift over time. Furthermore, the computation time increases with the size of the training data.Following these observations, we present an incremental learning model (ILM) which learns action models incrementally over planning problems, under the framework of reinforcement learning. PPDDL, a planning language modelling probabilistic planning problems BID20 ) (see Figure 1) , is used for planning, and a rules-based representation (see FIG0 ) is used for the learning process. A parser translates between these two representations. Action models that were learned previously are provided to subsequent planning problems and are improved upon acquiring new training data; past training data are not used.We denote the models provided as prior action models. These could also be hand-coded, incomplete models serving as prior knowledge. Using prior knowledge has two advantages: (1) it biases the learning towards the prior action models, and (2) it reduces the amount of exploration required.While the learning progress cannot be determined without the true action models, we can estimate it empirically based on the results of learning and acting. This empirical estimate, or reliability, is used to guide the search in the space of possible models during learning and as an intrinsic motivation in reinforcement learning. When every action is sufficiently reliable, we instead exploit with Gourmand, a planner that solves finite-horizon Markov Decision Processes (MDP) problems online BID9 .Another major contribution of our work is its ability to learn from failure. Actions fail to be executed if their preconditions are not satisfied in the current state. This is common when the model is incorrect. Failed executions can have dire consequences in the real-world or cause irreversible changes such that goal states cannot be reached. ILM records failed executions and prevents any further attempts that would lead to similar failure. This reduces the number of failed executions and increases the efficiency of exploration.The rest of the paper is organized as follows. First, we review related work and then present the necessary background. Next, we provide details of ILM. Lastly, we evaluate ILM in three planning domains and discuss the significance of various algorithmic features introduced in this paper. We presented a domain-independent framework, ILM, for incremental learning over multiple planning problems of a domain without the use of past training data. We introduced a new measure, reliability, which serves as an empirical estimate of the learning progress and influences the processes of learning and planning. The relational counts are weighted with reliability to reduce the amount of exploration required for reliable action models. We also extended an existing rules learner to consider prior knowledge in the form of incomplete action models. ILM learns from failure by checking if an action is in a list of state-action pairs which represents actions that have failed to execute. We evaluated ILM on three benchmark domains. Experimental results showed that variational distances of learned action models decreased over each subsequent round. Learning from failure greatly reduces the number of failed executions leading to improved correctness and goal-directedness. For complex domains, more training data is required to learn action models. Using past training data would not work well for non-stationary domains and also increases the computation time for learning. The first issue could be resolved by learning distributions from the current training data only. The second issue could be resolved by maintaining a fixed size of training data by replacing older experiences while maximizing the exposure, or variability, of the training data. These will be explored in the future.
Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.
{"src": "scitldr-AIC_train", "paper_id": "B1eZxbU9DE"}