Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI. This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities. We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS), consisting of a multi-target sub-policy that acts on visual inputs, and a Bayesian model over semantic structures. When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations. We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects. LEAPS outperforms strong baselines that do not explicitly plan using the semantic content. Deep reinforcement learning (DRL) has undoubtedly witnessed strong achievements in recent years BID7 Mnih et al., 2015; BID9 . However, training an agent to solve tasks in a new unseen scenario, usually referred to as its generalization ability, remains a challenging problem (Geffner, 2018; Lake et al., 2017) . In model-free RL, the agent is trained to reactively make decisions from the observations, e.g., first-person view, via a black-box policy approximator. However the generalization ability of agents trained by model-free RL is limited, and is even more evident on tasks that require extensive planning BID9 Kansky et al., 2017) . On the other hand, model-based RL learns a dynamics model, predicting the next observation when taking an action. With the model, sequential decisions can be made via planning. However, learning a model for complex tasks and with high dimensional observations, such as images, is challenging. Current approaches for learning action-conditional models from video are only accurate for very short horizons BID3 Oh et al., 2015) . Moreover, it is not clear how to efficiently adapt such models to changes in the domain.In this work, we aim to improve the generalization of RL agents in domains that involve highdimensional observations. Our insight is that in many realistic settings, building a pixel-accurate model of the dynamics is not necessary for planning high-level decisions. There are semantic structures and properties that are shared in real-world man-made environments. For example, rooms in indoor scenes are often arranged by their mutual functionality (e.g. , bathroom next to bedroom, dining room next to kitchen). Similarly, objects in rooms are placed at locations of practical significance (e.g. , nightstand next to bed, chair next to table). Humans often make use of such structural priors when exploring a new scene, or when making a high-level plan of actions in the domain. However, pixel-level details are still necessary for carrying out the high-level plan. For example, we need high-fidelity observations to locate and interact with objects, open doors, etc.Based on this observation, we propose a hybrid framework, LEArning and Planning with Semantics (LEAPS), which consists of a model-based component that works on the semantic level to pursue a high-level target, and a model-free component that executes the target by acting on pixel-level inputs. Concretely, we (1) train model-free multi-target subpolicies in the form of neural networks that take the first-person views as input and sequentially execute sub-targets towards the final goal; (2) build a semantic model in the form of a latent variable model that only takes semantic signals, i.e., low-dimensional binary vectors, as input and is dynamically updated to plan the next sub-target. LEAPS has following advantages: (1) via model-based planning, generalization ability is improved; (2) by learning the prior distribution of the latent variable model, we capture the semantic consistency among the environments; (3) the semantic model can be efficiently updated by posterior inference when the agent is exploring the unseen environment, which is effective even with very few exploration experiences thanks to the Bayes rule; and (4) the semantic model is lightweight and fully interpretable.Our approach requires observations that are composed of both pixel-level data and a list of semantic properties of the scene. In general, automatically extracting high-level semantic structure from data is difficult. As a first step, in this work we focus on domains where obtaining semantics is easy. In particular, we consider environments which resemble the real-world and have strong object detectors available (He et al., 2017 ). An example of such environments is House3D which contains 45k human-designed 3D scenes BID12 . House3D provides a diverse set of scene layouts, object types, sizes and connectivity, which all conform to a consistent "natural" semantics. Within these complex scenes, we tackle navigation tasks within novel indoor scenes. Note that this problem is extremely challenging as the agent needs to reach far-away targets which can only be completed effectively if it can successfully reason about the overall structure of the new scenario. Lastly, we emphasize that although we consider navigation as a concrete example in this work, our approach is general and can be applied to other tasks for which semantic structures and signals are availableOur extensive experiments show that our LEAPS framework outperforms strong model-free RL approaches, even when the semantic signals are given as input to the policy. Furthermore, the relative improvements of LEAPS over baselines become more significant when the targets are further away from the agent's birthplace, indicating the effectiveness of planning on the learned semantic model. In this work, we proposed LEAPS to improve generalization of RL agents in unseen environments with diverse room layouts and object arrangements, while the underlying semantic information is opt plan-steps 1 2 3 4 5 overall Horizon H = 300 random 20.5 / 15.9 6.9 / 16.7 3.8 / 10.7 1.6 / 4.2 3.0 / 8.8 7.2 / 13.6 pure µ(θ) 49.4 / 47.6 11.8 / 27.6 2.0 / 4.8 2.6 / 10.8 4.2 / 13.2 13.1 / 22.9 aug.µ S (θ) 47.8 / 45.3 11.4 / 23.1 3.0 / 7.8 3.4 / 8.1 4.4 / 11.2 13.0 / 20.5 RNN control. 52.7 / 45.2 13.6 / 23.6 3.4 / 9.6 3.4 / 10.2 6.0 / 17.6 14.9 / 21.9 LEAPS 53.4 / 58.4 15.6 / 31.5 4.5 / 12.5 3.6 / 6.6 7.0 / 18.0 16.4 / 27.9 Horizon H = 500 random 21.9 / 16.9 9.3 / 18.3 5.2 / 12.1 3.6 / 6.1 4.2 / 9.9 9.1 / 15.1 pure µ(θ) 54.0 / 57.5 15.9 / 25.6 3.8 / 7.7 2.8 / 6.4 4.8 / 8.6 16.2 / 22.9 aug.µ S (θ) 54.1 / 51.8 15.5 / 26.5 4.6 / 8. Our LEAPS agents have the highest success rates for all the cases requiring planning computations, i.e., plan-steps larger than 1. For SPL metric, LEAPS agents have the highest overall SPL value over all baseline methods (rightmost column). More importantly, as the horizon increases, LEAPS agents outperforms best baselines more. LEAPS requires a relatively longer horizon for the best practical performances since the semantic model is updated every fixed N = 30 steps, which may potentially increase the episode length for short horizons. More discussions are in Sec. 6.4.shared with the environments in which the agent is trained on. We adopt a graphical model over semantic signals, which are low-dimensional binary vectors. During evaluation, starting from a prior obtained from the training set, the agent plans on model, explores the unknown environment, and keeps updating the semantic model after new information arrives. For exploration, sub-policies that focus on multiple targets are pre-trained to execute primitive actions from visual input. The semantic model in LEAPS is lightweight, interpretable and can be updated dynamically with little explorations. As illustrated in the House3D environment, LEAPS works well for environments with semantic consistencies -typical of realistic domains. On random environments, e.g., random mazes, LEAPS degenerates to exhaustive search.Our approach is general and can be applied to other tasks, such as robotics manipulations where semantic signals can be status of robot arms and object locations, or video games where we can plan on semantic signals such as the game status or current resources. In future work we will investigate models for more complex semantic structures.
We propose a hybrid model-based & model-free approach using semantic information to improve DRL generalization in man-made environments.
{"src": "scitldr-AIC_train", "paper_id": "SJgs1n05YQ"}