In this paper, we show how novel transfer reinforcement learning techniques can be applied to the complex task of target-driven navigation using the photorealisticAI2THOR simulator. Specifically, we build on the concept of Universal SuccessorFeatures with an A3C agent. We introduce the novel architectural1contribution of a Successor Feature Dependent Policy (SFDP) and adopt the concept of VariationalInformation Bottlenecks to achieve state of the art performance.VUSFA, our final architecture, is a straightforward approach that can be implemented using our open source repository. Our approach is generalizable, showed greater stability in training, and outperformed recent approaches in terms of transfer learning ability. The human's ability of navigating unknown spaces (e.g. a firefighter finding the fire hydrant very quickly) primarily relies on visual perception, as well as on previous experience and heavy training (Ramirez et al., 2009 ). In robotics, we would like to mimic this human behaviour. The advancement of visual navigation algorithms essentially contribute to the prevalence and mobility in robotics and therefore, many different approaches are being explored. Previous research has studied map-based, map-building, and map-less approaches (Bonin-Font et al., 2008; Oriolo et al., 1995; Borenstein & Koren, 1991) . In the past, map-based and map-building approaches have been favoured. However, they heavily depend on an accurate mapping of the environment. Also, it requires a carefully executed human-guided training phase which limits its generalizability (Filliat & Meyer, 2003) . With recent advances in Deep Reinforcement Learning (DRL) (Mnih et al., 2015; , map-less navigation has experienced major advancements Mirowski et al., 2018) . It has been demonstrated that DRL-based methods are now able to solve navigation tasks in a more human-like manner (Fan et al., 2018) . Research has shown that DRL-based navigation, in particular target driven visual navigation, is still a challenging task especially when targets are represented in the form of visual information that is highly dynamic. In previous navigation paradigms, the agent navigates to a target demonstrating specific properties (e.g. a yellow cone, such as in the case of Zhang et al. (2017) ), whose location may change over time. In contrast, in target driven visual navigation, the agent should be able to learn to navigate in a persistent state space to a dynamic set of goals. The agent is required to learn to navigate when both the goal and the current state are presented as visual images. A current challenge for DRL algorithms is learning new tasks or goals that vary from what the agent was initially trained for. This ability is called transfer learning. There are two popular strategies for achieving transfer learning in DRL, either by using the concept of General Value Functions (GVF) (Sutton et al., 2011) or by using Successor Feature Approximation (SFA) (Dayan, 1993) . For the task of target driven visual navigation, demonstrated that an A3C agent using the concept of GVF can improve the transfer learning ability. GVF does not however allow us to easily see the underlining process of learning the dynamics of tasks and GVF agents also frequently struggle in complex environments (Sutton et al., 2018) . The second strategy, applying SFA, enables us to capture the dynamics of the environment by attempting to learn future state visitations, although these also encounter limitations when facing multiple tasks. Universal Successor Features Approximators (USFA) , which is an extension of SFA, is able to consider multiple tasks and can improve the transfer learning ability of the agent. In summary, our research contribution is threefold: • For the first time in the literature, we apply Universal Successor Feature Approximators (USFA) for the complex task of target driven visual navigation. Our new approach provides a stable training mechanism and enhances the transfer reinforcement learning ability in complex environments. • We introduce the concept of a Successor Feature Dependant Policy (SFDP), a novel architectural contribution in which the policy can directly make use of the information presented by USFA (an abstract map in our case). This important add-on significantly improves the transfer learning ability of the DRL agent. • Finally, we contribute Variational Universal Successor Feature Approximators (VUSFA), by adopting the concept of Variational Information Bottlenecks. We show that this combination works stably with complex tasks such as target driven visual navigation in the photo-realistic AI2THOR environment . Besides stable convergence, our approach shows possible ways in which transfer learning could be improved in the future. Our second contribution is the addition of a Successor Feature Dependant Policy (SFDP) to the USFA implementation. As mentioned before, ψ g (s t ) can be seen as an abstract representation of the cumulative sum of the future states the agent will visit by following an optimal policy (Dayan, 1993; Barreto et al., 2017) . Traditionally, successor features are not directly consulted when determining an action (Ma et al., 2018b) . However, we hypothesise that feeding the abstract map of future states could be useful in determining the next action. USF can be described as representing the cumulutive sum of discounted future states the agent visits following an optimal policy. This property by itself helps with transfer learning because eventhough different goals have different optimal paths, they can share some common sub-paths. For example, when tasked with finding the microwave and sink in a kitchen, the initial steps of the agent in going to the kitchen will be similar for both tasks. We hypothesised that if the policy has direct access to the USF (see Equation 7), the agent will be able to learn from these similar paths. By directly concatenating ψ g with the final layer of the policy head naively results in ψ g being updated with gradients from the conventional bellman optimality Equation 3 and the policy gradients Figure 1: Proposed Network Architecture "VUSFA": The model's input is the current state of the agent s t and the goal location g as images. These go through a shared simaese encoder E(z|s t ). The reparametrized output z is used to train the ω vector. The policy is conditioned on the USF vector (dotted line indicates gradients do not flow from policy to the USFA head). The USFA ψ is trained with the temporal difference error using φ to give the expected future state occupancies. The discounted episode return is used to train both ω and USFA vectors. of the A3C agent. This can harm the true USF representation and can reduce the transfer learning capabilities of the agent. Therefore in the final model, we stopped the gradient flow from the policy head to the USF branch. The stopping of policy gradients for the USF branch is illustrated in Figure 1 with dotted lines. We proposed Variational Universal Successor Features Approximator (VUSFA) to solve rather complex tasks, such as target driven visual navigation in photorealistic environments using the AI2THOR simulator. To our knowledge, this is the first time the Deep Variational Information Bottleneck theory has been applied with Universal Successor Features in Deep Reinforcement Learning. Our results indicate that VUSFA is able to improve the transfer learning ability in respect to previous state-of-the-art GVF and USF-RL based research . Our approach is generalizable and can be easily adapted to various tasks other than navigation. For re-implementation, we provide the source code via our github repository 1 . Our approach introduces a new perspective and should be considered in future research aiming to improve transfer learning for Deep Reinforcement Learning. In particular, further research could look into exploration of the semantical impacts of φ , ω, and ψ.
We present an improved version of Universal Successor Features based DRL method which can improve the transfer learning of agents.
{"src": "scitldr-AIC_train", "paper_id": "BygXY34FDr"}