Intrinsic rewards are introduced to simulate how human intelligence works; they are usually evaluated by intrinsically-motivated play, i.e., playing games without extrinsic rewards but evaluated with extrinsic rewards. However, none of the existing intrinsic reward approaches can achieve human-level performance under this very challenging setting of intrinsically-motivated play. In this work, we propose a novel megalomania-driven intrinsic reward (called ), which, to our knowledge, is the first approach that achieves human-level performance in intrinsically-motivated play. Intuitively, mega-reward comes from the observation that infants' intelligence develops when they try to gain more control on entities in an environment; therefore, mega-reward aims to maximize the control capabilities of agents on given entities in a given environment. To formalize mega-reward, a relational transition model is proposed to bridge the gaps between direct and latent control. Experimental studies show that mega-reward can (i) greatly outperform all state-of-the-art intrinsic reward approaches, (ii) generally achieves the same level of performance as Ex-PPO and professional human-level scores; and (iii) has also superior performance when it is incorporated with extrinsic reward.
The concept of originally comes from psychology @cite_16 @cite_0 , where infants are proved to be aware that the entities in the state are potentially related to their actions. The idea was first introduced into AI by @cite_39 . More recently, the discovery of @cite_20 , a neuroscience finding that supports the psychology concept of contingency awareness, trigged the interests of applying grid cells in AI agents @cite_3 @cite_18 . Another popular idea developed from contingency awareness is the one of , which are used to learn representations that contain the necessary information about action-related changes in states @cite_26 , or generate attention masks about which part of the states is action-related @cite_34 . Other ideas following contingency awareness include controllable feature learning @cite_25 @cite_31 , tool use discovery @cite_36 and sensing guidance @cite_1 @cite_5 @cite_15 . However, we formalize contingency awareness into a powerful intrinsic reward (mega-reward) for human-level intrinsically-motivated play. Besides, existing works are only capable of figuring out what is under the agent's direct control, while we build the awareness of latent control and show that the awareness of latent control is the key to achieving powerful intrinsic reward.
@cite_18 @cite_26 @cite_15 @cite_36 @cite_1 @cite_3 @cite_39 @cite_0 @cite_34 @cite_5 @cite_31 @cite_16 @cite_25 @cite_20
2804040390 2751973545 2111927824 2565678376 1554318634 1512866498 2084588252 2963126744 1591713425 2963201535 1718985298 2744921630 2142449474
A central problem to understanding intelligence is the concept of generalisation. This allows previously learnt structure to be exploited to solve tasks in novel situations differing in their particularities. We take inspiration from neuroscience, specifically the hippocampal-entorhinal system known to be important for generalisation. We propose that to generalise structural knowledge, the representations of the structure of the world, i.e. how entities in the world relate to each other, need to be separated from representations of the entities themselves. We show, under these principles, artificial neural networks embedded with hierarchy and fast Hebbian memory, can learn the statistics of memories and generalise structural knowledge. Spatial neuronal representations mirroring those found in the brain emerge, suggesting spatial cognition is an instance of more general organising principles. We further unify many entorhinal cell types as basis functions for constructing transition graphs, and show these representations effectively utilise memories. We experimentally support model assumptions, showing a preserved relationship between entorhinal grid and hippocampal place cells across environments. The central resource processed by the sensorimotor system of an organism is information. We propose an information-based quantity that allows one to characterize the efficiency of the perception-action loop of an abstract organism model. It measures the potential of the organism to imprint information on the environment via its actuators in a way that can be recaptured by its sensors, essentially quantifying the options available and visible to the organism. Various scenarios suggest that such a quantity could identify the preferred direction of evolution or adaptation of the sensorimotor loop of organisms. This article studies algorithms used by a learner to explore high-dimensional structured sensorimotor spaces such as in tool use discovery. In particular, we consider goal babbling architectures that were designed to explore and learn solutions to fields of sensorimotor problems, i.e. to acquire inverse models mapping a space of parameterized sensorimotor problems effects to a corresponding space of parameterized motor primitives. However, so far these architectures have not been used in high-dimensional spaces of effects. Here, we show the limits of existing goal babbling architectures for efficient exploration in such spaces, and introduce a novel exploration architecture called Model Babbling (MB). MB exploits efficiently a modular representation of the space of parameterized problems effects. We also study an active version of Model Babbling (the MACOB architecture). These architectures are compared in a simulated experimental setup with an arm that can discover and learn how to move objects using two tools with different properties, embedding structured high-dimensional continuous motor and sensory spaces. The classical approach to using utility functions suffers from the drawback of having to design and tweak the functions on a case by case basis. Inspired by examples from the animal kingdom, social sciences and games we propose empowerment, a rather universal function, defined as the information-theoretic capacity of an agent’s actuation channel. The concept applies to any sensorimotoric apparatus. Empowerment as a measure reflects the properties of the apparatus as long as they are observable due to the coupling of sensors and actuators via the environment. Contingency awareness is the recognition that some aspects of a future observation are under an agent's control while others are solely determined by the environment. This paper explores the idea of contingency awareness in reinforcement learning using the platform of Atari 2600 games. We introduce a technique for accurately identifying contingent regions and describe how to exploit this knowledge to generate improved features for value function approximation. We evaluate the performance of our techniques empirically, using 46 unseen, diverse, and challenging games for the Atari 2600 console. Our results suggest that contingency awareness is a generally useful concept for model-free reinforcement learning agents. Abstract According to Martin and Levey (1987) evaluative conditioning is different from signal learning, i.e. the acquisition of knowledge about predictive relations between environmental events. The hypothesis was tested that evaluative conditioning, unlike signal learning, does not require awareness of the CS-US contingency. In three pilot experiments it was demonstrated that pairing neutral stimuli with either liked or disliked stimuli is sufficient to change neutral stimuli into a positive or negative direction. As indicated by postconditioning recognition questionnaires, this evaluative shift did not require and was not even influenced by contingency awareness. These findings were replicated and corroborated in an experiment, using a concurrent awareness assessment procedure and more fine-grained evaluative response measurements. The relevance of this conditioning without contingency awareness is discussed in the context of recent information processing models of Pavlovian conditioning. The classical approach to using utility functions suffers from the drawback of having to design and tweak the functions on a case by case basis. Inspired by examples from the animal kingdom, social sciences and games we propose empowerment, a rather universal function, defined as the information-theoretic capacity of an agent's actuation channel. The concept applies to any sensorimotor apparatus. Empowerment as a measure reflects the properties of the apparatus as long as they are observable due to the coupling of sensors and actuators via the environment. Using two simple experiments we also demonstrate how empowerment influences sensor-actuator evolution Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment. The code used in the experiments is available at https: github.com flowersteam Curiosity_Driven_Goal_Exploration. An improved suspension clip particularly suited for use by viticulturists in tying vines. The invention is characterized by a resilient body formed of a length of an elastic met allic wire and includes a pair of arcuate end portions of inverted U-shaped configurations adapted to receive a suspension wire, and an arcuate mid-portion conforming substantially to an incomplete annulus having a throat defined between said end portions for accommodating a reception of an arm of a grapevine. The body is responsive to manipulation for assuming a stressed configuration wherein the mid-portion defines a first closed loop for supporting an arm of a grapevine and the end portions thereof are disposed in an interfering, interlocking relationship and define a second closed loop for confining a suspension wire. Intrinsically motivated spontaneous exploration is a key enabler of autonomous lifelong learning in human children. It allows them to discover and acquire large repertoires of skills through self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present the unsupervised multi-goal reinforcement learning formal framework as well as an algorithmic approach called intrinsically motivated goal exploration processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP algorithmic architecture relies on several principles: 1) self-generation of goals as parameterized reinforcement learning problems; 2) selection of goals based on intrinsic rewards; 3) exploration with parameterized time-bounded policies and fast incremental goal-parameterized policy search; 4) systematic reuse of information acquired when targeting a goal for improving other goals. We present a particularly efficient form of IMGEP that uses a modular representation of goal spaces as well as intrinsic rewards based on learning progress. We show how IMGEPs automatically generate a learning curriculum within an experimental setup where a real humanoid robot can explore multiple spaces of goals with several hundred continuous dimensions. While no particular target goal is provided to the system beforehand, this curriculum allows the discovery of skills of increasing complexity, that act as stepping stone for learning more complex skills (like nested tool use). We show that learning several spaces of diverse problems can be more efficient for learning complex skills than only trying to directly learn these complex skills. We illustrate the computational efficiency of IMGEPs as these robotic experiments use a simple memory-based low-level policy representations and search algorithm, enabling the whole system to learn online and incrementally on a Raspberry Pi 3. The hippocampal system is critical for storage and retrieval of declarative memories, including memories for locations and events that take place at those locations. Spatial memories place high demands on capacity. Memories must be distinct to be recalled without interference and encoding must be fast. Recent studies have indicated that hippocampal networks allow for fast storage of large quantities of uncorrelated spatial information. The aim of the this article is to review and discuss some of this work, taking as a starting point the discovery of multiple functionally specialized cell types of the hippocampal–entorhinal circuit, such as place, grid, and border cells. We will show that grid cells provide the hippocampus with a metric, as well as a putative mechanism for decorrelation of representations, that the formation of environment-specific place maps depends on mechanisms for long-term plasticity in the hippocampus, and that long-term spatiotemporal memory storage may depend on offline consolidation processes related to sharp-wave ripple activity in the hippocampus. The multitude of representations generated through interactions between a variety of functionally specialized cell types in the entorhinal–hippocampal circuit may be at the heart of the mechanism for declarative memory formation.