Intrinsic rewards are introduced to simulate how human intelligence works; they are usually evaluated by intrinsically-motivated play, i.e., playing games without extrinsic rewards but evaluated with extrinsic rewards. However, none of the existing intrinsic reward approaches can achieve human-level performance under this very challenging setting of intrinsically-motivated play. In this work, we propose a novel megalomania-driven intrinsic reward (called ), which, to our knowledge, is the first approach that achieves human-level performance in intrinsically-motivated play. Intuitively, mega-reward comes from the observation that infants' intelligence develops when they try to gain more control on entities in an environment; therefore, mega-reward aims to maximize the control capabilities of agents on given entities in a given environment. To formalize mega-reward, a relational transition model is proposed to bridge the gaps between direct and latent control. Experimental studies show that mega-reward can (i) greatly outperform all state-of-the-art intrinsic reward approaches, (ii) generally achieves the same level of performance as Ex-PPO and professional human-level scores; and (iii) has also superior performance when it is incorporated with extrinsic reward.
The concept of originally comes from psychology @cite_16 @cite_0 , where infants are proved to be aware that the entities in the state are potentially related to their actions. The idea was first introduced into AI by @cite_39 . More recently, the discovery of @cite_20 , a neuroscience finding that supports the psychology concept of contingency awareness, trigged the interests of applying grid cells in AI agents @cite_3 @cite_18 . Another popular idea developed from contingency awareness is the one of , which are used to learn representations that contain the necessary information about action-related changes in states @cite_26 , or generate attention masks about which part of the states is action-related @cite_34 . Other ideas following contingency awareness include controllable feature learning @cite_25 @cite_31 , tool use discovery @cite_36 and sensing guidance @cite_1 @cite_5 @cite_15 . However, we formalize contingency awareness into a powerful intrinsic reward (mega-reward) for human-level intrinsically-motivated play. Besides, existing works are only capable of figuring out what is under the agent's direct control, while we build the awareness of latent control and show that the awareness of latent control is the key to achieving powerful intrinsic reward.
@cite_18 @cite_26 @cite_15 @cite_36 @cite_1 @cite_3 @cite_39 @cite_0 @cite_34 @cite_5 @cite_31 @cite_16 @cite_25 @cite_20
1512866498 2151516755 2542071751 2011348940
Contingency awareness is the recognition that some aspects of a future observation are under an agent's control while others are solely determined by the environment. This paper explores the idea of contingency awareness in reinforcement learning using the platform of Atari 2600 games. We introduce a technique for accurately identifying contingent regions and describe how to exploit this knowledge to generate improved features for value function approximation. We evaluate the performance of our techniques empirically, using 46 unseen, diverse, and challenging games for the Atari 2600 console. Our results suggest that contingency awareness is a generally useful concept for model-free reinforcement learning agents. Humans are adept at inferring the mental states underlying other agents’ actions, such as goals, beliefs, desires, emotions and other thoughts. We propose a computational framework based on Bayesian inverse planning for modeling human action understanding. The framework represents an intuitive theory of intentional agents’ behavior based on the principle of rationality: the expectation that agents will plan approximately rationally to achieve their goals, given their beliefs about the world. The mental states that caused an agent’s behavior are inferred by inverting this model of rational planning using Bayesian inference, integrating the likelihood of the observed actions with the prior over mental states. This approach formalizes in precise probabilistic terms the essence of previous qualitative approaches to action understanding based on an ‘‘intentional stance” [Dennett, D. C. (1987). The intentional stance. Cambridge, MA: MIT Press] or a ‘‘teleological stance” [Gergely, G., Nadasdy, Z., Csibra, G., & Biro, S. (1995). Taking the intentional stance at 12 months of age. Cognition, 56, 165–193]. In three psychophysical experiments using animated stimuli of agents moving in simple mazes, we assess how well different inverse planning models based on different goal priors can predict human goal inferences. The results provide quantitative evidence for an approximately rational inference mechanism in human goal inference within our simplified stimulus paradigm, and for the flexible nature of goal representations that human observers can adopt. We discuss the implications of our experimental results for human action understanding in real-world contexts, and suggest how our framework might be extended to capture other kinds of mental state inferences, such as inferences about beliefs, or inferring whether an entity is an intentional agent. Motivated by online recommendation and advertising systems, we consider a causal model for stochastic contextual bandits with a latent low-dimensional confounder. In our model, there are @math observed contexts and @math arms of the bandit. The observed context influences the reward obtained through a latent confounder variable with cardinality @math ( @math ). The arm choice and the latent confounder causally determines the reward while the observed context is correlated with the confounder. Under this model, the @math mean reward matrix @math (for each context in @math and each arm in @math ) factorizes into non-negative factors @math ( @math ) and @math ( @math ). This insight enables us to propose an @math -greedy NMF-Bandit algorithm that designs a sequence of interventions (selecting specific arms), that achieves a balance between learning this low-dimensional structure and selecting the best arm to minimize regret. Our algorithm achieves a regret of @math at time @math , as compared to @math for conventional contextual bandits, assuming a constant gap between the best arm and the rest for each context. These guarantees are obtained under mild sufficiency conditions on the factors that are weaker versions of the well-known Statistical RIP condition. We further propose a class of generative models that satisfy our sufficient conditions, and derive a lower bound of @math . These are the first regret guarantees for online matrix completion with bandit feedback, when the rank is greater than one. We further compare the performance of our algorithm with the state of the art, on synthetic and real world data-sets. Abstract In this article we propose an account of the early development of children's knowledge about the mind and report two studies designed to test a part of it. According to this “connections-representations” account, young children begin their discovery of the mental world by learning that they and other people have internal experiences or mental states that connect them cognitively to external objects and events-experiences such as seeing them, hearing them, and wanting them. Later, they realize that the same object can be seriously (other than in pretense) mentally represented in different, seemingly contradictory ways: for example, as A in appearance, B in reality, C according to their perceptual or conceptual perspective, and D according to another person's. The results of both studies confirmed the prediction that 3-year-olds would perform well on appearance-reality and perceptual perspective-taking tasks requiring only an understanding of cognitive connections, but perform poorly on tasks requiring an understanding of seemingly-contrary-to-fact mental representations. To illustrate, children of this age had little difficulty determining that they could hear but could not see a noise-making object located on the experimenter's side of a barrier, and that the experimenter could see it (connections-level tasks). In contrast, they were largely unable to say, for instance, that a toy bear held behind a large elephant mask and emitting a cat sound looked like an elephant, sounded like a cat, and really was a toy bear (representations-level tasks)—even though the experimenter had actually told them previously what it looked like, sounded like, and really was. The article concludes with speculations about the possible origins of connections and representations knowledge and observations about the significance of these acquisitions for the child's development.