Deep reinforcement learning has seen great success across a breadth of tasks such as in game playing and robotic manipulation. However, the modern practice of attempting to learn tabula rasa disregards the logical structure of many domains and the wealth of readily-available human domain experts' knowledge that could help warm start'' the learning process. Further, learning from demonstration techniques are not yet sufficient to infer this knowledge through sampling-based mechanisms in large state and action spaces, or require immense amounts of data. We present a new reinforcement learning architecture that can encode expert knowledge, in the form of propositional logic, directly into a neural, tree-like structure of fuzzy propositions that are amenable to gradient descent. We show that our novel architecture is able to outperform reinforcement and imitation learning techniques across an array of canonical challenge problems for artificial intelligence.
In this paper, we describe a novel approach to imitation learning that infers latent policies directly from state observations. We introduce a method that characterizes the causal effects of latent actions on observations while simultaneously predicting their likelihood. We then outline an action alignment procedure that leverages a small amount of environment interactions to determine a mapping between the latent and real-world actions. We show that this corrected labeling can be used for imitating the observed behavior, even though no expert actions are given. We evaluate our approach within classic control environments and a platform game and demonstrate that it performs better than standard approaches. Code for this work is available at this https URL. Intelligent systems that learn interactively from their end-users are quickly becoming widespread. Until recently, this progress has been fueled mostly by advances in machine learning; however, more and more researchers are realizing the importance of studying users of these systems. In this article we promote this approach and demonstrate how it can result in better user experiences and more effective learning systems. We present a number of case studies that characterize the impact of interactivity, demonstrate ways in which some existing systems fail to account for the user, and explore new ways for learning systems to interact with their users. We argue that the design process for interactive machine learning systems should involve users at all stages: explorations that reveal human interaction patterns and inspire novel interaction methods, as well as refinement stages to tune details of the interface and choose among alternatives. After giving a glimpse of the progress that has been made so far, we discuss the challenges that we face in moving the field forward. Contents: Preface. An Applied Information-Processing Psychology. Part I: Science Base. The Human Information-Processor. Part II: Text-Editing. System and User Variability. An Exercise in Task Analysis. The GOMS Model of Manuscript Editing. Extensions of the GOMS Analysis. Models of Devices for Text Selection. Part III: Engineering Models. The Keystroke-Level Model. The Unit-Task Level of Analysis. Part IV: Extensions and Generalizations. An Exploration into Circuit Design. Cognitive Skill. Applying Psychology to Design Reprise. Sequential prediction problems such as imitation learning, where future observations depend on previous predictions (actions), violate the common i.i.d. assumptions made in statistical learning. This leads to poor performance in theory and often in practice. Some recent approaches (Daume , 2009; Ross and Bagnell, 2010) provide stronger guarantees in this setting, but remain somewhat unsatisfactory as they train either non-stationary or stochastic policies and require a large number of iterations. In this paper, we propose a new iterative algorithm, which trains a stationary deterministic policy, that can be seen as a no regret algorithm in an online learning setting. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance under the distribution of observations it induces in such sequential settings. We demonstrate that this new approach outperforms previous approaches on two challenging imitation learning problems and a benchmark sequence labeling problem. We have developed learning and interaction algorithms to support a human teaching hierarchical task models to a robot using a single demonstration in the context of a mixedinitiative interaction with bi-directional communication. In particular, we have identified and implemented two important heuristics for suggesting task groupings based on the physical structure of the manipulated artifact and on the data flow between tasks. We have evaluated our algorithms with users in a simulated environment and shown both that the overall approach is usable and that the grouping suggestions significantly improve the learning and interaction. Categories and Subject Descriptors I.2.9 [Artificial Intelligence]: Robotics Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments. Cognitive Task Analysis (CTA) helps researchers understand how cognitive skills and strategies make it possible for people to act effectively and get things done. CTA can yield information people need -- employers faced with personnel issues, market researchers who want to understand the thought processes of consumers, trainers and others who design instructional systems, health care professionals who want to apply lessons learned from errors and accidents, systems analysts developing user specifications, and many other professionals. CTA can show what makes the workplace work -- and what keeps it from working as well as it might. Working Minds is a true handbook, offering a set of tools for doing CTA: methods for collecting data about cognitive processes and events, analyzing them, and communicating them effectively. It covers both the "why" and the "how" of CTA methods, providing examples, guidance, and stories from the authors' own experiences as CTA practitioners. Because effective use of CTA depends on some conceptual grounding in cognitive theory and research -- on knowing what a cognitive perspective can offer -- the book also offers an overview of current research on cognition. The book provides detailed guidance for planning and carrying out CTA, with chapters on capturing knowledge and capturing the way people reason. It discusses studying cognition in real-world settings and the challenges of rapidly changing technology. And it describes key issues in applying CTA findings in a variety of fields. Working Minds makes the methodology of CTA accessible and the skills involved attainable.