Accelerating Multiagent Reinforcement Learning through Transfer Learning
Reinforcement Learning (RL) is a widely used solution for sequential decision making problems and has been used in many complex domains. However, RL algorithms suffer from scalability issues, especially when multiple agents are acting in a shared environment. This research intends to accelerate learning in multiagent sequential decision making tasks by reusing previous knowledge, both from past solutions and advising between agents. We intend to contribute a Transfer Learning framework focused on Multiagent RL, requiring as few domain specific hand coded parameters as possible. Context and Motivation Reinforcement Learning (RL) (Littman 2015) has been extensively used to solve sequential decision making problems with minimal input data. Despite the recent successes in many complex domains, the classical RL approach is not scalable and suffers from the curse of dimensionality. Scalability issues are further intensified in Multiagent domains, as the decision maker may need to reason over the other agents in the environment. In a world where more and more devices have computing power, many tasks can be solved by Multiagent Systems (sometimes must) Therefore, scalable and robust techniques are of utmost importance for Multiagent RL algorithms. One approach to accelerate the learning process and scale RL to complex domains is to reuse previous knowledge in the new task. Transfer Learning (TL) methods have been used to profit from previous knowledge in many ways (Taylor and Stone 2009) For example, a robot learning how to play soccer may reuse previously acquired skills, such as walking while carrying a ball. Additionally, a more experienced teammate could advice the agent on when to shoot to maximize the goal chance. Also, the agent may observe its teammates' behavior and imitate strategies and moves to learn faster than when learning from scratch without any previous knowledge or guidance. In this research we aim at accelerating learning in Multiagent RL Systems by reusing knowledge from multiple *We gratefully acknowledge financial support from CNPq (grant 311608/2014 0) and Sao Paulo Research Foundation (FAPESP) grant 2015/16310 4. Copyright c (c) 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org) All rights reserved. sources. Even though we here focus on Multiagent RL Systems, the main ideas of our proposal are applicable in the Multiagent Systems, Reinforcement Learning, and Machine Learning areas in general. TL has already been used in Multiagent RL and presented promising results. However, there are no consensual answers to many aspects that must be defined to specify a suitable TL algorithm. Especially, we want to develop algorithms that require as few domain specific hand coded definitions as possible. This is a challenging goal, as it is hard to autonomously generalize knowledge, define task similarities, and reuse task solutions. Research Goals and Expected Contributions This research aims to propose a Transfer Learning framework to accelerate Multiagent Reinforcement Learning through knowledge reuse. Figure 1 depicts the proposed framework. The agent has an Abstract Knowledge Base from which it can extract previous solutions of tasks that are similar to the new one. Additionally, an Advisor Agent may provide advice helping the agent to achieve its goals. The agent then is expected to learn the task faster than without TL. When the task is solved, the new task solution can be abstracted and stored in the knowledge base for latter use. Specifying an algorithm that integrates this framework into the learning processes requires the definition of: (i) A model which allows knowledge generalization; (ii) What information is transferred through tasks or agents; (iii) How to identify which previous tasks are similar enough to be used; (iv) How to define when the knowledge of a given agent must be transferred to another; and (v) How to define if an agent giving advice is trustworthy. Background and Related Work RL is a popular solution for sequential decision making tasks modeled as Markov Decision Processes (MDP) An MDP is described by the tuple <S,A, T,R> where S is the set of environment states, A is the set of available actions, T is the transition function, and R is the reward function, which gives a feedback toward task completion. At each decision step, the agent observes the state s and chooses an action a (among the applicable ones in s) Then the next state is defined by T The agent must learn a policy p that maps the best action for each possible state. The solution of Proceedings of the Thirty First AAAI Conference on Artificial Intelligence (AAAI 17)
AAAI