One of the long-standing challenges in Artificial Intelligence for learning goal-directed behavior is to build a single agent which can solve multiple tasks. Recent progress in multi-task learning for goal-directed sequential problems has been in the form of distillation based learning wherein a student network learns from multiple task-specific expert networks by mimicking the task-specific policies of the expert networks. While such approaches offer a promising solution to the multi-task learning problem, they require supervision from large expert networks which require extensive data and computation time for training.
 In this work, we propose an efficient multi-task learning framework which solves multiple goal-directed tasks in an on-line setup without the need for expert supervision. Our work uses active learning principles to achieve multi-task learning by sampling the harder tasks more than the easier ones. We propose three distinct models under our active sampling framework. An adaptive method with extremely competitive multi-tasking performance. A UCB-based meta-learner which casts the problem of picking the next task to train on as a multi-armed bandit problem. A meta-learning method that casts the next-task picking problem as a full Reinforcement Learning problem and uses actor-critic methods for optimizing the multi-tasking performance directly. We demonstrate results in the Atari 2600 domain on seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance. Deep Reinforcement Learning (DRL) arises from the combination of the representation power of Deep learning (DL) BID10 BID3 ) with the use of Reinforcement Learning (RL) BID28 objective functions. DRL agents can solve complex visual control tasks directly from raw pixels BID6 BID12 BID24 BID11 BID23 BID13 BID29 BID30 BID2 BID26 BID7 . However, models trained using such algorithms tend to be task-specific because they train a different network for different tasks, however similar the tasks are. This inability of the AI agents to generalize across tasks motivates the field of multi-task learning which seeks to find a single agent (in the case of DRL algorithms, a single deep neural network) which can perform well on all the tasks. Training a neural network with a multi-task learning (MTL) algorithm on any fixed set of tasks (which we call a multi tasking instance (MTI)) leads to an instantiation of a multi-tasking agent (MTA) (we use the terms Multi-Tasking Network (MTN) and MTA interchangeably). Such an MTA would possess the ability to learn task-agnostic representations and thus generalize learning across different tasks. Successful DRL approaches to the goal-directed MTL problem fall into two categories. First, there are approaches that seek to extract the prowess of multiple task-specific expert networks into a single student network. The Policy Distillation framework BID20 and Actor-Mimic Networks BID16 fall into this category. These works train k task-specific expert networks (DQNs ) and then distill the individual task-specific policies learned by the expert networks into a single student network which is trained using supervised learning. While these approaches eventually produce a single a network that solves multiple tasks, individual expert networks must first be trained, and this training tends to be extremely computation and data intensive.The second set of DRL approaches to multi-tasking are related to the field of transfer learning. Many recent DRL works BID16 BID21 BID18 BID4 attempt to solve the transfer learning problem. Progressive networks BID21 ) is one such framework which can be adapted to the MTL problem. Progressive networks iteratively learn to solve each successive task that is presented. Thus, they are not a truly on-line learning algorithm. Progressive Networks instantiate a task-specific column network for each new task. This implies that the number of parameters they require grows as a large linear factor with each new task. This limits the scalability of the approach with the results presented in the work being limited to a maximum of four tasks only. Another important limitation of this approach is that one has to decide the order in which the network trains on the tasks. In this work we propose a fully on-line multi-task DRL approach that uses networks that are comparable in size to the single-task networks.In particular, our contributions are the following: 1) We propose the first successful on-line multi-task learning framework which operates on MTIs that have many tasks with very visually different high-dimensional state spaces (See FIG0 for a visual depiction of the 21 tasks that constitute our largest multi-tasking instance). 2) We present three concrete instantiations of our MTL framework: an adaptive method, a UCB-based meta-learning method and a A3C based meta-learning method. 3) We propose a family of robust evaluation metrics for the multi-tasking problem and demonstrate that they evaluate a multi-tasking algorithm in a more sensible manner than existing metrics. 4) We provide extensive analyses of the abstract features learned by our methods and argue that most of the features help in generalization across tasks because they are task-agnostic. 5) We report results on seven distinct MTIs: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance. Previous works have only reported results on a single MTI. Our largest MTI has more than double the number of tasks present in the largest MTI on which results have been published in the Deep RL literature BID20 . 6) We hence demonstrate how hyper-parameters tuned for an MTI (an instance with six tasks) generalize to other MTIs (with up to 21 tasks). We propose a framework for training MTNs which , through a form of active learning succeeds in learning to perform on-line multi-task learning. The key insight in our work is that by choosing the task to train on, an MTA can choose to concentrate its resources on tasks in which it currently performs poorly. While we do not claim that our method solves the problem of on-line multi-task reinforcement learning definitively, we believe it is an important first step. Our method is complementary to many Figure 6 : Turn Off analysis heap-maps for the all agents. For BA3C since the agent scored 0 on one of the games, normalization along the neuron was done only across the other 5 games.of the existing works in the field of multi-task learning such as: BID20 and BID16 . These methods could potentially benefit from our work. Another possible direction for future work could be to explicitly force the learned abstract representations to be task-agnostic by imposing objective function based regularizations. One possible regularization could be to force the average firing rate of a neuron to be the same across the different tasks.
Letting a meta-learner decide the task to train on for an agent in a multi-task setting improves multi-tasking ability substantially
{"src": "scitldr-AIC_train", "paper_id": "B1nZ1weCZ"}