We investigate methods to efficiently learn diverse strategies in reinforcement learning for a generative structured prediction problem: query reformulation. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as
 an ensemble of agents trained on the full data. We evaluate on the tasks of document retrieval and question answering. The
 improved performance seems due to the increased diversity of reformulation strategies. This suggests that multi-agent, hierarchical approaches might play an important role in structured prediction tasks of this kind. However, we also find that it is not obvious how to characterize diversity in this context, and a first attempt based on clustering did not produce good results. Furthermore, reinforcement learning for the reformulation task is hard in high-performance regimes. At best, it only marginally improves over the state of the art, which highlights the complexity of training models in this framework for end-to-end language understanding problems. Reinforcement learning (RL) has proven effective in several language tasks, such as machine translation (Wu et al., 2016; Ranzato et al., 2015; BID1 , question-answering BID12 Hu et al., 2017) , and text summarization (Paulus et al., 2017) . In RL efficient exploration is key to achieve good performance. The ability to explore in parallel a diverse set of strategies often speeds up training and leads to a better policy (Mnih et al., 2016; Osband et al., 2016) .In this work, we propose a simple method to achieve efficient parallelized exploration of diverse policies, inspired by hierarchical reinforcement learning BID7 Lin, 1993; Dietterich, 2000; Dayan & Hinton, 1993) . We structure the agent into multiple sub-agents, which are trained on disjoint subsets of the training data. Sub-agents are co-ordinated by a meta-agent, called aggregator, that groups and scores answers from the sub-agents for each given input. Unlike sub-agents , the aggregator is a generalist since it learns a policy for the entire training set. We argue that it is easier to train multiple sub-agents than a single generalist one since each sub-agent only needs to learn a policy that performs well for a subset of examples. Moreover, specializing agents on different partitions of the data encourages them to learn distinct policies, thus giving the aggregator the possibility to see answers from a population of diverse agents. Learning a single policy that results in an equally diverse strategy is more challenging. Since each sub-agent is trained on a fraction of the data, and there is no communication between them, training can be done faster than training a single agent on the full data. Additionally, it is easier to parallelize than applying existing distributed algorithms such as asynchronous SGD or A3C (Mnih et al., 2016) , as the sub-agents do not need to exchange weights or gradients. After training the sub-agents , only their actions need to be sent to the aggregator.We build upon the works of Nogueira & Cho (2017) and Buck et al. (2018b) . Hence, we evaluate our method on the same tasks: query reformulation for document retrieval and question-answering. We show that it outperforms a strong baseline of an ensemble of agents trained on the full dataset. We also found that performance and reformulation diversity are correlated (Sec. 5.5). Our main contributions are the following:• A simple method to achieve more diverse strategies and better generalization performance than a model average ensemble.• Training can be easily parallelized in the proposed method.• An interesting finding that contradicts our, perhaps naive, intuition: specializing agents on semantically similar data does not work as well as random partitioning. An explanation is given in Appendix F.• We report new state-of-the art results on several datasets using BERT (Devlin et al., 2018) .However results improve marginally using reinforcement learning and on the question answering task we see no improvements. 3 A query is the title of a paper and the ground-truth answer consists of the papers cited within. Each document in the corpus consists of its title and abstract. We proposed a method to build a better query reformulation system by training multiple sub-agents on partitions of the data using reinforcement learning and an aggregator that learns to combine the answers of the multiple agents given a new query. We showed the effectiveness and efficiency of the proposed approach on the tasks of document retrieval and question answering. We also found that a first attempt based on semantic clustering did not produce good results, and that diversity was an important but hard to characterize reason for improved performance. One interesting orthogonal extension would be to introduce diversity on the beam search decoder BID11 Li et al., 2016) , thus shedding light on the question of whether the gains come from the increased capacity of the system due to the use of the multiple agents, the diversity of reformulations, or both. Furthermore, we found that reinforcement learning for the reformulation task is hard when the underlying system already performs extremely well on the task. This might be due to the tasks being too constrained (which makes it possible for machines to almost reach human performance), and requires further exploration. AGGREGATOR: The encoder f q0 is a word-level two-layer CNN with filter sizes of 9 and 3, respectively, and 128 and 256 kernels, respectively. D = 512. No dropout is used. ADAM is the optimizer with learning rate of 10 −4 and mini-batch of size 64. It is trained for 100 epochs.
We use reinforcement learning for query reformulation on two tasks and surprisingly find that when training multiple agents diversity of the reformulations is more important than specialisation.
{"src": "scitldr-AIC_train", "paper_id": "BJeypMU5wE"}