We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers an easy-to-hard curriculum when learning interactively from an oracle. Our LBA generated data consistently matches or outperforms the CLEVR train data and is more sample efficient. We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.
centers on settings in which an agent explores the environment to acquire supervision ; it has been studied in the context of, among others, computer games and navigation , multi-user games , inverse kinematics @cite_3 , and motion planning for humanoids . Exploratory learning problems are generally framed with reinforcement learning in which the agent receives (delayed) rewards, which are used to learn a policy that maximizes the expected rewards. A key difference in the LBA setting is that it does have sparse delayed rewards. Contextual multi-armed bandits are another class of reinforcement learning algorithms that more closely resemble the LBA setting. However, unlike bandits, online performance is irrelevant in our setting: our aim is not to minimize regret, but to minimize the error of the final VQA model produced by the learner.
@cite_3
2004303440
We consider the problem of learning in single-player and multiplayer multiarmed bandit models. Bandit problems are classes of online learning problems that capture exploration versus exploitation tradeoffs. In a multiarmed bandit model, players can pick among many arms, and each play of an arm generates an i.i.d. reward from an unknown distribution. The objective is to design a policy that maximizes the expected reward over a time horizon for a single player setting and the sum of expected rewards for the multiplayer setting. In the multiplayer setting, arms may give different rewards to different players. There is no separate channel for coordination among the players. Any attempt at communication is costly and adds to regret. We propose two decentralizable policies, @math ( @math - @math ) and @math - @math , that can be used in both single player and multiplayer settings. These policies are shown to yield expected regret that grows at most as O( @math ). It is well known that @math is the lower bound on the rate of growth of regret even in a centralized case. The proposed algorithms improve on prior work where regret grew at O( @math ). More fundamentally, these policies address the question of additional cost incurred in decentralized online learning, suggesting that there is at most an @math -factor cost in terms of order of regret. This solves a problem of relevance in many domains and had been open for a while.