We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers an easy-to-hard curriculum when learning interactively from an oracle. Our LBA generated data consistently matches or outperforms the CLEVR train data and is more sample efficient. We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.
centers on settings in which an agent explores the environment to acquire supervision ; it has been studied in the context of, among others, computer games and navigation , multi-user games , inverse kinematics @cite_3 , and motion planning for humanoids . Exploratory learning problems are generally framed with reinforcement learning in which the agent receives (delayed) rewards, which are used to learn a policy that maximizes the expected rewards. A key difference in the LBA setting is that it does have sparse delayed rewards. Contextual multi-armed bandits are another class of reinforcement learning algorithms that more closely resemble the LBA setting. However, unlike bandits, online performance is irrelevant in our setting: our aim is not to minimize regret, but to minimize the error of the final VQA model produced by the learner.
@cite_3
2280442445
We study networks of communicating learning agents that cooperate to solve a common nonstochastic bandit problem. Agents use an underlying communication network to get messages about actions selected by other agents, and drop messages that took more than d hops to arrive, whered is a delay parameter. We introduce EXP3-COOP, a cooperative version of the EXP3 algorithm and prove that with K actions and N agents the average per-agent regret after T rounds is at most of order q d + 1 + K d (T lnK), where d is the independence number of the d-th power of the communication graphG. We then show that for any connected graph, ford = p K the regret bound isK 1=4 p T , strictly better than the minimax regret p KT for noncooperating agents. More informed choices ofd lead to bounds which are arbitrarily close to the full information minimax regret p T lnK when G is dense. When G has sparse components, we show that a variant of EXP3-COOP, allowing agents to choose their parameters according to their centrality inG, strictly improves the regret. Finally, as a by-product of our analysis, we provide the first characterization of the minimax regret for bandit learning with delay.