Contextual Information-Directed Sampling
Information-directed sampling (IDS) has recently demonstrated its potential
as a data-efficient reinforcement learning algorithm. However, it is still
unclear what is the right form of information ratio to optimize when contextual
information is available. We investigate the IDS design through two contextual
bandit problems: contextual bandits with graph feedback and sparse linear
contextual bandits. We provably demonstrate the advantage of contextual IDS
over conditional IDS and emphasize the importance of considering the context
distribution. The main message is that an intelligent agent should invest more
on the actions that are beneficial for the future unseen contexts while the
conditional IDS can be myopic. We further propose a computationally-efficient
version of contextual IDS based on Actor-Critic and evaluate it empirically on
a neural network contextual bandit.