Optimistic Whittle Index Policy: Online Learning for Restless Bandits
Restless multi-armed bandits (RMABs) extend multi-armed bandits to allow for
stateful arms, where the state of each arm evolves restlessly with different
transitions depending on whether that arm is pulled. However, solving RMABs
requires information on transition dynamics, which is often not available
upfront. To plan in RMAB settings with unknown transitions, we propose the
first online learning algorithm based on the Whittle index policy, using an
upper confidence bound (UCB) approach to learn transition dynamics.
Specifically, we formulate a bilinear program to compute the optimistic Whittle
index from the confidence bounds in transition dynamics. Our algorithm,
UCWhittle, achieves sublinear $O(\sqrt{T \log T})$ frequentist regret to solve
RMABs with unknown transitions. Empirically, we demonstrate that UCWhittle
leverages the structure of RMABs and the Whittle index policy solution to
achieve better performance than existing online learning baselines across three
domains, including on real-world maternal and childcare data aimed at reducing
maternal mortality.