Multi-Armed Bandit Problem with Temporally-Partitioned Rewards: When Partial Feedback Counts
There is a rising interest in industrial online applications where data
becomes available sequentially. Inspired by the recommendation of playlists to
users where their preferences can be collected during the listening of the
entire playlist, we study a novel bandit setting, namely Multi-Armed Bandit
with Temporally-Partitioned Rewards (TP-MAB), in which the stochastic reward
associated with the pull of an arm is partitioned over a finite number of
consecutive rounds following the pull. This setting, unexplored so far to the
best of our knowledge, is a natural extension of delayed-feedback bandits to
the case in which rewards may be dilated over a finite-time span after the pull
instead of being fully disclosed in a single, potentially delayed round. We
provide two algorithms to address TP-MAB problems, namely, TP-UCB-FR and
TP-UCB-EW, which exploit the partial information disclosed by the reward
collected over time. We show that our algorithms provide better asymptotical
regret upper bounds than delayed-feedback bandit algorithms when a property
characterizing a broad set of reward structures of practical interest, namely
alpha-smoothness, holds. We also empirically evaluate their performance across
a wide range of settings, both synthetically generated and from a real-world
media recommendation problem.