Nonstochastic bandit models [1]} allow the sequence of rewards for each arm to be generated by an adversary. See [2]} for an overview. The decision-maker's goal is to select a sequence of actions that perform nearly as well in as any single fixed action would have on the reward sequence.
