Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning
We expose the danger of reward poisoning in offline multi-agent reinforcement
learning (MARL), whereby an attacker can modify the reward vectors to different
learners in an offline data set while incurring a poisoning cost. Based on the
poisoned data set, all rational learners using some confidence-bound-based MARL
algorithm will infer that a target policy - chosen by the attacker and not
necessarily a solution concept originally - is the Markov perfect dominant
strategy equilibrium for the underlying Markov Game, hence they will adopt this
potentially damaging target policy in the future. We characterize the exact
conditions under which the attacker can install a target policy. We further
show how the attacker can formulate a linear program to minimize its poisoning
cost. Our work shows the need for robust MARL against adversarial attacks.