Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner
Inverse reinforcement learning (IRL) deals with estimating an agent's utility
function from its actions. In this paper, we consider how an agent can hide its
strategy and mitigate an adversarial IRL attack; we call this inverse IRL
(I-IRL). How should the decision maker choose its response to ensure a poor
reconstruction of its strategy by an adversary performing IRL to estimate the
agent's strategy? This paper comprises four results: First, we present an
adversarial IRL algorithm that estimates the agent's strategy while controlling
the agent's utility function. Our second result for I-IRL result spoofs the IRL
algorithm used by the adversary. Our I-IRL results are based on revealed
preference theory in micro-economics. The key idea is for the agent to
deliberately choose sub-optimal responses that sufficiently masks its true
strategy. Third, we give a sample complexity result for our main I-IRL result
when the agent has noisy estimates of the adversary specified utility function.
Finally, we illustrate our I-IRL scheme in a radar problem where a
meta-cognitive radar is trying to mitigate an adversarial target.