Learning Security Strategies through Game Play and Optimal Stopping
We study automated intrusion prevention using reinforcement learning.
Following a novel approach, we formulate the interaction between an attacker
and a defender as an optimal stopping game and let attack and defense
strategies evolve through reinforcement learning and self-play. The
game-theoretic perspective allows us to find defender strategies that are
effective against dynamic attackers. The optimal stopping formulation gives us
insight into the structure of optimal strategies, which we show to have
threshold properties. To obtain the optimal defender strategies, we introduce
T-FP, a fictitious self-play algorithm that learns Nash equilibria through
stochastic approximation. We show that T-FP outperforms a state-of-the-art
algorithm for our use case. Our overall method for learning and evaluating
strategies includes two systems: a simulation system where defender strategies
are incrementally learned and an emulation system where statistics are produced
that drive simulation runs and where learned strategies are evaluated. We
conclude that this approach can produce effective defender strategies for a
practical IT infrastructure.