This state of affairs changed dramatically in 2013, with the report of the Deep Q Network (DQN), the first deep RL system that learned to play classic Atari video games [1]}, [2]}. Although DQN was widely noted for attaining better-than-human performance on many games, the real breakthrough was simply in getting deep RL to work in a reliably stable way. It incorporated several mechanisms that reduced nonstationarity, treating the RL problem more like a series of supervised learning problems, upon which the tools of deep learning could be more reliably applied. One example is `experience replay' [3]}, in which past state-action-reward-next-state transitions were stored away and intermittently re-presented in random order in order to mimic the random sampling of training examples that occurs in supervised learning. This helped to greatly reduce variance and stabilize the updates.
