The core idea that gives rise to the Q-Network is using neural networks to approximate the Q-function. Whereas a traditional Q-learning attempts to fill in every single value for the Q-function in increasing accuracy in a look-up table manner, a Q-Network attempts to train a neural network that approximates this function. The Q-Network is also able to interpolate after training, since the network attempts to approximate the Q-function continuously, filling in for the gaps in the dataset. The benefits of such an approach have been demonstrated in detail in DeepMind's Atari Network [1]}. However, while attempting to be more efficient and general than a naive Q-agent, Q-Networks sacrifice a lot of stability. Re-training a Q-Network with a newly discovered dataset can be destructive to already existing features of the network. In order to remedy the destructive re-training issue, especially while training through large datasets, (deep) Q-Networks make use of “experience replay” [1]}. A Q-Network agent that uses experience replay will store training data as it comes, and backpropagates that data across the network in occasional intervals, as opposed to some novel data-point.
