Although backpropagation was developed well over thirty years ago [1]}, [2]}, until recently it was employed almost exclusively for supervised learning, as defined above, or for unsupervised learning, where only inputs are presented, and the task is to learn a ‘good’ representation of those inputs based on some function evaluating representational structure, as is done for example in clustering algorithms. Importantly, both of these learning problems differ fundamentally from RL. In particular, unlike supervised and unsupervised learning, RL requires exploration, since the learner is responsible for discovering actions that increase reward. Furthermore, exploration must be balanced against leveraging action-value information already acquired, or as it is conventionally put, exploration must be weighed against ‘exploitation.’ Unlike with most traditional supervised and unsupervised learning problems, a standard assumption in RL is that the actions of the learning system affect its inputs on the next time-step, creating a sensory-motor feedback loop, and potential difficulties due to nonstationarity in the training data. This creates a situation in which target behaviors or outputs involve multi-step decision processes, rather than single input-output mappings. Until very recently, applying deep learning to RL settings has stood as a frustratingly impenetrable problem.
