A notable early precursor to modern-day successes with deep RL occurred in the early 1990’s, with a system nicknamed ‘TD-Gammon,’ which combined neural networks with RL to learn how to play backgammon competitively with top human players [1]}. More specifically, TD-Gammon used a temporal difference RL algorithm, which computed an estimate for each encountered board position of how likely the system was to win (a state-value estimate). The system then computed a reward-prediction error (RPE) – essentially an indication of positive surprise or disappointment – based on subsequent events. The RPE was fed as an error signal into the backpropagation algorithm, which updated the network’s weights so as to yield more accurate state value estimates. Actions could then be selected so as to maximize the state value for the next board state. In order to generate many games on which to train, TD-gammon used self-play, in which the algorithm would play moves against itself until one side won.
