Function approximation has played an essential role in the reinforcement learning (RL) literature to overcome the limitations of the Q-learning method. The authors of [1]}, [2]} have built the foundation of function approximation RL. Therein, the convergence of function approximations which are linearly combined from basis functions over a state space are established. This result opens up a wide range applications of RL as it only requires a compact representation of the cost-to-go function, in which there are fewer number of parameters than states.
In [3]}, a neural network based on reinforcement learning, namely neural fitted Q (NFQ), is proposed. NFQ comprises of a multi-layer perceptron which is able to store and reuse the transition experience. It is shown therein that NFQ can effectively train a model-free Q-value and achieve the control policy after few communications rounds.
The authors of [4]} consider similar fitted Q-iteration applied to continuous state and continuous actions batch reinforcement learning. The goal is to achieve a good policy generalized from a sufficient number of generated trajectories. The authors have developed first finite-time bound for value-function based algorithms applied to continuous state and action problems.
The authors of [5]} extend the temporal-difference learning proposed in [1]} to stochastic control settings via convergence analysis of several variations of Q-learning under function approximation. Therein, the condition for the approximate methods to converge with probability one is identified.
The advantages of approximate RL have been successfully demonstrated in real-world environments in Atari game in [7]}, [8]}. Therein, the deep Q-network was able learn the policies directly from the pixels and the game score and outperforms all existing learning models.
