Remember our approach does not use the Probability Transition Matrix, i.e., it is model-free, hence the agent learns the control policy by sampling the environment. However, training a network from consecutive samples directly obtained from the environment is inefficient because of the strong correlations in this data, which can cause a high variance in the network parameter updates [1]}.
