Our goal is to find the optimal state-action value function \(Q^*(s,a)=max_{\pi } Q^\pi (s,a)\) , from which we may at each state derive the optimal policy \(\pi ^*(s)=argmax_a Q^*(s,a)\) . A well-known RL approach to solve this problem is Q-learning [1]}. Tabular Q-learning maintains a table of Q-value estimates \(\hat{Q}(s,a)\) , collects transition tuples \((s_t,a_t,r_t,s_{t+1})\) , and subsequently updates the tabular estimates according to

Q(st,at) Q(st,at) + [rt+a Q(st+1,a) - Q(st,at)]

 where \(\alpha \in [0,1]\)  denotes a learning rate. Under a policy that is greedy in the limit with infinite exploration (GLIE) this algorithm converges on the optimal value function [1]}.
