value iteration policy iteration
Assuming full knowledge of the MDP, the two basic approaches to compute the optimal action-value function are value iteration and policy iteration. Both algorithms compute a sequence of functions 




Q

k




{\displaystyle Q_{k}}

 (



k
=
0
,
1
,
2
,
…


{\displaystyle k=0,1,2,\ldots }

) that converge to 




Q

∗




{\displaystyle Q^{*}}

. Computing these functions involves computing expectations over the whole state-space, which is impractical for all but the smallest (finite) MDPs. In reinforcement learning methods, expectations are approximated by averaging over samples and using function approximation techniques to cope with the need to represent value functions over large state-action spaces.
