Consider a fixed episode \(k\)  and step \(h\) . We define \(=\lbrace f_\theta :f_\theta (s,a)=\phi (s,a)^\top \theta \rbrace \)  where \(\theta \in \mathbb {R}^d\) , \(= \lbrace (s_h^\tau ,a_h^\tau ,r_h^\tau )\rbrace _{\tau \in [k-1]}\) , and \(R(f_\theta )=\Vert \theta \Vert ^2=\sum _{j=1}^d p_j(f_\theta )^2\)  where \(p_j(f_\theta )=e_j^\top \theta \)  with \(e_j\)  being the \(j\) -th standard basis vector. It is well known that linear function class satisfies Assumption REF  in linear MDP [1]}, [2]}.
We set \(\widehat{f}=\operatornamewithlimits{arg\,min}_{f\in } L(f\,|\,)^2+\lambda R(f)\)  to be \(f_{\widehat{\theta }}\) . Then we have
\(\widehat{\theta }&=\operatornamewithlimits{arg\,min}_\theta \sum _{\tau =1}^{k-1} (\phi (s_h^\tau ,a_h^\tau )^\top \theta -r_h^\tau )^2+\lambda \Vert \theta \Vert ^2 \\ &=(\Lambda ^k_h)^{-1}\sum _{\tau =1}^{k-1} r_h^\tau \phi (s_h^\tau ,a_h^\tau ),\) 
