Lemma C.4 (Lemma B.1 in [1]}) 
Under Definition  REF  of linear MDP, for any fixed policy \(\pi \) , let \(\lbrace \theta _h^\pi \rbrace _{h \in [H]}\)  be the corresponding weights such that \(Q_h^\pi (s,a) = \langle \phi (s,a),\theta _h^\pi \rangle \)  for all \((s,a,h)\in \times \times [H]\) . Then for all \(h\in [H]\) , we have
\(\Vert \theta _h^\pi \Vert  \le 2H\sqrt{d}.\) 

