Definition 4.3 (Linear MDP, [1]}, [2]}) 
We consider a linear Markov decision process, \(\mathrm {MDP}(, , H, \mathbb {P}, r)\)  with a feature map \(\phi : \times \rightarrow \mathbb {R}^d\) , where for any \((h,k) \in [H] \times [K]\) , there exist \(d\)  unknown (signed) measures \(\mu _h = (\mu _h^{(1)},\cdots , \mu _h^{(d)})\)  over \(\)  and an unknown vector \(w_h \in \mathbb {R}^d\) , such that for any \((s,a) \in \times \) , the following holds:
\(\mathbb {P}_h(s^{\prime }| s,a) = \langle \phi (s,a), \mu _h(s^{\prime })\rangle , \quad r_h(s,a) = \langle \phi (s,a), w_h \rangle .\) 
