Formally, let \(\Pi _t\)  be the arm selection policy space at time \(t \in [T]\) , and \(\pi \in \Pi _t\)  be a specific policy. The implementation of policy \(\pi \)  is modeled by a soft intervention. Denoting by \(R(\pi )\)  the post-interventional value of the reward after performing the intervention, the expected reward under policy \(\pi \) , denoted by \(\mu _{\pi }\) , is given by \(\mathbb {E}[R(\pi )| \mathbf {x}_t]\) . According to the \(\sigma \) -calculus [1]}, it can be further decomposed as follows:
\(\begin{split}\mu _{\pi } = \mathbb {E}[R(\pi )| \mathbf {x}_{t}] = \sum _{\mathbf {a}} P_{\pi }(\mathbf {a}|\mathbf {x}_{t}) \cdot \mathbb {E}[R(\mathbf {a})|\mathbf {x}_{t}] = \mathbb {E}_{\mathbf {a}\sim \pi } \left[ \mathbb {E}[R(\mathbf {a})|\mathbf {x}_{t}] \right]\end{split}\) 
