For our posterior sampling algorithm, we use Bayesian Linear Regression (BLR) with action centering for our reward approximating function [1]}. BLR with action centering has a closed-form posterior update, only requires access to treatment effect features at decision times, and is robust to miss-specification of the correctness of the baseline reward model [1]}.
Specifically, the BLR with action centering posterior sampling algorithm uses the following model of the reward:
\(R_{i, t} = m(S_{i, t})^T \alpha _0 + \pi _{i,t} f(S_{i, t})^T \alpha _1 \\+ (A_{i, t} - \pi _{i, t}) f(S_{i, t})^T \beta + \epsilon \) 
