Once an action \(a\)  is applied to the state \(s\) , the simulator outputs the values of \(Q_p\) , \(kW_p\) , and \(H_p\)  for the respective timestep \(t\)  (see details in  REF ). These values can then used to calculate a reward \(r(t)\) . Furthermore, a new tank level is observed at \(t+1\) , corresponding to state \(s^{\prime }\) . Thus we have a loop in which transitions are generated and stored in the Experience Replay [1]}, implemented with prioritization mechanism [2]}. Finally, we can use these experiences to learn a new policy \(\pi _a\) , different from the behavioral policy \(\pi _b\)  which generated the dataset. To do so, we provide an implementation of Random Ensemble Mixture (REM) [3]} as a learning algorithm. REM is dropout inspired convex combination of \(K\)  parametric Q-functions aiming to mitigate overestimation in offline RL settings. The underlying idea is for each policy update, generate a set of \(k\)  random weights \(\alpha \)  such that \(Q(s,a)^{k}_{\theta } = \lbrace \alpha _{1}Q(s,a)_{\theta _{1}} + \alpha _{2}Q(s,a)_{\theta _{2}} + \alpha _{3}Q(s,a)_{\theta _{3}}+ ... + \alpha _{k}Q(s,a)_{\theta _{k}}\rbrace \)  and \(\sum _{k}\alpha _{k} = 1\) .
