The environment is identical to that in [1]}, with reward function parameters \((R_0,R_1,R_2)=(10^{-3},0.5,2)\)  for the standard variant of the grid and \((10^{-4},1.0,3.0)\)  for the harder variant. The models giving logits of \(P_F(-|s)\)  and \(P_B(-|s)\) , as well as \(\log F(s)\) , are MLPs of the same architecture as in [2]}, taking a one-hot representation of the coordinates of \(s\)  as input and sharing all layers except the last. The initial state flow \(\log Z=\log F(s_0)\)  is an independent parameter whose learning rate is set to \(10\times \)  the learning rate of other parameters.
