Searching in LeGEM leverages the collective intelligence [1]}, [2]} in OBMAS. We utilize TD learning to train MARL policies. The TD error is the difference between the TD target and the prediction. TD targets can be estimated with \(n\) -step target, TD(\(\lambda \) ) and other techniques [3]}, [4]}. Unfortunately, current \(n\) -step target and TD(\(\lambda \) ) methods are far from accurate estimating TD targets. They even incur underestimation with off-beat trajectories. In essence, to train MARL policies in OBMAS, one should accurately estimate the TD target where the reward plays the key role [5]}, [6]}. We resolve the aforementioned conundrum by redistributing rewards to their pivot timesteps. The key idea is that we can pull the outcome of one joint off-beat action back to the timestep when it was committed to the environment, which can dramatically enhance learning despite the long-term reward delays incurred by off-beat actions.
We utilize \(e_t\)  to update the reward of the transit \((s^{e_t}, \mathbf {\tilde{\text{$u$}}}^{e_t}, r^{e_t}, s^{e_t+1})\) :

\(\hat{r}^{e_t}=\mathbb {1}(e_t \ge t) \cdot r^{e_t}+ \mathbb {1}(e_t < t) \cdot r_{t}, ~\text{and}~ r_t=(1-\mathbb {1}(e_t < t) \cdot (1-\beta )) \cdot r_t. \) 
