Inspired by backward induction to solve for SPEs [1]}, we train two RL policies, in two stages: in the first stage, for \(i \in [N]\) , each agents independently learns “subgame" policies \(\pi ^s_i\)  given a randomly sampled contracts \(\theta \sim P(\Theta )\)  (in our experiments, we take \(P\)  to be the uniform distribution on \(\Theta \) , since all contracts below are closed intervals in \(\mathbb {R}\) ). Then, we freeze the policy profile \(\pi ^s\) , and, taking \(i=1\)  as the proposing agent, we then train “negotiation" policies \(\pi ^n_1\)  to propose a contract \(\theta \in \Theta \) , and \(\pi ^n_i\)  for \(i > 1\)  to accept or reject the offered contract. The rewards for this second stage are computed by rolling out one episode of \(\pi ^s\)  from the chosen contract \(\theta \in \Theta \)  (or \(\emptyset \)  if \(\theta \)  was rejected). The overall algorithm is then assessed using the rewards gained by \(\pi ^n\)  at the end of training. In order to not use too many additional timesteps, we limit second stage training to only use 10% of the environment timesteps as in the first stage, or until convergence, whichever comes first. Algorithm  in apx:pseudocode shows the pseudocode for subgame learning.
