We study the synthetic hypergrid environment introduced in [1]}. The set of interior states is a \(d\) -dimensional hypergrid of size \(H\times H\times \dots \times H\)  with a multimodal reward function concentrated near each of the \(2^d\)  corners of the hypergrid (see [1]}, [3]} and Fig. REF ). The initial state is \((0, 0, \dots , 0)\) , and each action is a step that increments one of the \(d\)  coordinates by 1 without leaving the grid. A special termination action is also allowed from each state.
This environment is designed to challenge a learning agent to infer and discover new modes from those that have been already been visited.
