We consider the Cleanup domain, again from [1]}. Here, agents similarly move along a square grid to consume apples with reward 1, but here, apples are not spawned by having nearby apples, but by first clearing out pollution from a nearby river (a task which is not given any reward). Thus, the dilemma comes from other agents eating apples while you an agent is cleaning the river in order to generate those apples. Again, we use engineered features here (specifically, we use agent position + orientation, closest agent position + orientation, closest apple position, closest waste position, and the number of current apples and current waste points on the map), and again we do not use a punishing beam.
