
Offers a simple way to recognize and remain on an exploration `frontier.' Repeatedly visiting states that have not yet been visited many times can mean staying on the frontier of a part of the world that the agent has yet to explore. By frontier, we mean states of the world from which, if the agent takes a particular action, they can end up in a state of the environment that they have never experienced before. If the agent occasionally takes a random action,Many reinforcement learning agents occasionally take random actions [1]}. Such agents learn about the world and develop estimates about which actions will let them accumulate the best return, and take a best action (according to that metric) most of the time. However, the agent occasionally takes one of its other possible actions, just in case its estimates were wrong. This design element to take a random action is considered a type of exploration strategy, and has good properties for ensuring the agent tries all possible actions from any state an infinite number of times [2]}, at least if the agent has infinite time! Two popular examples are \(\epsilon \) -greedy exploration and soft-max/Boltzmann policy exploration. staying on a frontier makes it more likely that it will end up visiting unexplored parts of the world via such a random actionOr, with carefully designed search control, the agent could be biased to take actions it has never taken from a given state before. than if the agent largely stayed in the middle of the part of the world already explored.

Offers a way to check if an action results in a consistent reaction. Another perceived benefit of doing the same thing repeatedly is to check for consistency.
In Section REF , we described repeating a test of the floorboard to decide if it was the source of a peculiar noise. Similarly, for the experimental section of this paper, we completed multiple trials of each experiment because we are interested in patterns that hold over time, rather than one-time outliers. This benefit relates to an important assumption in many uses of reinforcement learning: that the world is a little bit random. When we want to estimate the value of a particular state, we are really interested in an average so we must observe a state multiple times to form a reasonable estimate. Because of this assumption, many exploration methods try to ensure the agent visits each state of its domain multiple times.
One way algorithm designers have encouraged agents to make exploratory visits to each state multiple times is through intrinsic rewards that decay over visits. This is an important area of study in exploration for reinforcement learning, and some notable approaches include Upper-Confidence bounds for Reinforcement Learning (UCRL, [3]} and UCRL2, [4]}), Model-based Interval Estimation (MBIE, [5]}, [6]}), and Random Network Distillation (RND, [7]})—even the early curiosity system by [8]} is based on a decaying bonus (assuming it is applied in a deterministic environment). The purpose of the decay is to only temporarily encourage visits to any given state, enough to obtain sufficient samples.

