Exploration is a fundamental aspect of Reinforcement Learning, typically implemented using stochastic action-selection. Exploration, however, can be more efficient if directed toward gaining new world knowledge. Visit-counters have been proven useful both in practice and in theory for directed exploration. However, a major limitation of counters is their locality. While there are a few model-based solutions to this shortcoming, a model-free approach is still missing.
 We propose $E$-values, a generalization of counters that can be used to evaluate the propagating exploratory value over state-action trajectories. We compare our approach to commonly used RL techniques, and show that using $E$-values improves learning and performance over traditional counters. We also show how our method can be implemented with function approximation to efficiently learn continuous MDPs. We demonstrate this by showing that our approach surpasses state of the art performance in the Freeway Atari 2600 game. "If there's a place you gotta go -I'm the one you need to know."(Map, Dora The Explorer)We consider Reinforcement Learning in a Markov Decision Process (MDP). An MDP is a fivetuple M = (S, A, P, R, γ) where S is a set of states and A is a set of actions. The dynamics of the process is given by P (s |s, a) which denotes the transition probability from state s to state s following action a. Each such transition also has a distribution R (r|s, a) from which the reward for such transitions is sampled. Given a policy π : S → A, a function -possibly stochastic -deciding which actions to take in each of the states, the state-action value function Q π : S × A → R satisfies: r,s ∼R×P (·|s,a) [r + γQ π (s , π (s ))] DISPLAYFORM0 where γ is the discount factor. The agent's goal is to find an optimal policy π * that maximizes Q π (s, π (s)). For brevity, Q π * Q * . There are two main approaches for learning π * . The first is a model-based approach, where the agent learns an internal model of the MDP (namely P and R). Given a model, the optimal policy could be found using dynamic programming methods such as Value Iteration BID19 . The alternative is a model-free approach, where the agent learns only the value function of states or state-action pairs, without learning a model BID5 1 .The ideas put forward in this paper are relevant to any model-free learning of MDPs. For concreteness, we focus on a particular example, Q-Learning BID23 BID19 . Q-Learning is a common method for learning Q * , where the agent iteratively updates its values of Q (s, a) by performing actions and observing their outcomes. At each step the agent takes action a t then it is transferred from s t to s t+1 and observe reward r. Then it applies the update rule regulated by a learning rate α: Q (s t , a t ) ← (1 − α) Q (s t , a t ) + α r + γ max a Q (s t+1 , a) .
We propose a generalization of visit-counters that evaluate the propagating exploratory value over trajectories, enabling efficient exploration for model-free RL
{"src": "scitldr-AIC_train", "paper_id": "ry1arUgCW"}