In reinforcement learning, it is common to let an agent interact with its environment for a fixed amount of time before resetting the environment and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed amount of time, or (ii) an indefinite period where the time limit is only used during training. In this paper, we investigate theoretically how time limits could effectively be handled in each of the two cases. In the first one, we argue that the terminations due to time limits are in fact part of the environment, and propose to include a notion of the remaining time as part of the agent's input. In the second case, the time limits are not part of the environment and are only used to facilitate learning. We argue that such terminations should not be treated as environmental ones and propose a method, specific to value-based algorithms, that incorporates this insight by continuing to bootstrap at the end of each partial episode. To illustrate the significance of our proposals, we perform several experiments on a range of environments from simple few-state transition graphs to complex control tasks, including novel and standard benchmark domains. Our results show that the proposed methods improve the performance and stability of existing reinforcement learning algorithms. The reinforcement learning framework BID19 BID2 BID21 BID10 involves a sequential interaction between an agent and its environment. At every time step t, the agent receives a representation S t of the environment's state, selects an action A t that is executed in the environment which in turn provides a representation S t+1 of the successor state and a reward signal R t+1 . An individual reward received by the agent does not directly indicate the quality of its latest action as some rewards may indeed be the consequence of a series of actions taken far in advance. Thus, the goal of the agent is to learn a good policy by maximizing the discounted sum of future rewards also known as return: DISPLAYFORM0 A discount factor 0 ≤ γ < 1 is necessary to exponentially decay the future rewards ensuring bounded returns. While the series is infinite, it is common to use this expression even in the case of possible terminations. Indeed, episode terminations can be considered to be the entering of an absorbing state that transitions only to itself and generates zero rewards thereafter. However, when the maximum length of an episode is predefined, it is easier to rewrite the expression above by explicitly including the time limit T : DISPLAYFORM1 Optimizing for the expectation of the return specified in Equation 2 is suitable for naturally timelimited tasks where the agent has to maximize its expected return G 0:T over a fixed episode length only. In this case, since the return is bounded, a discount factor of γ = 1 can be used. However, in practice it is still common to keep γ smaller than 1 in order to give more priority to short-term rewards. Under this optimality model, the objective of the agent does not go beyond the time limit. Therefore, an agent optimizing under this model should ideally learn to take more risky actions that reveal higher expected return than safer ones as approaching the end of the time limit. In Section 2, we study this case and illustrate that due to the presence of the time limit, the remaining time is present in the environment's state and is essential to its Markov property BID19 . Therefore, we propose to include a notion of the remaining time in the agent's input, an approach that we refer to as time-awareness. We describe various general scenarios where lacking a notion of the remaining time can lead to suboptimal policies and instability, and demonstrate significant performance improvements for time-aware agents.Optimizing for the expectation of the return specified by Equation 1 is relevant for time-unlimited tasks where the interaction is not limited in time by nature. In this case, the agent has to maximize its expected return over an indefinite (e.g. infinite) period. However, it is desirable to use time limits in order to diversify the agent's experience. For example, starting from highly diverse states can avoid converging to suboptimal policies that are limited to a fraction of the state space. In Section 3, we show that in order to learn good policies that continue beyond the time limit, it is important to differentiate between the terminations that are due to time limits and those from the environment. Specifically, for value-based algorithms, we propose to continue bootstrapping at states where termination is due to the time limit, or generally any other causes other than the environmental ones. We refer to this method as partial-episode bootstrapping. We describe various scenarios where having a time limit can facilitate learning, but where the aim is to learn optimal policies for indefinite periods, and demonstrate that our method can significantly improve performance.We evaluate the impact of the proposed methods on a range of novel and popular benchmark domains using a deep reinforcement learning BID0 BID9 algorithm called the Proximal Policy Optimization (PPO), one which has recently been used to achieve stateof-the-art performance in many domains BID17 BID8 . We use the OpenAI Baselines 1 implementation of the PPO algorithm with the hyperparameters reported by BID17 , unless explicitly specified. All novel environments are implemented using the OpenAI Gym framework BID3 and the standard benchmark domains are from the MuJoCo BID22 Gym collection. We modified the TimeLimit wrapper to include remaining time in the observations for the proposed time-aware agent and a flag to separate timeout terminations from environmental ones for the proposed partial-episode bootstrapping agent. For every task involving PPO, to have perfect reproducibility, we used the same 40 seeds from 0 to 39 to initialize the pseudo-random number generators for the agents and environments. Every 5 training cycles (i.e. 10240 time steps), we perform an evaluation on a complete episode and store the sums of rewards, discounted returns, and estimated state-values. For generating the performance plots, we average the values across all runs and then apply smoothing with a sliding window of size 10. The performance graphs show these smoothed averages as well as their standard error.We empirically show that time-awareness significantly improves the performance and stability of PPO for the time-limited tasks and can sometimes result in quite interesting behaviors. For example, in the Hopper-v1 domain with T = 300, our agent learns to efficiently jump forward and fall towards the end of its time in order to maximize its travelled distance and achieve a "photo finish". For the time-unlimited tasks, we show that bootstrapping at the end of partial episodes allows to significantly outperform the standard PPO. In particular, on Hopper-v1, even if trained with episodes of only 200 steps, our agent manages to learn to hop for at least 10 6 time steps, resulting in more than two hours of rendered video. Detailed results for all variants of the tasks using PPO with and without the proposed methods are available in the Appendix. The source code will be made publicly available shortly. A visual depiction of highlights of the learned behaviors can be viewed at the address sites.google.com/view/time-limits-in-rl. We showed in Section 2 that time-awareness is required for correct credit assignment in domains where the agent has to optimize its performance over a time-limited horizon. However, even without the knowledge of the remaining time, reinforcement learning agents still often manage to perform relatively well. This could be due to several reasons including: (1) If the time limit is sufficiently long that terminations due to time limits are hardly ever experienced-for instance, in the Arcade Learning Environment (ALE) BID1 BID13 BID12 . In deep learning BID11 BID15 , it is highly common to use a stack of previous observations or recurrent neural networks (RNNs) BID6 to address scenarios with partial observations BID25 ). These solutions may to an extent help when the remaining time is not included as part of the agent's input. However, they are much more complex architectures and are only next-best solutions, while including a notion of the remaining time is quite simple and allows better diagnosis of the learned policies. The proposed approach is quite generic and can potentially be applied to domains with varying time limits where the agent has to learn to generalize as the remaining time approaches zero. In real-world applications such as robotics the proposed approach could easily be adapted by using the real time instead of simulation time steps.In order for the proposed partial-episode bootstrapping in Section 3 to work, as is the case for valuebased methods in general, the agent needs to bootstrap from reliable estimated predictions. This is in general resolved by enabling sufficient exploration. However, when the interactions are limited in time, exploration of the full state-space may not be feasible from some fixed starting states. Thus, a good way to allow appropriate exploration in such domains is to sufficiently randomize the initial states. It is worth noting that the proposed partial-episode bootstrapping is quite generic in that it is not restricted to partial episodes caused only due to time limits. In fact, this approach is valid for any early termination causes. For instance, it is common in the curriculum learning literature to start from near the goal states (easier tasks), and gradually expand to further states (more difficult tasks) BID5 . In this case, it can be helpful to stitch the learned values by terminating the episodes and bootstrapping as soon as the agent enters a state that is already well known.Since the proposed methods were shown to enable to better optimize for the time-limited and timeunlimited domains, we believe that they have the potential to improve the performance and stability of a large number of existing reinforcement learning algorithms. We also propose that, since reinforcement learning agents are in fact optimizing for the expected returns, and not the undiscounted sum of rewards, it is more appropriate to consider this measure for performance evaluation. We considered the problem of learning optimal policies in time-limited and time-unlimited domains using time-limited interactions. We showed that when learning policies for time-limited tasks, it is important to include a notion of the remaining time as part of the agent's input. Not doing so can cause state-aliasing which in turn can result in suboptimal policies, instability, and slower convergence. We then showed that, when learning policies that are optimal for time-unlimited tasks, it is more appropriate to continue bootstrapping at the end of the partial episodes when termination is due to time limits, or any other early termination causes other than environmental ones. In both cases, we illustrated that our proposed methods can significantly improve the performance of PPO and allow to optimize more directly, and accurately, for either of the optimality models. Reacher-v1, = 1.0, training limit = 50, evaluation limit = 50
We consider the problem of learning optimal policies in time-limited and time-unlimited domains using time-limited interactions.
{"src": "scitldr-AIC_train", "paper_id": "HyDAQl-AW"}