Adaptive Lambda Least Squares Temporal Difference Learning
Temporal Difference learning or TD(\lambda$ is a fundamental algorithm in the field of reinforcement learning. However, setting TD's \lambda$ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the \lambda$ selection problem as a bias variance trade off where the solution is the value of \lambda$ that leads to the smallest Mean Squared Value Error (MSVE) To solve this trade off we suggest applying Leave One Trajectory Out Cross Validation (LOTO CV) to search the space of \lambda$ values. Unfortunately, this approach is too computationally expensive for most practical applications. For Least Squares TD (LSTD) we show that LOTO CV can be implemented efficiently to automatically tune \lambda$ and apply function optimization methods to efficiently search the space of \lambda$ values. The resulting algorithm, ALLSTD, is parameter free and our experiments demonstrate that ALLSTD is significantly computationally faster than the na\"{i}ve LOTO CV implementation while achieving similar performance.
ArXiv