Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
Offline reinforcement learning (RL) aims at learning an optimal strategy
using a pre-collected dataset without further interactions with the
environment. While various algorithms have been proposed for offline RL in the
previous literature, the minimax optimal performance has only been (nearly)
achieved for tabular Markov decision processes (MDPs). In this paper, we focus
on offline RL with linear function approximation and propose two new
algorithms, SPEVI+ and SPMVI+, for single-agent MDPs and two-player zero-sum
Markov games (MGs), respectively. The proposed algorithms feature carefully
crafted data splitting mechanisms and novel variance-reduction pessimistic
estimators. Theoretical analysis demonstrates that they are capable of matching
the performance lower bounds up to logarithmic factors. As a byproduct, a new
performance lower bound is established for MGs, which tightens the existing
results. To the best of our knowledge, these are the first computationally
efficient and nearly minimax optimal algorithms for offline single-agent MDPs
and MGs with linear function approximation.