An Empirical Study of Implicit Regularization in Deep Offline RL
Deep neural networks are the most commonly used function approximators in
offline Reinforcement Learning these days. Prior works have shown that neural
nets trained with TD-learning and gradient descent can exhibit implicit
regularization that can be characterized by under-parameterization of these
networks. Specifically, the rank of the penultimate feature layer, also called
\textit{effective rank}, has been observed to drastically collapse during the
training. In turn, this collapse has been argued to reduce the model's ability
to further adapt in later stages of learning, leading to the diminished final
performance. Such an association between the effective rank and performance
makes effective rank compelling for offline RL, primarily for offline policy
evaluation. In this work, we conduct a careful empirical study on the relation
between effective rank and performance on three offline RL datasets : bsuite,
Atari, and DeepMind lab. We observe that a direct association exists only in
restricted settings and disappears in the more extensive hyperparameter sweeps.
Also, we empirically identify three phases of learning that explain the impact
of implicit regularization on the learning dynamics and found that
bootstrapping alone is insufficient to explain the collapse of the effective
rank. Further, we show that several other factors could confound the
relationship between effective rank and performance and conclude that studying
this association under simplistic assumptions could be highly misleading.