Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning
Deep reinforcement learning (RL) algorithms suffer severe performance
degradation when the interaction data is scarce, which limits their real-world
application. Recently, visual representation learning has been shown to be
effective and promising for boosting sample efficiency in RL. These methods
usually rely on contrastive learning and data augmentation to train a
transition model for state prediction, which is different from how the model is
used in RL--performing value-based planning. Accordingly, the learned model may
not be able to align well with the environment and generate consistent value
predictions, especially when the state transition is not deterministic. To
address this issue, we propose a novel method, called value-consistent
representation learning (VCR), to learn representations that are directly
related to decision-making. More specifically, VCR trains a model to predict
the future state (also referred to as the ''imagined state'') based on the
current one and a sequence of actions. Instead of aligning this imagined state
with a real state returned by the environment, VCR applies a $Q$-value head on
both states and obtains two distributions of action values. Then a distance is
computed and minimized to force the imagined state to produce a similar action
value prediction as that by the real state. We develop two implementations of
the above idea for the discrete and continuous action spaces respectively. We
conduct experiments on Atari 100K and DeepMind Control Suite benchmarks to
validate their effectiveness for improving sample efficiency. It has been
demonstrated that our methods achieve new state-of-the-art performance for
search-free RL algorithms.