Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
World models in model-based reinforcement learning usually face unrealistic
long-time-horizon prediction issues due to compounding errors as the prediction
errors accumulate over timesteps. Recent works in graph-structured world models
improve the long-horizon reasoning ability via building a graph to represent
the environment, but they are designed in a goal-conditioned setting and cannot
guide the agent to maximize episode returns in a traditional reinforcement
learning setting without externally given target states. To overcome this
limitation, we design a graph-structured world model in offline reinforcement
learning by building a directed-graph-based Markov decision process (MDP) with
rewards allocated to each directed edge as an abstraction of the original
continuous environment. As our world model has small and finite state/action
spaces compared to the original environment, value iteration can be easily
applied here to estimate state values on the graph and figure out the best
future. Unlike previous graph-structured world models that requires externally
provided targets, our world model, dubbed Value Memory Graph (VMG), can provide
the desired targets with high values by itself. VMG can be used to guide
low-level goal-conditioned policies that are trained via supervised learning to
maximize episode returns. Experiments on the D4RL benchmark show that VMG can
outperform state-of-the-art methods in several tasks where long horizon
reasoning ability is crucial. Code will be made publicly available.