Learning to Plan from Pixels.
Another related work is learning dynamics models for
model-based RL [1]}, [2]}.
Recent works have shown that deep networks can learn to plan directly from pixel observations in domains such as table-top manipulation [3]}, [4]},
navigation in VizDoom [5]}, and locomotion in joint space [6]}.
Universal Planning Networks (UPN) [4]} assumes the action space to be differentiable and uses a gradient descent planner to learn representations from expert demonstrations.
Prior work [8]} proposes the conjugate dynamics model to expedite the latent space learning, but suffers from compounding error. Without using explicit action supervision, causal InfoGAN [3]} extracts state representations by learning salient features.
Instead, our method builds a transformer-based model that is amenable to long-horizon planning, operating directly on real-world videos and handling the semantics of actions with sequential learning.
