KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images
We present KeyCLD, a framework to learn Lagrangian dynamics from images.
Learned keypoints represent semantic landmarks in images and can directly
represent state dynamics. Interpreting this state as Cartesian coordinates
coupled with explicit holonomic constraints, allows expressing the dynamics
with a constrained Lagrangian. Our method explicitly models kinetic and
potential energy, thus allowing energy based control. We are the first to
demonstrate learning of Lagrangian dynamics from images on the dm_control
pendulum, cartpole and acrobot environments. This is a step forward towards
learning Lagrangian dynamics from real-world images, since previous work in
literature was only applied to minimalistic images with monochromatic shapes on
empty backgrounds. Please refer to our project page for code and additional
results: https://rdaems.github.io/keycld/