Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control
Learned models and policies can generalize effectively when evaluated within
the distribution of the training data, but can produce unpredictable and
erroneous outputs on out-of-distribution inputs. In order to avoid distribution
shift when deploying learning-based control algorithms, we seek a mechanism to
constrain the agent to states and actions that resemble those that it was
trained on. In control theory, Lyapunov stability and control-invariant sets
allow us to make guarantees about controllers that stabilize the system around
specific states, while in machine learning, density models allow us to estimate
the training data distribution. Can we combine these two concepts, producing
learning-based control algorithms that constrain the system to in-distribution
states using only in-distribution actions? In this work, we propose to do this
by combining concepts from Lyapunov stability and density estimation,
introducing Lyapunov density models: a generalization of control Lyapunov
functions and density models that provides guarantees on an agent's ability to
stay in-distribution over its entire trajectory.