Stochastic Langevin Differential Inclusions with Applications to Machine Learning
Stochastic differential equations of Langevin-diffusion form have received
significant recent, thanks to their foundational role in both Bayesian sampling
algorithms and optimization in machine learning. In the latter, they serve as a
conceptual model of the stochastic gradient flow in training over-parametrized
models. However, the literature typically assumes smoothness of the potential,
whose gradient is the drift term. Nevertheless, there are many problems, for
which the potential function is not continuously differentiable, and hence the
drift is not Lipschitz-continuous everywhere. This is exemplified by robust
losses and Rectified Linear Units in regression problems. In this paper, we
show some foundational results regarding the flow and asymptotic properties of
Langevin-type Stochastic Differential Inclusions under assumptions appropriate
to the machine-learning settings. In particular, we show strong existence of
the solution, as well as asymptotic minimization of the canonical Free Energy
Functional.