Learning Algorithms from Bayesian Principles
Machine learning algorithms are commonly derived using ideas from optimization and statistics, followed by an extensive empirical efforts to make them practical as there is a lack of underlying principles to guide this process. In this paper, we present a learning rule derived from Bayesian principles, which enables us to connect a wide variety of learning algorithms. Using this rule, we can derive a wide range of learning algorithms in fields such as probabilistic graphical models, continuous optimization, and deep learning. This includes classical algorithms such as least squares, Newton's method, and Kalman filter, as well as modern deep learning algorithms such as stochastic gradient descent, RMSprop and Adam. Overall, we show that Bayesian principles not only unify, generalize, and improve existing learning algorithms, but also help us design new ones. [This is a working draft and a work in progress]