In SGLD, we add Gaussian noise to each step, \(\eta _t\) . This induces noise along with the learning rate that will then decay with each iteration. The idea is that SGLD allows the optimization to locate the mode (i.e., MAP); however, it never actually collapses to it due to the added noise. Moreover, it also does not allow the solution to leave the locality of the mode because the learning rate is decaying. Eventually, SGLD iterations will be randomly walking around the MAP collecting samples (which are theoretically proven to belong to the true posterior distribution; see proof in [1]}. Accordingly, there are two phases for the SGLD: (i) optimization in which iterations locate the mode, and (ii) sampling in which iterations walk around the mode. Ultimately, this allows us to collect parameter values during the sampling phase and thus enabling Bayesian learning to estimate the posterior distribution. Further, another nice aspect about SGLD is that we can do all this through an optimization framework with mini-batch data (i.e., subset of data). For this reason, SGLD is very fast, computationally effective, and overcomes curse of dimensionality.
