<commit_before>---
layout: post-light-feature
title: How does physics connect to machine learning?
description: "Did <NAME> help seed a key machine learning technique in the 60s?"
category: articles
tags: [physics, mean field methods, probabilistic models, probability models, ising models, exponential families, variational inference, autoencoder, neural networks, graphical models, inference, deep learning, machine learning]
image:
  feature: physics-machine-learning-figure.svg
  thumb: physics-machine-learning-thumb.png
published: true
---

I struggled to learn machine learning. I was used to variational tricks, MCMC samplers, and discreet Taylor expansions from years of physics training. Now the concepts were mixed up. The intuitive models of physical systems were replaced by abstract models of ‘data’ and amechanical patterns of cause and effect.

I had to fit these fields together. Physics and machine learning are intricately connected, but it is taking me years to make the overlaps precise. This process requires representing the new with the familiar, mapping jargon from one field to another.

A simple model of magnets---the Ising model---will help illustrate the rich connection between these fields. We first analyze this model with physics intuition. Then we derive the variational principle in physics and show that it recovers the same solution.

We then discover how that very same variational principle in physics opens a window into machine learning. We identify Boltzmann distributions as exponential families to make the mapping transparent, and show how approximate posterior inference is scaled to massive data thanks to the variational principle.

If you have a physics background, I hope you will have a better sense of machine learning and be able to read papers in the field. If you are a machine learner, I hope you will have the context to read a statistical physics paper about mean-field theory and the Ising model.

*If this article is confusing, falls short of these goals, or could be improved in any way please [email me](mailto:<EMAIL>), [@ me](https://twitter.com/thejaan), or [submit a pull request](https://github.com/altosaar/jaan.io/blob/master/_posts/blog/2017-08-11-how-does-physics-connect-machine-learning.md).*

### The Ising model, a physics perspective

Consider a lattice of spins that point up or down:

<center>
<figure>
{% asset ising-model.svg %}
</figure>
</center>

What features might make this a convincing model of magnetism?

Think about playing with magnets---if you put them close together, they pull each other closer. They repulse each other when their poles oppose, and if they’re far apart, they don't attract.

This means neighboring spins should affect each other in our model: if the spins around $$s_i$$ point upward, it should also want to point upward.

Let’s refer to the spin at location $$i$$ as $$s_i$$. A spin can be in one of two **states**: a spin can point up ($$s_i=+1$$) or down ($$s_i=-1$$).

We can capture our intuition about spins being attracted to each other (they want to point in the same direction) or repulsed (they want to point in opposite directions) by introducing a parameter $$J$$. This interaction parameter captures the **interaction strength** between spin $$i$$ and spin $$j$$.

If two neighboring spins point in the same direction, we'll have them contribute a term $$-J$$ to the total energy; if they point in opposing directions, they will contribute $$J$$.

This lets us write the **energy function**, or Hamiltonian, of the system:

$$E(s_1, s_2,...,s_N) = -\frac{1}{2}\sum_{i=1}^N\sum_{j=1}^NJ_{ij} s_i s_j$$

Here $$J_{ij} = J$$ if spins $$i$$ and $$j$$ are neighbors, and $$J_{ij} = 0$$ otherwise. The factor of $$\frac{1}{2}$$ in front is to account for double counting from the sums over both $$i$$ and $$j$$. Note that the system has finitely many spins ($$N$$ spins). 

A spin **configuration** or state of the system is a specific setting of values for all spins. The set $$\{s_1=+1, s_2=+1, s_3=-1, ..., s_N=+1\}$$ is an example of a configuration.

The second law of thermodynamics says that at a fixed temperature and entropy, a system will seek configurations that minimize its energy. This lets us reason about interactions.

If the interaction strength $$J$$ is zero, the spins do not interact and the system has the same energy, zero, for all configurations (i.e. the energy is trivially minimized). But if the interaction strength $$J$$ is positive, the spins will tend to align to minimize the energy of the system $$E(s_1, s_2,...,s_N)$$. This corresponds to minimization because of the minus sign convention in front of the sum in the energy function.

Now let’s introduce a **magnetic field** $$H$$. Imagine the lattice of spins immersed in a magnetic field, perhaps the ambient field from the earth's crust. The magnetic field affects every spin independently, and each spin will try to align with the field. We can include the magnetic field in the energy of the system by summing the independent contributions for each spin:

$$E(s_1, s_2,...,s_N) = -\frac{1}{2}\sum_{i, j} J_{ij}s_i s_j - H\sum_i s_i$$

We can reason about the magnetic field strength $$H$$ by imagining what happens if it is large or small (strong or weak). If $$H$$ is large and the interactions between spins are weak, the magnetic field term will dominate and the spins will align with the magnetic field to minimize energy. But if the magnetic field is small, it is more difficult to reason about.

Now that we have defined the Ising model and its characteristics, let’s think about our goals. What questions can we answer about this Ising model? For example, if we observe the system, what state will it be in---what are the most likely spin configurations? What is the average magnetization?


### The Boltzmann distribution

Can we make our goals more precise and make math from words? To do this, we need to define a distribution over spin configurations. It is straightforward to derive the probability of finding the system in an equilibrium state [^1]:

[^1]: [Derivation](http://www.physics.udel.edu/~glyde/PHYS813/Lectures/chapter_3.pdf)

$$p(s_1, s_2,...,s_N) = \frac{e^{-\beta E(s_1, s_2,...,s_N)}}{Z}$$

This is the **Boltzmann distribution**. The numerator is called the Boltzmann factor for a particular configuration. This factor gives high or low weight to a specific state of the system according to the energy for that state.

We query the Boltzmann distribution at a specific configuration of spins to get the probability of finding the system in this state.

For example, say the first spins in our configuration happen to be up, up, down, etc. We plug this in and get  $$p(s_1=+1, s_2=+1, s_3=-1,...,s_N=+1)=0.7321$$. This means this state was pretty likely.

This distribution behaves intuitively: low energy states are more probable than configurations with high energy. For example, if $$J=+1$$, the spins will align, and the state where all spins point in the same direction is most probable. Why? Because it leads to the most negative energy function, which corresponds to the Boltzmann factor with the largest weight.

The parameter $$\beta$$ is proportional to the inverse temperature, $$\beta = \frac{1}{k_BT}$$ and is used for notational convenience. (Specifically, it includes the constant $$k_B$$ to make the probability density dimensionless.) Temperature affects the model by controlling how important the interactions are. If $$T\rightarrow \infty$$ we are at a high temperature, and the inverse temperature is small with $$\beta \ll 1$$, so the interaction strength $$J$$ is not important. But at low temperatures, the inverse temperature is small and $$J$$ divided by a small number is big, so interactions have a large effect on the system’s behavior.

### The partition function

The denominator $$Z$$ is of utmost importance. It ensures that the distribution integrates to $$1$$ and is thus a valid probability distribution. We need this normalization to calculate properties of the system. Calculating mean values and other moments can only be done with a probability mass function. The name of $$Z$$ is "**partition function**" or "normalizing constant". It is the sum of each state's Boltzmann factor:

$$Z = \sum_{s_1=\pm1}\sum_{s_2=\pm1}...\sum_{s_N=\pm1}e^{-\beta E(s_1, s_2, ..., s_N)}$$

I explicitly wrote out the sum to illustrate why we can’t evaluate this distribution: we need to sum over all possible configurations. Each spin has two states, and there are $$N$$ spins. This leads to $$2^N$$ terms in the sum. For a small system with a hundred spins, this is already greater than the number of atoms in the universe so we can never hope to calculate it [^2].

[^2]: For a tiny system, e.g. with three spins, we have $$8$$ states and the sum is doable - but the system is uninteresting.

### Using the Boltzmann distribution to calculate properties of the system

We arrived at a probability distribution describing which states of the system are likely, however we were stumped by the intractable partition function. Let's temporarily assume we have infinite computation and *can* calculate the Boltzmann distribution's partition function. What are some interesting things we can learn about the system from it's Boltzmann distribution?

This distribution lets us to calculate properties of the system as a whole by taking **expectations** (i.e. calculating observable quantities). For example, the magnetization $$m$$ is the average magnetization over all spins:

$$m = \frac{1}{N} \langle  s_1 + s_1 + ... + s_N \rangle = \langle  s_i \rangle$$

Why should we care about this magnetization? It tells us about the system as a whole, the macrostate, rather than a specific microstate. We lose specificity because we can't say anything about the first spin $$s_1$$, but we learn about how it behaves across all possible states of the rest of the spins. <commit_msg>fix parameters<commit_after>---
layout: post-light-feature
title: How does physics connect to machine learning?
description: "Did <NAME> help seed a key machine learning technique in the 60s?"
category: articles
tags: [physics, mean field methods, probabilistic models, probability models, ising models, exponential families, variational inference, autoencoder, neural networks, graphical models, inference, deep learning, machine learning]
image:
  feature: physics-machine-learning-figure.svg
  thumb: physics-machine-learning-thumb.png
published: true
---

_Mandarin translation available - 用普通话阅读这篇文章: [WeChat](http://mp.weixin.qq.com/s?__biz=MjM5MTQzNzU2NA==&mid=2651658123&idx=1&sn=5f0c4bdfcb11e623e93874387f5e6d19&chksm=bd4c30188a3bb90e536612140d0ccf52e511aaaee148b0e7e7ce351c198938d5ff6b855d613c&scene=0#rd)_

I struggled to learn machine learning. I was used to variational tricks, MCMC samplers, and discreet Taylor expansions from years of physics training. Now the concepts were mixed up. The intuitive models of physical systems were replaced by abstract models of ‘data’ and amechanical patterns of cause and effect.

I had to fit these fields together. Physics and machine learning are intricately connected, but it is taking me years to make the overlaps precise. This process requires representing the new with the familiar, mapping jargon from one field to another.

A simple model of magnets---the Ising model---will help illustrate the rich connection between these fields. We first analyze this model with physics intuition. Then we derive the variational principle in physics and show that it recovers the same solution.

We then discover how that very same variational principle in physics opens a window into machine learning. We identify Boltzmann distributions as exponential families to make the mapping transparent, and show how approximate posterior inference is scaled to massive data thanks to the variational principle.

If you have a physics background, I hope you will have a better sense of machine learning and be able to read papers in the field. If you are a machine learner, I hope you will have the context to read a statistical physics paper about mean-field theory and the Ising model.

*If this article is confusing, falls short of these goals, or could be improved in any way please [email me](mailto:<EMAIL>), [@ me](https://twitter.com/thejaan), or [submit a pull request](https://github.com/altosaar/jaan.io/blob/master/_posts/blog/2017-08-11-how-does-physics-connect-machine-learning.md).*

### The Ising model, a physics perspective

Consider a lattice of spins that point up or down:

<center>
<figure>
{% asset ising-model.svg %}
</figure>
</center>

What features might make this a convincing model of magnetism?

Think about playing with magnets---if you put them close together, they pull each other closer. They repulse each other when their poles oppose, and if they’re far apart, they don't attract.

This means neighboring spins should affect each other in our model: if the spins around $$s_i$$ point upward, it should also want to point upward.

Let’s refer to the spin at location $$i$$ as $$s_i$$. A spin can be in one of two **states**: a spin can point up ($$s_i=+1$$) or down ($$s_i=-1$$).

We can capture our intuition about spins being attracted to each other (they want to point in the same direction) or repulsed (they want to point in opposite directions) by introducing a parameter $$J$$. This interaction parameter captures the **interaction strength** between spin $$i$$ and spin $$j$$.

If two neighboring spins point in the same direction, we'll have them contribute a term $$-J$$ to the total energy; if they point in opposing directions, they will contribute $$J$$.

This lets us write the **energy function**, or Hamiltonian, of the system:

$$E(s_1, s_2,...,s_N) = -\frac{1}{2}\sum_{i=1}^N\sum_{j=1}^NJ_{ij} s_i s_j$$

Here $$J_{ij} = J$$ if spins $$i$$ and $$j$$ are neighbors, and $$J_{ij} = 0$$ otherwise. The factor of $$\frac{1}{2}$$ in front is to account for double counting from the sums over both $$i$$ and $$j$$. Note that the system has finitely many spins ($$N$$ spins). 

A spin **configuration** or state of the system is a specific setting of values for all spins. The set $$\{s_1=+1, s_2=+1, s_3=-1, ..., s_N=+1\}$$ is an example of a configuration.

The second law of thermodynamics says that at a fixed temperature and entropy, a system will seek configurations that minimize its energy. This lets us reason about interactions.

If the interaction strength $$J$$ is zero, the spins do not interact and the system has the same energy, zero, for all configurations (i.e. the energy is trivially minimized). But if the interaction strength $$J$$ is positive, the spins will tend to align to minimize the energy of the system $$E(s_1, s_2,...,s_N)$$. This corresponds to minimization because of the minus sign convention in front of the sum in the energy function.

Now let’s introduce a **magnetic field** $$H$$. Imagine the lattice of spins immersed in a magnetic field, perhaps the ambient field from the earth's crust. The magnetic field affects every spin independently, and each spin will try to align with the field. We can include the magnetic field in the energy of the system by summing the independent contributions for each spin:

$$E(s_1, s_2,...,s_N) = -\frac{1}{2}\sum_{i, j} J_{ij}s_i s_j - H\sum_i s_i$$

We can reason about the magnetic field strength $$H$$ by imagining what happens if it is large or small (strong or weak). If $$H$$ is large and the interactions between spins are weak, the magnetic field term will dominate and the spins will align with the magnetic field to minimize energy. But if the magnetic field is small, it is more difficult to reason about.

Now that we have defined the Ising model and its characteristics, let’s think about our goals. What questions can we answer about this Ising model? For example, if we observe the system, what state will it be in---what are the most likely spin configurations? What is the average magnetization?


### The Boltzmann distribution

Can we make our goals more precise and make math from words? To do this, we need to define a distribution over spin configurations. It is straightforward to derive the probability of finding the system in an equilibrium state [^1]:

[^1]: [Derivation](http://www.physics.udel.edu/~glyde/PHYS813/Lectures/chapter_3.pdf)

$$p(s_1, s_2,...,s_N) = \frac{e^{-\beta E(s_1, s_2,...,s_N)}}{Z}$$

This is the **Boltzmann distribution**. The numerator is called the Boltzmann factor for a particular configuration. This factor gives high or low weight to a specific state of the system according to the energy for that state.

We query the Boltzmann distribution at a specific configuration of spins to get the probability of finding the system in this state.

For example, say the first spins in our configuration happen to be up, up, down, etc. We plug this in and get  $$p(s_1=+1, s_2=+1, s_3=-1,...,s_N=+1)=0.7321$$. This means this state was pretty likely.

This distribution behaves intuitively: low energy states are more probable than configurations with high energy. For example, if $$J=+1$$, the spins will align, and the state where all spins point in the same direction is most probable. Why? Because it leads to the most negative energy function, which corresponds to the Boltzmann factor with the largest weight.

The parameter $$\beta$$ is proportional to the inverse temperature, $$\beta = \frac{1}{k_BT}$$ and is used for notational convenience. (Specifically, it includes the constant $$k_B$$ to make the probability density dimensionless.) Temperature affects the model by controlling how important the interactions are. If $$T\rightarrow \infty$$ we are at a high temperature, and the inverse temperature is small with $$\beta \ll 1$$, so the interaction strength $$J$$ is not important and has little effect. But at low temperatures, the inverse temperature is large, so interactions have a large effect on the system’s behavior.

### The partition function

The denominator $$Z$$ is of utmost importance. It ensures that the distribution integrates to $$1$$ and is thus a valid probability distribution. We need this normalization to calculate properties of the system. Calculating mean values and other moments can only be done with a probability mass function. The name of $$Z$$ is "**partition function**" or "normalizing constant". It is the sum of each state's Boltzmann factor:

$$Z = \sum_{s_1=\pm1}\sum_{s_2=\pm1}...\sum_{s_N=\pm1}e^{-\beta E(s_1, s_2, ..., s_N)}$$

I explicitly wrote out the sum to illustrate why we can’t evaluate this distribution: we need to sum over all possible configurations. Each spin has two states, and there are $$N$$ spins. This leads to $$2^N$$ terms in the sum. For a small system with a hundred spins, this is already greater than the number of atoms in the universe so we can never hope to calculate it [^2].

[^2]: For a tiny system, e.g. with three spins, we have $$8$$ states and the sum is doable - but the system is uninteresting.

### Using the Boltzmann distribution to calculate properties of the system

We arrived at a probability distribution describing which states of the system are likely, however we were stumped by the intractable partition function. Let's temporarily assume we have infinite computation and *can* calculate the Boltzmann distribution's partition function. What are some interesting things we can learn about the system from it's Boltzmann distribution?

This distribution lets us to calculate properties of the system as a whole by taking **expectations** (i.e. calculating observable quantities). For example, the magnetization $$m$$ is the average magnetization over all spins:

$$m = \frac{1}{N} \langle  s_1 + s_1 + ... + s_N \rangle = \langle  s_i \rangle$$

Why should we care about this magnetization? It tells us about the system as a whole, the macrostate, rather than a specific microstate. We lose specificity because we can't say anything about the first spin $$s_1$$, but we learn about how it behaves across all possible states of the rest of the spins. 
{'fraction_non_alphanumeric': 0.07881381231963379, 'fraction_numerical': 0.011493680963279929, 'mean_word_length': 4.609545073960368, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4264529', 'n_tokens_mistral': 5385, 'n_tokens_neox': 5002, 'n_words': 2987}