<jupyter_start><jupyter_text>Machine learning the phase transition in the 2D Ising model*Authors: <NAME> (University of California, Berkeley)*![ML model](https://raw.githubusercontent.com/enze-chen/learning_modules/master/fig/ML_Ising_schematic.png)This notebook teaches you how to use machine learning (ML) models to learn the phases in the 2D ferromagnetic Ising model on a square lattice. In particular we'll use **logistic regression** and a **feed-forward neural network** (FFNN). This notebook will cover the entire pipeline, including:1. Generating the training data using Monte Carlo (MC).1. Constructing and training the ML models. 1. Making predictions and visualizations with the ML models.I have tried to keep the code simple and explanations plentiful so that someone who is comfortable with Python and computational MSE can understand everything. Apologies that the notebook is quite long. While the *implementation* of the code isn't necessarily hard thanks to several wonderful Python libraries, the *theory* behind it—particularly the ML portions—can be a little tricky.  How to run this notebookIf you are viewing this notebook on Google Colaboratory, then all the software is already set up for you (hooray). If you want to run the notebook locally, make sure all the Python libraries in the [`requirements.txt`](https://github.com/enze-chen/learning_modules/blob/master/requirements.txt) file are installed.For pedagogical reasons, there are a few sections for you to complete the code in order to run the simulation. These are delineated with the dashed lines as follows, and you should **only change what's inside**. You don't have to edit the text or code anywhere else. I've also included "**TODO**" to separate the background context from the actual instructions.```python ----------------------  YOUR CODE HERE ---------------------- ```If you edit the code in a cell, just press `Shift+Enter` to run it again. You have to execute **all** the code cells in this notebook from top to bottom (so don't skip around). A number `[]` will appear to the left of the code cell once it's done executing. When all done successfully, you should be able to see a few accuracy scores of your model and a few probability plots like the one in the upper-right of this notebook. AcknowledgementsThis notebook was inspired by the recent work of [<NAME> <NAME>. *Nature Physics*, **13**, 2017](https://www.nature.com/articles/nphys4035) and [Mehta et al. *arXiv:1803.08823*, 2018](https://arxiv.org/abs/1803.08823). I also drew inspiration from [<NAME>'s tutorial in Julia](https://juliaphysics.github.io/PhysicsTutorials.jl/tutorials/machine_learning/ml_ising/ml_ising.html). I also thank my advisor [Prof. <NAME>](https://mse.berkeley.edu/people_new/asta/) for encouraging me in my education-related pursuits. An interactive version of this notebook can be found online at [Google Colaboratory](https://colab.research.google.com/github/enze-chen/learning_modules/blob/master/mse/Machine_learning_Ising_model.ipynb).   Important equations Ising model I assume that you're familiar with the Ising model and how to simulate the phase transition using Monte Carlo. If not, you can check out [my other notebook](https://github.com/enze-chen/learning_modules/blob/master/mse/Monte_Carlo_Ising_model.ipynb) or countless other resources, such as the textbooks by [Newman and Barkema](https://global.oup.com/academic/product/monte-carlo-methods-in-statistical-physics-9780198517979?cc=us&lang=en&) or [Landau and Binder](https://www.cambridge.org/core/books/guide-to-monte-carlo-simulations-in-statistical-physics/2522172663AF92943C625056C14F6055).The most important takeaway is that in the 2D Ising model on a square lattice, there is a critical temperature for a magnetic phase transition that has a theoretical value of:$$ T_c = \frac{2J}{k_B \ln \left( 1 + \sqrt{2} \right)} \approx 2.269 $$This phase transition can be simulated fairly accurately using Monte Carlo in the limit of large system sizes and accurate sampling. Therefore, we can then ask ourselves the following question: **Is it possible to use ML to predict the phase of the system purely from the raw spin configurations?** The goal of this notebook is to show you that the answer is "yes."First, we will use a standard Metropolis-Hastings MC algorithm to generate some data. Then we will split the data into ordered ($T \ll T_C$), disordered ($T \gg T_c$), and critical ($T \approx T_c$) subsets. We'll use most of the ordered and disordered data for the training data and we'll use everything else for the test data.   Machine learning![Sigmoid](https://raw.githubusercontent.com/enze-chen/learning_modules/master/fig/sigmoid.png)The first ML model we will try is [**logistic regression**](https://en.wikipedia.org/wiki/Logistic_regression), which is one of the simplest [linear models](https://en.wikipedia.org/wiki/Generalized_linear_model) for binary classification. It is parameterized by a set of weights $\vec{w}$ and outputs a probability $g \in (0, 1)$ according to the **logistic function** (also known as the **sigmoid function**), which looks like the image above and is parameterized as follows:$$ g(\vec{x}; \vec{w}) = \dfrac{1}{1 + \exp \left( -\vec{w}^{\top} \vec{x} \right)} $$The predicted label is then $0$ if $g < 0.5$ and $1$ otherwise. Training the logistic regression model involves supplying labeled data with labels in the set $\{0, 1\}$ and optimizing the parameters $\vec{w}$.![Neuron math](https://raw.githubusercontent.com/enze-chen/learning_modules/master/fig/neuron_math.png)The second ML model we will try is a [**feed-forward neural network**](https://en.wikipedia.org/wiki/Feedforward_neural_network) (also known as a **multilayer perceptron**), which is one of the simplest neural network (NN) models. A NN consists of layers of neurons (a single neuron is shown above) that pass information from one layer to the next according to two steps:1. The inputs $\vec{x}^{[i-1]}$ from the previous layer $\ell_{i-1}$ are multiplied by a matrix of weights $W^{[i-1]}$ and summed with a vector of biases $\vec{b}^{[i-1]}$ to produce a vector $\vec{z}^{[i]}$ in the current layer $\ell_{i}$ according to:$$ \vec{z}^{[i]} = W^{[i-1]} \vec{x}^{[i-1]} + \vec{b}^{[i-1]} $$2. The result is then passed through a **non-linear** activation function $g(\cdot)$ in the current layer $\ell_{i}$ according to:$$\vec{a}^{[i]} = g(\vec{z}^{[i]})$$where $\vec{a}^{[i]}$ then becomes the inputs $\vec{x}^{[i]}$ for the next layer $\ell_{i+1}$. The square bracket superscripts $[i]$ denote the layer that the variable is associated with. In this notebook, $g(\cdot)$ will be the [rectified linear unit (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) that is applied element-wise. Note that the above image and the math contained within is for a *single neuron*, whereas the math presented in Markdown is for a *layer of neurons* and so the scalar/vector/matrix expressions are slightly different.Training the NN model involves supplying the same labeled data as for logistic regression and optimizing the parameters $W^{[\cdot]}$ and $\vec{b}^{[\cdot]}$ associated with **all the layers**. We'll provide more details on the specific NN structure (also known as the *NN architecture*) in the relevant sections below. If you would like some background reading, [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) by <NAME> is a fantastic introduction (it's also online and free).  Known issues* As the code isn't heavily optimized, it will be slow if you run it for too many iterations or for too large of a system. Please be gentle. ❤ Import Python librariesThese are all the required Python libraries. `sklearn` is the library name for the popular [Scikit-learn](https://scikit-learn.org/stable/index.html) machine learning library in Python that we'll be using for convenience. We also seed the random number generator so we can all get reproducible results.**TODO**: Run this cell with `Shift+Enter` before starting the others.<jupyter_code># General libraries
import os

# Scientific computing libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Machine learning libraries
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split

# Set random seed
seed = 2020
np.random.seed(seed)<jupyter_output><empty_output><jupyter_text>Generating the data with MCWe will use some standard MC code to generate the data for the 2D Ising model. The function is provided below. Note that we assign a label of $1$ for the ordered state ($T  T_c$). We'll also store the temperature of the simulation so we can reference it later.<jupyter_code>Tc = 2 / (np.log(1 + np.sqrt(2)))
def mc_sweep(spins, beta):
    n = len(spins)
    for _ in range(n**2):
        i = np.random.randint(0, n)
        j = np.random.randint(0, n)
        nb_sum = spins[(i + 1) % n, j] + spins[(i - 1) % n, j] + \
                 spins[i, (j + 1) % n] + spins[i, (j - 1) % n]
        dE = 2 * spins[i, j] * nb_sum
        if np.random.random() < np.exp(-dE * beta):
            spins[i, j] *= -1
    return spins

def mc_run(Ts, L=8, eqsteps=2000, mcsteps=200, dt=100):
    data = []
    labels = []
    temps = []
    for T in Ts:
        spins = np.random.choice([1, -1], size=(L, L))
        beta = 1 / T
        for _ in range(eqsteps):
            mc_sweep(spins, beta)
        for i in range(mcsteps):
            mc_sweep(spins, beta)
            if i % dt == 0:
                temps.append(T)
                data.append(spins.flatten())
                if T < Tc:
                    labels.append(1)
                else:
                    labels.append(0)
        print(f'Finished generating data for T = {T}.')
    return (np.array(temps), np.array(data), np.array(labels))<jupyter_output><empty_output><jupyter_text>We'll choose a $12 \times 12$ lattice and sample every $500$ MC steps to balance statistics with computational speed, though these are just my *unproven heuristics*. This results in $1000$ samples for each $T$. But first, we check to see if a dataset already exists so that we don't have to run the previous function. We'll attempt to load the data from disk (GitHub directory) first; if that fails, then we'll just grab it from the online GitHub repository using `pd.read_csv()`.**TODO**: Please add two more lines of code to finish the `except` block of code so we can get the `labels` and `temps`. Note that you should add `.ravel()` after the `to_numpy()` method to [flatten these two arrays](https://numpy.org/doc/1.18/reference/generated/numpy.ravel.html).<jupyter_code>datapath = os.path.join('..', 'data', 'mc_data.csv')
labelpath = os.path.join('..', 'data', 'mc_labels.csv')
tempspath = os.path.join('..', 'data', 'mc_temps.csv')
dataurl = 'https://raw.githubusercontent.com/enze-chen/learning_modules/master/data/mc_data.csv'
labelurl = 'https://raw.githubusercontent.com/enze-chen/learning_modules/master/data/mc_labels.csv'
tempsurl = 'https://raw.githubusercontent.com/enze-chen/learning_modules/master/data/mc_temps.csv'

if "I don't want to wait forever,":
    try:
        temps = np.loadtxt(tempspath, delimiter=',')
        data = np.loadtxt(datapath, delimiter=',')
        labels = np.loadtxt(labelpath, delimiter=',')
        print('Loading data from disk...')
    except:
        data = pd.read_csv(dataurl, header=None).to_numpy()
        
        # ---------------------- #
        # YOUR CODE HERE

        # ---------------------- #
        
        print('Loading data from online...')
    print(f'Found existing data for {data.shape[0]} examples, ' + \
          f'{data.shape[1]} features, and {len(np.unique(temps))} temperatures.')
else:
    Ts = np.linspace(1.25, 3.25, 9)
    temps, data, labels = mc_run(Ts=Ts, L=12, eqsteps=5000, mcsteps=500000, dt=500)
    np.savetxt(tempspath, temps, delimiter=',')
    np.savetxt(datapath, data, delimiter=',')
    np.savetxt(labelpath, labels, delimiter=',')<jupyter_output><empty_output><jupyter_text>Data augmentationIn ML, you'll common hear the phrase, "You need more data." While this isn't always the solution to your problem, it typically won't hurt. **Data augmentation** refers to the process of artificially creating more training data from existing training data. Here, we'll take advantage of the special symmetry of the 2D Ising model Hamiltonian to negate all the spins (effectively doubling `data`) without changing their energy (we just duplicate the same `temps` and `labels`). Nifty!<jupyter_code>data = np.concatenate((data, -data), axis=0)
temps = np.concatenate((temps, temps), axis=0)
labels = np.concatenate((labels, labels), axis=0)
print(f'Now we have {len(data)} examples!')<jupyter_output><empty_output><jupyter_text>Splitting the data into training, validation, and test setsNow we have to split the data. There are many ways to do this, but we'll choose to **stratify the data based on** $T$ to create a harder test case by grouping together the examples with $2.0 \le T \le 2.5$. These might easily get misclassified since they lie close to the transition temperature. Furthermore, since materials data tends to be highly clustered, there's been a growing trend to stratify the splits for training and test data rather than use random splits from the entire dataset (see [<NAME> al. *Molecular Systems and Design Engineering*, **3**, 2018](https://pubs.rsc.org/en/content/articlelanding/2018/me/c8me00012c!divAbstract)).**TODO**: Split the `data` and `labels` arrays into ordered/disordered and critical subsets based on the `temps`. You'll combine the ordered and disordered subsets into one for subsequent use. There are many ways to do this, and I suggest one way here:1. You need to get the relevant row indices which you can then use to select the rows from `data`. To find the *indices* corresponding to the array elements that satisfy a criteria, we can use the comparison operators ``, and `=`. For example, if we had an array `arr = [0, 5, 1, 4, 2, 3]` and wanted the *indices* of the elements that were less than $2$, we can type ```pythonind = arr < 2.0```which would set```pythonind = [1, 0, 1, 0, 0, 0]```Note that this new result is no longer an integer array but rather a **Boolean array**, where `0` corresponds to False and `1` corresponds to True. Therefore, passing this result into the row argument (slice) of a 2D array would select the first and third rows from that 2D array.2. Since these comparison results are Boolean arrays, we can use the `&` and `|` operators to get their intersection and union, respectively. For example, if we had```pythonind1 = arr < 2.0ind2 = arr > 4.0inter = ind1 & ind2union = ind1 | ind2```then the results would be ```inter = [0, 0, 0, 0, 0, 0]union = [1, 1, 1, 0, 0, 0]```3. Use your Boolean arrays to slice the correct rows from `data` and `labels` and store them into variables. I suggest writing some print statements at the end to confirm you obtained the correct arrays (e.g. [check their dimensions](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.shape.html)).<jupyter_code># ---------------------- #
# YOUR CODE HERE

# ---------------------- #<jupyter_output><empty_output><jupyter_text>Next we'll use Scikit-learn's [`train_test_split()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function to create our training and validation data sets from the ordered/disordered data and labels you created above.**TODO**: Write one line of code to return `X_train, X_val, y_train, y_val`. See the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)for `train_test_split()` to see how it works. Set a `test_size` of `0.3` (pretty standard).<jupyter_code># ---------------------- #
# YOUR CODE HERE

# ---------------------- #
print(f'There are {X_train.shape[0]} training examples.')
print(f'There are {X_val.shape[0]} validation examples.')
print(f'There are {X_test.shape[0]} test examples.')<jupyter_output><empty_output><jupyter_text>Train an ML modelNow that we have the data prepared, it's time to begin building and training our ML models! We'll start with the [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html?highlight=dummyclassifiersklearn.dummy.DummyClassifier) in Scikit-learn as a baseline mode that simply calculates the mode in the training data and predicts that every time.**TODO**: Finish the following code to create and train a `DummyClassifier`. Some hints are:* Create a `DummyClassifier()` using its constructor and store it in the variable `dummy_clf`.* To train the classifier, use the method [`dummy_clf.fit(training_data, training_labels)`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html?highlight=dummyclassifiersklearn.dummy.DummyClassifier.fit) where the arguments are the names you used above.* The [`score()`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html?highlight=dummyclassifiersklearn.dummy.DummyClassifier.score) method uses the trained classifier to make predictions on the arguments and evaluates the accuracy. *Write two more lines* to obtain the accuracy on the validation and test sets.<jupyter_code># ---------------------- #
# YOUR CODE HERE
# Initialize and train the dummy classifier


# Print accuracy of dummy predictions 
print(f'The accuracy on the training set is {dummy_clf.score(X_train, y_train):.4f}')

# ---------------------- #<jupyter_output><empty_output><jupyter_text>Do the results above agree with your expectations?-------------------------- Logistic regressionSince the `DummyClassifier` isn't all that predictive, let's go with a different model that actually tries to learn the relationship between your inputs and outputs. For this, we'll turn to Scikit-learn's [`LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier. Please see the documentation for more information on its input arguments. If you would like a little more background on the theory, please see [page 16 of these notes](https://drive.google.com/file/d/0By9sqCDYwU7xZE02YVl0SEd2WWM/view?usp=sharing) from <NAME>'s CS 229 course at Stanford University.**TODO** Complete the code for logistic regression below. You'll notice that the structure is very similar to the `DummyClassifier` code you wrote above, which is the intention of the developers behind Scikit-learn. For now, we'll use most of the default parameters of `LogisticRegression()` with the following tweak:* `max_iter` describes the maximum number of iterations the solver will take to obtain convergence. `100` is the default, but let's use `1e4` here.Don't forget to test your trained model by reporting some accuracy scores.<jupyter_code># ---------------------- #
# YOUR CODE HERE

# ---------------------- #<jupyter_output><empty_output><jupyter_text>Our scores look a little more promising. But often times reporting just an accuracy isn't sufficient because it's not clear if the model has actually *learned* anything. For example, some of the results might just be due to data imbalance and the classifier predicting the majority label every time.We'd like some more solid statistics and visualizations. Let's create a plot of the classifier's predictions (probabilities with error bars) as a function of temperature. Our intention is to see how "certain" our model is at predicting the phase at each temperature and if the cross-over point between the two phases indeed occurs at $T_c$. Since the outputs of logistic regression are actual probabilities, it justifies making this type of analysis.**TODO**: Finish the code below, which we've started for you by creating four empty lists to hold the mean and uncertainties (standard deviations) of the classifier's predictions at each temperature. Some hints for you:* You'll want to create an array `Ts` of unique temperature values in your dataset that you can then loop through. To obtain all the unique values in a NumPy array, you can use the function `np.unique(my_array)`.* For each temperature, you'll want to find the indices in `temps` that matches that temperature. You might find the `==` operator handy.* To get the probabilities, we can use the `predict_proba()` method. The argument should be rows from the *entire dataset* `data` since we're querying at every temperature.* You can get the mean and standard deviation of an array with `np.mean()` and `np.std()`, repsectively. Note that you'll want to pass the `axis=0` parameter into both methods to get the column-wise statistics (but this is also the default).* The two columns correspond to the class labels in *increasing* arithmetic order.<jupyter_code>mean_ord_lr = []
err_ord_lr = []
mean_dis_lr = []
err_dis_lr = []

# ---------------------- #
# YOUR CODE HERE

# ---------------------- #<jupyter_output><empty_output><jupyter_text>If you got the above code working correctly, then we can plot the probabilities for both states as a function of temperature. We'll also add a vertical line in the background corresponding to the critical temperature $T_c \approx 2.269$.<jupyter_code>plt.rcParams.update({'figure.figsize':(7,5), 'lines.linewidth':4, \
                     'axes.linewidth':2, 'lines.markersize':14, 'font.size':16})
fig, ax = plt.subplots()
ax.axvline(x=Tc, ymin=0, ymax=1, lw=3, ls='--', c='C2', alpha=0.7)
ax.errorbar(Ts, mean_ord_lr, err_ord_lr, color='C0', fmt='-o', \
            capsize=8, capthick=3, elinewidth=3, ecolor='#c0c0c0dd', label='ordered')
ax.errorbar(Ts, mean_dis_lr, err_dis_lr, color='C1', fmt='-o', \
            capsize=8, capthick=3, elinewidth=3, ecolor='#c0c0c0dd', label='disodered')
ax.set_xlim(min(Ts) - 0.1, max(Ts) + 0.1)
ax.set_ylim(0, 1)
ax.set_xlabel('Temperature')
ax.set_ylabel('Probability')
ax.set_title('Logistic regression')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Wow! Good thing we checked, because [on my computer, at least] it looks like we hardly learned anything at all. This might be a little disheartening for our first ML model, but I wouldn't despair. There could be many reasons for this lack of predictive power, chief among them is that a linear hyperplane (decision boundary) simply doesn't exist in the high dimensional space. At this point, we could do a number of different things: tune the model, get more data, engineer better features, etc. But for demonstration purposes, we'll actually just switch to a new model altogether, one that can maybe capture the non-linearity in the data, as suggested by these preliminary results.--------------------- Neural networkNext we will try to use a NN model, specifically the [`MLPClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.htmlsklearn.neural_network.MLPClassifier) from Scikit-learn, which has a pretty good [overview page on NNs](https://scikit-learn.org/stable/modules/neural_networks_supervised.html). The more popular deep learning (DL) libraries like [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), and [Keras](https://keras.io/) are certainly good to know and a lot more powerful, but they also require a bit more setup than I would like for this exercise, so I opted to stick with Scikit-learn.Now, if you're new to DL, you probably realized from my blurb at the top (unless you didn't read it; that's fine) that a NN has **a ton** of parameters. Like, seriously. Way more than I have time to cover here. So to make things simple, we'll just change one of the default parameters in the `MLPClassifier()` constructor for now, and the [Extensions](Extensions) section lists some more things you can try. You're also welcome to read the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.htmlsklearn.neural_network.MLPClassifier) which explains these concepts quite well. Again, if you feel like you're missing the core theoretical background, I recommend [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) by <NAME>.**TODO**: Finish the code by constructing the `MLPClassifier()` with a `hidden_layer_sizes=(20,)` neurons (or however many you want). After you've trained the classifier, write some print statements to show the model's accuracy. Again, you will find that the code will be structured very similarly to what you wrote previously.<jupyter_code># ---------------------- #
# YOUR CODE HERE

# ---------------------- #<jupyter_output><empty_output><jupyter_text>As you can see, the structure of your code was pretty much identical to the `LogisticRegression` classifier. This was one of the design principles behind Scikit-learn that makes it very user-friendly. Moreover, the Scikit-learn implementation only uses [**cross-entropy loss**](https://datascience.stackexchange.com/a/20301), which returns a probability as the output similar to logistic regression. As such, we'll also try and make a plot of the predicted probabilities below.<jupyter_code>mean_ord_mlp = []
err_ord_mlp = []
mean_dis_mlp = []
err_dis_mlp = []

# ---------------------- #
# YOUR CODE HERE

# ---------------------- #<jupyter_output><empty_output><jupyter_text>Now we plot the probabilities for both states as a function of temperature.<jupyter_code>fig, ax = plt.subplots()
ax.axvline(x=Tc, ymin=0, ymax=1, lw=3, ls='--', c='C2', alpha=0.7)
ax.errorbar(Ts, mean_ord_mlp, err_ord_mlp, color='C0', fmt='-o', \
            capsize=8, capthick=3, elinewidth=3, ecolor='#c0c0c0dd', label='ordered')
ax.errorbar(Ts, mean_dis_mlp, err_dis_mlp, color='C1', fmt='-o', \
            capsize=8, capthick=3, elinewidth=3, ecolor='#c0c0c0dd', label='disodered')
ax.set_xlim(min(Ts) - 0.1, max(Ts) + 0.1)
ax.set_ylim(0, 1)
ax.set_xlabel('Temperature')
ax.set_ylabel('Probability')
ax.set_title('Multilayer perceptron')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Yeet! How do the NN results compare with logistic regression? Note that it's likely that this behavior is suboptimal as we did nothing to optimize the **hyperparameters** of the NN.------------- Bonus: Unsupervised learning with t-SNEI decided to throw this section in last-minute as it suddenly came to my mind. Previous ML examples were all *supervised learning* in which we supplied our model both the data (spins) and the labels (ordered or disordered). Now we'll try an *unsupervised learning* method, in which we'll show the ML model just the data (spins) but not the labels. While you might scoff and think that hiding the labels invariably leads to a less informative model, I caution you that these two classes of ML algorithms do different things. With unsupervised learning, you're now more interested in revealing the inherent structure in your data, which could lead to suprising discoveries.The unsupervised learning algorithm we will use here is called **t-SNE** ("tee-snee"), which is short for "t-distributed stochastic neighbor embedding." It was discovered somewhat recently by [<NAME>. and <NAME>. *Journal of Machine Learning Research*, **9**, 2008](http://www.jmlr.org/papers/v9/vandermaaten08a.html), and it is a popular technique for **dimensionality reduction** and visualization. The key principle behind t-SNE that you need to know is that it is a **non-linear** algorithm that preserves **local structure**, meaning points that are close together in high-dimensional space remain close together in the lower-dimensional space. That's all. If phrases like "Student's t-distribution," "Kullback-Leibler divergence," and "manifold learning" get you excited, I advise you to check out their paper.We will once again use Scikit-learn and its implementation of [`TSNE`](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html). As we have data from ordered and disordered phases, the hope is that t-SNE will reveal two clusters differentiating the two phases that we can then visualize in 2D instead of 144D. Note that we're only running the t-SNE algorithm on the validation data because the full dataset would take too long to run.<jupyter_code>from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=seed)
# X_embed = tsne.fit_transform(X_val)   # uncomment this line and run these two cells<jupyter_output><empty_output><jupyter_text>As a word of warning, the results may or may not be good. In a way I'm just trying to share a (somewhat flashy) unsupervised learning technique; but more importantly, I'm trying reflect how an ML engineer might think about problems and the various aspects of your data that you should consider. Supervised learning and NNs are all the rage and quite powerful hammers, but not every problem is a nail. ☺<jupyter_code>plt.rcParams.update({'lines.markersize':8})
fig, ax = plt.subplots()
ax.scatter(X_embed[y_val==0, 0], X_embed[y_val==0, 1], label='disordered')
ax.scatter(X_embed[y_val==1, 0], X_embed[y_val==1, 1], label='ordered')
ax.tick_params(left=False, labelleft=False, bottom=False, labelbottom=False)
ax.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11419427326274216, 'fraction_numerical': 0.012325082564434306, 'mean_word_length': 4.389357798165138, 'pattern_counts': {'":': 1, '<': 77, '<?xml version=': 0, '>': 73, 'https://': 38, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '658007', 'license': 'MIT', 'n_tokens_mistral': 8685, 'n_tokens_neox': 8130, 'n_words': 3509, 'path': 'mse/Machine_learning_Ising_model.ipynb', 'repo_name': 'enze-chen/learning_modules', 'size': '29445'}