<jupyter_start><jupyter_text>Lab 9 - Convolutional Neural NetworksIn this lab you'll use [PyTorch](https://pytorch.org/) to train a *convolutional neural network* (CNN, or ConvNet) on synthetic and real data, and to inspect both the filters and the feature maps of your model to ensure you understand the inner workings (intermediate representations) that are computed during the feed-forward computation of a ConvNet.If you really want to understand convolutions in detail, you can also download the *convolution_layer_demo.zip* file in the course Moodle page and run the Jupyter notebook inside. (All the code in that notebook is done for you.)**Run the code cell below** to import the required packages.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import torch
import copy
np.set_printoptions(precision=3, suppress=True)  # Print array values as 0.0023 instead of 2.352e-3
torch.set_printoptions(precision=3, sci_mode=False)<jupyter_output><empty_output><jupyter_text>**Run the code cell below** to define a plotting function that is useful for visualizing the weights of neural networks and images.<jupyter_code>def plot_matrix_grid(V, cmap='bwr'):
    """
    Given an array V containing stacked matrices, plots them in a grid layout.
    V should have shape (K,M,N) where V[k] is a matrix of shape (M,N).
    The default cmap is "bwr" (blue-white-red) but can also be "gray".
    """
    if isinstance(V, torch.Tensor):
        V = V.detach().numpy()
    assert V.ndim == 3, "Expected V to have 3 dimensions, not %d" % V.ndim
    k, m, n = V.shape
    ncol = 8                                     # At most 8 columns
    nrow = min(4, (k + ncol - 1) // ncol)        # At most 4 rows
    V = V[:nrow*ncol]                            # Focus on just the matrices we'll actually plot
    figsize = (2*ncol, max(1, 2*nrow*(m/n)))     # Guess a good figure shape based on ncol, nrow
    fig, axes = plt.subplots(nrow, ncol, sharex=True, sharey=True, figsize=figsize)
    vmax = np.percentile(np.abs(V), [99.9])      # Show the main range of values, between 0.1%-99.9%
    for v, ax in zip(V, axes.flat):
        img = ax.matshow(v, vmin=-vmax, vmax=vmax, cmap=plt.get_cmap(cmap))
        ax.set_xticks([])
        ax.set_yticks([])
    for ax in axes.flat[len(V):]:
        ax.set_axis_off()
    fig.colorbar(img, cax=fig.add_axes([0.92, 0.25, 0.01, .5]))   # Add a colorbar on the right<jupyter_output><empty_output><jupyter_text>1. Convolutional Neural Networks on Synthetic DataExercise 1.1&ndash;1.4 ask you to generate a synthetic data set and to inspect how a 1D convolutional neural network learns "pattern detectors." Using a 1D data set makes it easy to plot all the training features, targets, and predictions, seeing them all at once.**Run the code cell below** to define a function that will be useful for plotting data and predictions over the synthetic training set.<jupyter_code>def plot_named_tensors(tensor_dict):
    """
    Given a dict of {name: tensor} pairs, plots the tensors side-by-side in a common
    color scale. The name of each tensor is shown above its plot.
    """
    n = len(tensor_dict)
    vmax = max(v.abs().max() for v in tensor_dict.values())
    figsize = (2*n, 6)
    fig, axes = plt.subplots(1, n, figsize=figsize,  constrained_layout=True, squeeze=True)
    axes = axes.flat if isinstance(axes, np.ndarray) else (axes,)
    for (name, v), ax in zip(tensor_dict.items(), axes):
        v = torch.squeeze(v.detach())   # Automatically convert (N,1,D) to (N,D)
        if v.ndim == 1:
            v = v.view(-1, 1)  # Automatically convert (N,) to (N,1)
        assert v.ndim == 2, "couldn't turn tensors[%d] with shape %s into 2D" % (i, v.shape)
        img = ax.matshow(v, vmin=-vmax, vmax=vmax, cmap=plt.get_cmap('bwr'))
        ax.set_xticks([])
        ax.set_yticks([])
        ax.set_title(name)
    fig.colorbar(img, cax=fig.add_axes([0.985, 0.25, 0.03, .5]))   # Add a colorbar on the right<jupyter_output><empty_output><jupyter_text>Exercise 1.1 &ndash; Build a synthetic 1D data setHere you are asked to write code that uses random numbers to build a synthetic training set $(\mathbf{X},\mathbf{y})$ with $N=75$ rows where the $i^\text{th}$ row $\mathbf{x}_i \in \mathbb{R}^{C \times W}$ represents a 1-dimensional "image" with $C=1$ color channels (grayscale) and width $W=20$. having binary class label $y_i \in \{0, 1\}$. (Note that $W$ here is just a number and that the use of symbol $W$ is a convension for specifying ConvNet tensor shapes; try not to confuse this with using symbol $\mathbf{W}$ to represent a matrix of parameters.)Since we will use this data set to train a PyTorch model, you will build PyTorch tensors rather than Numpy arrays. However, you will still use the Numpy random number functions because they're simpler.The idea is to build a classification data set, where each input is a 1-dimensional image, and the image is classified as positive if and only if it contains a specific sequence of values $\ldots,0,1,0,\ldots$ somewhere in the image. **The obvious way to classify those images as positive is to learn a little filter that 'activated' by the local pattern $[0,1,0]$ the sequence.**Specifically, initialize each synthetic training case from $i=0,\ldots,N-1$ will be initialially all zeros and then constructed exactly as follows:1. Sample a random integer $\text{size} \in \{1,2\}$, _i.e._ the size of the subrange of pixels that will be assigned.2. Sample a random integer $\text{start} \in \{0,\ldots,W-\text{size}-1\}$, _i.e._ the offset from the start of the image3. Sample a random choice $\text{value} \in \{-1, -\frac{1}{2}, \frac{1}{2}, 1\}$, _i.e._ the value to fill for the subrange4. Assign features $X[i,0, \text{start}:\text{start}+\text{size}] = \text{value}$, _i.e._ set either a 1- or 2-pixel subrange to be the 'value'.5. Assign target $y[i]$ to be 1 if both $\text{size}=1$ and $\text{value} = +1$, _i.e._ if $X[i,0,:]$ has a single "+1" pixel somewhere in it.The goal here is to get you to practice using the random number functions (e.g. *randint*, *choice*) and to build a synthetic data set that will be useful for "inspecting" a simple 1D ConvNet. Note that the *randint(a,b)* returns an integer strictly less than *b*.If PyTorch complains about not being able to convert *numpy.int32* or *numpy.bool_* to *torch.FloatTensor* you may need to explicitly convert the value to float using *float(value)*.The**[torch.nn.Conv1d](https://pytorch.org/docs/stable/nn.htmltorch.nn.Conv1d)** module expects its input features to be stored as a tensor $\mathbf{X} \in \mathbb{R}^{N \times C \times W}$, so that is the reason behind the strange shape of our feature tensor.**Complete the starter code below** by adding a for loop that implements the above instructions for building the synthetic data.<jupyter_code>np.random.seed(0)  # For reproducibility

N = 75  # N = number of training cases
C = 1   # C = number of channels (just 1 in our case)
W = 20  # W = width of the 1-dimensional input image

# We create feature tensor X in "(N,C,W) format" (the shape) so that it can
# be used directly as input to PyTorch's Conv1D module.
X = torch.zeros((N, C, W))   # Start with zeros. You need to assign some of these feature values!
y = torch.zeros((N, 1))      # Start with zeros. You need to assign some of these target values!

for i in range(N):
    size = np.random.choice([1,2])
    start = np.random.choice(W - size - 1)
    value = np.random.choice([-1, -0.5, 0.5, 1])
    X[i, 0, start : start + size] = value
    y[i] = (0, 1)[1 in X[i, 0, :]]<jupyter_output><empty_output><jupyter_text>**Check your answer** by running the code cell below. It will plot the entire training set as a heatmap, where blue indicates negative values and red indicates positive values. If you initialized your training set correctly (and kept `np.random.seed(0)` above) you should see exactly this training set:![image](img/synthetic_training_set.png)Here we have $N=75$ rows and $W=20$ columns shown for the $\mathbf{X}$ tensor. Notice that the red dots in the $\mathbf{y}$ vector correspond to the $y_i=1$ for which there is exactly a 1-pixel red dot (a $+1$) in the corresponding input image.<jupyter_code>plot_named_tensors({'X': X, 'y': y})<jupyter_output>torch.Size([75, 1, 20])<jupyter_text>Exercise 1.2 &ndash; Define a ConvNet architecture suitable for classifing your synthetic dataYou are asked to define a 1D convolutional neural network with the following feed-forward architecture built from the [PyTorch modules](https://pytorch.org/docs/stable/nn.html) mentioned. The architecture should transform an input tensor $X \in \mathbb{R}^{N \times C \times W}$ into a vector of scalar activations $\mathbf{a} \in \mathbb{R}^{N \times 1}$. ![image](img/conv1d_architecture.png)Comments on the architecture:* The *convolutional* layer is depicted as having exactly enough padding (implicitly zero) to ensure that the output feature maps have the same spatial length ($W=20$) as the input vector had. Notice that if you were to increase the filter size by 2, you need to increase the padding by 1 (padding appears on both ends) to keep the output of convolution the same length as before. See the documentation for the Conv1d module.* The *max pooling* layer is being used here to take the maximum value across each of the 3 feature maps shown. Since we want to take the max over the entire spatial extent of the feature map, we use a large *kernel_size*.* The 1-dimensional convolution and pooling operations require *spatial* data in $(N,C,W)$ format (where $W$ is the spatial dimension), so that the operators know how long the spatial dimension is. However the *linear* (fully-connected) layer doesn't know how to deal with spatial data, so the *flatten* operation simply reshapes the tensor from shape $(N,C,W)$ to shape $(N,D)$ where $D=C \times W$.* Even though we'll use this architecture for binary classification, we do not add an extra *sigmoid* operation directly to the output. This is done for same reasons as for the multi-class PyTorch neural network from Lab 8. After the model is defined, we can still use it to predict binary class probabilities $\hat{y}_i \in [0,1]$ by feeding features $X$ through the model and then feeding the resulting activations $\mathbf{a}$ through a sigmoid so that class predictions are $\mathbf{\hat{y}} = \sigma(\mathbf{a})$. **Add a few lines of code** to define a PyTorch model with the above architecture.<jupyter_code>torch.manual_seed(0) # Ensure model weights initialized with same random numbers

num_filter = 4   # The number of filters to learn
filter_size = 5  # The size of each filter

model = torch.nn.Sequential(
    torch.nn.Conv1d(1, num_filter, 5, padding=2),
    torch.nn.ReLU(),
    torch.nn.MaxPool1d(W),
    torch.nn.Flatten(),
    torch.nn.Linear(num_filter, 1)
)<jupyter_output><empty_output><jupyter_text>**Check your model architecture** by running the code cell below.<jupyter_code>assert len(model) == 5, "Should be 5 layers!"
assert isinstance(model[0], torch.nn.Conv1d), "layer 0 should be Conv1d"
assert model[0].in_channels == C, "layer 0 should expect C input channel"
assert model[0].out_channels == num_filter, "layer 0 should expect %d output channels" % num_filter
assert model[0].kernel_size[0] == filter_size, "layer 0 filter size should be %d" % filter_size
assert model[0].padding[0] == filter_size//2
assert isinstance(model[1], torch.nn.ReLU), "layer 1 should be ReLU"
assert isinstance(model[2], torch.nn.MaxPool1d), "layer 2 should be MaxPool1d"
assert model[2].kernel_size == W, "layer 2 should pool over the entire input feature map"
assert isinstance(model[3], torch.nn.Flatten), "layer 3 should by Flatten"
assert isinstance(model[4], torch.nn.Linear), "layer 4 should be Linear"
assert model[4].in_features == num_filter, "layer 4 should have accept %d inputs" % num_filter
assert model[4].out_features == 1, "layer 4 should have only 1 output"
print("Looks OK!")<jupyter_output>Looks OK!<jupyter_text>**Plot your model's initial predictions** (i.e. *before* training) by running the code cell below. Notice that the untrained model predicts arbitrary class probabilies that all look similar. This is because the initial weights of the Conv1d and Linear layers are small random values, so the output is only weakly sensitive to the input.<jupyter_code>y_pred = torch.sigmoid(model(X))   # Turn activations into probabilities by feeding through sigmoid
print(y_pred[:5])                  # Print the first few probabilities
plot_named_tensors({'X': X, '$\hat{y}$': y_pred, 'y': y})<jupyter_output>tensor([[0.379],
        [0.357],
        [0.381],
        [0.338],
        [0.371]], grad_fn=<SliceBackward>)<jupyter_text>Exercise 1.3 &ndash; Train the ConvNet on your synthetic dataHere you should train the ConvNet architecture from Exercise 1.2, much as you trained a PyTorch neural network in the Lab 8.First, **run the code cell below** to make a copy of the untrained model from Exercise 1.2. This will make it easy to re-run the training code cell multiple times, each time starting from an untrained model (rather than continuing to train the same model!).<jupyter_code>untrained_model = copy.deepcopy(model)<jupyter_output><empty_output><jupyter_text>Next you'll need to define a loss function and implement a standard PyTorch training loop to train a copy of the untrained model. The PyTorch training loop for a ConvNet is identical to that of a neural network, so use Lab 8 as a guide.However, for this exercise use the following configuration:* *No mini-batches:* Unlike Lab 8, for this exercise do not use mini-batches. Instead feed the whole $X$ matrix through your model, like `model(X)` to get $N=75$ outputs. In other words, your training loop doesn't need an inner "mini-batch" loop.* *Loss function:* Use the "binary cross entropy" loss that directly accepts activations ('logits'). In PyTorch this is implemented by the **[torch.nn.BCEWithLogitsLoss](https://pytorch.org/docs/stable/nn.htmltorch.nn.BCEWithLogitsLoss)** module. This is similar to the *CrossEntropyLoss* in Lab 8, except it's specialized to handle a single binary output.* *Optimizer:* This is the learning algorithm. Use the **[torch.optim.SGD](https://pytorch.org/docs/stable/optim.htmltorch.optim.SGD)** optimizer with learning rate $0.05$, momentum $0.9$, and weight decay (weight penalty) of $0.001$.Your training code should also print the current training loss for the first epoch and every 50 epochs after. The code for doing so is similar to in Lab 8. For example, if you computed your loss in variable `loss_value` then something like this would work:```pythonfor epoch in range(1, num_epoch+1):        ... your code to apply a step of gradient descent here        if epoch == 1 or epoch % 50 == 0:        print("Epoch %d had training loss %.4f" % (epoch, loss_value.item()))```where the `.item()` is a method that converts a scalar-valued PyTorch tensor into a standard Python value, like a *float*, so that it can be more easily formatted as part of the string.The output of your training loop should look exactly like this:```Epoch 1 had training loss 0.8562Epoch 50 had training loss 0.3519Epoch 100 had training loss 0.3281Epoch 150 had training loss 0.3212...```However, the loss should not go to zero for the current model architecture. You'll fix that shortly.<jupyter_code># Start from a copy of the untrained model, so that when we re-run this code
# cell it always resets to the beginning.
model = copy.deepcopy(untrained_model)

# The number of times to evaluate the full training data (in this case, number of gradient steps)
num_epoch = 500

# Your code for defining loss, optimizer, and training loop here. Aim for 10-12 lines.
loss = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, weight_decay=0.001)
for epoch in range(num_epoch):
    y_predict = model(X)
    loss_value = loss(y_predict, y)
    model.zero_grad()
    loss_value.backward()
    optimizer.step()
    if epoch == 1 or epoch % 50 == 0:
        print(f"Epoch {epoch} had training loss {loss_value}")<jupyter_output>/home/d2/.local/lib/python3.8/site-packages/torch/autograd/__init__.py:130: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  Variable._execution_engine.run_backward(<jupyter_text>**Plot your trained model's predictions** in the code cell below. You should see that the prediction vector has an interesting pattern now (unlike the initial model), but that the predictions still do not quite match the targets.<jupyter_code>y_pred = torch.sigmoid(model(X))   # Turn activations into probabilities by feeding through sigmoid
print(y_pred[:5])                  # Print the first few probabilities
plot_named_tensors({'X': X, '$\hat{y}$': y_pred, 'y': y})<jupyter_output>tensor([[    0.000],
        [    0.983],
        [    0.000],
        [    0.993],
        [    0.012]], grad_fn=<SliceBackward>)<jupyter_text>The mismatch between predictions and targets is why the loss did not get close to zero, despite the fact that your ConvNet architecture can, in principle, learn a perfect classifier.If your model architecture was able to get the training loss to zero, the predictions would look more like this:![image](img/synthetic_predictions.png)Currently,  your ConvNet training got stuck in a local minimum!To give your ConvNet architecture a better chance at "getting lucky" and finding a path from random weights to useful weights, **try going back to Exercise 1.2 and increase the number of filters from 3 to 4**. You will then have to re-run all the code cells up to this point. With an extra filter, your ConvNet is now "lucky" enough to find at least one filter that detects a $0,1,0$ pattern.  Exercise 1.4 Inspect the filters and weights of your ConvNetHere you should visualize both the filter weights and the linear (fully-connected) weights. Use the *plot_matrix_grid* function. If you managed to get your training loss to be small (close to zero) then at least one of your filters should look like this:![image](img/synthetic_filter_example.png)where red means positive and blue means negative. When this filter is convolved with a sequence containing pattern $\ldots,0,1,0,\ldots$ it will "activate" and have a large output at the position centered on the $1$. That activation then gets selected by the max pooling operation, and is then used to activate the output of the network, giving correct classification.**Write a few lines of plotting code** to visualize the weights of both layers. Do you see a correspondence between the filter that looks "right" and the positive weight(s) in the linear layer?<jupyter_code>W1, b1, W2, b2 = model.parameters()
print(W1.shape)  # The tensor of 4 filters, each with 1 channel and kernel size 5
print(W2.T.shape)  # The matrix of linear (fully-connected) weights that combine the filter responses
# Your plotting code here. Aim for 2-4 lines of code.
something = np.stack((W1, W2))
plot_matrix_grid(W1)<jupyter_output>torch.Size([4, 1, 5])
torch.Size([4, 1])<jupyter_text>2. Digit classification with Convolutional Neural NetworksExercise 2.1&ndash;2.2 expand on Lab 8, this time asking you to train a convolutional neural network on the MNIST data set rather than a fully-connected neural network.  Exercise 2.1 Load and preprocess MNIST**Implement the *load_mnist_for_convnet* function** below. Rather than normalizing the features with scikit-learn, perform a simple normalization by scaling the pixel intensities from range $[0,255]$ down to $[0,1]$. This will be good enough.<jupyter_code>def load_mnist_for_convnet(filename):
    """
    Loads the MNIST data from a Numpy NPZ file and returns two PyTorch tensors:
    X: a float tensor with shape (N,1,28,28) where N is the number of images in the file
    y: an int64 tensor with shape (N,) containing the class targets for the images.
    The pixels values are scaled to be in range [0,1] where 0 is black and 1 is white.
    """
    # Your code here. Aim for 7-10 lines.

X_train, y_train = load_mnist_for_convnet("mnist_train.npz")
X_test,  y_test  = load_mnist_for_convnet("mnist_test.npz")<jupyter_output><empty_output><jupyter_text>**Check your answer** by running the code cell below.<jupyter_code>assert isinstance(X_train, torch.FloatTensor), "Features should be float32!"
assert isinstance(y_train, torch.LongTensor), "Targets should be int64 or long!"
assert X_train.shape == (60000, 1, 28, 28), "X_train has wrong shape"
assert y_train.shape == (60000,), "y_train has wrong shape"
assert X_test.shape == (10000, 1, 28, 28), "X_test has wrong shape"
assert y_test.shape == (10000,), "y_test has wrong shape"
assert X_train.min() == 0 and X_train.max() == 1, "Features don't seem to be scaled right!"
print("Looks good!")<jupyter_output><empty_output><jupyter_text>Exericise 2.2 Train a Convolutional Neural Network on MNISTYou are asked to train a 2D ConvNet to classify MNIST digits. You'll need the *Conv2d* and *MaxPool2d* modules as described in the [PyTorch modules documentation](https://pytorch.org/docs/stable/nn.html). Your architecture will be similar to Part 1 of this lab, except rather than pooling over the entire input signal you will pool only over small 2x2 regions, preserving more spatial information for the subsequent linear (fully-connected) layer to use in classification.**Write a few lines of code** to define a 2D ConvNet with the following feed-forward architecture:1. Convolution with 8 filters each of size 5x5 and enough padding to ensure that each of the resulting feature maps has spatial dimensions 28x28, just like the input.2. Rectified linear transformation (ReLU).3. Max pooling with kernel size 2x2 and a non-overlapping stride (i.e. stride same as kernel size).4. Flatten the spatial information5. Linear (fully-connected) layer to use all the outputs of the max pooling layer as features to make a prediction.(To predict class probabilities you'll have to apply a softmax function to the output, but that is normal in many deep learning frameworks because it makes training more numerically stable.)Hint: the trickiest part<jupyter_code>torch.manual_seed(0) # Ensure model weights initialized with same random numbers

# Your code here. Aim for 8-11 lines.<jupyter_output><empty_output><jupyter_text>**Check your model architecture** by trying to feed some inputs through. If an error is raised, then something is mis-configured in your network. The most likely error is that you did not correctly calculate the expected number of *in_features* for the linear layer, leading to a "size mismatch" in the tensors.<jupyter_code>y_pred = model(X_train[:5])  # Check model architecture by trying to feed inputs through it.
assert y_pred.shape == (5, 10), "Expected a batch of 5 images to produce output of shape (5, 10)"<jupyter_output><empty_output><jupyter_text>**Plot your initial model's predictions** on a the first 30 training inputs by running the code cell below. Notice that once again all the predictions are pretty much the same, and are far from the targets.<jupyter_code># Feed the model's predictions on 30 training cases through a row-wise softmax.
y_pred = torch.softmax(model(X_train[:50]), dim=1)

# Convert the first 30 training targets from index format {0,...,9} to a 1-hot format,
# for easier side-by-side comparison with the 10-dimensional output prediction.
y_true = torch.zeros((50, 10))
y_true[torch.arange(50), y_train[:50]] = 1

plot_named_tensors({'$\hat{y}$': y_pred, '$y$': y_true})<jupyter_output><empty_output><jupyter_text>**Inspect your initial model's filters** by plotting them with the *plot_matrix_grid* function. They should look random and have small values roughly in range $[-0.1, 0.1]$ (Only plot the first layer filter weights, not the linear layer weights.)<jupyter_code># Your plotting code here. Aim for 1-2 lines of code.<jupyter_output><empty_output><jupyter_text>**Train your model** with the following configuration:* Use the *CrossEntropyLoss* (since this is multi-class classification, not binary)* Use the exact same optimizer configuration you used for Part 1 of this lab (learning rate $0.05$, etc).* Use mini-batch training like you did in Lab 8, with a batch size of $100$.* Train for 5 epochs (5 passes over the full training set)First, define the loss function and optimizer in a separate code cell below:<jupyter_code>batch_size = 100
num_epoch = 5

# Your code to define loss function and optimizer here. Aim for 2 lines.<jupyter_output><empty_output><jupyter_text>Then, write your training loop in a separate code cell below, so that you can re-run this code cell to "continue" training with the same optimizer object, if you want to train your model longer.I recommend you add a print statement to report progress like we did in Lab 8, such as:```Epoch 1 final minibatch had loss 0.2452Epoch 2 final minibatch had loss 0.2077...```<jupyter_code># Your mini-batch training loop here. Aim for 9-12 lines.<jupyter_output><empty_output><jupyter_text>**Plot your trained model's predictions**. The predictions should look different from before, and matching the targets better.<jupyter_code># Your code here. Aim for 2-3 lines. Re-use the y_true matrix from the earlier code cell.<jupyter_output><empty_output><jupyter_text>If you only trained your ConvNet for 5 epochs total, you should be able to identify at least one training case that appears to be "harder" than the others, *i.e.* the model has a harder time giving a confident prediction. For example, here is an easy "9" and a hard "9" side by side:![image](img/easy_and_hard.png)**Plot an 'easy' training example and a 'hard' training example side-by-side** using the *plot_matrix_grid* function. This will require you to pull out two separate rows of *X_train* and turn them into a tensor with shape (2,28,28) so that the *plot_matrix_grid* knows what to plot.<jupyter_code># You code here. Aim for 1-4 lines.<jupyter_output><empty_output><jupyter_text>**Inspect your trained model's filters** by plotting them with the *plot_matrix_grid* function. They should no longer be completely random and instead contain structures that look like little "edge" detectors, such as the "diagonal line" detector shown below![image](img/mnist_trained_filters.png)<jupyter_code># Your code here. Aim for 2-3 lines.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09386879445970868, 'fraction_numerical': 0.02073845458993564, 'mean_word_length': 4.5055946953999175, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '608611', 'license': 'MIT', 'n_tokens_mistral': 8148, 'n_tokens_neox': 7656, 'n_words': 3512, 'path': 'Lab 9/lab9-exercises.ipynb', 'repo_name': 'm-triassi/ml-exercises', 'size': '26569'}