<jupyter_start><jupyter_text>Start with a simple neural network for MNISTNote that there are 2 layers, one with 20 neurons, and one with 10.The 10-neuron layer is our final layer because we have 10 classes we want to classify.Train this, and you should see it get about 98% accuracy<jupyter_code>import tensorflow as tf
data = tf.keras.datasets.mnist

(training_images, training_labels), (val_images, val_labels) = data.load_data()

training_images  = training_images / 255.0
val_images = val_images / 255.0
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                                    tf.keras.layers.Dense(20, activation=tf.nn.relu),
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(training_images, training_labels, epochs=20, validation_data=(val_images, val_labels))<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
Epoch 1/20
1875/1875 [==============================] - 5s 2ms/step - loss: 0.7084 - accuracy: 0.7964 - val_loss: 0.2786 - val_accuracy: 0.9231
Epoch 2/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.2664 - accuracy: 0.9231 - val_loss: 0.2313 - val_accuracy: 0.9331
Epoch 3/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.2147 - accuracy: 0.9385 - val_loss: 0.1986 - val_accuracy: 0.9408
Epoch 4/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.1886 - accuracy: 0.9463 - val_loss: 0.1863 - val_accuracy: 0.9447
Epoch 5/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.1678 - accuracy: 0.9508 - val_loss: 0.1759 - val_accuracy: 0.9472
Epoch 6/20
1875/1875 [==============================] - 3s 2ms/step - loss: 0.1477 - accuracy: 0.9570 - val_loss: 0.1662 - [...]<jupyter_text>Examine the test dataUsing model.evaluate, you can get metrics for a test set. In this case we only have a training set and a validation set, so we can try it out with the validation set. The accuracy will be slightly lower, at maybe 96%.  This is because the model hasn't previously seen this data and may not be fully generalized for all data. Still it's a pretty good score.You can also predict images, and compare against their actual label. The [0] image in the set is a number 7, and here you can see that neuron 7 has a 9.9e-1 (99%+) probability, so it got it right!<jupyter_code>model.evaluate(val_images, val_labels)

classifications = model.predict(val_images)
print(classifications[0])
print(val_labels[0])<jupyter_output>313/313 [==============================] - 0s 2ms/step - loss: 0.1504 - accuracy: 0.9599
[5.3084786e-06 2.4565529e-14 1.0924864e-07 1.1004267e-04 8.2607105e-14
 7.5921825e-08 7.8746477e-16 9.9986339e-01 9.5606447e-07 2.0077236e-05]
7<jupyter_text>Modify to inspect learned valuesThis code is identical, except that the layers are named prior to adding to the sequential. This allows us to inspect their learned parameters later.<jupyter_code>import tensorflow as tf
data = tf.keras.datasets.mnist

(training_images, training_labels), (val_images, val_labels) = data.load_data()

training_images  = training_images / 255.0
val_images = val_images / 255.0
layer_1 = tf.keras.layers.Dense(20, activation=tf.nn.relu)
layer_2 = tf.keras.layers.Dense(10, activation=tf.nn.softmax)
model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),
                                    layer_1,
                                    layer_2])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(training_images, training_labels, epochs=20)

model.evaluate(val_images, val_labels)

classifications = model.predict(val_images)
print(classifications[0])
print(val_labels[0])<jupyter_output>Epoch 1/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.6839 - accuracy: 0.8013
Epoch 2/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.2260 - accuracy: 0.9357
Epoch 3/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1855 - accuracy: 0.9474
Epoch 4/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1586 - accuracy: 0.9535
Epoch 5/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1496 - accuracy: 0.9565
Epoch 6/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1378 - accuracy: 0.9590
Epoch 7/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1277 - accuracy: 0.9620
Epoch 8/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1210 - accuracy: 0.9639
Epoch 9/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.1106 - accuracy: 0.9666
Epoch 10/20
1875/1875 [==============================] - 3s 1ms/step - loss: 0.110[...]<jupyter_text>Inspect weightsIf you print layer_1.get_weights(), you'll see a lot of data. Let's unpack it. First, there are 2 arrays in the result, so let's look at the first one. In particular let's look at its size.<jupyter_code>print(layer_1.get_weights()[0].size)<jupyter_output>15680<jupyter_text>The above code should print 15680. Why?Recall that there are 20 neurons in the first layer.Recall also that the images are 28x28, which is 784.If you multiply 784 x 20 you get 15680.So...this layer has 20 neurons, and each neuron learns a W parameter for each pixel. So instead of y=Mx+c, we have y=M1X1+M2X2+M3X3+....+M784X784+C in every neuron! Every pixel has a weight in every neuron. Those weights are multiplied by the pixel value, summed up, and given a bias.<jupyter_code>print(layer_1.get_weights()[1].size)<jupyter_output>20<jupyter_text>The above code will give you 20 -- the get_weights()[1] contains the biases for each of the 20 neurons in this layer.  Inspecting layer 2Now let's look at layer 2. Printing the get_weights will give us 2 lists, the first a list of weights for the 10 neurons, and the second a list of biases for the 10 neuronsLet's look first at the weights:<jupyter_code>print(layer_2.get_weights()[0].size)<jupyter_output>200<jupyter_text>This should return 200. Again, consider why?There are 10 neurons in this layer, but there are 20 neurons in the previous layer. So, each neuron in this layer will learn a weight for the incoming value from the previous layer. So, for example, the if the first neuron in this layer is N21, and the neurons output from the previous layers are N11-N120, then this neuron will have 20 weights (W1-W20) and it will calculate its output to be:W1N11+W2N12+W3N13+...+W20N120+BiasSo each of these weights will be learned as will the bias, for every neuron.<jupyter_code>print(layer_2.get_weights()[1].size)<jupyter_output>10
{'fraction_non_alphanumeric': 0.17535750397226635, 'fraction_numerical': 0.10472338581539795, 'mean_word_length': 4.209932279909706, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '505911', 'license': 'MIT', 'n_tokens_mistral': 2766, 'n_tokens_neox': 2234, 'n_words': 708, 'path': 'ML/Exploring_Categorical.ipynb', 'repo_name': 'bhupendpatil/Practice', 'size': '6923'}