<jupyter_start><jupyter_text>MNIST digits classification with KerasWe don't expect you to code anything here because you've already solved it with TensorFlow.But you can appreciate how simpler it is with Keras.We'll be happy if you play around with the architecture though, there're some tips at the end.<jupyter_code>import numpy as np
from sklearn.metrics import accuracy_score
from matplotlib import pyplot as plt
%matplotlib inline
import tensorflow as tf
print("We're using TF", tf.__version__)
import keras
print("We are using Keras", keras.__version__)

import sys
sys.path.append("../..")
import keras_utils
# from keras_utils import reset_tf_session<jupyter_output>We're using TF 2.1.0
We are using Keras 2.3.1
Using TensorFlow backend.<jupyter_text>Look at the dataIn this task we have 50000 28x28 images of digits from 0 to 9.We will train a classifier on this data.<jupyter_code>import preprocessed_mnist
X_train, y_train, X_val, y_val, X_test, y_test = preprocessed_mnist.load_dataset()
# X contains rgb values divided by 255
print("X_train [shape %s] sample patch:\n" % (str(X_train.shape)), X_train[1, 15:20, 5:10])
print("A closeup of a sample patch:")
plt.imshow(X_train[1, 15:20, 5:10], cmap="Greys")
plt.show()
print("And the whole sample:")
plt.imshow(X_train[1], cmap="Greys")
plt.show()
print("y_train [shape %s] 10 samples:\n" % (str(y_train.shape)), y_train[:10])
# flatten images
X_train_flat = X_train.reshape((X_train.shape[0], -1))
print(X_train_flat.shape)

X_val_flat = X_val.reshape((X_val.shape[0], -1))
print(X_val_flat.shape)
# one-hot encode the target
y_train_oh = keras.utils.to_categorical(y_train, 10)
y_val_oh = keras.utils.to_categorical(y_val, 10)

print(y_train_oh.shape)
print(y_train_oh[:3], y_train[:3])
# building a model with keras
from keras.layers import Dense, Activation
from keras.models import Sequential

tf.keras.backend.clear_session()

model = Sequential()  # it is a feed-forward network without loops like in RNN
model.add(Dense(256, input_shape=(784,)))  # the first layer must specify the input shape (replacing placeholders)
model.add(Activation('sigmoid'))
model.add(Dense(256))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('softmax'))
# you can look at all layers and parameter count
model.summary()
# now we "compile" the model specifying the loss and optimizer
model.compile(
    loss='categorical_crossentropy', # this is our cross-entropy
    optimizer='adam',
    metrics=['accuracy']  # report accuracy during training
)
# and now we can fit the model with model.fit()
# and we don't have to write loops and batching manually as in TensorFlow
model.fit(
    X_train_flat, 
    y_train_oh,
    batch_size=512, 
    epochs=40,
    validation_data=(X_val_flat, y_val_oh),
    callbacks=[],
    verbose=2
)<jupyter_output>Train on 50000 samples, validate on 10000 samples
Epoch 1/40
 - 1s - loss: 1.2533 - accuracy: 0.6790 - val_loss: 0.4899 - val_accuracy: 0.8825
Epoch 2/40
 - 1s - loss: 0.3976 - accuracy: 0.8939 - val_loss: 0.2990 - val_accuracy: 0.9165
Epoch 3/40
 - 1s - loss: 0.2942 - accuracy: 0.9152 - val_loss: 0.2488 - val_accuracy: 0.9287
Epoch 4/40
 - 1s - loss: 0.2508 - accuracy: 0.9271 - val_loss: 0.2208 - val_accuracy: 0.9354
Epoch 5/40
 - 1s - loss: 0.2217 - accuracy: 0.9349 - val_loss: 0.2010 - val_accuracy: 0.9425
Epoch 6/40
 - 1s - loss: 0.1966 - accuracy: 0.9423 - val_loss: 0.1797 - val_accuracy: 0.9494
Epoch 7/40
 - 1s - loss: 0.1769 - accuracy: 0.9485 - val_loss: 0.1651 - val_accuracy: 0.9546
Epoch 8/40
 - 1s - loss: 0.1603 - accuracy: 0.9522 - val_loss: 0.1539 - val_accuracy: 0.9561
Epoch 9/40
 - 1s - loss: 0.1449 - accuracy: 0.9572 - val_loss: 0.1432 - val_accuracy: 0.9597
Epoch 10/40
 - 1s - loss: 0.1314 - accuracy: 0.9613 - val_loss: 0.1341 - val_accuracy: 0.9615
Epoch 11/40
 - 1s -[...]<jupyter_text>Here're the notes for those who want to play around hereHere are some tips on what you could do: * __Network size__   * More neurons,    * More layers, ([docs](https://keras.io/))   * Other nonlinearities in the hidden layers     * tanh, relu, leaky relu, etc   * Larger networks may take more epochs to train, so don't discard your net just because it could didn't beat the baseline in 5 epochs. * __Early Stopping__   * Training for 100 epochs regardless of anything is probably a bad idea.   * Some networks converge over 5 epochs, others - over 500.   * Way to go: stop when validation score is 10 iterations past maximum      * __Faster optimization__   * rmsprop, nesterov_momentum, adam, adagrad and so on.     * Converge faster and sometimes reach better optima     * It might make sense to tweak learning rate/momentum, other learning parameters, batch size and number of epochs * __Regularize__ to prevent overfitting   * Add some L2 weight norm to the loss function, theano will do the rest     * Can be done manually or via - https://keras.io/regularizers/       * __Data augmemntation__ - getting 5x as large dataset for free is a great deal   * https://keras.io/preprocessing/image/   * Zoom-in+slice = move   * Rotate+zoom(to remove black stripes)   * any other perturbations   * Simple way to do that (if you have PIL/Image):      * ```from scipy.misc import imrotate,imresize```     * and a few slicing   * Stay realistic. There's usually no point in flipping dogs upside down as that is not the way you usually see them.<jupyter_code>sess.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10438144329896908, 'fraction_numerical': 0.061855670103092786, 'mean_word_length': 4.1351606805293, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '231841', 'license': 'MIT', 'n_tokens_mistral': 2103, 'n_tokens_neox': 1830, 'n_words': 632, 'path': 'intro-to-dl/week2/v2/mnist_with_keras.ipynb', 'repo_name': 'Encryptioner/Coursera-HSE-Introduction-to-Deep-Learning', 'size': '5432'}