<jupyter_start><jupyter_text>An Introduction to KerasIn this notebook, I will provide a brief introduction to [Keras](https://keras.io/), a high-level neural networks python library; it is a wrapper ontop of either **Theano** or **Tensorflow**. In the first part, I will implement the simple perceptron model from the first homework. Then I will try the 'hello world' of neural net applications: hand-writing recognition from the [MNIST database](http://yann.lecun.com/exdb/mnist/).   MNIST DatabaseThe MNIST database of handwritten digits. It consists of a training set of **60,000 examples**, and a test set of **10,000 examples**. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.*From [Yann Lecun's Website](http://yann.lecun.com/exdb/mnist/)* First, let's import the necessary basic python libraries:<jupyter_code>%matplotlib inline

# Standard Libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

sns.set(style='ticks')<jupyter_output><empty_output><jupyter_text>Next, we let's load the **MNIST dataset**. Fortunately for us, this is already available in the **Keras** module (as well as many other common datasets). So we will import from **Keras** as follows:<jupyter_code># Load Dataset from Keras Libraries
from keras.datasets import mnist
from keras.utils import np_utils

(X_train, y_train), (X_test, y_test) = mnist.load_data()
nb_class = 10

# Visualize Some Examples
idx = np.random.choice(X_train.shape[0],6,replace=False)
for i,ix in enumerate(idx):
    plt.subplot(231+i)
    plt.title('Label is {0}'.format(y_train[ix]))
    plt.imshow(X_train[ix], cmap='gray')
    plt.axis('off')

plt.tight_layout()

# Reshape X data from 2D to 1D (28x28 to 784)
X_train = X_train.reshape(60000,784).astype('float32')
X_test  = X_test.reshape(10000,784).astype('float32')

# Convert Y labels to Categorical One-Hot Format
y_train = np_utils.to_categorical(y_train, nb_class)
y_test = np_utils.to_categorical(y_test, nb_class)<jupyter_output>Using TensorFlow backend.<jupyter_text>Neural Network ModelNow, we are goint to create our neural network model using Keras' high-level library. The big advantage of Keras is that it abstracts away the low level code of the basic elements of neural networks such as hidden units, layers, activations, optimizers, etc. In other words, you just call Keras functions to create these things for you, without hand coding the details yourself. Underneath it all, Keras is using **Tensorflow** or **Theano**, just like you would yourself.To demonstrate this, let's first decide on a network architecture for our MNIST dataset. Let's use the same architecture as in the `multilayer_perceptron.py` example we used in class to test our **Tensorflow** installation. Here is the network architecture: Network Architecture![Network Architecture](NetArch1.png)Now let's proceed to build this in Keras.<jupyter_code># Import Keras Libraries
from keras.models import Sequential
from keras.layers import Dense, Activation


# Define Model Parameters
nb_feat   = 28    # no. of features/columns of input
L1_units  = 256   # no. of nodes in Layer 1
L2_units  = 100   # no. of nodes in Layer 2
L3_units  = 50   # no. of nodes in Layer 2
nb_class  = 10    # no. of output classes

# Neural Network Model
model = Sequential()                             # Sequential network model description
model.add(Dense(L1_units,input_shape=(784,)))    # Add 1st Dense Layer
model.add(Activation("relu"))                    # Add activation function

model.add(Dense(L2_units))                       # Add 2nd Dense Layer
model.add(Activation("relu"))                    # Add activation function

model.add(Dense(L3_units))                       # Add 3nd Dense Layer
model.add(Activation("relu"))                    # Add activation function

model.add(Dense(nb_class))                       # Add 3rd Dense Layer, also the classification layer
model.add(Activation('softmax'))                 # Add sigmoid classification<jupyter_output>Using TensorFlow backend.<jupyter_text>Once you've described the model in a sequential model, you then proceed to compile the model. The model needs to be compiled because the actual Theano/Tensorflow backend uses C/C++ internally for faster implementations and runs. Also, while compiling, we defining the loss function (optimization criterion) and the kind of optimizer to use. Here I use the popular **RMSprop**, an adaptive variant of gradient descent.<jupyter_code>model.compile(loss='categorical_crossentropy',optimizer='rmsprop', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Finally, I proceed to train the model with a simple one line command:<jupyter_code>log = model.fit(X_train, y_train,nb_epoch=30, batch_size=32, verbose=2,validation_data=(X_test, y_test))
# Scikit-Learn Machine Learning Utilities
from sklearn.metrics import confusion_matrix

## Final Accuracy
score = model.evaluate(X_test, y_test)
print'Model Accuracy: {}%'.format(score[1]*100)
print ''
print ''

y_pred = model.predict_classes(X_test)
cm = confusion_matrix(np.argmax(y_test,axis=1),y_pred)  ## Confusion matrix
plot_confusion_matrix(cm, ['0','1','2','3','4','5','6','7','8','9'],normalize=True)
# summarize history for accuracy
plt.plot(log.history['acc'])
plt.plot(log.history['val_acc'])
plt.title('Model Accuracy'), plt.ylabel('Accuracy'), plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
# summarize history for loss
plt.plot(log.history['loss'])
plt.plot(log.history['val_loss'])
plt.title('Model Loss'), plt.ylabel('Loss'), plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    thresh = cm.max() / 2.

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10448423856741157, 'fraction_numerical': 0.01479946721918011, 'mean_word_length': 4.384860557768924, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '203617', 'license': 'Apache-2.0', 'n_tokens_mistral': 2126, 'n_tokens_neox': 2002, 'n_words': 680, 'path': 'notebooks/Keras Introduction.ipynb', 'repo_name': 'justiceamoh/engs108-vagrant-machine', 'size': '6757'}