<jupyter_start><jupyter_text>Getting deeper with KerasTensorflow is a powerful and flexible tool, but coding large neural architectures with it is tedious. To speed things up, there's plenty of deep learning toolkits that work on top of it like Slim, TFLearn, Sonnet, Keras.We'll be mostly focusing on Keras as it can be used both as auxiliary interface to TensorFlow and as standalone high-level library. We'll get you through on both of these in no time. With acknowelagements to <NAME><jupyter_code>import numpy as np
from mnist import load_dataset
import keras
X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()
y_train,y_val,y_test = map(keras.utils.np_utils.to_categorical,[y_train,y_val,y_test])
import matplotlib.pyplot as plt
%matplotlib inline
plt.imshow(X_train[0,0]);<jupyter_output><empty_output><jupyter_text>The pretty kerasThe high-level Keras interface lets you define deep learning architectures from high-level building blocks like "Dense layer with nonlinearity", "dropout" or "batch normalization". Keras even takes the responsibility to handle dropouts correctly when evaluating.This is convenient if you're building something commonplace like an image classifier. More delicate matters like sharing weight matrices or adversarial training (later) will get hacky.<jupyter_code>import tensorflow as tf
gpu_options = tf.GPUOptions(allow_growth=True,per_process_gpu_memory_fraction=0.1)
s = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))
import keras
from keras.models import Sequential
import keras.layers as ll

model = Sequential(name="cnn")

model.add(ll.InputLayer([1,28,28]))

model.add(ll.Flatten())

#network body
model.add(ll.Dense(25))
model.add(ll.Activation('linear'))

model.add(ll.Dropout(0.9))

model.add(ll.Dense(25))
model.add(ll.Activation('linear'))

#output layer: 10 neurons for each class with softmax
model.add(ll.Dense(10,activation='softmax'))

model.compile("adam","categorical_crossentropy",metrics=["accuracy"])
model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 1, 28, 28)         0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 784)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 25)                19625     
_________________________________________________________________
activation_1 (Activation)    (None, 25)                0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 25)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 25)                650       
_________________________________________________________________
activation[...]<jupyter_text>Model interfaceKeras models follow __Scikit-learn__'s interface of fit/predict with some notable extensions. Let's take a tour.<jupyter_code>#fit(X,y) with a neat automatic logging. Highly customizable under the hood.
model.fit(X_train, y_train, validation_data=(X_val,y_val), epochs=5)
#estimate probabilities P(y|x)
model.predict_proba(X_val[:2])
#Save trained weights
model.save("weights.h5")
print("\nLoss, Accuracy = ",model.evaluate(X_test,y_test))<jupyter_output>8096/10000 [=======================>......] - ETA: 0s
Loss, Accuracy =  [0.56361814851760861, 0.88039999999999996]<jupyter_text>Whoops!So far our model is staggeringly inefficient. It didn't even beat the linear model (val_acc ~0.92). Guess, why?There's an error that causes your model to underfit, and another error that prevents it from learning nonlinear dependencies. Try fixing both and play with the network architecture until you beat the thresholds below.<jupyter_code># Test score...
test_predictions = model.predict_proba(X_test).argmax(axis=-1)
test_answers = y_test.argmax(axis=-1)

test_accuracy = np.mean(test_predictions==test_answers)

print("\nTest accuracy: {} %".format(test_accuracy*100))

assert test_accuracy>=0.92,"Logistic regression can do better!"
assert test_accuracy>=0.975,"Your network can do better!"
print("Great job!")<jupyter_output>7488/10000 [=====================>........] - ETA: 0s
Test accuracy: 87.11 %<jupyter_text>Keras + tensorboardRemember the interactive graphs from Tensorboard one notebook ago? Thing is, Keras can use tensorboard to show you a lot of useful information about the learning progress. Just take a look!<jupyter_code>model = Sequential(name="cnn")

model.add(ll.InputLayer([1,28,28]))
model.add(ll.Flatten())

<Your architecture here>

#output layer: 10 neurons for each class with softmax
model.add(ll.Dense(10,activation='softmax'))

model.compile("adam","categorical_crossentropy",metrics=["accuracy"])

from keras.callbacks import TensorBoard
model.fit(X_train, y_train, validation_data=(X_val,y_val),epochs=10,
          callbacks=[TensorBoard("./logs")])
import os
port = 6000 + os.getuid()
print("Port: %d" % port)
#!killall tensorboard
os.system("tensorboard --logdir=./logs --port=%d &" % port)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11638418079096045, 'fraction_numerical': 0.02749529190207156, 'mean_word_length': 5.4375757575757575, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '279491', 'license': 'Apache-2.0', 'n_tokens_mistral': 1632, 'n_tokens_neox': 1490, 'n_words': 436, 'path': 'lecture_4/Keras.ipynb', 'repo_name': 'yandexdataschool/cern_summer_school_2017', 'size': '5318'}