<jupyter_start><jupyter_text>```<NAME> Deep Learning9 kwietnia 2020```<jupyter_code>import urllib3
urllib3.disable_warnings()

import tensorflow as tf
import tensorflow_datasets as tfds

import numpy as np
print(tf.__version__)
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
!nvidia-smi<jupyter_output>Thu Apr  9 19:19:43 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   56C    P0    N/A /  N/A |    522MiB /  4042MiB |     14%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU  [...]<jupyter_text>Dataset  CIFAR10 From `tensorflow_datasets`<jupyter_code>###########
# Option 1
# dataset = tfds.load('cifar10', as_supervised=True)
# ds_train, ds_test = dataset['train'], dataset['test']
###########


###########
# Option 2
dataset = tfds.image.cifar.Cifar10()

# Download dataset
dataset.download_and_prepare()

ds = dataset.as_dataset(as_supervised=True)
ds_train, ds_test = ds['train'], ds['test']<jupyter_output><empty_output><jupyter_text>Preprocess data<jupyter_code># Preprocess dataset (scale to [0, 1) and convert to float)

def preprocess_fn(image, label):
    return tf.image.convert_image_dtype(image, dtype=tf.float32), label

ds_train = ds_train.map(preprocess_fn)
ds_test  = ds_test.map(preprocess_fn)<jupyter_output><empty_output><jupyter_text>MLP models  model1 = MLP from tf.keras.models.Sequential with softmax activation<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense

model1 = Sequential([
    Flatten(input_shape=(32, 32, 3)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])<jupyter_output><empty_output><jupyter_text>Few experiments Lets take some batch from training data<jupyter_code>x_batch, y_batch = next( ds_train.batch(16).take(1).__iter__() )

print(f"The shape of x_batch is {x_batch.shape}")
print(f"The shape of y_batch is {y_batch.shape}")<jupyter_output>The shape of x_batch is (16, 32, 32, 3)
The shape of y_batch is (16,)<jupyter_text>We have two tensors `x_batch` and `y_batch`<jupyter_code>type(x_batch), type(y_batch)
y_batch.numpy()<jupyter_output><empty_output><jupyter_text>Lets feedforward the model with `x_batch`<jupyter_code>y_pred = model1(x_batch) # Tensor
y_pred.shape<jupyter_output><empty_output><jupyter_text>Since we used `softmax` at the end our predictions are actually *probaibilites* (not logits).<jupyter_code>y_pred1 = model1.predict(x_batch)
y_pred2 = model1.predict_proba(x_batch)

assert np.all( np.equal( y_pred1, y_pred2 ) )<jupyter_output><empty_output><jupyter_text>Once also ask for the classes<jupyter_code>model1.predict_classes(x_batch)<jupyter_output><empty_output><jupyter_text>One can try to compare this with labels from ds_train<jupyter_code>labels = next( ds_train.batch(16).take(1).map(lambda _, label: label).__iter__() )
labels
# Option 1 - using numpy
np.mean( labels.numpy() == model1.predict_classes(x_batch) )
# Option 2 - using only tensorflow

# Info: this is probably too complicated :-)
tf.reduce_mean(
    tf.cast(
        tf.equal(
            tf.argmax( model1(x_batch), axis=1 ), 
            labels
        ),
        dtype=tf.float32
    )
)<jupyter_output><empty_output><jupyter_text>model2 = model1 without softmax<jupyter_code>model2 = Sequential([
    Flatten(input_shape=(32, 32, 3)),
    Dense(128, activation='relu'),
    Dense(10) # no more softmax activation
])<jupyter_output><empty_output><jupyter_text>Few more experiments<jupyter_code>y_pred = model2(x_batch) # Tensor
y_pred.shape<jupyter_output><empty_output><jupyter_text>Since we didn't use `softmax` at the end our predictions are actually **logits**.<jupyter_code>y_pred1 = model2.predict(x_batch)
# Warning: one should not use predict_proba now
y_pred2 = model2.predict_proba(x_batch)
# but the assertion passes
assert np.all( np.equal( y_pred1, y_pred2 ) )<jupyter_output><empty_output><jupyter_text>Once also ask for the classes<jupyter_code>model2.predict_classes(x_batch)
np.mean( labels.numpy() == model2.predict_classes(x_batch) )<jupyter_output><empty_output><jupyter_text>Compile the models It is important to use `from_logits=False` (`True`) with `model1` (`model2`)<jupyter_code>model1.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
model2.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Training  Train models for 5 epochs Lets use `batch_size=16`.One can observe that `model2` is usually insignificantly better<jupyter_code>model1.fit(ds_train.batch(16), epochs=5)
model2.fit(ds_train.batch(16), epochs=5)<jupyter_output>Epoch 1/5
3125/3125 [==============================] - 8s 2ms/step - loss: 1.9414 - accuracy: 0.2840
Epoch 2/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.8522 - accuracy: 0.3285
Epoch 3/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.8235 - accuracy: 0.3436
Epoch 4/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.7982 - accuracy: 0.3526
Epoch 5/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.7750 - accuracy: 0.3654<jupyter_text>Train again and monitor the validation loss (and accuracy)<jupyter_code>model1.fit(ds_train.batch(16), epochs=5, validation_data=ds_test.batch(64))
model2.fit(ds_train.batch(16), epochs=5, validation_data=ds_test.batch(64))<jupyter_output>Epoch 1/5
3125/3125 [==============================] - 8s 2ms/step - loss: 1.7609 - accuracy: 0.3704 - val_loss: 1.7762 - val_accuracy: 0.3694
Epoch 2/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.7515 - accuracy: 0.3739 - val_loss: 1.7774 - val_accuracy: 0.3673
Epoch 3/5
3125/3125 [==============================] - 6s 2ms/step - loss: 1.7441 - accuracy: 0.3771 - val_loss: 1.7713 - val_accuracy: 0.3689
Epoch 4/5
3125/3125 [==============================] - 6s 2ms/step - loss: 1.7385 - accuracy: 0.3788 - val_loss: 1.7669 - val_accuracy: 0.3699
Epoch 5/5
3125/3125 [==============================] - 5s 2ms/step - loss: 1.7338 - accuracy: 0.3804 - val_loss: 1.7645 - val_accuracy: 0.3691<jupyter_text>One can see above the logged values of `val_loss` and `val_accuracy`  Analyse the confusion_matrix<jupyter_code>train_images = ds_train.map(lambda img,label: img)
train_labels = ds_train.map(lambda img,label: label)
predictions = model1.predict_classes( train_images.batch(32), batch_size=None )
tf.math.confusion_matrix( list(train_labels), predictions )
predictions = model2.predict_classes( train_images.batch(32), batch_size=None )<jupyter_output><empty_output><jupyter_text>Remember to use callbacks<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping, TensorBoard, ReduceLROnPlateau, ModelCheckpoint
# Lets start from scratch

model1 = Sequential([
    Flatten(input_shape=(32, 32, 3)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])
model2 = Sequential([
    Flatten(input_shape=(32, 32, 3)),
    Dense(128, activation='relu'),
    Dense(10) # no more softmax activation
])

model1.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])

model2.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
# stop training when 5 epochs does not improve validation loss
early_stopping = EarlyStopping(monitor='val_loss', verbose=1, patience=5)
# logging tensorboard values
tensorboard1 = TensorBoard('logs/mlp-model1')
tensorboard2 = TensorBoard('logs/mlp-model2')
# reduce learning rate when monitored values does not improve
reduce_lr = ReduceLROnPlateau(monitor='val_loss', verbose=1, patience=3)  # reduce_lr.patience should be smaller than early_stopping.patience

# save the best model
model1_checkpoint = ModelCheckpoint('model1.h5', save_best_only=True)
model2_checkpoint = ModelCheckpoint('model2.h5', save_best_only=True)
model1.fit(
    ds_train.batch(16), epochs=100, validation_data=ds_test.batch(64),
    callbacks=[early_stopping, tensorboard1, reduce_lr, model1_checkpoint]
)
model2.fit(
    ds_train.batch(16), epochs=100, validation_data=ds_test.batch(64),
    callbacks=[early_stopping, tensorboard2, reduce_lr, model2_checkpoint]
)<jupyter_output>Epoch 1/100
3125/3125 [==============================] - 9s 3ms/step - loss: 1.8821 - accuracy: 0.3217 - val_loss: 1.7720 - val_accuracy: 0.3623
Epoch 2/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.7468 - accuracy: 0.3718 - val_loss: 1.7085 - val_accuracy: 0.3880
Epoch 3/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.7001 - accuracy: 0.3894 - val_loss: 1.6659 - val_accuracy: 0.4019
Epoch 4/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.6699 - accuracy: 0.4014 - val_loss: 1.6607 - val_accuracy: 0.4017
Epoch 5/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.6505 - accuracy: 0.4078 - val_loss: 1.6543 - val_accuracy: 0.4017
Epoch 6/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.6351 - accuracy: 0.4143 - val_loss: 1.6397 - val_accuracy: 0.4101
Epoch 7/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.6229 - accuracy: 0.4169 - val_loss: 1.6385 - val_ac[...]<jupyter_text>Load the best models<jupyter_code>from tensorflow.keras.models import load_model
model1 = load_model('model1.h5')
model2 = load_model('model2.h5')<jupyter_output><empty_output><jupyter_text>... and evaluate them<jupyter_code>model1.evaluate(ds_train.batch(64)), model1.evaluate(ds_test.batch(64))
model2.evaluate(ds_train.batch(64)), model2.evaluate(ds_test.batch(64))<jupyter_output>157/Unknown - 0s 3ms/step - loss: 1.5542 - accuracy: 0.4481<jupyter_text>We have 43% (41%) accuracy on train (test) dataset  CNN model<jupyter_code>from tensorflow.keras.layers import Convolution2D, ReLU, MaxPooling2D

# Lets start with very poor convolutional network
model_cnn = Sequential([
    Convolution2D(filters=6, kernel_size=(3,3), padding='same', input_shape=(32,32,3)),
    ReLU(),
    MaxPooling2D(),
    Flatten(),
    Dense(128, activation='relu'),
    Dense(10) # no more softmax activation
])
model_cnn.summary()
model_cnn.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', verbose=1, patience=5)
tensorboard = TensorBoard('logs/cnn-model')
reduce_lr = ReduceLROnPlateau(monitor='val_loss', verbose=1, patience=3)
model_checkpoint = ModelCheckpoint('model_cnn.h5', save_best_only=True)

model_cnn.fit(
    ds_train.batch(16), epochs=100, validation_data=ds_test.batch(64),
    callbacks=[early_stopping, tensorboard, reduce_lr, model_checkpoint]
)<jupyter_output>Epoch 1/100
3125/3125 [==============================] - 10s 3ms/step - loss: 1.5026 - accuracy: 0.4649 - val_loss: 1.3326 - val_accuracy: 0.5195
Epoch 2/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.2612 - accuracy: 0.5539 - val_loss: 1.2675 - val_accuracy: 0.5508
Epoch 3/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.1595 - accuracy: 0.5905 - val_loss: 1.2391 - val_accuracy: 0.5618
Epoch 4/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.0829 - accuracy: 0.6189 - val_loss: 1.2250 - val_accuracy: 0.5742
Epoch 5/100
3125/3125 [==============================] - 7s 2ms/step - loss: 1.0220 - accuracy: 0.6411 - val_loss: 1.2286 - val_accuracy: 0.5772
Epoch 6/100
3125/3125 [==============================] - 7s 2ms/step - loss: 0.9703 - accuracy: 0.6592 - val_loss: 1.2194 - val_accuracy: 0.5835
Epoch 7/100
3125/3125 [==============================] - 7s 2ms/step - loss: 0.9215 - accuracy: 0.6777 - val_loss: 1.2311 - val_a[...]
{'fraction_non_alphanumeric': 0.20623538581449727, 'fraction_numerical': 0.08051441932969602, 'mean_word_length': 4.256452273658336, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '354538', 'license': 'MIT', 'n_tokens_mistral': 5147, 'n_tokens_neox': 4433, 'n_words': 890, 'path': 'NN-DL/TensorflowWorkshop/4 - Deep Learning workshop - Keras good practices.ipynb', 'repo_name': 'TheFebrin/Machine-Learning', 'size': '12843'}