<jupyter_start><jupyter_text>Load play datasets<jupyter_code>fashion_mnist = keras.datasets.fashion_mnist
(x_train_all, y_train_all), (x_test, y_test) = fashion_mnist.load_data()
x_valid, x_train = x_train_all[:5000], x_train_all[5000:]
y_valid, y_train = y_train_all[:5000], y_train_all[5000:]

print(x_valid.shape, y_valid.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)<jupyter_output>(5000, 28, 28) (5000,)
(55000, 28, 28) (55000,)
(10000, 28, 28) (10000,)<jupyter_text>Normalize inputdata by using Sklearn Standscalar- made x = (x - u) / std<jupyter_code>from sklearn.preprocessing import StandardScaler

# before normalization
print(np.max(x_train), np.min(x_train))

# perform normalization
scaler = StandardScaler()
# 1. data in x_train is int32, we need to convert them to float32 first 
# 2. convert x_train data from 
#    [None, 28, 28] -> [None, 784] 
#       -> after all reshape back to [None, 28, 28]
x_train_scaled = scaler.fit_transform(x_train.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)
x_valid_scaled = scaler.transform(x_valid.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)
x_test_scaled  = scaler.transform(x_test.astype(np.float32).reshape(-1, 1)).reshape(-1, 28, 28)

# after normalization
print(np.max(x_train_scaled), np.min(x_train_scaled))<jupyter_output>255 0
2.0231433 -0.8105136<jupyter_text>Build model with BatchNormalization and Dropout by using tf.keras high level APIUse keras.layers.BatchNormalization() between to net layers to enable batch normalize data. HowIn general there are two ways to implement batch normalization1. direct after convolutional layer (after activation function)2. before convolutional layer add a additional layer then batch normalization and explicit activation (keras.layers.Activation) is added behind batch normalizationIn order to avoid overfiting problem, is dropout often used in the last few layers. For example you can add 2 - 4 Dropout from 20 layers to reduce training parameters. There are two common Dropout API:1. keras.layers.Dropout  -> normal dropout could change the distribution of dataset2. keras.layers.AlphaDropout -> a enhanced version of Dropout with following features...    2.1 AlphaDroupout will keep standard deviation and standard variance        2.2 Cause of keeping std-dev and std-var after Dropout. You can combine some operations together alpha-dropout, batch-normalization and activation-function        2.3 It is recommended to set dropout rate to 0.5<jupyter_code>model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape = [28, 28]))

for _ in range(10):
    model.add(keras.layers.Dense(100, activation = "relu"))
    model.add(keras.layers.BatchNormalization())
    '''
    model.add(keras.layers.Dense(100))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Activation(activation = "relu"))
    '''
model.add(keras.layers.AlphaDropout(rate = 0.5))
model.add(keras.layers.Dense(10, activation = "softmax"))

# if y is one_hot vector then use categorical_crossentropy as loss function
model.compile(loss = "sparse_categorical_crossentropy",
              optimizer = "sgd",
              metrics = ["accuracy"])<jupyter_output><empty_output><jupyter_text>Train model with callback<jupyter_code># Tensorboard, EarlyStopping, ModelCheckPoint
# https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/callbacks

logdir = './dnn_batch_norm_dropout' # logdir for tensorboard logdata
if not os.path.exists(logdir):
    os.mkdir(logdir)
output_model_file = os.path.join(logdir, "fashion_mnist_model.h5")    

callbacks = [
    keras.callbacks.TensorBoard(log_dir=logdir),
    keras.callbacks.ModelCheckpoint(output_model_file, save_best_only = True),
    keras.callbacks.EarlyStopping(patience = 5, min_delta = 1e-3)
]

history = model.fit(x = x_train_scaled, y = y_train, epochs = 10, 
                    validation_data = (x_valid_scaled, y_valid),
                    callbacks = callbacks)

# once training is finished, you can use tensorboard to visualize results
# tensorboard --logdir=callbacks<jupyter_output>WARNING: Logging before flag parsing goes to stderr.
W0703 09:44:00.609814 140315510544192 deprecation.py:323] From /home/bai/.virtualenvs/tensorflow2/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>Result visualization<jupyter_code>from tensorflow.python.keras.callbacks import History

def plot_learning_curves(history: History):
    pd.DataFrame(history.history).plot(figsize = (8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1)
    plt.show()

plot_learning_curves(history)<jupyter_output><empty_output><jupyter_text>Evaluate model with test dataset<jupyter_code>test_loss, test_acc = model.evaluate(x_test_scaled, y_test)<jupyter_output>10000/10000 [==============================] - 1s 67us/sample - loss: 0.5558 - accuracy: 0.8699<jupyter_text>Made a simple prediction on test dataset<jupyter_code># one_hot encoded results
predictions = model.predict(x_test)

# compare ml-prediction to test label
for pred_index in range(20):
    print(np.argmax(predictions[pred_index]), y_test[pred_index])<jupyter_output>0 9
2 2
6 1
6 1
6 6
6 1
4 4
4 6
1 5
8 7
6 4
6 5
3 7
3 3
2 4
6 1
2 2
2 4
6 8
0 0
{'fraction_non_alphanumeric': 0.10993425858290723, 'fraction_numerical': 0.04967129291453616, 'mean_word_length': 5.058628318584071, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '357371', 'license': 'MIT', 'n_tokens_mistral': 2048, 'n_tokens_neox': 1835, 'n_words': 478, 'path': '7_tensorflow2_keras/4_classification_batch_norm_dropout.ipynb', 'repo_name': 'xiabai84/tensorflow_for_enginner', 'size': '5476'}