<jupyter_start><jupyter_text>Ensembling modelsImporting data<jupyter_code>import tensorflow as tf
mnist = tf.keras.datasets.mnist
_, (validation_data, validation_labels) = mnist.load_data()
validation_data = validation_data / 255
validation_data = validation_data.reshape(-1, 28, 28, 1)<jupyter_output><empty_output><jupyter_text>Importing trained models<jupyter_code>model_V4_3 = tf.keras.models.load_model("CNN_MNIST_V4_3.h5")
model_V4_5= tf.keras.models.load_model("CNN_MNIST_V4_5.h5")
model_V4_7 = tf.keras.models.load_model("CNN_MNIST_V4_7.h5")
print("Model V4_3 Accuracy")
model_V4_3.evaluate(validation_data, validation_labels)
print("Model V4_5 Accuracy")
model_V4_5.evaluate(validation_data, validation_labels)
print("Model V4_7 Accuracy")
model_V4_7.evaluate(validation_data, validation_labels)<jupyter_output>Model V4_3 Accuracy
313/313 [==============================] - 2s 5ms/step - loss: 0.0143 - accuracy: 0.9962
Model V4_5 Accuracy
313/313 [==============================] - 1s 4ms/step - loss: 0.0148 - accuracy: 0.9966
Model V4_7 Accuracy
313/313 [==============================] - 1s 4ms/step - loss: 0.0138 - accuracy: 0.9965<jupyter_text>Generating prediction tables for all models<jupyter_code>model_V4_3_predict = model_V4_3.predict(validation_data)
model_V4_5_predict = model_V4_5.predict(validation_data)
model_V4_7_predict = model_V4_7.predict(validation_data)<jupyter_output><empty_output><jupyter_text>Creating ensemble and computing ensemble accuracy. Note the methodology used to compute the accuracy is to take the majority agreement, but if there is a three-way-tie then take the prediction for which all models are collectively the most confident in by summing confidence.<jupyter_code>import numpy as np
num_examples = len(validation_data)
count = 0
incorrect = []
for i in range(num_examples):
    predict_3, predict_5, predict_7 = np.argmax(model_V4_3_predict[i]), np.argmax(model_V4_5_predict[i]), np.argmax(model_V4_7_predict[i])
    ensemble_predict = 0
    if predict_3 == predict_5 or predict_3 == predict_7:
        ensemble_predict = predict_3
    elif predict_5 == predict_7:
        ensemble_predict = predict_5
    else:
        ensemble_predict = np.argmax(model_V4_3_predict[i] + model_V4_5_predict[i] + model_V4_7_predict[i])

    if ensemble_predict == validation_labels[i]:
        count += 1
    else:
        incorrect.append(i)
print(count / num_examples)<jupyter_output>0.9973<jupyter_text>Printing examples that ensemble incorrectly predicted<jupyter_code>print(incorrect)
for i in incorrect:
    print(f"Example = {i},\tactual = {validation_labels[i]}, " +
          f"model_4_3 = {np.argmax(model_V4_3.predict(validation_data[i][np.newaxis]))}, " +
          f"model_4_5 = {np.argmax(model_V4_5.predict(validation_data[i][np.newaxis]))}, " +
          f"model_4_7 = {np.argmax(model_V4_7.predict(validation_data[i][np.newaxis]))}")<jupyter_output>Example = 193,	actual = 9, model_4_3 = 8, model_4_5 = 8, model_4_7 = 8
Example = 247,	actual = 4, model_4_3 = 4, model_4_5 = 2, model_4_7 = 2
Example = 445,	actual = 6, model_4_3 = 6, model_4_5 = 0, model_4_7 = 0
Example = 447,	actual = 4, model_4_3 = 9, model_4_5 = 9, model_4_7 = 4
Example = 582,	actual = 8, model_4_3 = 8, model_4_5 = 2, model_4_7 = 2
Example = 659,	actual = 2, model_4_3 = 2, model_4_5 = 7, model_4_7 = 7
Example = 1112,	actual = 4, model_4_3 = 6, model_4_5 = 6, model_4_7 = 6
Example = 1393,	actual = 5, model_4_3 = 7, model_4_5 = 7, model_4_7 = 7
Example = 1737,	actual = 5, model_4_3 = 5, model_4_5 = 3, model_4_7 = 3
Example = 1901,	actual = 9, model_4_3 = 4, model_4_5 = 4, model_4_7 = 4
Example = 2040,	actual = 5, model_4_3 = 6, model_4_5 = 6, model_4_7 = 6
Example = 2130,	actual = 4, model_4_3 = 9, model_4_5 = 9, model_4_7 = 9
Example = 2462,	actual = 2, model_4_3 = 2, model_4_5 = 0, model_4_7 = 0
Example = 2771,	actual = 4, model_4_3 = 9, model_4_5 = 9, model_4_7 = [...]
{'fraction_non_alphanumeric': 0.13295832267962157, 'fraction_numerical': 0.08693428790590642, 'mean_word_length': 5.309677419354839, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '399456', 'license': 'MIT', 'n_tokens_mistral': 1676, 'n_tokens_neox': 1505, 'n_words': 325, 'path': 'V4/ensemble.ipynb', 'repo_name': 'isaacwen/Digit-Classifier', 'size': '3911'}