<jupyter_start><jupyter_text>looking at 600 epochs of data augmented cnn runmodel is [v2.py](v2.py)The purpose is to check if using multiple models to predict the digit is better or not.Spoiler!!!: => I get about 99.79% accuracy on the 10,000 test data.Let's read the saved validation accuracy to check what was the median value<jupyter_code>import numpy as np
import glob<jupyter_output><empty_output><jupyter_text>read saved history dataThe graph shows that during data augmented training, the training and validation loss/accuracy really did not improve.<jupyter_code>loss = np.load('loss.npy')
val_loss = np.load('val_loss.npy')
acc = np.load('acc.npy')
val_acc=np.load('val_acc.npy')

from matplotlib import pyplot as plt
%matplotlib inline

# show procession of training...
plt.plot(loss)
plt.plot(val_loss)

plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.show()
plt.close()

plt.plot(acc)
plt.plot(val_acc)

plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='lower right')
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>histogram of the val_accUsing the histogram, you can identify what the "average" prediction rate was during data augmented training.Some dashboard claim very selective error rate vs citing what the highest frequency rate was.If someone is saving the "random seed" for his model, then you must suspect the modeller is seeking for that very special situation where the error rate was the lowest.During 600 epoch run, I only reach upto 0.9970 validation accuracy rate.Sheeeee.... Some of the ingenious people came up with "distortions" to apply to the training set of digits to mimic how the acutal validation set would look a like. In fact, if you look at the problematic digit labeling ( 9s that look a like 4s and 4s that look alike 9s), you could come up with some strategy to distort/transform/augment your training data to represent some of the unpredictable digits...Too much said.<jupyter_code>print('median:',  np.median(val_acc))
binwidth=0.0002
plt.hist(val_acc, bins=np.arange(min(val_acc), max(val_acc) + binwidth, binwidth))
plt.show()
plt.close()<jupyter_output>median: 0.9958<jupyter_text>let's read in the MNIST data<jupyter_code>from tensorflow import keras

# get mnist data
mnist = keras.datasets.mnist

print('loading MNIST data...')
# using path saves to the ~/.keras/data/path location so it's not downloaded next time
(x_train, y_train),(x_test, y_test) = mnist.load_data()

# shape data for CNN

# convert the shape of data depending on the image data format
is_channels_first = (keras.backend.image_data_format() == 'channels_first')

if is_channels_first :
    x_train2 = x_train.reshape(x_train.shape[0], 1, 28, 28)
    x_test2 = x_test.reshape(x_test.shape[0], 1, 28, 28)
else:
    x_train2 = x_train.reshape(x_train.shape[0], 28, 28, 1)
    x_test2 = x_test.reshape(x_test.shape[0], 28, 28, 1)
    input_shape = (28, 28, 1)

# get hot label output

y_train2 = keras.utils.to_categorical(y_train, num_classes=10)
y_test2 = keras.utils.to_categorical(y_test, num_classes=10)

# normalize the data
x_train2 = x_train2.astype('float32')
x_test2 = x_test2.astype('float32')

# convert the data from 0 to 1.0
x_train2, x_test2 = x_train2 / 255, x_test2 / 255

print('x train shape:',x_train2.shape)
print('y train shape:',y_train2.shape)
print('x test shape:',x_test2.shape)
print('y test shape:',y_test2.shape)<jupyter_output>loading MNIST data...
x train shape: (60000, 28, 28, 1)
y train shape: (60000, 10)
x test shape: (10000, 28, 28, 1)
y test shape: (10000, 10)<jupyter_text>let's read 3 good modelsThe purpose is to use consensus model:you have two experts (or slaves as used in this notebook) on digit prediction. If they agree on the guess, then the agreed number is the prediction valueIf they disagree, then the "master" picks his number as the correct prediction<jupyter_code>saved_dir='saved_models'

model_names = glob.glob(saved_dir + "/*.hdf5")
sorted_names = sorted(model_names , reverse=True)<jupyter_output><empty_output><jupyter_text>Let's see the result of 3 models working togetherThe model at 0 and 1 have 99.70% val_acc rate and both were achieved near the 500th epoch while the model2 was picked from around 290 ish epoch and has 99.69% accuracy rate<jupyter_code>print('master:', sorted_names[2])
print('slave1:', sorted_names[0])
print('slave2:', sorted_names[1])
model = keras.models.load_model(sorted_names[2])
slave1 = keras.models.load_model(sorted_names[0])
slave2 = keras.models.load_model(sorted_names[1])
# process data through 3 models and predict using the above 3 model protocol:
# if two slaves agree then that is the prediction, else, the master picks

predictions = model.predict(x_test2)
slave1_pred = slave1.predict(x_test2)
slave2_pred = slave2.predict(x_test2)

my_matrix = np.zeros( (10,10), dtype='int')

# count of good guesses
count_matrix = np.zeros( (10,), dtype='int')
good_matrix = np.zeros( (10,), dtype='int')
miss_count = 0;

missed_list = []

# iterate through 10,000 test data
for i in range(10000):
    count_matrix[y_test[i]] +=1
    guess = np.argmax(predictions[i])
    s1_guess = np.argmax(slave1_pred[i])
    s2_guess = np.argmax(slave2_pred[i])
    
    if s1_guess == s2_guess:
        guess = s1_guess
    
    if guess == y_test[i]:
        good_matrix[guess] +=1
    else:
        # increment [expected][guess] matrix
        my_matrix[y_test[i]][guess] += 1
        miss_count +=1
        info = [i,y_test[i],guess]
        missed_list.append(info)

# show good matrix
print('missed count:', miss_count , 'ouf of 10,000')
print('Good guesses:')
for i in range(10):
    percent = "( {:.2f}".format((good_matrix[i] * 100.0) / count_matrix[i]) + " %)"
    print('match count for:',i,'=', good_matrix[i] , '/',count_matrix[i] , percent, 'missed:', count_matrix[i] - good_matrix[i])<jupyter_output>missed count: 21 ouf of 10,000
Good guesses:
match count for: 0 = 980 / 980 ( 100.00 %) missed: 0
match count for: 1 = 1134 / 1135 ( 99.91 %) missed: 1
match count for: 2 = 1030 / 1032 ( 99.81 %) missed: 2
match count for: 3 = 1010 / 1010 ( 100.00 %) missed: 0
match count for: 4 = 980 / 982 ( 99.80 %) missed: 2
match count for: 5 = 889 / 892 ( 99.66 %) missed: 3
match count for: 6 = 956 / 958 ( 99.79 %) missed: 2
match count for: 7 = 1026 / 1028 ( 99.81 %) missed: 2
match count for: 8 = 970 / 974 ( 99.59 %) missed: 4
match count for: 9 = 1004 / 1009 ( 99.50 %) missed: 5<jupyter_text>show all the missed numbers<jupyter_code>max_ct = len(missed_list)
rows = (max_ct / 7)
if (max_ct % 7) > 0 :
    rows +=1

if rows > 5 :
    rows = 5
    max_ct = 35
    print('showing 35 out of', len(missed_list), 'mismatches')

#increase the figsize for better view

fig = plt.figure(figsize=(10,5))
for i in range(max_ct):
    plt.subplot(rows,7,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    info = missed_list[i]
    index = info[0]
    plt.imshow(x_test[index] , cmap='gray_r')
    plt.xlabel(str(info[1])+", guess:" + str(info[2]), fontsize='9')
    plt.text(0,3,str(index),fontsize='8')
    
plt.show()
plt.close()

print('\nConfusion Matrix')

fig = plt.figure()
plt.xticks( range(10))
plt.yticks( range(10))

for y in range(10):
    for x in range(10):
        if my_matrix[y][x] != 0:
            # put text
            plt.text( x-len(str(x)) * 0.2, y+0.1, str(my_matrix[y][x]))
plt.xlabel('prediction')
plt.ylabel('expected')
plt.imshow(my_matrix, cmap='YlOrRd')
plt.colorbar()
plt.show()
plt.close()
expected_number = 9
guessed_number = 4

print('Expected:', expected_number, ', predicted:', guessed_number)
non_match_list = []
for i in range(10000):
    if y_test[i] == expected_number:
        guess = np.argmax(predictions[i])
            
        s1_guess = np.argmax(slave1_pred[i])
        s2_guess = np.argmax(slave2_pred[i])

        if s1_guess == s2_guess:
            guess = s1_guess
            
        if guess == guessed_number:
            non_match_list.append(i)

fig = plt.figure( figsize = (10,2))

for i in range(len(non_match_list)):
    plt.subplot(1,20,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    index = non_match_list[i]
    plt.imshow(x_test[index], cmap='gray_r')
plt.show()
plt.close()<jupyter_output>Expected: 9 , predicted: 4
{'fraction_non_alphanumeric': 0.1293216891972724, 'fraction_numerical': 0.049288192367508074, 'mean_word_length': 4.338441890166028, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '581542', 'license': 'Apache-2.0', 'n_tokens_mistral': 3312, 'n_tokens_neox': 2973, 'n_words': 875, 'path': 'read_model.ipynb', 'repo_name': 'sspak9/ml', 'size': '8359'}