<jupyter_start><jupyter_text>MNIST DatasetI designed an 18 layer ResNet and used it to perform digit classification on the MNIST dataset. I ran a total of 16 Epochs on the training data. The batch size is 128, and I varied the learning rate and if regularization is used. The following is the summary of the results.| Epoch       | Learning rate | Regularization  | Optimizer | Test set accuracy | Train set accuracy ||:-----------:|:-------------:|:---------------:|:---------:|:-----------------:|:------------------:|| 1-4         | 0.025         | 0.0001          | Adam      | 0.8829            | 0.87615            || 5-8         | 0.025         | No Reg.         | Adam      | 0.9755            | 0.9701             || 9-12        | 0.01          | 0.0001          | Adam      | 0.9861            | 0.9918             || 13-16       | 0.005         | 0.0001          | Adam      | 0.9927            | 0.9995             |The following is the result of the last run.<jupyter_code>%run ../NNSolver/allModules.py
import time
import matplotlib.pyplot as pt
import sys
HPara = {}
HPara['BATCH_SIZE'] = 128           # Mini batch size
HPara['ALPHA'] = 0.005             # Learning rate
HPara['REGULARIZATION'] = True   # True or False
HPara['LAMBDA'] = 0.0001           # L2 regularization weight decay
HPara['OPTIMIZER'] = "Adam"      # 'Basic' - simple gradient descent, 'Momentum' - momentum, 'Adam'
HPara['BETAM'] = 0.9              # Momentum parameter
HPara['BETAR'] = 0.99
HPara['INITIALIZATION'] = 'He-2015'  
data = np.load('../Dataset/MNIST/mnistData.npz')
mnistSrc = ImageDataSource({'instanceName':'mnistSrc'})
mnistSrc.setTrainData(data['trainImages'],data['trainLabels'])
mnistSrc.setTestData(data['testImages'],data['testLabels'])
pSize = 7
rn18 = RN18({'instanceName':'rn18','c1OutChannel':16,'c2OutChannel':32,
          'rnb1OutChannel':32,'rnb4OutChannel':64,'pSize':pSize,'classNum':10})
imageSolver = Solver(mnistSrc,rn18)
Layer.setHyperParameters(HPara)
# number of past mean and std in normalization layer kept. used in inference mode
Normalize.numStats = 100
imageSolver.createStruct()
imageSolver.loadParameters("mnist-rn18-parameters3.npz")
#imageSolver.solveNMiniBatch(5)
Solver.miniCnt = 50
N = 4
imageSolver.solveNEpoch(N)
imageSolver.saveParameters('mnist-rn18-parameters4.npz')
loss = rn18.lossData()
pt.plot(loss)
testAccuracy = imageSolver.calcAccuracy('test')
trainAccuracy = imageSolver.calcAccuracy('train')<jupyter_output>0  50  100  150  200  250  300  350  400  450  
Time to calculate  train  data accuracy:  1211.8214211463928  seconds.
Accuracy:  0.9995333333333334  with data set size  60000
{'fraction_non_alphanumeric': 0.14838709677419354, 'fraction_numerical': 0.08159392789373814, 'mean_word_length': 2.6459197786998616, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '315995', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 991, 'n_tokens_neox': 833, 'n_words': 215, 'path': 'Notebooks/RN18-MNIST-Results.ipynb', 'repo_name': 'mkaywong/NNSolver', 'size': '2635'}