<jupyter_start><jupyter_text>CIFAR-10 DatasetI implemented an 24 layer ResNet and used it to perform image classification on the CIFAR-10 dataset. I ran a total of 36 Epochs on the training data. The batch size is 128, and I varied the learning rate, optimizer and if regularization is used. The following is the summary of the results.| Epoch       | Learning rate | Regularization  | Optimizer | Test set accuracy | Train set accuracy ||:-----------:|:-------------:|:---------------:|:---------:|:-----------------:|:------------------:|| 1-8         | 0.05          | 0.0001          | Adam      | 0.5128            | 0.527              || 9-16        | 0.025         | No Reg.         | Adam      | 0.6248            | 0.64496            || 16-20       | 0.02          | 0.0001          | Adam      | 0.7088            | 0.75986            || 21-24       | 0.01          | 0.0001          | Adam      | 0.7417            | 0.81488            || 25-28       | 0.005         | 0.0001          | Adam      | 0.7552            | 0.8407             || 29-36       | 0.01          | 0.0001          | Momentum  | 0.7618            | 0.85436            |The following is the result of the last run.<jupyter_code>%run ../NNSolver/allModules.py
import time
import matplotlib.pyplot as pt
import sys
HPara = {}
HPara['BATCH_SIZE'] = 128           # Mini batch size
HPara['ALPHA'] = 0.01             # Learning rate
HPara['REGULARIZATION'] = True   # True or False
HPara['LAMBDA'] = 0.0001           # L2 regularization weight decay
HPara['OPTIMIZER'] = "Momentum"      # 'Basic' - simple gradient descent, 'Momentum' - momentum, 'Adam'
HPara['BETAM'] = 0.9              # Momentum parameter
HPara['BETAR'] = 0.99
HPara['INITIALIZATION'] = 'He-2015'  
data = np.load('../Dataset/cifar-10/cifar10Data.npz')
imageSrc = ImageDataSource({'instanceName':'cifar10Src'})
imageSrc.setTrainData(data['trainImages'],data['trainLabels'])
imageSrc.setTestData(data['testImages'],data['testLabels'])
rn24 = RN24({'instanceName':'rn24','c1OutChannel':16,'c2OutChannel':32,
          'rnb1OutChannel':32,'rnb4OutChannel':64,'pSize':8,'classNum':10})
imageSolver = Solver(imageSrc,rn24)
Layer.setHyperParameters(HPara)
Normalize.numStats = 100
imageSolver.createStruct()
#imageSolver.printIOShape()
imageSolver.loadParameters("cifar10-rn24-parameters5.npz")
#imageSolver.solveNMiniBatch(10)
Solver.miniCnt = 50
N = 8
imageSolver.solveNEpoch(N)
imageSolver.saveParameters('cifar10-rn24-parameters6.npz')
loss = rn24.lossData()
pt.plot(loss)
testAccuracy = imageSolver.calcAccuracy('test')
trainAccuracy = imageSolver.calcAccuracy('train')<jupyter_output>0  50  100  150  200  250  300  350  
Time to calculate  train  data accuracy:  1696.2804639339447  seconds.
Accuracy:  0.85436  with data set size  50000
{'fraction_non_alphanumeric': 0.15080789946140036, 'fraction_numerical': 0.09299820466786356, 'mean_word_length': 2.3285543608124253, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '221061', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1072, 'n_tokens_neox': 886, 'n_words': 211, 'path': 'Notebooks/RN24-cifar10-Results.ipynb', 'repo_name': 'mkaywong/NNSolver', 'size': '2785'}