<jupyter_start><jupyter_text>User Params<jupyter_code>basepath = Path('/home/jupyter/data')
dataset_name = 'fashion_mnist_tiny'
n_epoch = 100<jupyter_output><empty_output><jupyter_text>Derived Params and Helpers<jupyter_code>path_img = basepath/'imagenet_style'/dataset_name
labels_file = basepath/dataset_name/'labels.csv'

def create_learner(data, savename='dummy', patience=8, modeltype=models.resnet34):
    learn = create_cnn(data, 
                       modeltype, 
                       metrics=error_rate, 
                       callback_fns=[partial(SaveModelCallback, name=savename), 
                                     partial(EarlyStoppingCallback, patience=patience),
                                     ShowGraph])   
    return learn<jupyter_output><empty_output><jupyter_text>Create dataset<jupyter_code>data = ImageDataBunch.from_folder(path_img, 
                                  train='train', 
                                  valid='valid', 
                                  test='test', 
                                  ds_tfms=get_transforms(), 
                                  size=224)
data.normalize(imagenet_stats)
print(data.classes)<jupyter_output>['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']<jupyter_text>Display dataset<jupyter_code>data.show_batch(rows=6)<jupyter_output><empty_output><jupyter_text>Train a standard transfer learning model (stage 1)<jupyter_code>learn = create_learner(data, 'stage1-bestmodel')
learn.fit_one_cycle(n_epoch)<jupyter_output><empty_output><jupyter_text>Show results from generic evaluation tool<jupyter_code>results_val = util.eval_rollup(labels_file, learn, evalset='val')
results_test, y_true, y_pred, scores, class_labels = util.eval_rollup(labels_file, learn, evalset='test')<jupyter_output>Dataset: test
{'sklearn': {'acc': 0.7923,
             'auc_avg': 0.9763430555555555,
             'auc_indiv': array([0.975064, 0.996667, 0.965624, 0.979573, 0.96401 , 0.994507, 0.908072, 0.990127, 0.995152, 0.994634]),
             'confmat': array([[748,   2,  37,  42, ..., 138,   0,  15,   0],
       [  0, 959,   4,  17, ...,  11,   0,   2,   1],
       [ 11,   1, 707,   5, ..., 126,   0,   8,   0],
       [ 41,  32,  28, 736, ..., 101,   0,   1,   2],
       ...,
       [140,   6, 100,  34, ..., 488,   0,  24,   2],
       [  0,   0,   1,   0, ...,   0, 764,   4,  87],
       [  2,   1,   6,   7, ...,   6,   4, 949,   3],
       [  2,   1,   2,   0, ...,   4,  56,   9, 906]]),
             'report': '             precision    recall  f1-score   support\n'
                       '\n'
                       '          0       0.79      0.75      0.77      1000\n'
                       '          1       0.96      0.96      0.96      1000\n'
                       '          2      [...]<jupyter_text>Show results using FastAI to confirm matches<jupyter_code># Confirm fastai reporting matching the generic one used for other platforms
# Some formatting work needed below to run on test and not validation dataset
y_true_num = tensor([data.classes.index(z) for z in y_true])  # Convert strings to numbers (tensor)
preds,y,losses = learn.get_preds(with_loss=True, is_test=True)
interp = ClassificationInterpretation(data, preds, tensor(y_true_num), losses)
#interp.plot_top_losses(15, figsize=(15,11))
interp.plot_confusion_matrix(figsize=(6,6), dpi=60)<jupyter_output><empty_output><jupyter_text>Fastai confusion matrix matches the generic one for the test dataset.  Unfreezing, fine-tuning, and learning rates First let's try just unfreezing the previously trained model and continue running some more training to see how well it can do.<jupyter_code>learn = create_learner(data, 'unfreeze-bestmodel')
learn.load('stage1-bestmodel')
learn.unfreeze()
learn.fit_one_cycle(n_epoch)
results_test, y_true, y_pred, scores, class_labels = util.eval_rollup(labels_file, learn, evalset='test')<jupyter_output>Dataset: test
{'sklearn': {'acc': 0.7877,
             'auc_avg': 0.9738390888888888,
             'auc_indiv': array([0.974002, 0.991656, 0.962743, 0.978037, 0.955261, 0.993845, 0.903029, 0.990936, 0.994095, 0.994787]),
             'confmat': array([[778,   1,  44,  88, ...,  52,   1,  12,   1],
       [  3, 917,   5,  59, ...,   7,   0,   2,   0],
       [ 15,   1, 629,   8, ...,  66,   0,   2,   0],
       [ 35,   6,  16, 849, ...,  48,   2,   6,   0],
       ...,
       [193,   3,  82,  75, ..., 306,   0,  25,   0],
       [  0,   0,   0,   0, ...,   0, 881,   4,  42],
       [  9,   1,  27,  17, ...,   8,   6, 895,  16],
       [  0,   0,   0,   4, ...,   0,  68,   3, 915]]),
             'report': '             precision    recall  f1-score   support\n'
                       '\n'
                       '          0       0.75      0.78      0.76      1000\n'
                       '          1       0.98      0.92      0.95      1000\n'
                       '          2      [...]<jupyter_text>Unfreeze but use identified learning rates Now let's go back to the original model that only trained the head again. But this time, we'll find the best learning rate to use for this, and then use those learning rates with the unfrozen model. First, find and plot the learning rates using lr_find.<jupyter_code>learn = create_learner(data)
learn.load('stage1-bestmodel')
learn.callback_fns = [Recorder]  # Need to flush callbacks or else lr_find crashes    
learn.lr_find()
learn.recorder.plot()<jupyter_output>LR Finder complete, type {learner_name}.recorder.plot() to see the graph.<jupyter_text>Now that we have the learning rate range, let's feed that into the model and rerun<jupyter_code># Reload with new savename
learn = create_learner(data, 'stage2-bestmodel')
learn.load('stage1-bestmodel')

# Unfreeze and kick off 
learn.unfreeze()
learn.fit_one_cycle(n_epoch, max_lr=slice(1e-5,1e-3))
results_test, y_true, y_pred, scores, class_labels = util.eval_rollup(labels_file, learn, evalset='test')<jupyter_output>Dataset: test
{'sklearn': {'acc': 0.8042,
             'auc_avg': 0.9783240888888889,
             'auc_indiv': array([0.978482, 0.996785, 0.966823, 0.981564, 0.965603, 0.995754, 0.913757, 0.992395, 0.995944, 0.996133]),
             'confmat': array([[726,   1,  20,  63, ..., 170,   0,  11,   0],
       [  0, 963,   4,  15, ...,  10,   0,   2,   0],
       [ 10,   0, 702,   6, ..., 146,   0,   1,   0],
       [ 14,  29,  19, 794, ..., 102,   0,   1,   3],
       ...,
       [118,   5,  94,  48, ..., 530,   0,  17,   3],
       [  0,   0,   1,   0, ...,   0, 816,   3,  87],
       [  4,   3,   5,   6, ...,  12,   6, 931,   5],
       [  1,   2,   0,   0, ...,   2,  45,   4, 931]]),
             'report': '             precision    recall  f1-score   support\n'
                       '\n'
                       '          0       0.83      0.73      0.77      1000\n'
                       '          1       0.96      0.96      0.96      1000\n'
                       '          2      [...]<jupyter_text>Try Resnet50<jupyter_code>data = ImageDataBunch.from_folder(path_img, 
                                  train='train', 
                                  valid='valid', 
                                  test='test', 
                                  ds_tfms=get_transforms(), 
                                  size=299,
                                  bs=32)
data.normalize(imagenet_stats)<jupyter_output><empty_output><jupyter_text>Try training just the head<jupyter_code>learn = create_learner(data, 'resnet50-stage1-bestmodel', modeltype=models.resnet50)
learn.fit_one_cycle(n_epoch)
results_test, y_true, y_pred, scores, class_labels = util.eval_rollup(labels_file, learn, evalset='test')<jupyter_output>Dataset: test
{'sklearn': {'acc': 0.7876,
             'auc_avg': 0.9731625777777777,
             'auc_indiv': array([0.974311, 0.995005, 0.961798, 0.968452, 0.953383, 0.99567 , 0.898416, 0.993842, 0.995918, 0.99483 ]),
             'confmat': array([[669,   1,  22,  72, ..., 191,   3,  18,   4],
       [  0, 946,   8,  14, ...,  25,   0,   4,   0],
       [  6,   0, 698,   3, ..., 161,   0,   4,   0],
       [ 21,  23,  18, 677, ..., 207,   1,   7,   3],
       ...,
       [101,   1,  56,  46, ..., 556,   0,  29,   1],
       [  0,   0,   0,   0, ...,   0, 907,   1,  35],
       [  4,   0,   6,   4, ...,  12,   7, 941,   5],
       [  1,   0,   1,   1, ...,   1,  67,   8, 883]]),
             'report': '             precision    recall  f1-score   support\n'
                       '\n'
                       '          0       0.83      0.67      0.74      1000\n'
                       '          1       0.97      0.95      0.96      1000\n'
                       '          2      [...]<jupyter_text>Unfreeze, finetune with learning rates Get the learning rate separately, like before<jupyter_code>learn = create_learner(data, modeltype=models.resnet50)
learn.load('resnet50-stage1-bestmodel')
learn.callback_fns = [Recorder]  # Need to flush callbacks or else lr_find crashes    
learn.lr_find()
learn.recorder.plot()<jupyter_output>LR Finder complete, type {learner_name}.recorder.plot() to see the graph.<jupyter_text>Reload the transfer learning results, unfreeze, and train using the range of learning rates.<jupyter_code># Reload with new savename
learn = create_learner(data, 'resnet50-stage2-bestmodel', modeltype=models.resnet50)
learn.load('resnet50-stage1-bestmodel')

# Unfreeze and kick off 
learn.unfreeze()
learn.fit_one_cycle(n_epoch, max_lr=slice(1e-6,1e-4))
results_test, y_true, y_pred, scores, class_labels = util.eval_rollup(labels_file, learn, evalset='test')<jupyter_output>Dataset: test
{'sklearn': {'acc': 0.8056,
             'auc_avg': 0.9766280277777778,
             'auc_indiv': array([0.976678, 0.994995, 0.965617, 0.976534, 0.958384, 0.996302, 0.910759, 0.994735, 0.996303, 0.995972]),
             'confmat': array([[733,   1,  23,  64, ..., 145,   0,  17,   3],
       [  0, 958,   7,  11, ...,  16,   0,   2,   1],
       [  8,   1, 691,   5, ..., 146,   0,   6,   0],
       [ 26,  21,  17, 742, ..., 141,   0,   8,   3],
       ...,
       [129,   2,  53,  46, ..., 545,   0,  31,   2],
       [  0,   0,   0,   0, ...,   0, 910,   1,  36],
       [  2,   1,   6,   8, ...,   9,   4, 951,   4],
       [  0,   0,   1,   1, ...,   1,  58,  10, 903]]),
             'report': '             precision    recall  f1-score   support\n'
                       '\n'
                       '          0       0.81      0.73      0.77      1000\n'
                       '          1       0.97      0.96      0.97      1000\n'
                       '          2      [...]
{'fraction_non_alphanumeric': 0.14404458297912534, 'fraction_numerical': 0.11202418059884764, 'mean_word_length': 1.7870492234798632, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 5, 'word_list_counts': {'cursed_substrings.json': 44, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '257610', 'license': 'Apache-2.0', 'n_tokens_mistral': 4472, 'n_tokens_neox': 3814, 'n_words': 883, 'path': 'fastai/fastai-fashion_mnist_tiny.ipynb', 'repo_name': 'tszumowski/custom-vision-study', 'size': '10587'}