<jupyter_start><jupyter_text>Import library<jupyter_code>from fastai import *
from fastai.vision import *<jupyter_output><empty_output><jupyter_text>Path to dataset<jupyter_code>path = Path("/notebooks/storage/dataset-resized"); path<jupyter_output><empty_output><jupyter_text>Create the fastai image data bunch<jupyter_code>data = ImageDataBunch.from_folder(path,train='train',valid='valid',test='test',ds_tfms=get_transforms(),bs=64,size=224).normalize(imagenet_stats); data<jupyter_output><empty_output><jupyter_text>Import the trained resnet 50 model<jupyter_code>learn = cnn_learner(data, models.resnet50, metrics=[error_rate])<jupyter_output><empty_output><jupyter_text>Find and plot the learning rate<jupyter_code>learn.lr_find()
learn.recorder.plot()<jupyter_output><empty_output><jupyter_text>Fit the model to the dataset<jupyter_code>learn.fit_one_cycle(10,max_lr=5e-04)
learn.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>Display the errors made on validation set<jupyter_code>interp = ClassificationInterpretation.from_learner(learn)
losses, idxs = interp.top_losses()
interp.plot_top_losses(10, figsize=(15,15))<jupyter_output><empty_output><jupyter_text>Plot the confusion matrix for validation set<jupyter_code>interp.plot_confusion_matrix()<jupyter_output><empty_output><jupyter_text>Export model alongwith all transformations, classes, normalization, etc on dataset<jupyter_code>learn.export('export-res50')<jupyter_output><empty_output><jupyter_text>Save the model weights<jupyter_code>learn.save("stage-3-res50",return_path = True)<jupyter_output><empty_output><jupyter_text>Get predictions on test set<jupyter_code>preds = learn.get_preds(ds_type=DatasetType.Test); preds[0]
max(preds[0][0]).item()
#reds[0][0]
output_list = []
for i in range (len(preds[0])):
    indx = np.argmax(preds[0][i]).item()
    prediction = str(preds[0][i][indx].item())
    output_list.append({data.classes[indx]:prediction})
print(len(output_list))<jupyter_output><empty_output><jupyter_text>Lets pick a random test image and see how well we have done.<jupyter_code>print(output_list[279])
learn.data.test_ds[279][0]<jupyter_output><empty_output><jupyter_text>Overload the image data bunch and supply test set along with labels as the valid set to get confusion matrix and all<jupyter_code>data_test = ImageDataBunch.from_folder(path, train='train', valid='test', bs=64, size=224, ds_tfms = get_transforms()).normalize(imagenet_stats)
learn_test = cnn_learner(data_test, models.resnet50, metrics=[error_rate])
learn_test.load('stage-3-res50')
interp_test = ClassificationInterpretation.from_learner(learn_test)

losses,idxs = interp_test.top_losses()<jupyter_output><empty_output><jupyter_text>Add the diagonal (going left to right downwards) numbers and divide by 435 to get accuracy = 88.505%<jupyter_code>interp_test.plot_confusion_matrix()
interp.plot_top_losses(10,figsize=(15,15))<jupyter_output><empty_output><jupyter_text>Now lets try to improve the model further. Lets now train all layers of resnet 50. Previously we were training only a couple of layers<jupyter_code>learn.unfreeze()
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(6, slice(1e-05, 8e-04))
learn.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>Validation error has reduced!!<jupyter_code>interp = ClassificationInterpretation.from_learner(learn)
losses, idxs = interp.top_losses()
interp.plot_confusion_matrix()
learn.save('stage-4-res50')
data_test = ImageDataBunch.from_folder(path, train='train', valid='test', bs=64, size=224, ds_tfms = get_transforms()).normalize(imagenet_stats)
learn_test = cnn_learner(data_test, models.resnet50, metrics=[error_rate])
learn_test.load('stage-4-res50')
interp_test = ClassificationInterpretation.from_learner(learn_test)
losses,idxs = interp_test.top_losses()
interp_test.plot_confusion_matrix()<jupyter_output><empty_output><jupyter_text>We achieved 91.034% accuracy on test set!!<jupyter_code>learn.export('stage-4-res50')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12319561971129915, 'fraction_numerical': 0.0233947237431558, 'mean_word_length': 11.032934131736527, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '351766', 'license': 'MIT', 'n_tokens_mistral': 1506, 'n_tokens_neox': 1447, 'n_words': 220, 'path': 'waste-classifier2.ipynb', 'repo_name': 'ParthJai/trashnet', 'size': '4018'}