<jupyter_start><jupyter_text>ImageDataBunch.from_folder takes Train dataset and depending on valid_pct has been provided or not it creates the validation set from train itself. If there is validation folder it puts the validation data from that folder and if nothing is provided train alone is created.<jupyter_code>np.random.seed(70)
data_res34 = ImageDataBunch.from_folder(path=train_path, train='TRAIN',valid='TEST', ds_tfms=get_transforms(flip_vert=True,), size=224, bs=64).normalize(imagenet_stats)
len(data_res34.train_dl.dl.dataset.x)
data_res34.show_batch(rows=3)
learn_res34 = create_cnn(data=data_res34, arch=models.resnet34, metrics=error_rate)
learn_res34.fit_one_cycle(3)
learn_res34.recorder.plot_losses()
learn_res34.save("learn_res34-3e")<jupyter_output><empty_output><jupyter_text>Above 2 epochs the error rate was increasing. Validation loss has increased in 2nd epoch but error rate has gone down.<jupyter_code>interpret_res34 = ClassificationInterpretation.from_learner(learn_res34)
interpret_res34.plot_confusion_matrix()
interpret_res34.most_confused()
interpret_res34.plot_top_losses()<jupyter_output><empty_output><jupyter_text>Check if we can have good lr for whole model or we can improve it.<jupyter_code>learn_res34.unfreeze()
learn_res34.fit_one_cycle(3)
learn_res34.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>It looks like the default lr is fine and we should make it more aggressive<jupyter_code>learn_res34.save("learn_res34_uf-3e")<jupyter_output><empty_output><jupyter_text>This clearly shows that we have better LR we will use the same LR and train on unfreezed model<jupyter_code>learn_res34.load('learn_res34-3e')
learn_res34.lr_find()
learn_res34.recorder.plot(skip_start=20)
learn_res34.unfreeze()
learn_res34.fit_one_cycle(5, max_lr=slice(9e-5,9e-4))
learn_res34.save("learn-uf-34-e5")
learn_res34.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>The losses are decreasing but they are far apart and also the validation loss is fluctuating<jupyter_code>interpret_res34_uf = ClassificationInterpretation.from_learner(learn_res34)
interpret_res34_uf.plot_confusion_matrix()
data2 = ImageDataBunch.single_from_classes(train_path, data_res34.classes, tfms=get_transforms(), size=224).normalize(imagenet_stats)
learn = create_cnn(data2, models.resnet34)
learn.load('learn-uf-34-e5')
preds = [learn.predict(open_image(img))[0] for img in orgfnames]
len(orgfnames)
org_pred.shape
res = pd.DataFrame({preds, org_pred.Category})
org_pred.Category
np.random.seed(70)
data50 = ImageDataBunch.from_folder(path=train_path, train='TRAIN',valid='TEST', ds_tfms=get_transforms(flip_vert=True,), size=299, bs=32).normalize(imagenet_stats)
learn50 = create_cnn(data50, arch=models.resnet50, metrics=[error_rate, accuracy])
learn50.fit_one_cycle(2)
learn50.save("lear50-1")
learn50.unfreeze()
learn50.fit_one_cycle(1)
learn50.load("lear50-1")
learn50.lr_find()
learn50.recorder.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10487394957983193, 'fraction_numerical': 0.04605042016806723, 'mean_word_length': 9.628571428571428, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '302843', 'license': 'MIT', 'n_tokens_mistral': 1142, 'n_tokens_neox': 1069, 'n_words': 194, 'path': 'Fastai_Projects/BloodCell_Clasification.ipynb', 'repo_name': 'rameshsinghds/intro-to-ds-with-python', 'size': '2975'}