<jupyter_start><jupyter_text>Plant Disease Classification with Monk  Setup<jupyter_code>!git clone https://github.com/Tessellate-Imaging/monk_v1.git
#!pip install -r /kaggle/working/monk_v1/installation/requirements_kaggle.txt
import sys
sys.path.append("./monk_v1/monk")
from pytorch_prototype import prototype<jupyter_output><empty_output><jupyter_text>Dataset The dataset was gathered by the awesome team at [Plant Village](https://plantvillage.org) The dataset is preprocessed and re-shared via dropbox.<jupyter_code>!wget https://www.dropbox.com/s/hgt9uystjlinzlp/plantVillage.zip
!unzip plantVillage.zip<jupyter_output><empty_output><jupyter_text>Create Experiment<jupyter_code># Step 1 - Create experiment
ptf = prototype(verbose=1);
ptf.Prototype("plant_disease", "exp1");
ptf.Default(dataset_path=["./dataset/train", "./dataset/val"], model_name="resnet18", freeze_base_network=True, num_epochs=5);<jupyter_output>Dataset Details
    Train path:     ./dataset/train
    Val path:       ./dataset/val
    CSV train path: None
    CSV val path:  None

Dataset Params
    Input Size:  224
    Batch Size:  4
    Data Shuffle: True
    Processors:   4

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Params
    Model name:           resnet18
    Use Gpu:              True
    Use pretrained:       True
    Freeze base network:  True

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           resnet18
        Num layers in model:  31
        Num trainable layers: 1

Optimizer
    Name:          sgd
    Learning rate: 0.01[...]<jupyter_text>Use Quick model finder<jupyter_code># Analysis - 1

# Analysis Project Name
analysis_name = "Model_Finder";

# Models to analyse
# First element in the list- Model Name
# Second element in the list - Boolean value to freeze base network or not
# Third element in the list - Boolean value to use pretrained model as the starting point or not
models = [["resnet34", True, True], ["resnet50", False, True], 
          ["densenet121", False, True], ["densenet169", True, True], ["densenet201", True, True]];  

# Num epochs for each experiment to run	
epochs=5;

# Percentage of original dataset to take in for experimentation
percent_data=10;

# "keep_all" - Keep all the sub experiments created
# "keep_non" - Delete all sub experiments created
ptf.Analyse_Models(analysis_name, models, percent_data, num_epochs=epochs, state="keep_none");<jupyter_output>Running Model analysis
Analysis Name      : Model_Finder

Running experiment : 1/5
Experiment name    : Model_resnet34_freeze_base_pretrained
Estimated time     : 2 min
Experiment Complete


Running experiment : 2/5
Experiment name    : Model_resnet50_unfreeze_base_pretrained
Estimated time     : 6 min
Experiment Complete


Running experiment : 3/5
Experiment name    : Model_densenet121_unfreeze_base_pretrained
Estimated time     : 8 min
Experiment Complete


Running experiment : 4/5
Experiment name    : Model_densenet169_freeze_base_pretrained
Estimated time     : 5 min
Experiment Complete


Running experiment : 5/5
Experiment name    : Model_densenet201_freeze_base_pretrained
Estimated time     : 6 min
Experiment Complete


Comparing Experiments
Comparison ID:      Comparison_Model_Finder
Generated statistics post all epochs
| Experiment Name                            |   Train Acc |   Val Acc |   Train Loss |   Val Loss |
|--------------------------------------------+-------------[...]<jupyter_text>Update the model<jupyter_code>## Update Model Architecture
ptf.update_model_name("densenet121");
ptf.update_freeze_base_network(True);
ptf.update_use_pretrained(True);
ptf.Reload();<jupyter_output>Update: Model name - densenet121

Update: Freeze Base Network - True

Update: Use pretrained - True

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           densenet121
        Num layers in model:  182
        Num trainable layers: 1<jupyter_text>Find the right batch size<jupyter_code>#Analysis - 2

# Analysis Project Name
analysis_name = "Batch_Size_Finder";

# Batch sizes to explore
batch_sizes = [4, 8, 16, 32];

# Num epochs for each experiment to run	
epochs = 10;

# Percentage of original dataset to take in for experimentation
percent_data = 10;

# "keep_all" - Keep all the sub experiments created
# "keep_non" - Delete all sub experiments created	
ptf.Analyse_Batch_Sizes(analysis_name, batch_sizes, percent_data, num_epochs=epochs, state="keep_none");<jupyter_output>Running Batch Size analysis
Analysis Name      : Batch_Size_Finder

Running experiment : 1/4
Experiment name    : Batch_Size_4
Estimated time     : 7 min
Epoch     5: reducing learning rate of group 0 to 1.0000e-03.
Epoch     8: reducing learning rate of group 0 to 1.0000e-04.
Experiment Complete


Running experiment : 2/4
Experiment name    : Batch_Size_8
Estimated time     : 5 min
Epoch     8: reducing learning rate of group 0 to 1.0000e-03.
Experiment Complete


Running experiment : 3/4
Experiment name    : Batch_Size_16
Estimated time     : 4 min
Experiment Complete


Running experiment : 4/4
Experiment name    : Batch_Size_32
Estimated time     : 4 min
Experiment Complete


Comparing Experiments
Comparison ID:      Comparison_Batch_Size_Finder
Generated statistics post all epochs
| Experiment Name   |   Train Acc |   Val Acc |   Train Loss |   Val Loss |
|-------------------+-------------+-----------+--------------+------------|
| Batch_Size_4      |    0.906043 |  0.930018 |    [...]<jupyter_text>Update batch size<jupyter_code>## Update Batch Size
ptf.update_batch_size(8);
ptf.Reload();<jupyter_output>Update: Batch size - 8

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           densenet121
        Num layers in model:  182
        Num trainable layers: 1<jupyter_text>Find the correct input dimension<jupyter_code>#Analysis - 3

# Analysis Project Name
analysis_name = "Input_Size_Finder";

# Input sizes to explore	
input_sizes = [224, 256, 512];

# Num epochs for each experiment to run	
epochs=5;

# Percentage of original dataset to take in for experimentation
percent_data=10;

# "keep_all" - Keep all the sub experiments created
# "keep_non" - Delete all sub experiments created	
ptf.Analyse_Input_Sizes(analysis_name, input_sizes, percent_data, num_epochs=epochs, state="keep_none");<jupyter_output>Running Input Size analysis
Analysis Name      : Input_Size_Finder

Running experiment : 1/3
Experiment name    : Input_Size_224
Estimated time     : 3 min
Experiment Complete


Running experiment : 2/3
Experiment name    : Input_Size_256
Estimated time     : 3 min
Experiment Complete


Running experiment : 3/3
Experiment name    : Input_Size_512
Estimated time     : 3 min
Experiment Complete


Comparing Experiments
Comparison ID:      Comparison_Input_Size_Finder
Generated statistics post all epochs
| Experiment Name   |   Train Acc |   Val Acc |   Train Loss |   Val Loss |
|-------------------+-------------+-----------+--------------+------------|
| Input_Size_224    |    0.903991 |  0.91989  |     0.418979 |   0.316026 |
| Input_Size_256    |    0.904903 |  0.899632 |     0.414027 |   0.364695 |
| Input_Size_512    |    0.905359 |  0.905157 |     0.418753 |   0.354991 |<jupyter_text>Update input size<jupyter_code>## Update Input Size

ptf.update_input_size(224);
ptf.Reload();<jupyter_output>Update: Input size - 224

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           densenet121
        Num layers in model:  182
        Num trainable layers: 1<jupyter_text>Findout the correct starting LR<jupyter_code>#Analysis - 4

# Analysis Project Name
analysis_name = "Learning_Rate_Finder"

# Learning rates to explore
lrs = [0.01, 0.005, 0.001, 0.0001];

# Num epochs for each experiment to run
epochs=5

# Percentage of original dataset to take in for experimentation
percent_data=10

# "keep_all" - Keep all the sub experiments created
# "keep_non" - Delete all sub experiments created
ptf.Analyse_Learning_Rates(analysis_name, lrs, percent_data, num_epochs=epochs, state="keep_none");<jupyter_output>Running Learning rate analysis
Analysis Name      : Learning_Rate_Finder

Running experiment : 1/4
Experiment name    : Learning_Rate_0.01
Estimated time     : 3 min
Experiment Complete


Running experiment : 2/4
Experiment name    : Learning_Rate_0.005
Estimated time     : 3 min
Experiment Complete


Running experiment : 3/4
Experiment name    : Learning_Rate_0.001
Estimated time     : 3 min
Experiment Complete


Running experiment : 4/4
Experiment name    : Learning_Rate_0.0001
Estimated time     : 3 min
Experiment Complete


Comparing Experiments
Comparison ID:      Comparison_Learning_Rate_Finder
Generated statistics post all epochs
| Experiment Name      |   Train Acc |   Val Acc |   Train Loss |   Val Loss |
|----------------------+-------------+-----------+--------------+------------|
| Learning_Rate_0.01   |    0.90764  |  0.895028 |     0.411713 |   0.362229 |
| Learning_Rate_0.005  |    0.877537 |  0.901473 |     0.611189 |   0.442408 |
| Learning_Rate_0.001  |    0.690992 |[...]<jupyter_text>Update LR<jupyter_code>## Update Learning Rate
ptf.update_learning_rate(0.01);
ptf.Reload();<jupyter_output>Update: Learning Rate - 0.01

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           densenet121
        Num layers in model:  182
        Num trainable layers: 1<jupyter_text>Optimiser hunting<jupyter_code># Analysis - 5
# Analysis Project Name
analysis_name = "Optimiser_Finder";

# Optimizers to explore
optimizers = ["sgd", "adam", "adamax", "rmsprop"];   #Model name, learning rate

# Num epochs for each experiment to run	
epochs = 5;

# Percentage of original dataset to take in for experimentation
percent_data = 10;

# "keep_all" - Keep all the sub experiments created
# "keep_non" - Delete all sub experiments created
ptf.Analyse_Optimizers(analysis_name, optimizers, percent_data, num_epochs=epochs, state="keep_none");<jupyter_output>Running Optimizer analysis
Analysis Name      : Optimiser_Finder

Running experiment : 1/4
Experiment name    : Optimizer_sgd
Estimated time     : 3 min
Experiment Complete


Running experiment : 2/4
Experiment name    : Optimizer_adam
Estimated time     : 3 min
Epoch     3: reducing learning rate of group 0 to 1.0000e-03.
Experiment Complete


Running experiment : 3/4
Experiment name    : Optimizer_adamax
Estimated time     : 3 min
Experiment Complete


Running experiment : 4/4
Experiment name    : Optimizer_rmsprop
Estimated time     : 3 min
Experiment Complete


Comparing Experiments
Comparison ID:      Comparison_Optimiser_Finder
Generated statistics post all epochs
| Experiment Name   |   Train Acc |   Val Acc |   Train Loss |   Val Loss |
|-------------------+-------------+-----------+--------------+------------|
| Optimizer_sgd     |    0.903079 |  0.904236 |     0.424674 |   0.344579 |
| Optimizer_adam    |    0.941163 |  0.939227 |     0.254955 |   0.351743 |
| Optimizer_adam[...]<jupyter_text>Update optimiser<jupyter_code>## Update Optimiser

ptf.optimizer_adamax(0.001);
ptf.Reload();<jupyter_output>Optimizer
    Name:          adamax
    Learning rate: 0.001
    Params:        {'lr': 0.001, 'dampening': 0, 'nesterov': False, 'weight_decay': 0, 'momentum': 0, 'betas': (0.9, 0.999), 'eps': 1e-08}

Pre-Composed Train Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Pre-Composed Val Transforms
[{'RandomHorizontalFlip': {'p': 0.8}}, {'Normalize': {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}}]

Dataset Numbers
    Num train images: 44014
    Num val images:   11004
    Num classes:      39

Model Details
    Loading pretrained model
    Model Loaded on device
        Model name:           densenet121
        Num layers in model:  182
        Num trainable layers: 1<jupyter_text>Set intermediate state saving to false This step reduces the size of our workspaces<jupyter_code>ptf.update_save_intermediate_models(False);<jupyter_output>Update: Save Intermediate models - False<jupyter_text>Train<jupyter_code>ptf.Train();<jupyter_output>Training Start
    Epoch 1/5
    ----------<jupyter_text>We can see that the losses can still go down. Next we create a new experiment using 'copy_experiment' and resume training to achieve further improvement.<jupyter_code>ptf = prototype(verbose=1);
ptf.Prototype("plant_disease", "exp2", copy_from=["plant_disease", "exp1"]);
ptf.Summary()
ptf.update_num_epochs(10)
ptf.Reload()
ptf.Train()<jupyter_output>Training Start
    Epoch 1/10
    ----------<jupyter_text>Compare<jupyter_code>from compare_prototype import compare
ctf = compare(verbose=1);
ctf.Comparison("plant_disease");
ctf.Add_Experiment("plant_disease", "exp1");
ctf.Add_Experiment("plant_disease", "exp2");
ctf.Generate_Statistics()<jupyter_output>Generating statistics...
Generated<jupyter_text>Inference<jupyter_code>import numpy as np
import PIL.Image as Image
import requests

test_url = "https://gardenerspath.com/wp-content/uploads/2019/08/Frogeye-spots-Botryosphaeria-obtusa-on-apple-leaf-FB.jpg"

# URL of the image to be downloaded is defined as image_url 
r = requests.get(test_url) # create HTTP response object 
with open('test.jpg','wb') as f: 
    f.write(r.content) 

test = Image.open('./test.jpg')
test
ptf.Prototype("plant_disease", "exp2", eval_infer=True);
img_name = "test.jpg";
predictions = ptf.Infer(img_name=img_name, return_raw=True);
print(predictions)
import csv

def read_labels():
    mydict = {}
    with open('./dataset/labels.csv', mode='r') as infile:
        reader = csv.reader(infile)
        with open('labels_new.csv', mode='w') as outfile:
            writer = csv.writer(outfile)
            mydict = {rows[0]:rows[1] for rows in reader}
    return mydict

def predict_label(predictions):
    pred_class = predictions['predicted_class']
    label_dict = read_labels()
    out_label = label_dict[pred_class]
    return out_label

print("Predicted class for test image is : {}".format(predict_label(predictions)))<jupyter_output>Predicted class for test image is : Apple___Black_rot
{'fraction_non_alphanumeric': 0.12469756188349153, 'fraction_numerical': 0.058998697189651964, 'mean_word_length': 3.5217391304347827, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '228920', 'license': 'Apache-2.0', 'n_tokens_mistral': 5869, 'n_tokens_neox': 5201, 'n_words': 1509, 'path': 'V) RoadMap 23 - Appendix 5 - Plant Disease Classification with Monk.ipynb', 'repo_name': 'Charlottecuc/Pytorch_Tutorial', 'size': '16119'}