<jupyter_start><jupyter_text>Coffee Leaf Rust detection ☕🌿[Hemileia vastatrix](https://en.wikipedia.org/wiki/Hemileia_vastatrix) a fungus which causes coffee leaf rust disease. This disease which reduces a plants ability to derive energy through photosynthesis, is extremely damaging to economics built on coffee cultivation as it can wipe out whole crops. The disease can be identified by the spores which cover the plants leaves.<jupyter_code>from datetime import datetime
import pathlib
import random
import shutil
from typing import List
import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import tensorflow as tf
from tensorflow import keras

from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import load_img

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Getting our Data 💾<jupyter_code>! pip install -q kaggle<jupyter_output><empty_output><jupyter_text>In order to use the Kaggle’s public API, you must first authenticate using an API token. From the site header, click on your user profile picture, then on “My Account” from the dropdown menu. This will take you to your account settings at https://www.kaggle.com/account. Scroll down to the section of the page labelled API:To create a new token, click on the “Create New API Token” button. This will download a fresh authentication token onto your machine in the form of a `kaggle.json`The cell below once run will require you to select this file for upload.<jupyter_code>from google.colab import files
files.upload()
! mkdir ~/.kaggle

! cp kaggle.json ~/.kaggle/
! chmod 600 ~/.kaggle/kaggle.json
! kaggle datasets download -d 'badasstechie/coffee-leaf-diseases'
! mkdir coffee-leaf-diseases

! unzip coffee-leaf-diseases.zip -d coffee-leaf-diseases &> /dev/null
# clean up downloaded zip
!rm -r coffee-leaf-diseases.zip<jupyter_output><empty_output><jupyter_text>Creating Train and Test data sets 🗂️The first task is to split up our data so we have images for both our two classes: `rust` and `other` (disease that is not identified as rust).<jupyter_code>train_data = pd.read_csv("coffee-leaf-diseases/train_classes.csv")
test_data = pd.read_csv("coffee-leaf-diseases/test_classes.csv")
train_data[train_data["rust"] == 1]<jupyter_output><empty_output><jupyter_text>We see here that images can be labelled both rust and miner. We kick out the images with double labels.<jupyter_code>train_rust = train_data[(train_data["rust"] == 1) & (train_data["miner"] != 1) & (train_data["phoma"] != 1)]
train_rust_ids = train_rust["id"].to_list()
other_train_ids = train_data[(train_data["rust"] == 0)]["id"].to_list()
test_rust = test_data[(test_data["rust"] == 1) & (test_data["miner"] != 1) & (test_data["phoma"] != 1)]
test_rust_ids = test_rust["id"].to_list()
other_test_ids = test_data[(test_data["rust"] == 0)]["id"].to_list()
print(f"Number of train rust images: {len(train_rust_ids)}")
print(f"Number of test rust images: {len(test_rust_ids)}")
os.mkdir("coffee-leaf-diseases/coffee-leaf-diseases/train/images/rust")
os.mkdir("coffee-leaf-diseases/coffee-leaf-diseases/train/images/other")

os.mkdir("coffee-leaf-diseases/coffee-leaf-diseases/test/images/rust")
os.mkdir("coffee-leaf-diseases/coffee-leaf-diseases/test/images/other")
for n in train_rust_ids:
    shutil.move(f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/{n}.jpg", f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/rust/")
for n in test_rust_ids:
    shutil.move(f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/{n}.jpg", f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/rust/")
for n in other_train_ids:
    shutil.move(f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/{n}.jpg", f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/other/")
for n in other_test_ids:
    shutil.move(f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/{n}.jpg", f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/other/")<jupyter_output><empty_output><jupyter_text>Let's take a look at the kind of image we have.<jupyter_code>img_path = f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/rust/{train_rust_ids[0]}.jpg"
rust_leaf = load_img(img_path, target_size=(299, 299))
print(rust_leaf)
rust_leaf
img_path = f"./coffee-leaf-diseases/coffee-leaf-diseases/train/images/other/{other_train_ids[0]}.jpg"
non_rust_leaf = load_img(img_path, target_size=(299, 299))
print(non_rust_leaf)
non_rust_leaf<jupyter_output><PIL.Image.Image image mode=RGB size=299x299 at 0x7F4EF20F0550><jupyter_text>When creating our data sets we did have images that were marked rust but also a different disease, we want to ignore these multip diseased images so we will remove them from all three datasets.<jupyter_code>!cd coffee-leaf-diseases/coffee-leaf-diseases/test/images && find -type f -not -path '*/other*' -not -path '*/rust*' -delete
!cd coffee-leaf-diseases/coffee-leaf-diseases/train/images && find -type f -not -path '*/other*' -not -path '*/rust*' -delete<jupyter_output><empty_output><jupyter_text>Training our model ⚙️<jupyter_code>train_data_path = "./coffee-leaf-diseases/coffee-leaf-diseases/train/images/"
batch_size = 32
img_height = 150
img_width = 150
input_shape=(150, 150, 3)
data_dir = pathlib.Path(train_data_path)
# allows us to read images from storage and use them for training
train_gen = ImageDataGenerator(preprocessing_function=preprocess_input, validation_split=0.2)

train_ds = train_gen.flow_from_directory(
    data_dir,
    subset="training",
    target_size=(img_height, img_width),
    batch_size=batch_size
)
# view the classes we have available in our training data
train_ds.class_indices
val_ds = train_gen.flow_from_directory(
    data_dir,
    subset="validation",
    target_size=(img_height, img_width),
    batch_size=batch_size
)
val_ds.class_indices
def make_model(input_size=150, learning_rate=0.01, size_inner=100,
               droprate=0.5):

    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(input_size, input_size, 3)
    )

    base_model.trainable = False

    #########################################

    inputs = keras.Input(shape=(input_size, input_size, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(2)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################

    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)

    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model
checkpoint = keras.callbacks.ModelCheckpoint(
    'xception_v4_1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)<jupyter_output><empty_output><jupyter_text>Tuning the learning rate 🔧Learning rate parameter is the most important thing to tune.- how fast a model learns- FAST: High learning rate - consume a lot of learning material quickly but not thorough- SLOW: Low learning rate - focus on less material but learn it much more thorough- if we use a high learning rate it could be we perform poorly (overfit) on validation where as a low learning rate may underfit so we have to find the right balance- we try different parameters to see which gives us the best result<jupyter_code>%%time

scores = {}

for lr in [0.0001, 0.001, 0.01, 0.1]:
    print(lr)

    model = make_model(learning_rate=lr)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds, callbacks=[checkpoint])
    scores[lr] = history.history

    print()
    print()
%%time

for lr in [0.0001, 0.001, 0.01, 0.1]:
  hist = scores[lr]
  plt.plot(hist['val_accuracy'], label=lr)


plt.legend()
learning_rate = 0.001<jupyter_output><empty_output><jupyter_text>Adding more layers<jupyter_code>%%time

scores = {}

for size in [10, 100, 1000]:
    print(size)

    model = make_model(learning_rate=learning_rate, size_inner=size)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[size] = history.history

    print()
    print()
%%time

for size in [10, 100, 1000]:
  hist = scores[size]
  plt.plot(hist['val_accuracy'], label=size)

plt.legend()
size = 100<jupyter_output><empty_output><jupyter_text>Regularisation and dropout<jupyter_code>scores = {}

for droprate in [0.0, 0.2, 0.5, 0.8]:
    print(droprate)

    model = make_model(
        learning_rate=learning_rate,
        size_inner=size,
        droprate=droprate
    )

    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[droprate] = history.history

    print()
    print()
for droprate, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % droprate))

plt.ylim(0.78, 0.925)
plt.legend()
hist = scores[0.0]
plt.plot(hist['val_accuracy'], label=0.0)

hist = scores[0.5]
plt.plot(hist['val_accuracy'], label=0.2)

plt.legend()
droprate = 0.2<jupyter_output><empty_output><jupyter_text>Data Augmentation  Training a larger model<jupyter_code>input_size = 299
train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    # shear_range=10,
    zoom_range=0.1,
    horizontal_flip=True,
    validation_split=0.2
)

train_ds = train_gen.flow_from_directory(
    data_dir,
    subset="training",
    target_size=(input_size, input_size),
    batch_size=32
)

val_ds = train_gen.flow_from_directory(
    data_dir,
    subset="validation",
    target_size=(input_size, input_size),
    batch_size=32,
    shuffle=False
)<jupyter_output>Found 973 images belonging to 2 classes.
Found 243 images belonging to 2 classes.<jupyter_text>Traing the final model<jupyter_code>print(f"""Final parameters: \n
    input_size={input_size}, \n
    learning_rate={learning_rate}, \n
    size_inner={size}, \n
    droprate={droprate} \n
""")
final_model = make_model(
    input_size=input_size,
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)

history = model.fit(train_ds, epochs=10, validation_data=val_ds,
                   callbacks=[checkpoint])
plt.plot(hist['accuracy'])
final_model.summary()
version = "1"
timestamp = datetime.utcnow().strftime("%Y-%m-%d_%H:%M")
model_name = f"xception_v{version}_{timestamp}.h5"
final_model.save(model_name)
print(f"Latest saved model: {model_name}")
files.download(model_name)<jupyter_output><empty_output><jupyter_text>Using our model<jupyter_code>model = keras.models.load_model(model_name)
test_data_path = "./coffee-leaf-diseases/coffee-leaf-diseases/test/images/"
batch_size = 32
img_height = 150
img_width = 150
input_shape=(150, 150, 3)
test_data_dir = pathlib.Path(test_data_path)
test_gen = ImageDataGenerator(preprocessing_function=preprocess_input)

test_ds = test_gen.flow_from_directory(
    test_data_dir,
    target_size=(img_height, img_width),
    batch_size=32,
    shuffle=False
)
final_model.evaluate(test_ds)
classes = list(test_ds.class_indices.keys())
classes
random_idx = random.randint(0, len(test_rust_ids))
img_path = f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/rust/{test_rust_ids[random_idx]}.jpg"
example_rust_leaf = load_img(img_path, target_size=(input_size, input_size))
print(example_rust_leaf)
example_rust_leaf
x = np.array(example_rust_leaf)
X = np.array([x])
X = preprocess_input(X)

pred = final_model.predict(X)

dict(zip(classes, pred[0]))
random_idx = random.randint(0, len(other_test_ids))
img_path = f"./coffee-leaf-diseases/coffee-leaf-diseases/test/images/other/{other_test_ids[random_idx]}.jpg"
example_other_leaf = load_img(img_path, target_size=(input_size, input_size))
print(example_other_leaf)
example_other_leaf
x = np.array(example_other_leaf)
X = np.array([x])
X = preprocess_input(X)

pred = final_model.predict(X)

dict(zip(classes, pred[0]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12591195999672103, 'fraction_numerical': 0.017132551848512173, 'mean_word_length': 5.24360286591607, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '513703', 'license': 'CC0-1.0', 'n_tokens_mistral': 4494, 'n_tokens_neox': 4235, 'n_words': 909, 'path': 'analysis/notebooks/leaf_rust_detection_exploration.ipynb', 'repo_name': 'sleepypioneer/coffee-leaf-rust-predictor', 'size': '12199'}