<jupyter_start><jupyter_text>"PLANT PATHOLOGY">"Using Computer Vision  to detect different plant diseases at different stages"- toc: true- branch: master- badges: true- comments: true- categories: [Computer vision, jupyter]- search_exclude: true  **Problem Statement**>Misdiagnosis of the many diseases impacting agricultural crops can lead to misuse of chemicals leading to the emergence of resistant pathogen strains, increased input costs, and more outbreaks with significant economic loss and environmental impacts. Current disease diagnosis based on human scouting is time-consuming and expensive, and although computer-vision based models have the promise to increase efficiency, the great variance in symptoms due to age of infected tissues, genetic variations, and light conditions within trees decreases the accuracy of detection.>**Specific Objectives**>Objectives of ‘Plant Pathology Challenge’ are to train a model using images of training dataset to 1) Accurately classify a given image from testing dataset into different diseased category or a healthy leaf; 2) Accurately distinguish between many diseases, sometimes more than one on a single leaf; 3) Deal with rare classes and novel symptoms; 4) Address depth perception—angle, light, shade, physiological age of the leaf; and 5) Incorporate expert knowledge in identification, annotation, quantification, and guiding computer vision to search for relevant features during learning.**Resources**If you use the dataset for your project, please cite the preprint https://arxiv.org/abs/2004.11958**Acknowledgments**We acknowledge financial support from Cornell Initiative for Digital Agriculture (CIDA) and special thanks to <NAME> for help with data collection.Kaggle is excited to partner with research groups to push forward the frontier of machine learning. Research competitions make use of Kaggle's platform and experience, but are largely organized by the research group's data science team. Any questions or concerns regarding the competition data, quality, or topic will be addressed by them.<jupyter_code>#collapse_hide
!pip install fastai2 graphviz ipywidgets matplotlib nbdev>=0.2.12 pandas scikit_learn azure-cognitiveservices-search-imagesearch sentencepiece<jupyter_output><empty_output><jupyter_text>Down grade the fascore library as the one installed above is not compatible with Fastai.Vision Library<jupyter_code>#collapse_hide
pip install fastcore==1.0.0
from fastai2.vision.all import*
from sklearn.metrics import roc_auc_score
data_path = Path("/content/drive/My Drive/Competition Datasets /Plant_Pathology_2020")
df = pd.read_csv(data_path/"train.csv")
df.head()
df.iloc[:, 1:].sum(axis=1).value_counts()
imglabels = list(df.columns[1:])
df["labels"] = df.apply(lambda x: imglabels[x.values[1:].argmax()], axis=1)
dls = ImageDataLoaders.from_df(df,
                               path=data_path, 
                               suff=".jpg", 
                               folder="images",
                               label_col="labels",
                               item_tfms=RandomResizedCrop(512, min_scale=0.5), # note that we use a bigger image size
                               batch_tfms=aug_transforms(),
                               valid_pct=0.05,
                               bs=16,
                               val_bs=16
                               )
dls.show_batch(max_n=16, nrows=2)
def mean_roc_auc(preds, targets, num_cols=4):
    """The competition metric
    
    Quoting: 'Submissions are evaluated on mean column-wise ROC AUC. 
    In other words, the score is the average of the individual AUCs 
    of each predicted column. '
    
    Unfortunately, we cannot use in validation, as it can happen that
    all files in a batch has the same label, and ROC is undefined
    """
    aucs = []
    preds = preds.detach().cpu().numpy()
    targets = targets.detach().cpu().numpy()
    
    for i in range(num_cols):
        # grab a column from the networks output
        cpreds = preds[:, i]
        # see which objects have the i-th label
        ctargets = [x == i for x in targets]
        aucs.append(roc_auc_score(ctargets, cpreds))
    return sum(aucs) / num_cols
learn = cnn_learner(dls, resnet50, metrics=[accuracy], model_dir="/content/drive/My Drive/Competition Datasets /Plant_Pathology_2020")
learn.lr_find()
learn.fit(4, lr=1e-3)
learn.unfreeze()
learn.lr_find()
learn.save("model")
learn.fit_one_cycle(16, lr_max=slice(1e-6,1e-5), cbs=[SaveModelCallback(every='epoch', monitor="accuracy")])
learn.load("model")
learn.load("model")
test_image_ids = [img.split(".")[0] for img in os.listdir(data_path/"images") if img.startswith("Test")]
test_images = [data_path/"images"/f"{img}.jpg" for img in test_image_ids]
preds = learn.get_preds(dl=dls.test_dl(test_images, shuffle=False, drop_last=False))
# ensure that the order of columns in preds matches the imglabels
preds = preds[0].cpu().numpy()
vocab = list(dls[0].dataset.vocab)
column_permutation = [vocab.index(l) for l in imglabels]
preds = preds[:, column_permutation]

submission = pd.DataFrame()
submission["image_id"] = test_image_ids
for i in range(len(imglabels)):
    submission[imglabels[i]] = preds[:, i]
submission.to_csv("submission.csv", index=False)
submission.head(10)
submission.to_csv("/content/drive/My Drive/Competition Datasets /Plant_Pathology_2020/submission.csv", index=False)
submission.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09112665684830633, 'fraction_numerical': 0.013254786450662739, 'mean_word_length': 3.979835013748854, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '233630', 'license': 'Apache-2.0', 'n_tokens_mistral': 1607, 'n_tokens_neox': 1531, 'n_words': 528, 'path': '_notebooks/2020_09_10_Fastai_Kaggle_Image_Classification (6).ipynb', 'repo_name': 'UnplugCharger/My-Data-Science-Playground', 'size': '5439'}