<jupyter_start><jupyter_text>Monk Library - https://github.com/Tessellate-Imaging/monk_v1¶ Monk is an opensource low-code tool for computer vision and deep learning Monk features    low-code    unified wrapper over major deep learning framework - keras, pytorch, gluoncv    syntax invariant wrapper Enables    to create, manage and version control deep learning experiments    to compare experiments across training metrics    to quickly find best hyper-parameters To contribute to Monk AI or Pytorch RoadMap repository raise an issue in the git-repo or dm us on linkedin    Abhishek - https://www.linkedin.com/in/abhishek-kumar-annamraju/    Akash - https://www.linkedin.com/in/akashdeepsingh01/  Competition - https://www.hackerearth.com/challenges/competitive/hackerearth-deep-learning-challenge-auto-tag-images-gala/machine-learning/auto-tag-images-of-the-gala-9e47fb31/  Score achieved with this tutorial: 83.54047 Monk team's final score: 85.785  Table of contents Install Monk Using pretrained model for adding tags to gala imagery Training a classifier from scratch  Install Monk  - git clone https://github.com/Tessellate-Imaging/monk_v1.git  - cd monk_v1/installation/Linux && pip install -r requirements_cu9.txt        (Select the requirements file as per OS and CUDA version)<jupyter_code>! git clone https://github.com/Tessellate-Imaging/monk_v1.git
# If using Colab install using the commands below
! cd monk_v1/installation/Misc && pip install -r requirements_colab.txt

# If using Kaggle uncomment the following command
#! cd monk_v1/installation/Misc && pip install -r requirements_kaggle.txt

# Select the requirements file as per OS and CUDA version when using a local system or cloud
#! cd monk_v1/installation/Linux && pip install -r requirements_cu9.txt<jupyter_output><empty_output><jupyter_text>Used trained classifier for demo<jupyter_code># Import monk

import os
import sys
sys.path.append("monk_v1/monk/");
# Download trained weights
! wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1cHoSP0UJK3sI5oPWtWMoDpdbpwy2wo5z' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1cHoSP0UJK3sI5oPWtWMoDpdbpwy2wo5z" -O cls_autogala_trained.zip && rm -rf /tmp/cookies.txt
! unzip -qq cls_autogala_trained.zip 
ls workspace/Project-Autogala
# Gluon project
from gluon_prototype import prototype
# Load project in inference mode

gtf = prototype(verbose=1);
gtf.Prototype("Project-Autogala", "Gluon-resnet50_v2", eval_infer=True);

#Other trained models - uncomment 
#gtf.Prototype("Project-Autogala", "Gluon-resnet101_v2", eval_infer=True);
#gtf.Prototype("Project-Autogala", "Gluon-resnet152_v2", eval_infer=True);
#Infer
img_name = "workspace/test/1.jpg"
predictions = gtf.Infer(img_name=img_name);
from IPython.display import Image
Image(filename=img_name)
img_name = "workspace/test/2.jpg"
predictions = gtf.Infer(img_name=img_name);
from IPython.display import Image
Image(filename=img_name)
img_name = "workspace/test/3.jpg"
predictions = gtf.Infer(img_name=img_name);
from IPython.display import Image
Image(filename=img_name)<jupyter_output><empty_output><jupyter_text>Training custom classifier from scratch  Goals    - Train a classifier on an expanded, manually augmented dataset    - Use the final model from the first classifier as a pretrained model for the seond classifier    - Train this second classifier on original training set    - Compare the experiments    - Run inference to generate test script  Table of contents [0. Install Monk](0) [1. Train on Augmented Set](1) [2. Copy the experiment and train on Original Set](2) [3. Compare the three experiments](3) [4. Run inference using final model from 2nd experiment](4)  Install Monk   - git clone https://github.com/Tessellate-Imaging/monk_v1.git  - cd monk_v1/installation && pip install -r requirements_cu9.txt     - (Select the requirements file as per OS and CUDA version)<jupyter_code>!git clone https://github.com/Tessellate-Imaging/monk_v1.git
# If using Colab install using the commands below
!cd monk_v1/installation/Misc && pip install -r requirements_colab.txt

# If using Kaggle uncomment the following command
#!cd monk_v1/installation/Misc && pip install -r requirements_kaggle.txt

# Select the requirements file as per OS and CUDA version when using a local system or cloud
#!cd monk_v1/installation/Linux && pip install -r requirements_cu9.txt<jupyter_output><empty_output><jupyter_text>Dataset - Credits: https://www.hackerearth.com/challenges/competitive/hackerearth-deep-learning-challenge-auto-tag-images-gala/machine-learning/auto-tag-images-of-the-gala-9e47fb31/<jupyter_code>! wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1wUEvnf_x4hpCBXrdbSEFeG-Gyv_fOP3a' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1wUEvnf_x4hpCBXrdbSEFeG-Gyv_fOP3a" -O gala_dataset.zip && rm -rf /tmp/cookies.txt
! unzip -qq gala_dataset.zip<jupyter_output><empty_output><jupyter_text>Experiment - 1 - On manually augmented data<jupyter_code>import os
import sys
sys.path.append("monk_v1/monk/")
# Using Mxnet Backend
from gluon_prototype import prototype


# To use pytorch backend
#from pytorch_prototype import prototype


# To use keras backend
#from keras_prototype import prototype<jupyter_output><empty_output><jupyter_text>Creating and managing experiments    - Provide project name    - Provide experiment name    - For a specific data create a single project    - Inside each project multiple experiments can be created    - Every experiment can be have diferent hyper-parameters attached to it<jupyter_code>gtf = prototype(verbose=1);
gtf.Prototype("Project", "Experiment-On-Augmented-Data");<jupyter_output><empty_output><jupyter_text>This creates files and directories as per the following structure            workspace        |        |--------Project (Project name can be different)                    |                    |                    |-----Experiment-On-Augmented-Data                                |                                |-----experiment-state.json                                |                                |-----output                                        |                                        |------logs (All training logs and graphs saved here)                                        |                                        |------models (all trained models saved here)   Load the data and the model  - Docs on  quick mode loading of data and model: https://github.com/Tessellate-Imaging/monk_v14  - Tutorials on Monk: https://github.com/Tessellate-Imaging/monk_v1/tree/master/study_roadmaps/1_getting_started_roadmap<jupyter_code>gtf.Default(dataset_path="dataset/Train_Images_Foldered", 
                    model_name="resnet152_v2", 
                    freeze_base_network=False, 
                    num_epochs=10);<jupyter_output><empty_output><jupyter_text>Docs on how to update hyper-parameters  - Update hyperparams using update mode - https://clever-noyce-f9d43f.netlify.com//update_mode/update_dataset  - Tutorials on how to update hyper-params - https://github.com/Tessellate-Imaging/monk_v1/tree/master/study_roadmaps/1_getting_started_roadmap/5_update_hyperparams<jupyter_code>gtf.update_batch_size(12)
gtf.optimizer_sgd(0.01);
gtf.lr_multistep_decrease([7, 9]);
gtf.update_save_intermediate_models(False)


# Very important to reload post updates
gtf.Reload();<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>gtf.Train();<jupyter_output><empty_output><jupyter_text>Experiment - 2 - Further training on original data<jupyter_code>from gluon_prototype import prototype
## Copy the previous experiment to use the final model of experiment 1 as pretrained model in experiment 2
gtf = prototype(verbose=1);
gtf.Prototype("Project", "Experiment-On-Original-Data",
             copy_from=["Project", "Experiment-On-Augmented-Data"]);<jupyter_output><empty_output><jupyter_text>This creates files and directories as per the following structure            workspace        |        |--------Project (Project name can be different)                    |                    |                    |-----Experiment-On-Augmented-Data (Already created)                                |                                |-----experiment-state.json                                |                                |-----output                                        |                                        |------logs (All training logs and graphs saved here)                                        |                                        |------models (all trained models saved here)                     |                     |                     |-----Experiment-On-Original-Data (Created Now)                                |                                |-----experiment-state.json                                |                                |-----output                                        |                                        |------logs (All training logs and graphs saved here)                                        |                                        |------models (all trained models saved here)<jupyter_code>gtf.update_dataset(dataset_path="dataset/Train_Images",
                  path_to_csv="dataset/train.csv");

# Very important to reload post Updates
gtf.Reload();
gtf.update_batch_size(16)
gtf.optimizer_sgd(0.001);
gtf.lr_multistep_decrease([8, 9]);
gtf.update_save_intermediate_models(False)


# Very important to reload post updates
gtf.Reload();<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>gtf.Train();<jupyter_output><empty_output><jupyter_text>Compare experiments<jupyter_code># Invoke the comparison class
from compare_prototype import compare<jupyter_output><empty_output><jupyter_text>Creating and managing comparison experiments  - Provide project name<jupyter_code># Create a project 
gtf = compare(verbose=1);
gtf.Comparison("Analysis");<jupyter_output><empty_output><jupyter_text>This creates files and directories as per the following structure        workspace        |        |--------comparison                        |                        |                        |-----Analysis                                    |                                    |------stats_best_val_acc.png                                    |------stats_max_gpu_usage.png                                    |------stats_training_time.png                                    |------train_accuracy.png                                    |------train_loss.png                                    |------val_accuracy.png                                    |------val_loss.png                                                            |                        |-----comparison.csv (Contains necessary details of all experiments)  Add experiments<jupyter_code>gtf.Add_Experiment("Project", "Experiment-On-Original-Data");
gtf.Add_Experiment("Project", "Experiment-On-Augmented-Data");<jupyter_output><empty_output><jupyter_text>Run Analysis<jupyter_code>gtf.Generate_Statistics();<jupyter_output><empty_output><jupyter_text>Visualize comparisons  Training Accuracy Curves<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/train_accuracy.png")<jupyter_output><empty_output><jupyter_text>Training Loss Curves<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/train_loss.png")<jupyter_output><empty_output><jupyter_text>Validation Accuracy Curves<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/val_accuracy.png")<jupyter_output><empty_output><jupyter_text>Validation loss curves<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/val_loss.png")<jupyter_output><empty_output><jupyter_text>Training time curves<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/stats_training_time.png")<jupyter_output><empty_output><jupyter_text>Best Validation accuracies<jupyter_code>from IPython.display import Image
Image(filename="workspace/comparison/Analysis/stats_best_val_acc.png")<jupyter_output><empty_output><jupyter_text>Inferencing on Experiment - 2<jupyter_code>from gluon_prototype import prototype

gtf = prototype(verbose=0);
# To load experiment in evaluation mode, set eval_infer can be set as True
gtf.Prototype("Project", "Experiment-On-Original-Data", eval_infer=True);
import pandas as pd
df = pd.read_csv("dataset/test.csv");
columns = list(df.columns)
combined = [];
from tqdm import tqdm_notebook as tqdm

for i in tqdm(range(len(df))):
    img_name = df[columns[0]][i];
    
    #Monk inference Engine
    predictions = gtf.Infer(img_name="dataset/Test_Images/" + img_name, return_raw=False);
    
    combined.append([img_name, predictions['predicted_class']])
df = pd.DataFrame(combined, columns = ['Image', 'Class'])  
df.to_csv("submission.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1029677613739078, 'fraction_numerical': 0.010093401626996084, 'mean_word_length': 2.9715824110080766, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 17, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 56, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '500442', 'license': 'Apache-2.0', 'n_tokens_mistral': 4049, 'n_tokens_neox': 3854, 'n_words': 865, 'path': 'study_roadmaps/4_image_classification_zoo/Classifier - Auto tag gala Hackerearth Dataset .ipynb', 'repo_name': 'shubham7169/monk_v1', 'size': '13276'}