<jupyter_start><jupyter_text>Octopod Ensemble Model Training Pipeline As the seventh step of this tutorial, we will train an ensemble model using the two image models and one text model that we already trained.This notebook was run on an AWS p3.2xlarge<jupyter_code>%load_ext autoreload

%autoreload 2
import sys
sys.path.append('../../')
import joblib
import pandas as pd

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.utils.data import DataLoader
from transformers import AdamW, BertTokenizer, get_cosine_schedule_with_warmup

from octopod.learner import MultiTaskLearner, MultiInputMultiTaskLearner
from octopod.dataloader import MultiDatasetLoader
from octopod.ensemble import OctopodEnsembleDataset, BertResnetEnsembleForMultiTaskClassification<jupyter_output><empty_output><jupyter_text>Load in train and validation datasets First we load in the csv's we created in Step 1.Remember to change the path if you stored your data somewhere other than the default.<jupyter_code>TRAIN_GENDER_DF = pd.read_csv('/home/ec2-user/fashion_dataset/gender_train.csv')
VALID_GENDER_DF = pd.read_csv('/home/ec2-user/fashion_dataset/gender_valid.csv')
TRAIN_SEASON_DF = pd.read_csv('/home/ec2-user/fashion_dataset/season_train.csv')
VALID_SEASON_DF = pd.read_csv('/home/ec2-user/fashion_dataset/season_valid.csv')<jupyter_output><empty_output><jupyter_text>You will most likely have to alter this to however big your batches can be on your machine<jupyter_code>batch_size = 128
bert_tok = BertTokenizer.from_pretrained(
    'bert-base-uncased',
    do_lower_case=True
)

max_seq_length = 128 
gender_train_dataset = OctopodEnsembleDataset(
    text_inputs=TRAIN_GENDER_DF['productDisplayName'],
    img_inputs=TRAIN_GENDER_DF['image_urls'],
    y=TRAIN_GENDER_DF['gender_cat'],
    tokenizer=bert_tok,
    max_seq_length=max_seq_length,
    transform='train',
    crop_transform='train'

)
gender_valid_dataset = OctopodEnsembleDataset(
    text_inputs=VALID_GENDER_DF['productDisplayName'],
    img_inputs=VALID_GENDER_DF['image_urls'],
    y=VALID_GENDER_DF['gender_cat'],
    tokenizer=bert_tok,
    max_seq_length=max_seq_length,
    transform='val',
    crop_transform='val'

)

season_train_dataset = OctopodEnsembleDataset(
    text_inputs=TRAIN_SEASON_DF['productDisplayName'],
    img_inputs=TRAIN_SEASON_DF['image_urls'],
    y=TRAIN_SEASON_DF['season_cat'],
    tokenizer=bert_tok,
    max_seq_length=max_seq_length,
    transform='train',
    crop_transform='train'

)
season_valid_dataset = OctopodEnsembleDataset(
    text_inputs=VALID_SEASON_DF['productDisplayName'],
    img_inputs=VALID_SEASON_DF['image_urls'],
    y=VALID_SEASON_DF['season_cat'],
    tokenizer=bert_tok,
    max_seq_length=max_seq_length,
    transform='val',
    crop_transform='val'
)<jupyter_output><empty_output><jupyter_text>We then put the datasets into a dictionary of dataloaders.Each task is a key.<jupyter_code>train_dataloaders_dict = {
    'gender': DataLoader(gender_train_dataset, batch_size=batch_size, shuffle=True, num_workers=6),
    'season': DataLoader(season_train_dataset, batch_size=batch_size, shuffle=True, num_workers=6),
}
valid_dataloaders_dict = {
    'gender': DataLoader(gender_valid_dataset, batch_size=batch_size, shuffle=False, num_workers=6),
    'season': DataLoader(season_valid_dataset, batch_size=batch_size, shuffle=False, num_workers=6),
}
TrainLoader = MultiDatasetLoader(loader_dict=train_dataloaders_dict)
len(TrainLoader)
ValidLoader = MultiDatasetLoader(
    loader_dict=valid_dataloaders_dict,
    shuffle=False
)
len(ValidLoader)<jupyter_output><empty_output><jupyter_text>Create Model and Learner=== Since the image model could potentially have multiple Resnets for different subsets of tasks, we need to create an `image_task_dict` that splits up the tasks grouped by the Resnet they use.This version uses different resnets for gender and season.<jupyter_code>image_task_dict = {
    'gender': {
        'gender': TRAIN_GENDER_DF['gender_cat'].nunique()    
    },
    'season': {
        'season': TRAIN_SEASON_DF['season_cat'].nunique()
    }  
}<jupyter_output><empty_output><jupyter_text>We still need to create the `new_task_dict` for the learner.<jupyter_code>new_task_dict = {
    'gender': TRAIN_GENDER_DF['gender_cat'].nunique(),
    'season': TRAIN_SEASON_DF['season_cat'].nunique()
}
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)<jupyter_output>cuda:0<jupyter_text>We first initialize the model by setting up the right shape with the image_task_dict.<jupyter_code>model = BertResnetEnsembleForMultiTaskClassification(
    image_task_dict=image_task_dict
)<jupyter_output><empty_output><jupyter_text>We then load in the existing models by specifying the folder where the models live and their id's.<jupyter_code>resnet_model_id_dict = {
    'gender': 'GENDER_IMAGE_MODEL1',
    'season': 'SEASON_IMAGE_MODEL1'
}
model.load_core_models(
    folder='/home/ec2-user/fashion_dataset/models/',
    bert_model_id='TEXT_MODEL1',
    resnet_model_id_dict=resnet_model_id_dict
)<jupyter_output><empty_output><jupyter_text>We've set some helper methods that will freeze the core bert and resnets for you if you only want to train the new layers. As with all other aspects of training, this is likely to require some experimentation to determine what works for your problem. You will likely need to explore different values in this section to find some that workfor your particular model.<jupyter_code>model.freeze_bert()
model.freeze_resnets()

lr_last = 1e-3
lr_main = 1e-5

lr_list = [
    {'params': model.bert.parameters(), 'lr': lr_main},
    {'params': model.dropout.parameters(), 'lr': lr_main},   
    {'params': model.image_resnets.parameters(), 'lr': lr_main},
    {'params': model.image_dense_layers.parameters(), 'lr': lr_main},
    {'params': model.ensemble_layers.parameters(), 'lr': lr_last},
    {'params': model.classifiers.parameters(), 'lr': lr_last},
]

optimizer = optim.Adam(lr_list)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size= 4, gamma= 0.1)
loss_function_dict = {'gender': 'categorical_cross_entropy', 'season': 'categorical_cross_entropy'}
metric_function_dict = {'gender': 'multi_class_acc', 'season': 'multi_class_acc'}
learn = MultiInputMultiTaskLearner(model, TrainLoader, ValidLoader, new_task_dict, loss_function_dict, metric_function_dict)<jupyter_output><empty_output><jupyter_text>Train Model=== As your model trains, you can see some output of how the model is performing overall and how it is doing on each individual task.<jupyter_code>learn.fit(
    num_epochs=10,
    scheduler=exp_lr_scheduler,
    step_scheduler_on_batch=False,
    optimizer=optimizer,
    device=device,
    best_model=True
)<jupyter_output><empty_output><jupyter_text>You will need to check your specific use case to determine whether it is better to train all of your image tasks in one resnet or in multiple resnets. You won't necessarily need one resnet per task, particularly if some of your tasks are related. Checking validation data=== We provide a method on the learner called `get_val_preds`, which makes predictions on the validation data. You can then use this to analyze your model's performance in more detail.<jupyter_code>pred_dict = learn.get_val_preds(device)
pred_dict<jupyter_output><empty_output><jupyter_text>Save/Export Model=== The ensemble model can also be saved or exported.<jupyter_code>model.save(folder='/home/ec2-user/fashion_dataset/models/', model_id='ENSEMBLE_MODEL2')
model.export(folder='/home/ec2-user/fashion_dataset/models/', model_id='ENSEMBLE_MODEL2')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09558536267743195, 'fraction_numerical': 0.0048183357207969785, 'mean_word_length': 6.492682926829269, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '402224', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2570, 'n_tokens_neox': 2523, 'n_words': 581, 'path': 'notebooks/fashion_data/Optional_Step7_Train_ensemble_model_with_two_resnets.ipynb', 'repo_name': 'sunggeunkim2/octopod', 'size': '7679'}