<jupyter_start><jupyter_text>1. Regression with Images and TextIn this notebook we will go through a series of examples on how to combine all Wide & Deep components, the Wide component, the stack of dense layers for the "categorical embeddings" and numerical column (deepdense), the text data (deeptext) and images (deepimage). To that aim I will use the Airbnb listings dataset for London, which you can download from [here](http://insideairbnb.com/get-the-data.html). I have taken a sample of 1000 listings to keep the data tractable in this notebook. Also, I have preprocess the data and prepared it for this excercise. All preprocessing steps can be found in the notebook `airbnb_data_preprocessing.ipynb` in this `examples` folder. Note that you do not need to go through that notebook to get an understanding on how to use this library.<jupyter_code>import numpy as np
import pandas as pd
import os
import torch

from pytorch_widedeep.preprocessing import WidePreprocessor, DeepPreprocessor, TextPreprocessor, ImagePreprocessor
from pytorch_widedeep.models import Wide, DeepDense, DeepText, DeepImage, WideDeep
from pytorch_widedeep.initializers import *
from pytorch_widedeep.callbacks import *
from pytorch_widedeep.optim import RAdam
df = pd.read_csv('data/airbnb/airbnb_sample.csv')
df.head()<jupyter_output><empty_output><jupyter_text>1.1 Regression with the defaults<jupyter_code># There are a number of columns that are already binary. Therefore, no need to one hot encode them
crossed_cols = (['property_type', 'room_type'],)
already_dummies = [c for c in df.columns if 'amenity' in c] + ['has_house_rules']
wide_cols = ['is_location_exact', 'property_type', 'room_type', 'host_gender',
'instant_bookable'] + already_dummies
cat_embed_cols = [(c, 16) for c in df.columns if 'catg' in c] + \
    [('neighbourhood_cleansed', 64), ('cancellation_policy', 16)]
continuous_cols = ['latitude', 'longitude', 'security_deposit', 'extra_people']
# it does not make sense to standarised Latitude and Longitude
already_standard = ['latitude', 'longitude']
# text and image colnames
text_col = 'description'
img_col = 'id'
# path to pretrained word embeddings and the images
word_vectors_path = 'data/glove.6B/glove.6B.100d.txt'
img_path = 'data/airbnb/property_picture'
# target
target_col = 'yield'<jupyter_output><empty_output><jupyter_text>1.1.1 Prepare the dataI will focus here on how to prepare the data and run the model. Check notebooks 1 and 2 to see what's going on behind the scencesPreparing the data is rather simple<jupyter_code>target = df[target_col].values
wide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)
X_wide = wide_preprocessor.fit_transform(df)
deep_preprocessor = DeepPreprocessor(embed_cols=cat_embed_cols, continuous_cols=continuous_cols)
X_deep = deep_preprocessor.fit_transform(df)
text_preprocessor = TextPreprocessor(word_vectors_path=word_vectors_path)
X_text = text_preprocessor.fit_transform(df, text_col)
image_processor = ImagePreprocessor()
X_images = image_processor.fit_transform(df, img_col, img_path)<jupyter_output>Reading Images from data/airbnb/property_picture<jupyter_text>1.1.2. Build the model components<jupyter_code># Linear model
wide = Wide(wide_dim=X_wide.shape[1], output_dim=1)
# DeepDense: 2 Dense layers
deepdense = DeepDense(hidden_layers=[128,64], dropout=[0.5, 0.5], 
                      deep_column_idx=deep_preprocessor.deep_column_idx,
                      embed_input=deep_preprocessor.embeddings_input,
                      continuous_cols=continuous_cols)
# DeepText: 2 LSTMs
deeptext = DeepText(vocab_size=len(text_preprocessor.vocab.itos), hidden_dim=64, 
                    n_layers=2, rnn_dropout=0.5, 
                    embedding_matrix=text_preprocessor.embedding_matrix)
# Pretrained Resnet 18 (default is all but last 2 conv blocks) plus a FC-Head 512->256->128
deepimage = DeepImage(pretrained=True, head_layers=[512, 256, 128])
model = WideDeep(wide=wide, deepdense=deepdense, deeptext=deeptext, deepimage=deepimage)<jupyter_output><empty_output><jupyter_text>1.1.3. Compile and fit<jupyter_code>model.compile(method='regression')
model.fit(X_wide=X_wide, X_deep=X_deep, X_text=X_text, X_img=X_images,
    target=target, n_epochs=1, batch_size=32, val_split=0.2)<jupyter_output>epoch 1:  96%|█████████▌| 25/26 [02:04<00:04,  4.99s/it, loss=117]
valid: 100%|██████████| 7/7 [00:14<00:00,  2.06s/it, loss=98.9]<jupyter_text>1.2 Regression with varying parameters and a FC-Head receiving the deep sideThis would be the second architecture shown in the README file<jupyter_code>wide = Wide(wide_dim=X_wide.shape[1], output_dim=1)
deepdense = DeepDense(hidden_layers=[128,64], dropout=[0.5, 0.5], 
                      deep_column_idx=deep_preprocessor.deep_column_idx,
                      embed_input=deep_preprocessor.embeddings_input,
                      continuous_cols=continuous_cols)
deeptext = DeepText(vocab_size=len(text_preprocessor.vocab.itos), hidden_dim=128, 
                    n_layers=2, rnn_dropout=0.5, 
                    embedding_matrix=text_preprocessor.embedding_matrix)
deepimage = DeepImage(pretrained=True, head_layers=[512, 256, 128])
model = WideDeep(wide=wide, deepdense=deepdense, deeptext=deeptext, deepimage=deepimage, head_layers=[128, 64])<jupyter_output><empty_output><jupyter_text>Let's have a look to the model Both, the Text and Image components allow FC-heads on their own (referred very creatively as `texthead` and `imagehead`). Following this nomenclature, the FC-head that receives the concatenation of the whole deep component is called `deephead`. Now let's go "kaggle crazy". Let's use different optimizers, initializers and schedulers for different components. Moreover, let's use a different learning rate for different parameter groups, for the `DeepDense` component<jupyter_code>deep_params = []
for childname, child in model.named_children():
    if childname == 'deepdense':
        for n,p in child.named_parameters():
            if "emb_layer" in n: deep_params.append({'params': p, 'lr': 1e-4})
            else: deep_params.append({'params': p, 'lr': 1e-3})
wide_opt = torch.optim.Adam(model.wide.parameters())
deep_opt = torch.optim.Adam(deep_params)
text_opt = RAdam(model.deeptext.parameters())
img_opt  = RAdam(model.deepimage.parameters())
head_opt = torch.optim.Adam(model.deephead.parameters())
wide_sch = torch.optim.lr_scheduler.StepLR(wide_opt, step_size=5)
deep_sch = torch.optim.lr_scheduler.MultiStepLR(deep_opt, milestones=[3,8])
text_sch = torch.optim.lr_scheduler.StepLR(text_opt, step_size=5)
img_sch  = torch.optim.lr_scheduler.MultiStepLR(deep_opt, milestones=[3,8])
head_sch = torch.optim.lr_scheduler.StepLR(head_opt, step_size=5)
# remember, one optimizer per model components, for lr_schedures and initializers is not neccesary
optimizers = {'wide': wide_opt, 'deepdense':deep_opt, 'deeptext':text_opt, 'deepimage': img_opt, 'deephead': head_opt}
schedulers = {'wide': wide_sch, 'deepdense':deep_sch, 'deeptext':text_sch, 'deepimage': img_sch, 'deephead': head_sch}

# Now...we have used pretrained word embeddings, so you do not want to
# initialise these  embeddings. However you might still want to initialise the
# other layers in the DeepText component. No probs, you can do that with the
# parameter pattern and your knowledge on regular  expressions. Here we are
# telling to the KaimingNormal initializer to NOT touch the  parameters whose
# name contains the string word_embed. 
initializers = {'wide': KaimingNormal, 'deepdense':KaimingNormal, 
                'deeptext':KaimingNormal(pattern=r"^(?!.*word_embed).*$"), 
                'deepimage':KaimingNormal}

mean = [0.406, 0.456, 0.485]  #BGR
std =  [0.225, 0.224, 0.229]  #BGR
transforms = [ToTensor, Normalize(mean=mean, std=std)]
callbacks = [LRHistory(n_epochs=10), EarlyStopping, ModelCheckpoint(filepath='model_weights/wd_out')]
model.compile(method='regression', initializers=initializers, optimizers=optimizers,
    lr_schedulers=schedulers, callbacks=callbacks, transforms=transforms)
model.fit(X_wide=X_wide, X_deep=X_deep, X_text=X_text, X_img=X_images,
    target=target, n_epochs=1, batch_size=32, val_split=0.2)
# we have only run one epoch, but let's check that the LRHistory callback records the lr values for each group
model.lr_history<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10246588460617669, 'fraction_numerical': 0.022504189609767775, 'mean_word_length': 5.604743083003952, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '64628', 'license': 'MIT', 'n_tokens_mistral': 2891, 'n_tokens_neox': 2706, 'n_words': 725, 'path': 'examples/04_Regression_with_Images_and_Text.ipynb', 'repo_name': 'awesome-archive/pytorch-widedeep', 'size': '8354'}