<jupyter_start><jupyter_text>Model ComponentsThe 5 main components of a `WideDeep` model are:1. `Wide`2. `DeepDense` or `DeepDenseResnet`3. `DeepText`4. `DeepImage`5. `deephead`The first 4 of them will be collected and combined by the `WideDeep` collector class, while the 5th one can be optionally added to the `WideDeep` model through its corresponding parameters: `deephead` or alternatively `head_layers`, `head_dropout` and `head_batchnorm`  1. WideThe wide component is a Linear layer "plugged" into the output neuron(s)The only particularity of our implementation is that we have implemented the linear layer via an Embedding layer plus a bias. While the implementations are equivalent, the latter is faster and far more memory efficient, since we do not need to one hot encode the categorical features. Let's assume we the following dataset:<jupyter_code>import torch
import pandas as pd
import numpy as np

from torch import nn
df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
df.head()<jupyter_output><empty_output><jupyter_text>one hot encoded, the first observation would be<jupyter_code>obs_0_oh = (np.array([1., 0., 0., 1., 0., 0.])).astype('float32')<jupyter_output><empty_output><jupyter_text>if we simply numerically encode (label encode or `le`) the values:<jupyter_code>obs_0_le = (np.array([0, 3])).astype('int64')<jupyter_output><empty_output><jupyter_text>Note that in the functioning implementation of the package we start from 1, saving 0 for padding, i.e. unseen values. Now, let's see if the two implementations are equivalent<jupyter_code># we have 6 different values. Let's assume we are performing a regression, so pred_dim = 1
lin = nn.Linear(6, 1)
emb = nn.Embedding(6, 1) 
emb.weight = nn.Parameter(lin.weight.reshape_as(emb.weight))
lin(torch.tensor(obs_0_oh))
emb(torch.tensor(obs_0_le)).sum() + lin.bias<jupyter_output><empty_output><jupyter_text>And this is precisely how the linear component `Wide` is implemented<jupyter_code>from pytorch_widedeep.models import Wide
?Wide
wide = Wide(wide_dim=10, pred_dim=1)
wide<jupyter_output><empty_output><jupyter_text>Note that even though the input dim is 10, the Embedding layer has 11 weights. Again, this is because we save 0 for padding, which is used for unseen values during the encoding process  2. DeepDenseThere are two alternatives for the so called `deepdense` component of the model: `DeepDense` and `DeepDenseResnet`. `DeepDense` is comprised by a stack of dense layers that receive the embedding representation of the categorical features concatenated with numerical continuous features. For those familiar with Fastai's tabular API, DeepDense is almost identical to their tabular model, although `DeepDense` allows for more flexibility when defining the embedding dimensions. `DeepDenseResnet` is similar to `DeepDense` but instead of dense layers, the embedding representation of the categorical features concatenated with numerical continuous features are passed through a series of dense ResNet layers. Each basic block comprises the following operations:   Let's have a look first to `DeepDense`:<jupyter_code>from pytorch_widedeep.models import DeepDense
# fake dataset
X_deep = torch.cat((torch.empty(5, 4).random_(4), torch.rand(5, 1)), axis=1)
colnames = ['a', 'b', 'c', 'd', 'e']
embed_input = [(u,i,j) for u,i,j in zip(colnames[:4], [4]*4, [8]*4)]
deep_column_idx = {k:v for v,k in enumerate(colnames)}
continuous_cols = ['e']
?DeepDense
# my advice would be to not use dropout in the last layer, but I add the option because you never 
# know..there is crazy people everywhere.
deepdense = DeepDense(hidden_layers=[16,8], dropout=[0.5, 0.], batchnorm=True, deep_column_idx=deep_column_idx,
                      embed_input=embed_input, continuous_cols=continuous_cols)
deepdense
deepdense(X_deep)<jupyter_output><empty_output><jupyter_text>Let's now have a look to `DeepDenseResnet`:<jupyter_code>from pytorch_widedeep.models import DeepDenseResnet
?DeepDenseResnet
deepdense = DeepDenseResnet(blocks=[16,8], dropout=0.5, deep_column_idx=deep_column_idx,
                      embed_input=embed_input, continuous_cols=continuous_cols)
deepdense
deepdense(X_deep)<jupyter_output><empty_output><jupyter_text>3. DeepTextThe `DeepText` class within the `WideDeep` package is a standard and simple stack of LSTMs on top of word embeddings. You could also add a FC-Head on top of the LSTMs. The word embeddings can be pre-trained. In the future I aim to include some simple pretrained models so that the combination between text and images is fair.  *While I recommend using the `Wide` and `DeepDense` classes within this package when building the corresponding model components, it is very likely that the user will want to use custom text and image models. That is perfectly possible. Simply, build them and pass them as the corresponding parameters. Note that the custom models MUST return a last layer of activations (i.e. not the final prediction) so that  these activations are collected by WideDeep and combined accordingly. In  addition, the models MUST also contain an attribute `output_dim` with the size of these last layers of activations.*Let's have a look to the `DeepText` class<jupyter_code>import torch
from pytorch_widedeep.models import DeepText
?DeepText
X_text = torch.cat((torch.zeros([5,1]), torch.empty(5, 4).random_(1,4)), axis=1)
deeptext = DeepText(vocab_size=4, hidden_dim=4, n_layers=1, padding_idx=0, embed_dim=4)
deeptext<jupyter_output><empty_output><jupyter_text>You could, if you wanted, add a Fully Connected Head (FC-Head) on top of it<jupyter_code>deeptext = DeepText(vocab_size=4, hidden_dim=8, n_layers=1, padding_idx=0, embed_dim=4, head_layers=[8,4], 
                    head_batchnorm=True, head_dropout=[0.5, 0.5])
deeptext<jupyter_output><empty_output><jupyter_text>Note that since the FC-Head will receive the activations from the last hidden layer of the stack of RNNs, the corresponding dimensions must be consistent.   4. DeepImageThe `DeepImage` class within the `WideDeep` package is either a pretrained ResNet (18, 34, or 50. Default is 18) or a stack of CNNs, to which one can add a FC-Head. If is a pretrained ResNet, you can chose how many layers you want to defrost deep into the network with the parameter `freeze`<jupyter_code>from pytorch_widedeep.models import DeepImage
?DeepImage
X_img = torch.rand((2,3,224,224))
deepimage = DeepImage(head_layers=[512, 64, 8])
deepimage
deepimage(X_img)<jupyter_output><empty_output><jupyter_text>if `pretrained=False` then a stack of 4 CNNs are used<jupyter_code>deepimage = DeepImage(pretrained=False, head_layers=[512, 64, 8])
deepimage<jupyter_output><empty_output><jupyter_text>5. deepheadNote that I do not use uppercase here. This is because, by default, the `deephead` is not defined outside `WideDeep` as the rest of the components. When defining the `WideDeep` model there is a parameter called `head_layers` (and the corresponding `head_dropout`, and `head_batchnorm`) that define the FC-head on top of `DeeDense`, `DeepText` and `DeepImage`. Of course, you could also chose to define it yourself externally and pass it using the parameter `deephead`. Have a look<jupyter_code>from pytorch_widedeep.models import WideDeep
?WideDeep<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08668221094500068, 'fraction_numerical': 0.016321492250720066, 'mean_word_length': 5.951382268827455, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '499849', 'license': 'MIT', 'n_tokens_mistral': 2312, 'n_tokens_neox': 2213, 'n_words': 856, 'path': 'examples/02_Model_Components.ipynb', 'repo_name': 'FlyingWing/pytorch-widedeep', 'size': '7291'}