<jupyter_start><jupyter_text>00:00:00 - Fast.AI student article recap* [Structured Deep Learning](https://towardsdatascience.com/structured-deep-learning-b8ca4138b848?gi=47b0bbe61a30)* [Fun with small image data-sets (Part 2)](https://towardsdatascience.com/fun-with-small-image-data-sets-part-2-54d683ca8c96)  * Vendors claimed that huge datasets and massive compute is required to do deep learning, but this post proves that's not the case.* [How do we train neural networks ?](https://towardsdatascience.com/how-do-we-train-neural-networks-edd985562b73)  00:07:48 - 2nd half of course overview* First half:  * Learn about applications.  * Learn best practises.* Second half:  * Learn what the Fast.ai libraries are doing under the hood.  00:09:01 - Collaborative filtering model from scratch<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.fastai.learner import *
from fastai.fastai.column_data import *
PATH = 'data/ml-latest-small/'
!mkdir {PATH}<jupyter_output>mkdir: data/ml-latest-small/: File exists<jupyter_text>* Download Movielens dataset:<jupyter_code>!wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip --directory-prefix={PATH}
!cd {PATH} && unzip ml-latest-small.zip
!ls -l {PATH}<jupyter_output>total 7936
-rw-r--r--  1 lex  staff     8364 18 Oct  2016 README.txt
-rw-r--r--  1 lex  staff   183372 18 Oct  2016 links.csv
drwxr-xr-x  7 lex  staff      238 18 Oct  2016 [34mml-latest-small[m[m
-rw-r--r--  1 lex  staff   918269 18 Oct  2016 ml-latest-small.zip
drwxr-xr-x  2 lex  staff       68 23 Apr 17:31 [34mmodels[m[m
-rw-r--r--  1 lex  staff   458390 18 Oct  2016 movies.csv
-rw-r--r--  1 lex  staff  2438266 18 Oct  2016 ratings.csv
-rw-r--r--  1 lex  staff    41902 18 Oct  2016 tags.csv
drwxr-xr-x  2 lex  staff       68 23 Apr 17:31 [34mtmp[m[m<jupyter_text>* Each dataset row contarins:  * Rating  * User id  * Movie id  * Timestamp (not used in model)<jupyter_code>ratings = pd.read_csv(f'{PATH}ratings.csv')
ratings.head()<jupyter_output><empty_output><jupyter_text>* Load in movie names for display purposes:<jupyter_code>movies = pd.read_csv(f'{PATH}movies.csv')
movies.head()<jupyter_output><empty_output><jupyter_text>00:10:50 - Solve MovieLens in Excel * Create crosstab of most popular movies and users with most ratings for Excel:<jupyter_code>g = ratings.groupby('userId')['rating'].count()
top_users = g.sort_values(ascending=False)[:15]

g = ratings.groupby('movieId')['rating'].count()
top_movies = g.sort_values(ascending=False)[:15]

top_r = ratings.join(top_users, rsuffix='_r', how='inner', on='userId')
top_r = top_r.join(top_movies, rsuffix='_r', how='inner', on='movieId')

pd.crosstab(top_r.userId, top_r.movieId, top_r.rating, aggfunc=np.sum)<jupyter_output><empty_output><jupyter_text>* Excel can perform gradient descent with the Solver Add On (I couldn't get it to work in Google Sheets):    1. Initialise 5 random numbers for each movie, and each user.    2. Solver then performs gradient descent on those numbers, attempting to minimise the loss (RMSE).* Intuition for the "latent factors" (random number) is basically each number refers to some feature (how much user like action films) which are learned.* Called collaborative filtering using "probabilistic matrix factorization".  00:21:32 - Audience questions* Q1: Should we cap the factorization outputs to between 0 and 5?* A1: Yes. We'll do that later.* Q2: How do you figure out how many latent factors you need?* A2: Since you're building an embedding matrix, the same rules apply: cardinality of class / 2 (try a few things and see what works).* Q3: What does it mean when the factor is negative?* A3: In the movie case: the movie is not dialog driven at all (as an example), in the user case: user actively dislikes some genre.* Q4: How can we trust these numbers?* A4: We are using gradient descent to find the best numbers, so there could be other numbers but these provide the lowest loss.* Q5: If we have a new user and new movie, do we have to retrain the model?* A5: You could have a new user and new movie model to support stuff temporarily, but over time you'd need to retrain your model.  00:26:02 - Solving with Fast.AI * Create validation set:<jupyter_code>val_idxs = get_cv_idxs(len(ratings))<jupyter_output><empty_output><jupyter_text>* Set weight decay (aka l2 regularisation):<jupyter_code>wd = 2e-4<jupyter_output><empty_output><jupyter_text>* How big is the embedding matrix?<jupyter_code>n_factors = 50
cf = CollabFilterDataset.from_csv(PATH, 'ratings.csv', 'userId', 'movieId', 'rating')
learn = cf.get_learner(n_factors, val_idxs, 64, opt_fn=optim.Adam)
learn.fit(1e-2, 2, wds=wd, cycle_len=1, cycle_mult=2)<jupyter_output><empty_output><jupyter_text>* What collaborative filtering is doing:  * For this user and movie, what other users are similar to them, based on their preference? * Take sqrt of RMSE:<jupyter_code>math.sqrt(0.764737)<jupyter_output><empty_output><jupyter_text>00:31:30 - Analyse predictions<jupyter_code>preds = learn.predict()
y = learn.data.val_y<jupyter_output><empty_output><jupyter_text>* Plot a summary of the data using the [Seaborn](https://seaborn.pydata.org) library:<jupyter_code>sns.jointplot(preds, y, kind='hex', stat_func=None);<jupyter_output>//anaconda/envs/fastai-cpu/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>00:32:16 - Audience questions* Q1: Why is n_factors set to 50?* A1: Jeremy tried a few and it seemed to work the best.  00:33:55 - Dot product in PyTorch * Can create a Tensor using the Fast.AI `T` function:<jupyter_code>T
a = T([[1., 2], [3, 4]])
b = T([[2., 2], [10, 10]])
a, b<jupyter_output><empty_output><jupyter_text>* Equivalent to `torch.FloatTensor`:<jupyter_code>torch.FloatTensor(np.array([[1., 2], [3., 4]]))<jupyter_output><empty_output><jupyter_text>* Can multiple element-wise using the Python multplication operators:<jupyter_code>a * b<jupyter_output><empty_output><jupyter_text>* The columns can be summed, as follows:<jupyter_code>(a * b).sum(1)<jupyter_output><empty_output><jupyter_text>00:37:17  - PyTorch module * PyTorch module can be used as a layer in a neural net.  * Can call the object with some tensors, and it will perform a calculation:<jupyter_code>class DotProduct(nn.Module):
    def forward(self, u, m):
        return (u * m).sum(1)
model = DotProduct()
model(a, b)<jupyter_output><empty_output><jupyter_text>* Can take the derivative of a Module and stack/compose into a larger model.  00:38:54 - Audience questions * Q1: If you have a lot of data can you use Fast.ai library?* A1: Yes, because it uses SGD which iterates through a dataset one batch at a time.  * Pandas dataframes do need to live in memory, but you could save it as a bcolz array or a [Dask](http://dask.pydata.org/) dataframe.  00:40:17 - `forward` method * Calculations in a `Module` must be performed in the `forward` method.  00:41:34 - Collaborative filtering in PyTorch * User id and movie id may not be contiguous, so we have to convert them into contiguous values, as follows:<jupyter_code># Get unique users
u_uniq = ratings.userId.unique()
# Great id to position dict
user2idx = {o:i for i, o in enumerate(u_uniq)}
# Convert user id to idx
ratings.userId = ratings.userId.apply(lambda x: user2idx[x])
# Get unique movies
m_uniq = ratings.movieId.unique()
# Get movie id to position dict
movie2idx = {o:i for i,o in enumerate(m_uniq)}
ratings.movieId = ratings.movieId.apply(lambda x: movie2idx[x])
n_users = int(ratings.userId.nunique())
n_movies = int(ratings.movieId.nunique())<jupyter_output><empty_output><jupyter_text>* Module constructor can take any arguments, you just need to ensure the parent constructor is also run (`super().__init__`)  * We also ensure that the random initialisation are around the value we need:<jupyter_code>class EmbeddingDot(nn.Module):
    def __init__(self, n_users, n_movies):
        super().__init__()
        
        # Create 2 embedding matrixes, one vector of n_factors for each user.
        self.u = nn.Embedding(n_users, n_factors)
        self.m = nn.Embedding(n_movies, n_factors)
        
        # Generate a uniform random matrix between 0 and 0.05
        self.u.weight.data.uniform_(0, 0.05)
        self.m.weight.data.uniform_(0, 0.05)

    def forward(self, cats, cont):
        # Get user and movie minibatch
        users, movies = cats[:,0], cats[:,1]
        
        # Look up minibatch in self.u and self.m
        u, m = self.u(users), self.m(movies)
        
        # Perform a dot product
        return (u*m).sum(1)<jupyter_output><empty_output><jupyter_text>* PyTorch convention for inplace methods it to include a `_` after it: `.uniform` vs `.uniform_`. * Create columnar data from a data frame as before:<jupyter_code>x = ratings.drop(['rating', 'timestamp'], axis=1)
y = ratings['rating']
data = ColumnarModelData.from_data_frame(PATH, val_idxs, x, y, ['userId', 'movieId'], 64)<jupyter_output><empty_output><jupyter_text>* Create an optimizer (the thing that will update the weights):  * In PyTorch weights are refered to as `parameters`:<jupyter_code>wd = 1e-5
model = EmbeddingDot(n_users, n_movies)
opt = optim.SGD(model.parameters(), 1e-1, weight_decay=wd, momentum=0.9)<jupyter_output><empty_output><jupyter_text>* Training loop is the thing that iterates through each minibatch and updates the weights.* Fast.ai has it's own training loop:<jupyter_code>fit<jupyter_output><empty_output><jupyter_text>* Fast.ai is designed to allow you to pick whatever layer of abstraction you need.<jupyter_code>fit(model, data, 3, opt, F.mse_loss)<jupyter_output><empty_output><jupyter_text>* Can see that we still aren't getting the sort of loss we were getting before. Few more things required to get there.* Can change the learning rate with `set_lrs` function:<jupyter_code>set_lrs(opt, 0.01)
fit(model, data, 3, opt, F.mse_loss)<jupyter_output><empty_output><jupyter_text>00:58:39 - Walk through of model.py * Where `fit` lives.  01:00:32 - Adding bias to embedding model * Some users may be super enthusiatic and tend to rate movies high, similarly, some movies may just be popular.* Want to introduce a constant to deal with this aka **bias**.* Excel example:  * Add extra row bias row to movie vectors and a column to the user vectors, which is just added to the calculation: `MMULT($B28:$F28,H$20:H$24)+H$19+$A28` * Get min and max rating, to limit output of rating predictions:<jupyter_code>min_rating, max_rating = ratings.rating.min(), ratings.rating.max()
min_rating, max_rating
def get_emb(num_inputs, num_factors):
    e = nn.Embedding(num_inputs, num_factors)
    e.weight.data.uniform_(0, 0.05)
    return e


class EmbeddingDotBias(nn.Module):
    def __init__(self, n_users, n_movies):
        super().__init__()
        self.user_emb = get_emb(n_users, n_factors)
        self.movies_emb = get_emb(n_movies, n_factors)
        
        # Add a bias and movies embedding with 1 factor.
        self.user_bias_emb = get_emb(n_users, 1)
        self.movie_bias_emb = get_emb(n_movies, 1)
        
    def forward(self, cats, conts):
        users, movies = cats[:,0], cats[:,1]
        um = (self.user_emb(users) * self.movies_emb(movies)).sum(1)
        res = (
            um +
            self.user_bias_emb(users).squeeze() +
            self.movie_bias_emb(movies).squeeze()
        )
        
        # Squish all rating to be between 1 and 5.
        res = F.sigmoid(res) * (max_rating - min_rating) + min_rating
        return res<jupyter_output><empty_output><jupyter_text>* ``squeeze`` removes all dimensions of size 1 to be removed.<jupyter_code>T([[[1,2, 3], [4,5,6]]]).shape
T([[[1,2,3], [4,5,6]]]).squeeze().shape<jupyter_output><empty_output><jupyter_text>* In PyTorch all operators you can do to Tensors are in `torch.nn.functional` which is usually imported as `F`:<jupyter_code>import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>* Fitting the model:<jupyter_code>wd=2e-4
model = EmbeddingDotBias(cf.n_users, cf.n_items)
opt = optim.SGD(model.parameters(), 1e-1, weight_decay=wd, momentum=0.9)
fit(model, data, 3, opt, F.mse_loss)
set_lrs(opt, 1e-2)
fit(model, data, 3, opt, F.mse_loss)<jupyter_output><empty_output><jupyter_text>* That is how collaborative filtering is generally done.* What's done is similar to a matrix factorization, however, it had to be modified slightly to allow for empty values.  * Only calculates loss for user + movie combinations that actually exist.  * "Probabilistic Matrix Factorization"  Audience questions - 01:16:52* Q1: Could we improve the model using the date?* A1: Possibly - worth a try. Could take the `EmbeddingDotBias` class out of Fast.ai and rework it.  01:17:20 - Neural Network version of CF 1. Take the user embedding and the movie embedding, then concat them to get a shape of `(1, 10)`2. Feed them into a weight matrix that's `(10, `3. Push it through a ReLU.4. Feed those outputs into a `(, 1)` linear layer.<jupyter_code>class EmbeddingNet(nn.Module):
    def __init__(self, n_users, n_movies):
        super().__init__()
        self.user_emb = get_emb(n_users, n_factors)
        self.movies_emb = get_emb(n_movies, n_factors)
        
        # Note: linear layers include bias.
        self.lin1 = nn.Linear(n_factors * 2, 10)
        self.lin2 = nn.Linear(10, 1)
        
    def forward(self, cats, conts):
        users, movies = cats[:,0], cats[:,1]
        x = F.dropout(
            torch.cat([
                self.user_emb(users),
                self.movies_emb(movies)], dim=1), 0.75)
        x = F.dropout(F.relu(self.lin1(x)), 0.75)
        return F.sigmoid(
            self.lin2(x)) * (max_rating - min_rating + 1) + min_rating - 0.5
wd=5e-4
model = EmbeddingNet(n_users, n_movies)
opt = optim.SGD(model.parameters(), 1e-2, weight_decay=wd, momentum=0.9)
# Note that we're passing the loss function to `fit`.
fit(model, data, 3, opt, F.mse_loss)
set_lrs(opt, 1e-2)
fit(model, data, 3, opt, F.mse_loss)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11492632024254389, 'fraction_numerical': 0.03123457660579567, 'mean_word_length': 4.3890577507598785, 'pattern_counts': {'":': 0, '<': 110, '<?xml version=': 0, '>': 110, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '614711', 'license': 'MIT', 'n_tokens_mistral': 5188, 'n_tokens_neox': 4832, 'n_words': 1499, 'path': 'notes/reference/moocs/fast.ai/dl-2018/lesson05.ipynb', 'repo_name': 'aav789/study-notes', 'size': '14183'}