<jupyter_start><jupyter_text>fastai collab model for jesterhello children. today we're going to build a joke recommendation engine.we're going to use the jester dataset from http://eigentaste.berkeley.edu/dataset/ - dataset 1 contains 100 jokes and ~4 million recommendations from ~73,400 users which should be plenty to play with.we're going to train a collab filtering model on the data and use it to power a joke recomendation engine which will use the ratings of previous users to match the user with jokes they're likely to rate highly.we will do this by putting jokes in front of them  and asking them to rate them, but we will use the ratings they give us each time to make more acurate predictions about the way they are likely to rate each joke. as we continue putting new jokes in front of them they should become less random, and more "up their street".<jupyter_code>import pandas as pd
from fastai.collab import *
from fastai.tabular import *

import shutil
from google.colab import drive

drive.mount('/content/drive')
DRIVE_DIR = "/content/drive/My Drive/fastai-v3/jester/"<jupyter_output><empty_output><jupyter_text>data cleaningthe jokes come as 100 seperate html files and the ratings are in 3 different files so lets fix that.ratings files 1 and 2 both contain users who have rated >= 36 jokes, file 3 is users who have rated 15 to 35.the ratings files are N x 101, one row per user, the first col is the number of jokes that user has rated, the other 100 are the joke ratings. ratings go from -10 up to 10. 99 means null / not rated.the sub-matrix containing cols [5, 7, 8, 13, 15, 16, 17, 18, 19, 20] is dense and contains jokes which almost all users have rated. this is useful information to cold-start our recommendation engine once we have built a model. feeding them jokes hardly anyone has rated won't provide you with useful information. (luckily this dataset is pretty dense anyway)another cold-start idea might be to try starting with some of the more contentious jokes. we'll see...<jupyter_code>!wget -q http://eigentaste.berkeley.edu/dataset/jester_dataset_1_joke_texts.zip
!wget -q http://eigentaste.berkeley.edu/dataset/jester_dataset_1_1.zip
# !wget -q http://eigentaste.berkeley.edu/dataset/jester_dataset_1_2.zip
# !wget -q http://eigentaste.berkeley.edu/dataset/jester_dataset_1_3.zip
!unzip -q jester_dataset_1_joke_texts.zip 'jokes/*'
!unzip -q jester_dataset_1_1.zip
# !unzip -q jester_dataset_1_2.zip
# !unzip -q jester_dataset_1_3.zip<jupyter_output><empty_output><jupyter_text>Clean ratingsFirst we concat them together. Next we need to flatten them out because we have a dense matrix which is n_users x n_jokes and fastai wants user,thing,rating rows.There are ~~a little over 4 million ratings so flattening it does take around 2 mins to run~~ a lot less now since I'm only using 1 file of 3.We keep both versions because we're going to use the matrix later.<jupyter_code>cols = ["n_rated"]
for i in range(1, 101): cols.append("joke_" + str(i))

# frames = []
# for i in range (1,4):
#   frames.append(pd.read_excel("jester-data-{}.xls".format(i), 
#                               header=None, names=cols))

# df_ratings = pd.concat(frames); frames = None
# df_ratings.reset_index(drop=True, inplace=True)

df_ratings = pd.read_excel("jester-data-1.xls", header=None, names=cols)
df_ratings.to_csv("ratings-matrix.csv", index=False)
df_ratings.tail()
def flatten_ratings(df_dense):
  rows = []
  
  for index, row in df_dense.iterrows():
    for i in range(1, 101):
      rating = row["joke_{}".format(i)]
      if(rating != 99):
        rows.append({"user_id":index, "joke_id":i, "rating":rating})

  df = pd.DataFrame(rows)
  return df

df_flattened = flatten_ratings(df_ratings)
df_flattened.to_csv("ratings-flattened.csv", index=False)
df_flattened.tail()
!rm *.zip
!rm *.xls<jupyter_output><empty_output><jupyter_text>Clean jokesthe joke files are html documents and as such contain \ tags for formatting the jokes so they display nicely. i've left them in because i intend to display them with a jupyter html widget anyway. you may wish to remove them.<jupyter_code>import re

ptn = "(:?<!--begin of joke -->\n)([\w\W]*)(:?<!--end of joke -->)"
p = re.compile(ptn, re.IGNORECASE + re.MULTILINE)

jokes = []

for i in range (1, 101):
  with open("jokes/init{}.html".format(i), mode="r") as fs:
    html = fs.read()
    m = p.search(html)
    assert m is not None, "i fail to find anything funny here. #dadjokes"
    jokes.append({"joke_num":i, "joke":m.group(2)})
    
df_jokes = pd.DataFrame(jokes)
df_jokes.to_csv("jokes.csv")
df_jokes.head()
!rm -r jokes<jupyter_output><empty_output><jupyter_text>Build a collab filtering model<jupyter_code># training with no val set due to a current issue with predictions afterwards.  
data = CollabDataBunch.from_df(df_flattened, seed=123, valid_pct=0)
learn = collab_learner(data, n_factors=50, y_range=[-10.5, 10.5], wd=1e-1)
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(3, 5e-3)
learn.save("stage1")
learn.export(DRIVE_DIR + "jester.pkl")
shutil.copyfile("jokes.csv", DRIVE_DIR + "jokes.csv")
shutil.copyfile("ratings-flattened.csv", DRIVE_DIR + "ratings-flattened.csv")
shutil.copyfile("ratings-matrix.csv", DRIVE_DIR + "ratings-matrix.csv")<jupyter_output><empty_output><jupyter_text>Test predictions<jupyter_code>learn.export("model.pkl")
learn = load_learner("model.pkl", ".")
import functools
import time

def timed(func):
  """Time the function and print the run time"""
  @functools.wraps(func)
  def wrapper(*args, **kwargs):
    start = time.perf_counter()
    retval = func(*args, **kwargs)
    end = time.perf_counter(); rt = end - start
    print("@timed:", func.__name__, "took", "{:.3f}".format(rt), "secs")
    return retval
  return wrapper

def pred_string_batch(user):
  rows = [{"user_id":user, "joke_id":joke, "rating":0.} for joke in range(1,101)]
  df = pd.DataFrame(rows); s = ""

  learn.data = CollabDataBunch.from_df(df, test=df, no_check=True)
  preds,_ = learn.get_preds(DatasetType.Test)
  for pred in preds:
    s += str(pred.item()) + "/"
    
  return s

def pred_string_1by1(user):
  rows = [{"user_id":user, "joke_id":joke, "rating":0.} for joke in range(1,101)]
  df = pd.DataFrame(rows); s = ""

  for index, row in df.iterrows():
    _,_,pred = learn.predict(df.loc[index])
    s += str(pred.item()) + "/"
    
  return s

@timed
def test(func, times, user_id=0):
  preds = {}

  for i in range(0,times):
    if(user_id == 0): s = func(i)
    else:             s = func(user_id)

    if(s in preds): preds[s] += 1
    else:           preds[s] = 1
    if((i+1) % 100 == 0): print("Done", i+1)

  return preds

preds = test(pred_string_batch, 100)
print(len(preds), "unique sets of preds (different users)")
preds = test(pred_string_batch, 100, user_id = 50)
print(len(preds), "unique sets of preds (same user)")
preds = test(pred_string_1by1, 10)
print(len(preds), "unique sets of preds (different users)")
preds = test(pred_string_1by1, 10, user_id=50)
print(len(preds), "unique sets of preds (same user)")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12201666431295015, 'fraction_numerical': 0.021042225674339783, 'mean_word_length': 4.498447204968944, 'pattern_counts': {'":': 11, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '562110', 'license': 'MIT', 'n_tokens_mistral': 2550, 'n_tokens_neox': 2422, 'n_words': 774, 'path': 'JesterModel.ipynb', 'repo_name': 'joedockrill/jester-collab-filtering', 'size': '7081'}