<jupyter_start><jupyter_text>Notebook demonstrating the Implemented Bellkor AlgorithmDate: 07/01/2018<jupyter_code># imports
import os
import datetime
import logging.config
import time

import pandas as pd
import numpy as np
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
                    datefmt="%Y-%m-%d %H:%M:%S")
logger = logging.getLogger("BellKorDemo")
# location of the ml-20 dataset 
#    for this demo I've downloaded and placed the data in the same location as this Notebook
data_location = os.getcwd()
# Get all the .csv files in the 'data_location'
datasets = [f for f in os.listdir(data_location) if f.endswith(".csv")]
print("Files in the data location: ", datasets)<jupyter_output>Files in the data location:  ['tags.csv', 'links.csv', 'ratings.csv', 'movies.csv']<jupyter_text>Load each of the Datasets into Scope<jupyter_code>tags = pd.read_csv("{}/tags.csv".format(data_location))
tags.head(n=3)
links = pd.read_csv("{}/links.csv".format(data_location))
links.head(n=3)
ratings = pd.read_csv("{}/ratings.csv".format(data_location))
ratings.head(n=3)
movies = pd.read_csv("{}/movies.csv".format(data_location))
movies.head(n=3)<jupyter_output><empty_output><jupyter_text>Prepare the data for the Bellkor Algorithm For this demo I'm not splitting the dataset into the typical Train / Test / Validation datasets as you'd normally expect when doing ML. The reason for this is to demonstrate the algorithm and not ML.    1.  Find the Global Mean    2.  Number of Users    3.  Number of Users    4.  Start Time, End Time    5.  Get the Average time mapping per User<jupyter_code>ratings.columns<jupyter_output><empty_output><jupyter_text>1. Finding the Global Mean<jupyter_code>global_mean = ratings["rating"].mean()
print("Global Mean Rating: {} starts".format(global_mean))<jupyter_output>Global Mean Rating: 3.5255285642993797 starts<jupyter_text>2. Number of Users<jupyter_code>distinct_users = ratings["userId"].unique()
user_count = distinct_users.shape[0]
print("Number of Users found: {}".format(user_count))
# Remap to an index starting from 0
user_mapping = dict(zip(distinct_users, np.arange(user_count)))
ratings.loc[:, "UserId"] = ratings.loc[:, "userId"].apply(lambda x: user_mapping[x])<jupyter_output><empty_output><jupyter_text>3.  Number of Movies<jupyter_code>distinct_movies = ratings["movieId"].unique()
movie_count = distinct_movies.shape[0]
print("Number of Movies found: {}".format(movie_count))
# Remap to an index starting from 0
movie_mapping = dict(zip(distinct_movies, np.arange(movie_count)))
ratings.loc[:, "MovieId"] = ratings.loc[:, "movieId"].apply(lambda x: movie_mapping[x])<jupyter_output><empty_output><jupyter_text>4.  Start Time, End Time<jupyter_code>start_time = ratings["timestamp"].min()
end_time = ratings["timestamp"].max()
print("Start Timestamp {}, End Timestamp {}".format(start_time, end_time))
print(datetime.datetime.fromtimestamp(start_time))
print(datetime.datetime.fromtimestamp(end_time))
adjusted_start_day = int(time.mktime(datetime.datetime.fromtimestamp(start_time).date().timetuple()))
adjusted_end_day = int(time.mktime(datetime.datetime.fromtimestamp(end_time).date().timetuple())) + 86400<jupyter_output><empty_output><jupyter_text>5.  Get the Average time mapping per User<jupyter_code>average_df = ratings.groupby("UserId")["timestamp"].mean().reset_index()
average_df.head(n=3)
average_times = pd.Series(average_df.timestamp.values, index=average_df.UserId).to_dict()<jupyter_output><empty_output><jupyter_text>View DatasetRe-mapped UserID and MovieId, Date as Date w.o time as a timestamp<jupyter_code>ratings.info()
ratings.describe()
# Checking if any nulls or NaN exist in any of the columns
ratings.count() == ratings.shape[0]
ratings.head(n=5)
ratings.tail(n=5)<jupyter_output><empty_output><jupyter_text>Bellkor Algorithm    1. Initilisation    2. Prepare dataset (done)    3. Train Model    4. Basic stats on the Parameters    5. Prediction    6. Save results  1. Initialisation<jupyter_code>from Bellkor.Algorithm import BellkorAlgorithm
print(BellkorAlgorithm.__doc__)
print(BellkorAlgorithm.__init__.__doc__)
calibrator = BellkorAlgorithm(n_items=movie_count, 
                              n_users=user_count, 
                              global_mean=global_mean,
                              time_setting=dict(Start=adjusted_start_day, 
                                                End=adjusted_end_day))<jupyter_output>2018-01-07 20:03:26 - Bellkor.Utils.Decorators.timeit - INFO - Method: __init__ - 3.133376359939575 seconds<jupyter_text>2. Prepare dataset expected columns: [Index, TimePeriod, User, Item, BaseRating]<jupyter_code>ratings.loc[:, "Index"] = ratings.index
ratings.head(n=3)
X = ratings.loc[:, ["Index", "timestamp", "UserId", "MovieId", "rating"]].as_matrix()
# show a few rows
print(X[0:3, :])<jupyter_output>[[0.00000000e+00 1.11248603e+09 0.00000000e+00 0.00000000e+00
  3.50000000e+00]
 [1.00000000e+00 1.11248468e+09 0.00000000e+00 1.00000000e+00
  3.50000000e+00]
 [2.00000000e+00 1.11248482e+09 0.00000000e+00 2.00000000e+00
  3.50000000e+00]]<jupyter_text>3. Train the Model<jupyter_code># When you running this properly increase the sample_size and number of iterations
cost, error = calibrator.train(x=X, average_times=average_times, sample_size=10, iterations=10)<jupyter_output>2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 1
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Epoch: 0 took: 0.0035970211029052734s
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 2
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Epoch: 1 took: 0.0024919509887695312s
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 3
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Epoch: 2 took: 0.0022182464599609375s
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 4
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Epoch: 3 took: 0.0029098987579345703s
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 5
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Epoch: 4 took: 0.0027086734771728516s
2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.train - INFO - Running Epoch: 6
2018-01-07 20:03:29 - Bell[...]<jupyter_text>Review the Parameters  4. Basic stats on the Parameters<jupyter_code># TODO<jupyter_output><empty_output><jupyter_text>5. Prediction<jupyter_code>preds = calibrator.predict(x=X[0:1000, :], average_times=average_times)
predictions = pd.DataFrame(data=preds, columns=["Index", "Prediction"])
predictions.head(n=3)<jupyter_output><empty_output><jupyter_text>6. Save results<jupyter_code>calibrator.pickle_parameters(file_name="DEMO_OUTPUT")<jupyter_output>2018-01-07 20:03:29 - BellKor.BellkorAlgorithm.pickle_parameters - INFO - Save the Parameters @ /home/dan/PycharmProjects/BellkorAlgorithm/resources/MODEL_PARAMS/DEMO_OUTPUT.pickle
2018-01-07 20:03:29 - Bellkor.Utils.Decorators.timeit - INFO - Method: pickle_parameters - 0.0007581710815429688 seconds<jupyter_text>Concluding Remarks<jupyter_code># TODO<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12607014636840652, 'fraction_numerical': 0.08298812482739575, 'mean_word_length': 5.4727435210008935, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '218449', 'license': 'MIT', 'n_tokens_mistral': 2930, 'n_tokens_neox': 2519, 'n_words': 518, 'path': 'resources/Demo.ipynb', 'repo_name': 'FunctorML/BellkorAlgorithm', 'size': '7242'}