<jupyter_start><jupyter_text>ExperimentsThis notebook contains experiments ran on the entire dataset. It contains a log of all the recommenders we discuss during our final presentation. We utilise both the v2 and v3 dataset. The hyperparamater optimisation and other experiments that do not need to be run on the entire dataset.<jupyter_code>from src.recommenders import ImprovedRecommender, PopBasedRecommender
import numpy as np
import pickle
def save_pickle(save_path: str, data:object) -> None:
    """Save object to pickle file

    Args:
        save_path (str): filepath to save location
        data (object): object to be saved
    """
    with open(save_path, "wb") as handle:
        pickle.dump(data, handle, protocol=pickle.HIGHEST_PROTOCOL)
def invert_dict(dicts:list) -> dict:
    """Converts list of dicts to dict of lists

    Args:
        dicts (list): list containing dicts to be inverted

    Returns:
        dict: inverted dict that contains a list of all values in the previous dicts (in order)
    """
    new_dict = dict()
    for i, d in enumerate(dicts):
        for key in d.keys():
            ar = new_dict.get(key, np.zeros(len(dicts)))
            ar[i] = d[key]
            new_dict[key] = ar
    return new_dict<jupyter_output><empty_output><jupyter_text>ParametersThe generation parameters all our recommenders will use.<jupyter_code>am_splits = 5           # amount of splits to consider
am_recommended = 20     # amount of items to recommend
k_values = [5, 10, 20]  # k values for metrics to be considered, max should be lesser or equal to am_recommended<jupyter_output><empty_output><jupyter_text>Old data (v2)The data used for these test is the complete dataset, as provided by group 10.- Duplicate items are dropped- Unowned items are dropped- Users with less than 3 items are droppedWe need at least 3 items for each users to ensure that none of the train/validation/test splits are empty.<jupyter_code>data_basepath = "./data/v2/"
quant_output_path = "./evaluation/v2/quantitative/"
qual_output_path = "./evaluation/v2/qualitative/"
items_path = data_basepath + "games.pkl"
reviews_path = data_basepath + "reviews.parquet"
data_path = data_basepath + "interactions_splits_"<jupyter_output><empty_output><jupyter_text>Popularity based recommenderFirst we create our baseline, this recommender will simply give the first `am_recommended` most frequently occurring items the user doesn't already have in it's training dataset.<jupyter_code>evaluations = list()

for split in range(am_splits):
    rec = PopBasedRecommender(train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet", val_path=f"{data_path}{split}_val.parquet")
    rec.generate_recommendations(amount=am_recommended)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
        
evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "PopBased.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Base recommenderRuns our recommender at the same settings used in the first presentation. This will be used to compare against our previous results, since our BaseRecommender is no longer compatible with the new data structure.<jupyter_code>evaluations = list()
use_data = ["genres", "tags"]
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, tfidf="smooth", normalize=True, columns=use_data, weighting_scheme=None)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"BaseRecommender_{split}.csv")

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "BaseRecommender.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Improved recommenderWe explore the effect of different "improvements" we are making.First we increase the data available in our item representations, we utilise the optimal combination we have found for our dataset which were:- specs- publisher- developer- tags<jupyter_code>evaluations = list()
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_none_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_none.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>A second improvement we explored was adding explicit feedback, in the form of game reviews. Not every user has reviewed every game they own, in fact only a tiny fraction of games is actually been reviewed. We append this explicit feedback from the user with sentiment rating, this sentiment rating is the average review score for a game. We explore two different sentiment metrics; mixed and ratingIn mixed we use a combination of the average rating and the amount of users that own the game, this will introduce a bias to more popular games.<jupyter_code>evaluations = list()
weighting_scheme = {"playtime": False, "sentiment": "mixed", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_sentiment_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_sentiment.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>For the rating only, we only use the average rating.<jupyter_code>evaluations = list()
weighting_scheme = {"playtime": False, "sentiment": "rating", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_sentiment_onlyrating_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_sentiment_onlyrating.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Lastly we can include some implicit feedback from the user, this comes in the form of playtime. We use a logarithmic base 10 function to remove extremely large outliers, not doing this would make the recommender focus almost solely on that one game. By using logarithmic time, we can ensure that more games are being considered.<jupyter_code>evaluations = list()
weighting_scheme = {"playtime": True, "sentiment": "mixed", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_allweighting_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_allweighting.pickle.gz", evaluations)
evaluations
evaluations = list()
weighting_scheme = {"playtime": True, "sentiment": "rating", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_allweighting_onlyrating_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_allweighting_onlyrating.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>New data (v3)We have observed that the previous dataset there occurs a heavy bias towards popular items. To combat this bias the top 1% of games has been removed.<jupyter_code>data_basepath = "./data/v3/"
quant_output_path = "./evaluation/v3/quantitative/"
qual_output_path = "./evaluation/v3/qualitative/"
items_path = data_basepath + "games.pkl.gz"
reviews_path = data_basepath + "reviews.parquet"
data_path = data_basepath + "interactions_splits_"<jupyter_output><empty_output><jupyter_text>Popularity based recommenderSince top 1% has been removed, the popularity based recommender suffers a major performance hit.<jupyter_code>evaluations = list()

for split in range(am_splits):
    rec = PopBasedRecommender(train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet", val_path=f"{data_path}{split}_val.parquet")
    rec.generate_recommendations(amount=am_recommended)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
        
evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "PopBased.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Base recommender<jupyter_code>evaluations = list()
use_data = ["genres", "tags"]
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, tfidf="smooth", normalize=True, columns=use_data, weighting_scheme=None)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"BaseRecommender_{split}.csv")

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "BaseRecommender.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Improved recommenderIncreased item representation on v3 dataset<jupyter_code>evaluations = list()
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_none_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_none.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Adding explicit feedback from reviews, appended with sentiment rating.<jupyter_code>evaluations = list()
weighting_scheme = {"playtime": False, "sentiment": "rating", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_sentiment_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_sentiment.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output><jupyter_text>Adding play time as implicit feedback<jupyter_code>evaluations = list()
weighting_scheme = {"playtime": True, "sentiment": "rating", "reviews": True}
for split in range(am_splits):
    rec = ImprovedRecommender(items_path, train_path=f"{data_path}{split}_train.parquet", test_path=f"{data_path}{split}_test.parquet",
                              val_path=f"{data_path}{split}_val.parquet", reviews_path=reviews_path, sparse=True, tfidf="smooth", normalize=True, weighting_scheme=weighting_scheme)
    rec.generate_recommendations(amount=am_recommended, silence=True)
    evaluation = dict()
    for k in k_values:
        evaluation.update(rec.evaluate(val=False, k=k))
    evaluations.append(evaluation)
    rec.qualitative_evaluation(export_path=qual_output_path + f"ImprovedRec_allweighting_{split}.csv")    

evaluations = invert_dict(evaluations)
save_pickle(quant_output_path + "ImprovedRec_allweighting.pickle.gz", evaluations)
evaluations<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10045448469455955, 'fraction_numerical': 0.0018045715813393931, 'mean_word_length': 5.761409850881157, 'pattern_counts': {'":': 18, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '428699', 'license': 'MIT', 'n_tokens_mistral': 4757, 'n_tokens_neox': 4617, 'n_words': 1018, 'path': 'experiments_history.ipynb', 'repo_name': 'lderoeck/AI-Project', 'size': '14962'}