<jupyter_start><jupyter_text>Load Data<jupyter_code># set path to where repo is located
REPO_PATH = os.path.join("..", "..")

# setting path
DATA_PATH = os.path.join(REPO_PATH, "data", "evaluation")

# category to train
CATEGORY = "Pet_Supplies"

# training parameters: training epochs, learning, and regularisation rate
PARAMS = {"n_epochs": 5,
          "lr_all": 0.005,
          "reg_all": 0.1}

# reproducibility checks
SEED = 42
np.random.seed(42)

# load train/test dataset
train = pd.read_csv(f"{DATA_PATH}/{CATEGORY}_train.csv")
test = pd.read_csv(f"{DATA_PATH}/{CATEGORY}_test.csv")
# checking train dataframe
train.head().append(train.tail())
test.head().append(test.tail())
# generating test history
test_user_history = (pd.DataFrame(test.groupby(['reviewerID'])['asin']
                                  .apply(list).reset_index()))
print(test_user_history)<jupyter_output>reviewerID                                  asin
0      A04173782GDZSQ91AJ7OD              [B0090Z9AYS, B00CPDWT2M]
1      A042274212BJJVOBS4Q85              [B005AZ4M3Q, B00771WQIY]
2       A0436342QLT4257JODYJ  [B0018CDR68, B003SJTM8Q, B00474A3DY]
3      A04795073FIBKY8GSLZYI              [B001PKT30M, B005DGI2RY]
4      A06658082A27F4VB5UG8E              [B000TZ1TTM, B0019VUHH0]
...                      ...                                   ...
18993          AZYJE40XW6MFG              [B00HVAKJZS, B00IDZT294]
18994          AZZ56WF4X19G2                          [B004A7X218]
18995          AZZNK89PXD006  [B0002DHV16, B005BP8MQ8, B009RTX4SU]
18996          AZZV9PDNMCOZW              [B007EQL390, B00ISBWVT6]
18997          AZZYW4YOE1B6E  [B0002AQPA2, B0002AQPA2, B0002ARQV4]

[18998 rows x 2 columns]<jupyter_text>Generate N-Recommendations = {5, 10, 15, 20}  Instantiate FunkSVD (Matrix Factorization)<jupyter_code># instantiating funksvd 
# funk_svd = algorithms.FunkMF(n_epochs=N_EPOCHS,
#                              lr_all=LR_ALL,
#                              reg_all=BETA)

funk_svd = algorithms.FunkMF(**PARAMS)
%%time
# fitting to the training data
funk_svd.fit(train)
%%time
# creating testset for prediction
testset = funk_svd.trainset.build_anti_testset()
%%time
# generate candidate times
candidate_items = funk_svd.test(testset, verbose=False)<jupyter_output>CPU times: user 11min 7s, sys: 6min 19s, total: 17min 26s
Wall time: 20min 52s<jupyter_text>Loop through N = {5, 10, 15, 20}For each top-N setting, we will generate candidates items up to *N*-items and run metrics evaluation of `Recall@N` and `Novelty@N` on all users.<jupyter_code># generate item popularity
item_popularity = evaluate_model.generate_item_popularity(train)
%%time
n_recommendations = {}
for n in [5, 10, 15, 20]:
    # retrieve the top-n items based on similarities
    top_ns = funk_svd.get_top_n(n)
    # evaluate how well the recommended items predicted the future purchases
    n_recommended_items = (evaluate_model.
                           evaluate_recommendations(model_name="FUNK-SVD",
                                                    top_ns=top_ns,
                                                    user_rating_history=test_user_history, 
                                                    item_popularity=item_popularity, 
                                                    n=n,
                                                    mf_based=True))
    # saving the n-value and recommended items
    n_recommendations[n] = (top_ns, n_recommended_items)<jupyter_output>The FUNK-SVD has an average recall@5: 0.00507, average novelty@5: 0.79858
The FUNK-SVD has an average recall@10: 0.00893, average novelty@10: 0.79893
The FUNK-SVD has an average recall@15: 0.01312, average novelty@15: 0.80896
The FUNK-SVD has an average recall@20: 0.01613, average novelty@20: 0.82021
CPU times: user 6min 4s, sys: 3min 20s, total: 9min 24s
Wall time: 12min 49s<jupyter_text>Evaluate N-RecommendationsFor each top-N recommendation list, we pick a randomly sampled user to look at their *N*-number of recommendations based on their purchase history.  N=5<jupyter_code>top_ns_05 = n_recommendations[5][0]
utilities.retrieve_recommendations(train, top_ns_05, mf_based=True)<jupyter_output>For user: AM7YDJ5OF3P5P:
Purchase History:
             asin                                              title
40436  B000PKSW5A  Precious Cat Dr. Elsey's Kitten Attract Scoopa...
50006  B001K5NTOS                   Our Pets SmartScoop Litter Mat-p
57972  B003R0LLIK  Purina Pro Plan Focus Chicken &amp; Rice Formu...
66811  B008MZILMY           Blue Tasty Chicken Kitty Yums Cat Treats

Recommending:

         asin                                              title
0  B001LNSSH2  ZYMOX Pet King Brand Otic Pet Ear Treatment wi...
1  B000F4AVPA                                Chuckit! Ultra Ball
2  B003JFRQQ4  Scaredy Cut Tiny Trim by Small Pet Grooming Sa...
3  B0002AS1CC                 Bergan Turbo Scratcher Accessories
4  B000255NCI                               API Master Test Kits<jupyter_text>N=10<jupyter_code>top_ns_10 = n_recommendations[10][0]
utilities.retrieve_recommendations(train, top_ns_10, mf_based=True)<jupyter_output>For user: AUIQTTPIMCVQ4:
Purchase History:
             asin                                              title
40981  B000ROR8Z4  Sojos Good Dog Crunchy Natural Dog Treats, 8-O...
60885  B004ULYZII                         Bully Pizzle, 6 Inch 25 Pk
67348  B00AEJANCW  KONG Chase-It Wand Squeaking Dog Toy, Assorted...

Recommending:

         asin                                              title
0  B003JFRQQ4  Scaredy Cut Tiny Trim by Small Pet Grooming Sa...
1  B000K9JRH8  GoCat DaBird Feather Refill, Assorted Colors, ...
2  B00025Z6YI  TetraFin Balanced Diet Goldfish Flake Food for...
3  B0002DJONY                            Vittles Vault Stackable
4  B000255NCI                               API Master Test Kits
5  B000F4AVPA                                Chuckit! Ultra Ball
6  B0002AS1CC                 Bergan Turbo Scratcher Accessories
7  B000255OIG       Stewart Freeze Dried Treats 14 oz Beef Liver
8  B0002AQI9K                     Marina Airline Tubing, 20 Feet
9  B000QSON4K  [...]<jupyter_text>N=15<jupyter_code>top_ns_15 = n_recommendations[15][0]
utilities.retrieve_recommendations(train, top_ns_15, mf_based=True)<jupyter_output>For user: A2YORE61EX6ASQ:
Purchase History:
             asin                             title
24664  B0002XUIRA  Fashion Pet Suede Shearling Coat
24707  B0002XUIRA  Fashion Pet Suede Shearling Coat

Recommending:

          asin                                              title
0   B0002DJONY                            Vittles Vault Stackable
1   B000F4AVPA                                Chuckit! Ultra Ball
2   B0002AS1CC                 Bergan Turbo Scratcher Accessories
3   B001LNSSH2  ZYMOX Pet King Brand Otic Pet Ear Treatment wi...
4   B003JFRQQ4  Scaredy Cut Tiny Trim by Small Pet Grooming Sa...
5   B000K9JRH8  GoCat DaBird Feather Refill, Assorted Colors, ...
6   B000255NCI                               API Master Test Kits
7   B00025Z6YI  TetraFin Balanced Diet Goldfish Flake Food for...
8   B0010B6IFY  Purina Fancy Feast Grilled Collection Wet Cat ...
9   B003BYQ1C8                            Armarkat Cat Tree Model
10  B0002AQKIO        Fluval Carbon, 100-gram Nylon Bags -[...]<jupyter_text>N=20<jupyter_code>top_ns_20 = n_recommendations[20][0]
utilities.retrieve_recommendations(train, top_ns_20, mf_based=True)<jupyter_output>For user: A3PDSVXNSBA9ZQ:
Purchase History:
             asin                                              title
20851  B0002DK2DU              Roll n Clean Self Cleaning Litter Box
21049  B0002DK2DU              Roll n Clean Self Cleaning Litter Box
31120  B0009YWLCM  K&amp;H Pet Products Pet Bed Warmer Tan - MET ...
35405  B000G136RM                 Hartz Home Protection Gel Dog Pads

Recommending:

          asin                                              title
0   B001LNSSH2  ZYMOX Pet King Brand Otic Pet Ear Treatment wi...
1   B000F4AVPA                                Chuckit! Ultra Ball
2   B000K9JRH8  GoCat DaBird Feather Refill, Assorted Colors, ...
3   B003JFRQQ4  Scaredy Cut Tiny Trim by Small Pet Grooming Sa...
4   B0002AS1CC                 Bergan Turbo Scratcher Accessories
5   B0002563O0                           JW Comfy Perch for Birds
6   B00008JOL0                       Zuke's Hip Action Dog Treats
7   B0002DJONY                            Vittles Vault Stackable
8[...]<jupyter_text>Cross-Analysis for Cold-Start Users (<= 2 Purchased Items)For each top-N setting, we will generate candidates items up to *N*-items and run metrics evaluation of `Recall@N` and `Novelty@N` on cold-start users (e.g., users who purchased two or less items based on items per user in the training set).<jupyter_code>cold_start_users = utilities.generate_cold_start_users(train)
for n in tuple(zip([5, 10, 15, 20], [top_ns_05, top_ns_10, top_ns_15, top_ns_20])):
    cold_start_top_ns = dict(filter(lambda x: x[0] in cold_start_users, n[1].items()))
    # evaluate how well the recommended items predicted the future purchases
    # on cold start users
    n_recommended_items = (evaluate_model.
                           evaluate_recommendations(model_name="FUNK-SVD",
                                                    top_ns=cold_start_top_ns,
                                                    user_rating_history=test_user_history, 
                                                    item_popularity=item_popularity, 
                                                    n=n[0],
                                                    mf_based=True))<jupyter_output>The FUNK-SVD has an average recall@5: 0.00512, average novelty@5: 0.79950
The FUNK-SVD has an average recall@10: 0.00909, average novelty@10: 0.79886
The FUNK-SVD has an average recall@15: 0.01289, average novelty@15: 0.80873
The FUNK-SVD has an average recall@20: 0.01572, average novelty@20: 0.81989<jupyter_text>Generating Recommended Items DataFrame<jupyter_code>max_recommendations = (utilities
                       .generate_recommendations_df(train=train,
                                                    n_recommendations=n_recommendations,
                                                    algo_name="FUNK-SVD",
                                                    mf_based=True,
                                                    max_recommended=20))
max_recommendations<jupyter_output><empty_output><jupyter_text>Store in `SQLite` DB<jupyter_code># engine = create_engine("sqlite:///recommender.db", echo=True)
# max_recommendations.to_sql(f"{CATEGORY}", con=engine, if_exists="append")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0685124354763022, 'fraction_numerical': 0.07433129985922102, 'mean_word_length': 2.1003782368344486, 'pattern_counts': {'":': 3, '<': 34, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 50, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '196469', 'license': 'MIT', 'n_tokens_mistral': 3958, 'n_tokens_neox': 3353, 'n_words': 879, 'path': 'notebooks/experiments/02-lwt-experimental-setup-funk-svd-ps.ipynb', 'repo_name': 'wtlow003/recommendation-systems', 'size': '10655'}