<jupyter_start><jupyter_text>Load Data and Models<jupyter_code># set path to where repo is located
REPO_PATH = os.path.join("..", "..")

# setting path
DATA_PATH = os.path.join(REPO_PATH, "data", "evaluation")
D2V_PATH = os.path.join(REPO_PATH, "models", "d2v")

# category to train
CATEGORY = "Grocery_and_Gourmet_Food"

# reproducibility checks
SEED = 42
np.random.seed(42)

# load train/test dataset and doc2vec trained model
train = pd.read_csv(f"{DATA_PATH}/{CATEGORY}_train.csv")
test = pd.read_csv(f"{DATA_PATH}/{CATEGORY}_test.csv")
d2v = Doc2Vec.load(f"{D2V_PATH}/{CATEGORY}_item_50_10_d2v.model")
# checking train dataframe
train.head().append(train.tail())
test.head().append(test.tail())
# generating test history
test_user_history = (pd.DataFrame(test.groupby(['reviewerID'])['asin']
                                  .apply(list).reset_index()))
print(test_user_history)
# testing d2v models
d2v.dv[0]<jupyter_output><empty_output><jupyter_text>Generate N-Recommendations = {5, 10, 15, 20}  Instantiate Embedded Review CBF (Item-based)<jupyter_code>er_cbf = algorithms.EmbeddedReviewCBF(d2v)
%%time
# fit learning algorithm to training data
er_cbf.fit(train)
%%time
# generate n-number of candidates items (200)
candidate_items = er_cbf.test()<jupyter_output>100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 13397/13397 [00:05<00:00, 2587.19it/s]<jupyter_text>Loop through N = {5, 10, 15, 20}For each top-N setting, we will generate candidates items up to *N*-items and run metrics evaluation of `Recall@N` and `Novelty@N` on all users.<jupyter_code># generate item popularity
item_popularity = evaluate_model.generate_item_popularity(train)
%%time
n_recommendations = {}
for n in [5, 10, 15, 20]:
    # retrieve the top-n items based on similarities
    top_ns = er_cbf.get_top_n(n)
    # evaluate how well the recommended items predicted the future purchases
    n_recommended_items = (evaluate_model.
                           evaluate_recommendations(model_name="ER-CBF",
                                                    top_ns=top_ns,
                                                    user_rating_history=test_user_history, 
                                                    item_popularity=item_popularity, 
                                                    n=n,
                                                    mf_based=False))
    # saving the n-value and recommended items
    n_recommendations[n] = (top_ns, n_recommended_items)<jupyter_output>The ER-CBF has an average recall@5: 0.01361, average novelty@5: 0.93771
The ER-CBF has an average recall@10: 0.02177, average novelty@10: 0.94207
The ER-CBF has an average recall@15: 0.02927, average novelty@15: 0.94574
The ER-CBF has an average recall@20: 0.03584, average novelty@20: 0.94881
CPU times: user 48.3 s, sys: 657 ms, total: 49 s
Wall time: 48.8 s<jupyter_text>Evaluate N-RecommendationsFor each top-N recommendation list, we pick a randomly sampled user to look at their *N*-number of recommendations based on their purchase history.  N=5<jupyter_code>top_ns_05 = n_recommendations[5][0]
utilities.retrieve_recommendations(train, top_ns_05)<jupyter_output>For user: A2H84V55USFJQX:
Purchase History:
             asin                                              title
13844  B000ZSZ5S4  Blue Diamond Almonds, Bold Salt &amp; Vinegar,...
41088  B005K4Q1T0  Grove Square Hot Cocoa Dark Chocolate, 24 Sing...

Recommending:

         asin                                              title
0  B0033HPPIO  Donut House Collection Coffee Chocolate Glazed...
1  B001EO5Z6A   Rocamojo Organic Gourmet Coffee, Wild French ...
2  B002WENMX4  Suchard Hot Chocolate Syrup, 8-Count T-Discs f...
3  B002MV63EW  Gloria Jean's K-Cup, Holiday Traditions, 24-Co...
4  B005QKH1V0  Cameron's Coffee Roasted Ground Coffee, Variet...<jupyter_text>N=10<jupyter_code>top_ns_10 = n_recommendations[10][0]
utilities.retrieve_recommendations(train, top_ns_10)<jupyter_output>For user: A2W9B725TZBXOX:
Purchase History:
             asin                                              title
1645   B0005Z6LLW  Ghirardelli Hot Chocolate Mix , Chocolate Moch...
4683   B000ED9L6C  Bob's Red Mill Raw Shelled Sunflower Seeds (Ke...
8508   B000GATCRQ  Dream Foods International Volcano Lemon Burst,...
15333  B0014EOUAW  V8 V-Fusion Light Peach Mango Juice Drink, 46-...
18192  B001E52ZAS  Post Shredded Wheat Lightly Frosted Cereal, Sp...
21093  B001EPQV1W  Honey Bunches of Oats with Almonds, 14.5-Ounce...
22932  B001G8UC8K  Tootsie Roll Midgees Candy 5 Pound Value Bag 7...
23025  B001GVIRD4                     Grocery &amp; Gourmet Food" />
24945  B001NC8HQS  Idahoan Buttery homestyle flavored mashed pota...
25830  B001SAQ7WQ  Progresso Traditional Soup, Chickarina (Chicke...
26956  B0025UOMY8  Maruchan Yakisoba Spicy Vegetable, 3.79-Ounce ...
33467  B00421DMLM  Libby's Splenda Sliced Peaches, 23.5-Ounce Jar...
34141  B004727CL2  Snack Factory Pretzel Crisps Everything,[...]<jupyter_text>N=15<jupyter_code>top_ns_15 = n_recommendations[15][0]
utilities.retrieve_recommendations(train, top_ns_15)<jupyter_output>For user: A1G8ECTOF35O2R:
Purchase History:
             asin                                              title
9115   B000H26J7E  Lindt Excellence Bar, 70% Cocoa Smooth Dark Ch...
47677  B00EKLPLU4              Healthworks Cacao Powder Organic, 1lb

Recommending:

          asin                                              title
0   B0019ZHU2Q  Ghirardelli Chocolate Intense Dark Bar, Twilig...
1   B000KNB4WA  Ghirardelli Chocolate Baking Chips, Semi-Sweet...
2   B001G0MG2I   Ghirardelli Chocolate Intense Dark Bar, Midni...
3   B000OQ4A3S  Navitas Organics Cacao Nibs, 8 oz. Bag &mdash;...
4   B000Y0HCHK  Nevada Manna Sugar Free Semi Sweet Chocolate C...
5   B000H27PB8  Ritter Sport Bars, Dark Chocolate with Marzipa...
6   B0029J6QLM         Snickers Dark Chocolate Candy (Pack of 24)
7   B0029JASWA  Dove Dark Chocolate Promises, 9.5-Ounce Packag...
8   B000JSLYSU        Cadbury Flake Chocolate Bars 23.5g, 4-Count
9   B001H9N9RE  Ghirardelli Dark Chocolate Twilight Delight 72...
10  B00[...]<jupyter_text>N=20<jupyter_code>top_ns_20 = n_recommendations[20][0]
utilities.retrieve_recommendations(train, top_ns_20)<jupyter_output>For user: A1NEKFNO1E743Y:
Purchase History:
            asin                                              title
8695  B000GFYRHG  Bigelow Constant Comment Tea 20-Count Boxes (P...

Recommending:

          asin                                              title
0   B000F4DKAI  Twinings of London English Afternoon Black Tea...
1   B007JFXXJY  Lipton Herbal Pyramid Tea Bags, Orange Blossom...
2   B001E5E1WS  Rishi Tea Organic Silver Needle Jasmine Loose ...
3   B005CT9OEW  Lipton Green Tea Bags, Decaffeinated Blackberr...
4   B001BM62T4  Red Espresso Ground Rooibos Tea, 8.8-Ounce Pou...
5   B008YUL4CQ  Lipton Decaffeinated Green Tea, Superfruit Jas...
6   B002AR158W   Numi Organic Tea Magnolia Puerh, Full Leaf Gr...
7   B0061IUKDM  Higgins &amp; Burke, Green Tea, 20 Count (Pack...
8   B001E5E20O  Rishi Tea Organic Black Tea, Earl Grey Loose T...
9   B004051BO0  Yogi Tea - Vanilla Spice Perfect Energy - Ener...
10  B002HQCWYM  Twinings of London English Breakfast Tea K-Cup...
11  B000CQBZ[...]<jupyter_text>Cross-Analysis for Cold-Start Users (<= 2 Purchased Items)For each top-N setting, we will generate candidates items up to *N*-items and run metrics evaluation of `Recall@N` and `Novelty@N` on cold-start users (e.g., users who purchased two or less items based on items per user in the training set).<jupyter_code>cold_start_users = utilities.generate_cold_start_users(train)
for n in tuple(zip([5, 10, 15, 20], [top_ns_05, top_ns_10, top_ns_15, top_ns_20])):
    cold_start_top_ns = dict(filter(lambda x: x[0] in cold_start_users, n[1].items()))
    # evaluate how well the recommended items predicted the future purchases
    # on cold start users
    n_recommended_items = (evaluate_model.
                           evaluate_recommendations(model_name="ER-CBF",
                                                    top_ns=cold_start_top_ns,
                                                    user_rating_history=test_user_history, 
                                                    item_popularity=item_popularity, 
                                                    n=n[0],
                                                    mf_based=False))<jupyter_output>The ER-CBF has an average recall@5: 0.01585, average novelty@5: 0.93636
The ER-CBF has an average recall@10: 0.02408, average novelty@10: 0.94052
The ER-CBF has an average recall@15: 0.03251, average novelty@15: 0.94431
The ER-CBF has an average recall@20: 0.03967, average novelty@20: 0.94768<jupyter_text>Generating Recommended Items DataFrame<jupyter_code>max_recommendations = (utilities
                       .generate_recommendations_df(train=train,
                                                    n_recommendations=n_recommendations,
                                                    algo_name="ER-CBF",
                                                    mf_based=False,
                                                    max_recommended=20))
max_recommendations<jupyter_output><empty_output><jupyter_text>Store in `SQLite` DB<jupyter_code># engine = create_engine("sqlite:///recommender.db", echo=True)
# max_recommendations.to_sql(f"{CATEGORY}", con=engine, if_exists="append")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09334176148335441, 'fraction_numerical': 0.06784660766961652, 'mean_word_length': 2.679457364341085, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 43, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '295028', 'license': 'MIT', 'n_tokens_mistral': 3644, 'n_tokens_neox': 3090, 'n_words': 822, 'path': 'notebooks/experiments/11-lwt-experimental-setup-er-cbf-ggf.ipynb', 'repo_name': 'wtlow003/recommendation-systems', 'size': '9492'}