<jupyter_start><jupyter_text>Modelo de filtrado colaborativo items to items SummaryEn este notebook se exploró la estrategia de filtrado colaborativo items to items. Esta estrategia simple, tiene la ventaja de requerir poca capacidad de procesamiento, consiguiendo una buena relación entre performance y resultados, aunque quizá lo logre captar patrones más complejos.Se forman las siguientes asociaciones de productos a partir de las sesiones:* Asociación entre productos por visitas en una misma sesión* Asociación entre productos visitados y productos comprados.* Asociación entre términos de búsquedas y productos visitados.* Asociación entre términos de búsquedas y productos comprados.También se scorean los productos visitados con una función que toma en cuenta las posiciones de los page views del ítem en la sesión. Tomando la posición 1 a la más reciente a la la compra, el score se define como:\begin{equation*}score(item_i) = \sum_{pos}  \frac{1}{log_{10}(pos + 1)}\end{equation*}Solamente con esta estrategia de score de productos visitados, y utilizando como relleno los productos más vistos del dominio de la primera recomendación, se logra un **ndcg 0.2639** en test.Finalmente se pesan todos los scores de asociación consiguiendo un **ndcg 0.2889** en test y **0.2876** en submission.<jupyter_code>import pandas as pd
import numpy as np
import json
from collections import Counter, defaultdict
import heapq<jupyter_output><empty_output><jupyter_text>Lee del catatálogo los dominios de los productos (se usa en la evalaución)<jupyter_code>catalog = {}
with open("./data/item_data.jl", "rt") as fd:
    for line in fd:
        data = json.loads(line)
        item_id = data.pop("item_id")
        data.pop("title")
        catalog[item_id] = data
        # obtiene el país
        catalog[item_id]["country"] = data["category_id"][2]
ITEM_TO_DOMAIN = {}
for item_id, item_info in catalog.items():
    ITEM_TO_DOMAIN[item_id] = item_info["domain_id"]
IDCG = np.sum([(1 if i != 1 else 12) / np.log2(1 + i) for i in range(1, 11)])

def dcg(rec, y_item_id, n=10):
    y_domain = ITEM_TO_DOMAIN[y_item_id]
    
    return np.sum([(1 if yhat_item_id != y_item_id else 12) / np.log2(1 + i)\
                   for i, yhat_item_id in enumerate(rec[:n], 1)\
                  if (ITEM_TO_DOMAIN[yhat_item_id] == y_domain)])<jupyter_output><empty_output><jupyter_text>Los productos sin precio o dominio se excluyen de las recomendaciones<jupyter_code>BLACK_LIST = set()
for item_id, item_info in catalog.items():
    if (item_info["domain_id"] == None) or (item_info["price"] == None):
        BLACK_LIST.add(item_id) 
len(BLACK_LIST)<jupyter_output><empty_output><jupyter_text>Recomendaciones BaselineSe usa como relleno (cold-start)<jupyter_code>most_viewed_items = Counter()
most_sessions_items = Counter()
most_bought_items = Counter()

most_viewed_by_domain = {}

line_idx = 0
with open("./data/train_dataset-train_split.jl", "rt") as fd:
    for line in fd:
        line_idx += 1
        data = json.loads(line)
        view = [event["event_info"] for event in data["user_history"] if event["event_type"] == "view"]
        most_viewed_items.update(view)
        most_sessions_items.update(set(view))
        most_bought_items[data["item_bought"]] += 1
        
        for item_id in set(view):
            domain = ITEM_TO_DOMAIN[item_id]
            if not domain in most_viewed_by_domain:
                most_viewed_by_domain[domain] = Counter()
            most_viewed_by_domain[domain][item_id] +=1
most_viewed_items_br =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_viewed_items.items() if catalog[item_id]["country"] == "B" }).most_common(10)]

most_viewed_items_mx  =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_viewed_items.items() if catalog[item_id]["country"] == "M" }).most_common(10)]

most_viewed_items = [item for item, _ in most_viewed_items.most_common(10)]
most_sessions_items_br =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_sessions_items.items() if catalog[item_id]["country"] == "B" }).most_common(10)]

most_sessions_items_mx  =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_sessions_items.items() if catalog[item_id]["country"] == "M" }).most_common(10)]

most_sessions_items = [item for item, _ in most_sessions_items.most_common(10)]
most_bought_items_br =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_bought_items.items() if catalog[item_id]["country"] == "B" }).most_common(10)]

most_bought_items_mx  =[item_id for item_id, _ in
    Counter({item_id: count for item_id, count\
             in most_bought_items.items() if catalog[item_id]["country"] == "M" }).most_common(10)]

most_bought_items = [item for item, _ in most_bought_items.most_common(10)]
for domain, counter in most_viewed_by_domain.items():
    most_viewed_by_domain[domain] = [item for item, _ in counter.most_common(10)]
viewed_count = 0
n_recs = 0

pop_view_sum_dcg = 0
pop_session_sum_dcg = 0
pop_bought_sum_dcg = 0


country_pop_view_sum_dcg = 0
country_pop_session_sum_dcg = 0
country_pop_bought_sum_dcg = 0

with open("./data/train_dataset-test_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1 

        data = json.loads(line)
        item_bought = data["item_bought"]
        
        pop_view_sum_dcg += dcg(most_viewed_items, item_bought)
        pop_session_sum_dcg += dcg(most_sessions_items, item_bought)
        pop_bought_sum_dcg += dcg(most_bought_items, item_bought)
        
        if catalog[item_bought]["country"] == "B":
            country_pop_view_sum_dcg += dcg(most_viewed_items_br, item_bought)
            country_pop_session_sum_dcg += dcg(most_sessions_items_br, item_bought)
            country_pop_bought_sum_dcg += dcg(most_bought_items_br, item_bought)
        elif catalog[item_bought]["country"] == "M":
            country_pop_view_sum_dcg += dcg(most_viewed_items_mx, item_bought)
            country_pop_session_sum_dcg += dcg(most_sessions_items_mx, item_bought)
            country_pop_bought_sum_dcg += dcg(most_bought_items_mx, item_bought)
        else:
            raise("Error")

print(f"NDCG (top visitas) : {pop_view_sum_dcg / (IDCG * n_recs): .4f}")
print(f"NDCG (top sesiones): {pop_session_sum_dcg / (IDCG * n_recs): .4f}")
print(f"NDCG (top compras) : {pop_bought_sum_dcg / (IDCG * n_recs): .4f}")
print(f"NDCG (top visitas por país) : {country_pop_view_sum_dcg / (IDCG * n_recs): .4f}")
print(f"NDCG (top sesiones por país): {country_pop_session_sum_dcg / (IDCG * n_recs): .4f}")
print(f"NDCG (top compras por país) : {country_pop_bought_sum_dcg / (IDCG * n_recs): .4f}")
def fill_rec(rec, fill, n=10 ):
    assert len(fill) >= n
    fill_index = 0
    while len(rec) < n:
        if fill[fill_index] not in rec:
            rec.append(fill[fill_index] )
        fill_index += 1
    return rec<jupyter_output><empty_output><jupyter_text>Score sobre productos visitadosDel análisis descriptivo se obtuvo que el 30% de los productos comprados fueron vistos en la sesión.Se analizan estrategias para rankear los productos visitados en la recomendación.* La mejor configuración fue contar las visitas pesando con la inversa del logaritmo de la posición, así más reciente mejor.<jupyter_code>n_recs = 0
sum_dcg = 0
model_sum_dcg = 0

with open("./data/train_dataset-test_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1
        # lee registro
        data = json.loads(line)
        item_bought = data["item_bought"]
        items_views = [event["event_info"] for event in data["user_history"] if event["event_type"] == "view"]
        
        # Ranking de items visitados
        items_pv_count = {}
        items_views = items_views[::-1]            
        for pos, item_view in enumerate(items_views, 1):
            items_pv_count[item_view] = items_pv_count.get(item_view,0) + 1 / np.log10(pos + 1)
        
        rec_scores = defaultdict(dict)
        # scores por visitas
        for item_view, pv_count in items_pv_count.items():
            # Asigna un score por item visitado
            rec_scores[item_view] = rec_scores.get(item_view, 0) + pv_count 
        
        
        # equivalent to counter.most_common(10)
        # https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity
        rec = [item for item, _ in heapq.nlargest(10, rec_scores.items(), key=lambda item: item[1])]

        # rellena en caso de no tener recomendaciones
        if len(rec) < 10:
            if len(rec):
                country = catalog[rec[0]]["country"]
                domain = ITEM_TO_DOMAIN[rec[0]]
                fill = most_viewed_by_domain.get(domain, []) +\
                        (most_viewed_items_br if country == "B" else most_viewed_items_mx)
            else: 
                fill = most_viewed_items
            rec = fill_rec(rec, fill)
            
        # evaluación
        model_sum_dcg += dcg(rec, item_bought)
        
        if (n_recs % 1000) == 0:
            print(f"NDCG: {model_sum_dcg / (IDCG * n_recs): .4f} ({n_recs} recomendaciones)")
 
print(f"NDCG (i2i) : {model_sum_dcg / (IDCG * n_recs): .4f}")<jupyter_output>NDCG (i2i) :  0.2639<jupyter_text>Asociaciones entre productos visitados y comprados Mejor configuración encontrada:* Tomar sesiones en el count de matriz* Normalizar count por el total de ventas del ítem vendido. Se aplica un smoothing la cantidad vendidos (suma 100 al denominador)* En la predicción se tiene en cuenta la cantidad de visitas y el orden de visita (más cercano mayor peso)* Se prioriza los visitados en la recomendación<jupyter_code>LOAD = True
if LOAD == False:
    user_idx = 0
    view2bought = {}

    counter_sessions = Counter()
    counter_bought = Counter()

    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            
            item_bought = data["item_bought"]
            country = catalog[item_bought]["country"]
            items_views = set([event["event_info"] for event in data["user_history"]\
                   if event["event_type"] == "view" and\
                   # exclude association in different countries
                   country == catalog[event["event_info"]]["country"]])

            counter_bought[item_bought] += 1

            items_pv_count = Counter(items_views)
            for item_view, count in items_pv_count.items():

                counter_sessions[item_view] += 1
                if item_view in view2bought:
                    view2bought[item_view][item_bought] += 1
                else:
                    counter = Counter()
                    counter[item_bought] += 1
                    view2bought[item_view] = counter
if LOAD == False:
    rows = []
    for item_i, counter in view2bought.items():
        for item_j, ses_count in counter.most_common():
            rows.append((item_i, item_j, counter_sessions[item_i], counter_bought[item_j], ses_count))

    df_bought_count = pd.DataFrame(rows, columns=("item_i", "item_j", "count_i", "count_j", "count_ij"))

    df_bought_count['item_i'] = df_bought_count.item_i.astype(np.uint32)
    df_bought_count['item_j'] = df_bought_count.item_j.astype(np.uint32)

    df_bought_count['count_i'] = df_bought_count.count_i.astype(np.uint16)
    df_bought_count['count_j'] = df_bought_count.count_j.astype(np.uint16)
    df_bought_count['count_ij'] = df_bought_count.count_ij.astype(np.uint16)

    del rows

    df_bought_count.to_pickle("./data/models/df_items_assosiation_bought_count.pkl")
else:
    df_bought_count = pd.read_pickle("./data/models/df_items_assosiation_bought_count.pkl")
df_bought_count.head()
LOAD = True
if LOAD == False:
    import itertools
    
    min_freq = 2
    counter_sessions = Counter()
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            item_bought = data["item_bought"]
            country = catalog[item_bought]["country"]
            items_views = set([event["event_info"] for event in data["user_history"]\
                   if event["event_type"] == "view" and\
                   # exclude association in different countries
                   country == catalog[event["event_info"]]["country"]])

            counter_sessions.update(items_views)
    item_set = set([k for k, v in counter_sessions.items() if  v >= min_freq])
    
    pair_items_counter = Counter()
    counter_sessions = Counter()
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            items_views = set([event["event_info"] for event in data["user_history"] if event["event_type"] == "view" and event["event_info"] in item_set])

            if len(items_views) > 1: 
                pair_items_counter.update(itertools.permutations(items_views, 2))
                counter_sessions.update(items_views)
if LOAD == False:
    rows = []
    for (item_i, item_j), ses_count in pair_items_counter.items():
        rows.append((item_i, item_j, counter_sessions[item_i], counter_sessions[item_j], ses_count))

    df_association_views = pd.DataFrame(rows, columns=("item_i", "item_j", "count_i", "count_j", "count_ij"))

    df_association_views['item_i'] = df_association_views.item_i.astype(np.uint32)
    df_association_views['item_j'] = df_association_views.item_j.astype(np.uint32)

    df_association_views['count_i'] = df_association_views.count_i.astype(np.uint16)
    df_association_views['count_j'] = df_association_views.count_j.astype(np.uint16)
    df_association_views['count_ij'] = df_association_views.count_ij.astype(np.uint16)

    del rows

    df_association_views.to_pickle("./data/models/df_items_assosiation_views_count.pkl")
else:
    df_association_views = pd.read_pickle("./data/models/df_items_assosiation_views_count.pkl")
df_association_views.head()
df_bought_count.head()
item2item = defaultdict(dict)
for row in df_association_views.itertuples():
    item2item[row.item_i][row.item_j] = row.count_ij / (10 + row.count_i)
from collections import defaultdict
view2bought = defaultdict(dict)
for row in df_bought_count.itertuples():
    view2bought[row.item_i][row.item_j] = row.count_ij  / (100 + row.count_j)
del df_association_views
del df_bought_count
n_recs = 0
sum_dcg = 0
model_sum_dcg = 0

W0 = 1
W1 = 1
W3 = 3

with open("./data/train_dataset-test_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1
        # lee registro
        data = json.loads(line)
        item_bought = data["item_bought"]
        items_views = [event["event_info"] for event in data["user_history"] if event["event_type"] == "view"]

        # Ranking de items visitados
        items_pv_count = {}
        items_views = items_views[::-1]            
        for pos, item_view in enumerate(items_views, 1):
            items_pv_count[item_view] = items_pv_count.get(item_view,0) + 1 / np.log10(pos + 1) 
        
            
        # recomendaciones por modelo
        rec_scores = {}
        for item_view, pv_count in items_pv_count.items():
            
            # Asigna un score por item visitado
            rec_scores[item_view] = rec_scores.get(item_view, 0) + np.log10(pv_count + 1) * W0 
            
            # Asigna scores por asociaciones de visitas
            if item_view in item2item:
                item_scores = item2item[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * W1 \
                                   for key in item_scores.keys()})
                
            # Asigna scores por de compras
            if item_view in view2bought:
                item_scores = view2bought[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * pv_count * W3 \
                                   for key in item_scores.keys()})
                
        # exclude items from black List
        rec_scores = {k: v for k, v in rec_scores.items() if k not in BLACK_LIST}
        
        # equivalent to counter.most_common(10)
        # https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity
        rec = [item for item, _ in heapq.nlargest(10, rec_scores.items(), key=lambda item: item[1])]

        # rellena en caso de no tener recomendaciones
        if len(rec) < 10:
            if len(rec):
                domain = ITEM_TO_DOMAIN[rec[0]]
                fill = most_viewed_by_domain.get(domain, []) + most_viewed_items
            else:
                fill = most_viewed_items

            rec = fill_rec(rec, fill)
            
        # evaluación
        model_sum_dcg += dcg(rec, item_bought)
        
        if (n_recs % 1000) == 0:
            print(f"NDCG: {model_sum_dcg / (IDCG * n_recs): .4f} ({n_recs} recomendaciones)")
        
print(f"NDCG (i2i) : {model_sum_dcg / (IDCG * n_recs): .4f}")<jupyter_output>NDCG (i2i) :  0.2851<jupyter_text>Asociaciones con search<jupyter_code>LOAD = True
# Contador de sesiones por busqueda y tokens
# También cuenta el df de token tomando como documentos los textos de búsquedas
if LOAD == False:
    counter_searchs = Counter()
    token_sessions = Counter()
    counter_bought = Counter()
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            searchs = set([event["event_info"] for event in data["user_history"] if event["event_type"] == "search"])
            item_bought = data["item_bought"]
            counter_searchs.update(searchs)
            counter_bought[item_bought] +=1

    token_df = Counter()
    for text in counter_searchs.keys():
        token_df.update(set(text.split()))

    token_sessions = Counter()
    for text, ses_count in counter_searchs.items():
        for token in set(text.split()):
            token_sessions[token] += ses_count
# Contador de asociaciones de tokens de búsquedas con productos comprados
if LOAD == False:
    token2bought = {}
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            searchs = set([event["event_info"] for event in data["user_history"] if event["event_type"] == "search"])
            item_bought = data["item_bought"]
            for text in searchs:
                for token in text.split():
                    if token in token2bought:
                        token2bought[token][item_bought] += 1
                    else:
                        counter = Counter()
                        counter[item_bought] += 1
                        token2bought[token] = counter
# Contador de asocaciones de busquedas con productos comprados
if LOAD == False:
    searchset = set([search for search, c in counter_searchs.items() if c > 1])
    searchbought = {}
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            searchs = set([event["event_info"] for event in data["user_history"] if event["event_type"] == "search" and event["event_info"] in searchset])
            item_bought = data["item_bought"]
            for search in searchs:
                if search in searchbought:
                    searchbought[search][item_bought] += 1
                else:
                    counter = Counter()
                    counter[item_bought] += 1
                    searchbought[search] = counter
# Contador de asociación entre busqueda y visitas

# se toma visitas posteriores a la búsqueda y previas de siguiente búsqueda
if LOAD == False:
    search2views = {}
    with open("./data/train_dataset-train_split.jl", "rt") as fd:
        for line in fd:
            data = json.loads(line)
            
            events = data["user_history"] 
            last_search = None
            search_itemset = {}
            for event in events:
                if event["event_type"] == "search":
                    last_search = event["event_info"]
                elif last_search:
                    if last_search in search_itemset:
                        search_itemset[last_search].add(event["event_info"])
                    else:
                        search_itemset[last_search] = set([event["event_info"]])

            for search, itemset in search_itemset.items():
                if search in search2views:
                    search2views[search].update(itemset)
                else:
                    search2views[search] = Counter(itemset)
                    

if LOAD == False:
    # contador de productos vistos luego de búsquedas
    counter_views_insearch = Counter()
    for counter in search2views.values():
        counter_views_insearch.update(dict(counter.items()))
        
    # Contador de asociación entre tokens y visitas
    token2views = {}
    for search, item_counter in search2views.items():
        for token in search.split():
            if token in token2views:
                token2views[token].update(dict(item_counter.items()))
            else:
                token2views[token] = Counter(dict(item_counter.items()))
# Dataset "Texto Búsqueda" -> "Productos comprados"
if LOAD == False:
    rows = []
    for search, items_count  in searchbought.items():
        for item, ses_count, in items_count.items():
            rows.append((search, item, counter_searchs[search], counter_bought[item], ses_count))

    df_search_bought = pd.DataFrame(rows, columns=("search", "item", "count_s", "count_i", "count_si"))

    df_search_bought['item'] = df_search_bought.item.astype(np.uint32)

    df_search_bought['count_i'] = df_search_bought.count_i.astype(np.uint16)
    df_search_bought['count_s'] = df_search_bought.count_s.astype(np.uint16)
    df_search_bought['count_si'] = df_search_bought.count_si.astype(np.uint16)

    del rows

    df_search_bought.to_pickle("./data/models/df_assosiation_search_text_bought.pkl")
else:
    df_search_bought = pd.read_pickle("./data/models/df_assosiation_search_text_bought.pkl")
df_search_bought.head()
# Dataset "Tokens búsquedas" -> "Productos comprados"
if LOAD == False:
    rows = []
    for token, items_count  in token2bought.items():
        for item, ses_count, in items_count.items():
            rows.append((token, item, token_sessions[token], counter_bought[item], ses_count))

    df_search_token_bought = pd.DataFrame(rows, columns=("token", "item", "count_t", "count_i", "count_ti"))

    df_search_token_bought['item'] = df_search_token_bought.item.astype(np.uint32)

    df_search_token_bought['count_i'] = df_search_token_bought.count_i.astype(np.uint16)
    df_search_token_bought['count_t'] = df_search_token_bought.count_t.astype(np.uint16)
    df_search_token_bought['count_ti'] = df_search_token_bought.count_ti.astype(np.uint16)

    del rows

    df_search_token_bought.to_pickle("./data/models/df_assosiation_search_token_bought.pkl")
else:
    df_search_token_bought = pd.read_pickle("./data/models/df_assosiation_search_token_bought.pkl")
df_search_token_bought.head()
# Dataset "Texto Búsqueda" -> "Productos visitados"
if LOAD == False:
    rows = []
    for search, items_count  in search2views.items():
        for item, ses_count, in items_count.items():
            rows.append((search, item, counter_searchs[search], counter_views_insearch[item], ses_count))

    df_search_views = pd.DataFrame(rows, columns=("search", "item", "count_s", "count_i", "count_si"))

    df_search_views['item'] = df_search_views.item.astype(np.uint32)

    df_search_views['count_i'] = df_search_views.count_i.astype(np.uint16)
    df_search_views['count_s'] = df_search_views.count_s.astype(np.uint16)
    df_search_views['count_si'] = df_search_views.count_si.astype(np.uint16)

    del rows

    df_search_views.to_pickle("./data/models/df_assosiation_search_text_views.pkl")
else:
    df_search_views = pd.read_pickle("./mei_challenger/data/models/df_assosiation_search_text_views.pkl")
df_search_views.head()
# Dataset "Tokens búsquedas" -> "Productos Visitados"
if LOAD == False:
    rows = []
    for token, items_count  in token2views.items():
        for item, ses_count, in items_count.items():
            rows.append((token, item, token_sessions[token], counter_views_insearch[item], ses_count))

    df_search_token_views = pd.DataFrame(rows, columns=("token", "item", "count_t", "count_i", "count_ti"))

    df_search_token_views['item'] = df_search_token_views.item.astype(np.uint32)

    df_search_token_views['count_i'] = df_search_token_views.count_i.astype(np.uint16)
    df_search_token_views['count_t'] = df_search_token_views.count_t.astype(np.uint16)
    df_search_token_views['count_ti'] = df_search_token_views.count_ti.astype(np.uint16)

    del rows

    df_search_token_views.to_pickle("./data/models/df_assosiation_search_token_views.pkl")
else:
    df_search_token_views = pd.read_pickle("./data/models/df_assosiation_search_token_views.pkl")
df_search_token_views.head()
from collections import defaultdict
token2bought = defaultdict(dict)
for row in df_search_token_bought.itertuples():
    token2bought[row.token][row.item] = (row.count_ti )
from collections import defaultdict
token2view = defaultdict(dict)
for row in df_search_token_views.itertuples():
    token2view[row.token][row.item] = (row.count_ti )
from collections import defaultdict
text2bought = defaultdict(dict)
for row in df_search_bought.itertuples():
    text2bought[row.search][row.item] = (row.count_si ) 
from collections import defaultdict
text2view = defaultdict(dict)
for row in df_search_views.itertuples():
    text2view[row.search][row.item] = (row.count_si ) 
del df_search_views
del df_search_token_bought
del df_search_token_views
del df_search_bought
# se queda con el top para optimizar consultas
n = 128
for text in text2view.keys():
    text2view[text] = {item: score for item, score in heapq.nlargest(n, text2view[text].items(), key=lambda item: item[1])}
    
for token in token2view.keys():
    token2view[token] = {item: score for item, score in heapq.nlargest(n, token2view[token].items(), key=lambda item: item[1])}
    
n = 256
for text in text2bought.keys():
    text2bought[text] = {item: score for item, score in heapq.nlargest(n, text2bought[text].items(), key=lambda item: item[1])}
    
for token in token2bought.keys():
    token2bought[token] = {item: score for item, score in heapq.nlargest(n, token2bought[token].items(), key=lambda item: item[1])}
counter_searchs = Counter()
with open("./data/train_dataset-train_split.jl", "rt") as fd:
    for line in fd:
        data = json.loads(line)
        searchs = set([event["event_info"] for event in data["user_history"] if event["event_type"] == "search"])
        counter_searchs.update(searchs)

token_df = Counter()
for text in counter_searchs.keys():
    token_df.update(set(text.split()))
del counter_searchs
n_recs = 0
sum_dcg = 0
model_sum_dcg = 0

Wb = 1
Wv = 1
with open("./data/train_dataset-test_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1
        # lee registro
        data = json.loads(line)
        item_bought = data["item_bought"]
        searchs = [event["event_info"] for event in data["user_history"] if event["event_type"] == "search"]
        

        # Ranking de palabras de búsqueda
        tf_counter = {}
        text_counter = {}
        searchs = searchs[::-1]            
        for pos, text in enumerate(searchs, 1):
            text_counter[text] = tf_counter.get(text, 0) + 1 / np.log10(pos + 1)
            for token in text.split():
                tf_counter[token] = tf_counter.get(token, 0) + 1 / np.log10(pos + 1)
            
        # recomendaciones por modelo
        rec_scores = {}
        
        for text, tf in text_counter.items():
            if text in text2bought:
                item_scores = text2bought[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * Wb\
                                  for key in item_scores.keys()})
        if text in  text2view:
            for text, tf in text_counter.items():
                item_scores = text2view[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * Wv\
                                  for key in item_scores.keys()})
            
        for token, tf in tf_counter.items():
            if len(token) < 2:
                continue
            if token in token2bought:
                item_scores = token2bought[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * Wb\
                                  for key in item_scores.keys()})
            if token in token2view:
                item_scores = token2view[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * Wv\
                                  for key in item_scores.keys()})
        if rec_scores:
            country_rec = catalog[max(rec_scores, key=rec_scores.get)]["country"]
            rec_scores = {k: v for k,v in rec_scores.items() if catalog[k]["country"] == country_rec}
         
        
        # equivalent to counter.most_common(10)
        # https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity
        rec = [item for item, _ in heapq.nlargest(10, rec_scores.items(), key=lambda item: item[1])]

        # rellena en caso de no tener recomendaciones
        if len(rec) < 10:
            if len(rec):
                domain = ITEM_TO_DOMAIN[rec[0]]
                fill = most_viewed_by_domain.get(domain, []) + most_viewed_items
            else:
                fill = most_viewed_items

            rec = fill_rec(rec, fill)
            
        # evaluación
        model_sum_dcg += dcg(rec, item_bought)
        
        if (n_recs % 1000) == 0:
            print(f"NDCG: {model_sum_dcg / (IDCG * n_recs): .4f} ({n_recs} recomendaciones)")
print(f"NDCG (i2i) : {model_sum_dcg / (IDCG * n_recs): .4f}")<jupyter_output>NDCG (i2i) :  0.1198<jupyter_text>Modelo Final: Se toman todas las asociacionesSe ensamblan todas las asociaciones usando pesos en los scores.<jupyter_code>n_recs = 0
sum_dcg = 0
model_sum_dcg = 0

W0 = 100 # ítems visitados en sesión
W1 = 100 # asociación con visitas
W3 = 300 # asociación con compras
W4 = 0.001 # asociación con búsquedas

fills_count = 0
with open("./data/train_dataset-test_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1
        # lee registro
        data = json.loads(line)
        item_bought = data["item_bought"]
        items_views = [event["event_info"] for event in data["user_history"] if event["event_type"] == "view"]
        searchs = [event["event_info"] for event in data["user_history"] if event["event_type"] == "search"]

        # Ranking de items visitados
        items_pv_count = {}
        items_views = items_views[::-1]            
        for pos, item_view in enumerate(items_views, 1):
            items_pv_count[item_view] = items_pv_count.get(item_view,0) + 1 / np.log10(pos + 1) 
        
            
        # recomendaciones por modelo
        rec_scores = {}
        for item_view, pv_count in items_pv_count.items():
            
            # Asigna un score por item visitado
            rec_scores[item_view] = rec_scores.get(item_view, 0) + np.log10(pv_count + 1) * W0 
            
            # Asigna scores por asociaciones de visitas
            if item_view in item2item:
                item_scores = item2item[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * W1 \
                                   for key in item_scores.keys()})
                
            # Asigna scores por de compras
            if item_view in view2bought:
                item_scores = view2bought[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * pv_count * W3 \
                                   for key in item_scores.keys()})
                
        
        # Ranking de palabras de búsqueda
        tf_counter = {}
        text_counter = {}
        searchs = searchs[::-1]            
        for pos, text in enumerate(searchs, 1):
            text_counter[text] = tf_counter.get(text, 0) + 1 / np.log10(pos + 1)
            for token in text.split():
                tf_counter[token] = tf_counter.get(token, 0) + 1 / np.log10(pos + 1)

        for text, tf in text_counter.items():
            if text in text2bought:
                item_scores = text2bought[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * W4\
                                  for key in item_scores.keys()})
        if text in  text2view:
            for text, tf in text_counter.items():
                item_scores = text2view[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * W4\
                                  for key in item_scores.keys()})

        for token, tf in tf_counter.items():
            if len(token) < 2:
                continue
            if token in token2bought:
                item_scores = token2bought[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * W4\
                                  for key in item_scores.keys()})
            if token in token2view:
                item_scores = token2view[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * W4\
                                  for key in item_scores.keys()})
        if rec_scores:
            country_rec = catalog[max(rec_scores, key=rec_scores.get)]["country"]
            rec_scores = {k: v for k,v in rec_scores.items() if catalog[k]["country"] == country_rec}
            # exclude items from black List
            rec_scores = {k: v for k, v in rec_scores.items() if k not in BLACK_LIST}
        
        # equivalent to counter.most_common(10)
        # https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity
        rec = [item for item, _ in heapq.nlargest(10, rec_scores.items(), key=lambda item: item[1])]

        
         # rellena en caso de no tener recomendaciones suficientes
        if len(rec) < 10:
            if len(rec):
                domain = ITEM_TO_DOMAIN[rec[0]]
                fill = most_viewed_by_domain.get(domain, []) +\
                        (most_viewed_items_br if country_rec == "B" else most_viewed_items_mx)
            else: 
                fill = most_viewed_items
            rec = fill_rec(rec, fill)
            
        # evaluación
        model_sum_dcg += dcg(rec, item_bought)
        
        if (n_recs % 1000) == 0:
            print(f"NDCG: {model_sum_dcg / (IDCG * n_recs): .4f} ({n_recs} recomendaciones)")
print(f"NDCG (i2i) : {model_sum_dcg / (IDCG * n_recs): .4f}")<jupyter_output>NDCG (i2i) :  0.2889<jupyter_text>Evalualción de validación:<jupyter_code>n_recs = 0
sum_dcg = 0
model_sum_dcg = 0

W0 = 100
W1 = 100
W3 = 300
W4 = 0.001

with open("./data/train_dataset-val_split.jl", "rt") as fd:
    for line in fd:
        n_recs += 1
        # lee registro
        try:
            data = json.loads(line)
        except:
            continue
        item_bought = data["item_bought"]
        items_views = [event["event_info"] for event in data["user_history"] if event["event_type"] == "view"]
        searchs = [event["event_info"] for event in data["user_history"] if event["event_type"] == "search"]

        # Ranking de items visitados
        items_pv_count = {}
        items_views = items_views[::-1]            
        for pos, item_view in enumerate(items_views, 1):
            items_pv_count[item_view] = items_pv_count.get(item_view,0) + 1 / np.log10(pos + 1) 
        
            
        # recomendaciones por modelo
        rec_scores = {}
        for item_view, pv_count in items_pv_count.items():
            
            # Asigna un score por item visitado
            rec_scores[item_view] = rec_scores.get(item_view, 0) + np.log10(pv_count + 1) * W0 
            
            # Asigna scores por asociaciones de visitas
            if item_view in item2item:
                item_scores = item2item[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * W1 \
                                   for key in item_scores.keys()})
                
            # Asigna scores por de compras
            if item_view in view2bought:
                item_scores = view2bought[item_view]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * pv_count * W3 \
                                   for key in item_scores.keys()})
                
        
        # Ranking de palabras de búsqueda
        tf_counter = {}
        text_counter = {}
        searchs = searchs[::-1]            
        for pos, text in enumerate(searchs, 1):
            text_counter[text] = tf_counter.get(text, 0) + 1 / np.log10(pos + 1)
            for token in text.split():
                tf_counter[token] = tf_counter.get(token, 0) + 1 / np.log10(pos + 1)
        
        for text, tf in text_counter.items():
            if text in text2bought:
                item_scores = text2bought[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * W4\
                                  for key in item_scores.keys()})
        if text in  text2view:
            for text, tf in text_counter.items():
                item_scores = text2view[text] 
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tf * W4\
                                  for key in item_scores.keys()})
            
        for token, tf in tf_counter.items():
            if len(token) < 2:
                continue
            if token in token2bought:
                item_scores = token2bought[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * W4\
                                  for key in item_scores.keys()})
            if token in token2view:
                item_scores = token2view[token] 
                tfidf =  tf  / token_df[token]
                rec_scores.update({key: rec_scores.get(key, 0) + item_scores.get(key, 0) * tfidf * W4\
                                  for key in item_scores.keys()})
                
        # equivalent to counter.most_common(10)
        # https://stackoverflow.com/questions/29240807/python-collections-counter-most-common-complexity
        rec = [item for item, _ in heapq.nlargest(10, rec_scores.items(), key=lambda item: item[1])]

        # rellena en caso de no tener recomendaciones suficientes
        if len(rec) < 10:
            if len(rec):
                domain = ITEM_TO_DOMAIN[rec[0]]
                fill = most_viewed_by_domain.get(domain, []) + most_viewed_items
            else:
                fill = most_viewed_items

            rec = fill_rec(rec, fill)
            
        # evaluación
        model_sum_dcg += dcg(rec, item_bought)
        
        if (n_recs % 1000) == 0:
            print(f"NDCG: {model_sum_dcg / (IDCG * n_recs): .4f} ({n_recs} recomendaciones)")
print(f"NDCG (i2i) : {model_sum_dcg / (IDCG * n_recs): .4f}")<jupyter_output>NDCG (i2i) :  0.2889
{'fraction_non_alphanumeric': 0.10795297892072085, 'fraction_numerical': 0.014938466551369777, 'mean_word_length': 2.561049625133881, 'pattern_counts': {'":': 3, '<': 41, '<?xml version=': 0, '>': 40, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '432978', 'license': 'MIT', 'n_tokens_mistral': 13839, 'n_tokens_neox': 13106, 'n_words': 2931, 'path': 'notebooks/01-Item2itemModel.ipynb', 'repo_name': 'leolnn/MeliDataChallenge2020', 'size': '39897'}