<jupyter_start><jupyter_text>**Dataset preparation**1.   Download the Dunnhumby The Complete Journey from official repository2.   Unpack the .zip file to the previously defined location<jupyter_code>!wget https://www.dunnhumby.com/wp-content/uploads/sourcefiles/dunnhumby_The-Complete-Journey.zip -P $PATH
!unzip -d $PATH -o -q $PATH/dunnhumby_The-Complete-Journey.zip<jupyter_output><empty_output><jupyter_text>**Installation of the Synerise Cleora platform**1.   Download the binary file of the latest released version of Cleory from the official Synerise repository. We are using version 1.1.1 in this notebook. You can find earlier and newer versions [here](https://github.com/Synerise/cleora/releases/).2.  Set the execute permissions of the previously downloaded Cleora binary file.<jupyter_code>!wget https://github.com/Synerise/cleora/releases/download/v1.1.1/cleora-v1.1.1-x86_64-unknown-linux-gnu -P $PATH 
!chmod +x $PATH/cleora-v1.1.1-x86_64-unknown-linux-gnu
df = pd.read_csv(DUNNHUMBY_JOURNEY_PATH+"dunnhumby - The Complete Journey CSV/transaction_data.csv", delimiter = ",")
products = pd.read_csv(DUNNHUMBY_JOURNEY_PATH+"dunnhumby - The Complete Journey CSV/product.csv", delimiter = ",", index_col=0)
df_grouped = df.groupby('BASKET_ID')<jupyter_output><empty_output><jupyter_text>**Clique expansion**1.   The **dunnhumby_cleora_input.txt** file has the structure of adjacency list.<jupyter_code>cleora_f_dict = open(PATH+'dunnhumby_cleora_input.txt', 'w') 
for name, group in df_grouped:
    product_ids = [str(p) for p in group['PRODUCT_ID'].tolist()]
    cleora_f_dict.write('{}\n'.format(' '.join(product_ids)))
cleora_f_dict.close()<jupyter_output><empty_output><jupyter_text>**Setting configuration before Train Cleora.**1.   cleora_n_iter 2.   cleora_dim3.   columnsand other if you want. More about configuration params you can find [here.](https://cleora.readthedocs.io/en/latest/running.htmlrun-options)<jupyter_code>config = {
    #embedding computation
    'cleora_n_iter': 4,
    'cleora_dim': 1024,
    'columns' : 'complex::reflexive::CliqueNode'
}<jupyter_output><empty_output><jupyter_text>The parameter -c "**complex::reflexive::CliqueNode**" means that edges will be created for all cominations of nodes from each line. This translates to clique expansion scheme.*    You can read more about **Clique Node** and other expansion methods you can read [here.](https://cleora.readthedocs.io/en/latest/algorithms.htmlclique-expansion)<jupyter_code>import subprocess

def train_cleora(dim, n_iter, columns, input_filename):
    command = [PATH+'cleora-v1.1.1-x86_64-unknown-linux-gnu',
                '--columns', columns,
                '--dimension', str(dim), 
                '-n', str(n_iter), 
                '--input', input_filename,
                '-o',OUTPUT_PATH,
               '-r', f'iterations_{n_iter}_'
               ]
    subprocess.run(command, check=True)
%%time
train_cleora(config['cleora_dim'], config['cleora_n_iter'], config['columns'], PATH+'dunnhumby_cleora_input.txt')<jupyter_output>CPU times: user 389 ms, sys: 52.7 ms, total: 442 ms
Wall time: 1min 44s<jupyter_text>**Prediction**<jupyter_code>prods_for_ar = []

for name, group in df_grouped:
    product_ids = [str(p) for p in group['PRODUCT_ID'].tolist()]
    prods_for_ar.append(product_ids)
vects_iter = np.loadtxt(OUTPUT_PATH+"__CliqueNode__CliqueNode.out", delimiter = " ", skiprows=1, usecols=[0]+list(range(2,1026)))
ids_iter = vects_iter[:, 0]
vects_iter = vects_iter[:, 1:]
dist = sklearn.metrics.pairwise.cosine_similarity(vects_iter[0].reshape(1, -1), vects_iter, dense_output=True)
ranking = (-dist).argsort()
for idx, v in enumerate(vects_iter):
    print('-------------base product:')
    curr_prod_id = ids_iter[idx]
    print(curr_prod_id)
    curr_product = products.loc[curr_prod_id]

    print(curr_product['COMMODITY_DESC'], curr_product['SUB_COMMODITY_DESC'], curr_product['CURR_SIZE_OF_PRODUCT'])
    dist = sklearn.metrics.pairwise.cosine_similarity(v.reshape(1, -1), vects_iter, dense_output=True)
    ranking = (-dist).argsort()[0]
    for r in ranking[:10]:
        suggested_prod_id = int(ids_iter[r])
        suggested_prod = products.loc[suggested_prod_id]
        print('suggested: ', suggested_prod_id, suggested_prod['COMMODITY_DESC'], suggested_prod['SUB_COMMODITY_DESC'], suggested_prod['CURR_SIZE_OF_PRODUCT'])<jupyter_output>[1;30;43mStreaming output truncated to the last 5000 lines.[0m
suggested:  16769752 PROCESSED DIPS 16 OZ
suggested:  915257 BEEF LOIN - STK/CHP/SLC  
suggested:  863447 BEEF CHOICE BEEF  
suggested:  12263300 BLEACH ALL FABRIC DRY BLEACH 1.5 LB
suggested:  955765 PROCESSED PROCESSED OTHER 4.5 OZ
-------------base product:
5564922.0
CRACKERS/MISC BKD FD CHEESE CRACKERS (CHEEZ-ITS/GOL 7.2 OZ
suggested:  5564922 CRACKERS/MISC BKD FD CHEESE CRACKERS (CHEEZ-ITS/GOL 7.2 OZ
suggested:  832312 IN-STORE PHOTOFINISHING OVERNIGHT PROCESSING  
suggested:  10285167 AUDIO/VIDEO PRODUCTS DVD S  
suggested:  7167711 ICE CREAM/MILK/SHERBTS PREMIUM 1/2 GAL
suggested:  1096449 LIQUOR LIQUEURS/SPECIALTIES (42 UNDER 750 ML
suggested:  15778515 PET CARE SUPPLIES DOG & CAT CHEMICALS  
suggested:  9297058 NO COMMODITY DESCRIPTION NO SUBCOMMODITY DESCRIPTION  
suggested:  7168701 STATIONERY & SCHOOL SUPPLIES CHILDREN S ACTIVITY 6 CT
suggested:  9445751 DIETARY AID PRODUCTS LIQUID NUTRITION FOR ADULTS 6PK/8OZ[...]
{'fraction_non_alphanumeric': 0.1212852897473997, 'fraction_numerical': 0.038261515601783064, 'mean_word_length': 4.8724100327153765, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '260157', 'license': 'MIT', 'n_tokens_mistral': 2113, 'n_tokens_neox': 1926, 'n_words': 421, 'path': 'example_dunnhumby_journey.ipynb', 'repo_name': 'mkrzywda/cleora', 'size': '5384'}