<jupyter_start><jupyter_text>PrefixLSH tests with domains data<jupyter_code>import sys
sys.path.append('..')

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import floc

from utils.data import read_weeks_machines_domains
import prefixLSH<jupyter_output><empty_output><jupyter_text>read in some pre-processed sessions data- starting domains list: take a list of domains with at least 20 domains (n_domains)- sampling domains list: take another 20 different domains to sample from.create 20 lists of domains, where each list i in 1,...20 duplicates list (i-1) and then modifies the list by replacing item i with a new domain from the sampling domains listdomains list 1 is the starting domains list<jupyter_code># read in the pre-processed sessions data 
# this maps week,machine_id -> domains set
weeks_machines_domains_fpath = '../output/weeks_machines_domains.csv'
weeks_machines_domains_df = read_weeks_machines_domains(weeks_machines_domains_fpath, nrows=100)
weeks_machines_domains_df.drop(['machine_id', 'domains'], axis=1).head()
n_domains = 20
starting_domains_list = weeks_machines_domains_df[
    (weeks_machines_domains_df.n_domains == n_domains)
].domains.values[2]
starting_domains_list = list(starting_domains_list)
starting_domains_list
# use most frequent domains from sampled rows as other domain samples
other_domains = weeks_machines_domains_df.domains.values
other_domains = [d for domains in other_domains for d in domains if d not in starting_domains_list]
other_domains = pd.Series(other_domains).value_counts().head(20).index
other_domains<jupyter_output><empty_output><jupyter_text>make a dataframe with columns```m, domains, simhash, ot_cohort, cohort_k2, cohort_k4, cohort_k8,```where - each m differs from previous m by one domain- ot_cohort generated via OT floc- cohort_k* generated using our prefixLSH with given khere the domains gradually transition from some unique user's domains to the most popularWe then expect similar cohort IDs to be clustered near each other in the list.<jupyter_code>m = list(range(20))
m
domains = [starting_domains_list]
for i in range(19):
    i1_domains = domains[i][:] # copy domains list from i
    i1_domains[i] = other_domains[i] # alter domains by one entry
    domains += [i1_domains]
test_df = pd.DataFrame({'m':m, 'domains':domains})
test_df['simhash'] = test_df.domains.apply(floc.hashes.sim_hash_string)
test_df['ot_cohort'] = test_df.domains.apply(floc.simulate)
for k in [2, 4, 8]:
    cohort_k = 'cohort_k%s'%k
    cohorts_dict = prefixLSH.get_cohorts_dict(test_df.simhash.astype(int), min_k=k)
    test_df[cohort_k] = test_df.simhash.map(cohorts_dict)
test_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08475836431226766, 'fraction_numerical': 0.012267657992565056, 'mean_word_length': 6.352459016393443, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '147326', 'license': 'MIT', 'n_tokens_mistral': 963, 'n_tokens_neox': 849, 'n_words': 259, 'path': 'notebooks/prefixLSH_tests_domains.ipynb', 'repo_name': 'aberke/floc-analysis', 'size': '2690'}