<jupyter_start><jupyter_text>Read data about the sites<jupyter_code>con = sqlite3.connect('2018-12-03_segmentation_pilot/2018-12-03_segmentation_pilot.sqlite')
sites = pd.read_sql_query('''SELECT * from site_visits''', con)
sites.shape
list(sites.columns.values)<jupyter_output><empty_output><jupyter_text>Read data about the segments<jupyter_code>segments = pd.read_sql_query('''SELECT * from segments''', con)
segments.shape
list(segments.columns.values)<jupyter_output><empty_output><jupyter_text>For this analysis, only consider those segments with text.<jupyter_code>segments = segments.loc[segments['inner_text'] != '']<jupyter_output><empty_output><jupyter_text>Let's also ignore all segments with body tags<jupyter_code>segments = segments.loc[segments['node_name'] != 'BODY']
segments.shape<jupyter_output><empty_output><jupyter_text>Let's perform some pre-processing. First, let's swap all numbers with DPNUM placeholder.<jupyter_code>segments['inner_text_processed'] = segments['inner_text'].str.replace(r'\d+', 'DPNUM')
segments['longest_text_processed'] = segments['longest_text'].str.replace(r'\d+', 'DPNUM')<jupyter_output><empty_output><jupyter_text>Next let's consider the individual nodes, particularly those that were updated<jupyter_code>def handle_node_update(gdata):
    return gdata.drop_duplicates(subset=['inner_text_processed', 'longest_text_processed'], keep='last')

segments = segments.groupby(['visit_id']).apply(handle_node_update)
list(segments.columns.values)
segments.shape<jupyter_output><empty_output><jupyter_text>Join the two dataframes<jupyter_code>dataset = segments.set_index('visit_id').join(sites.set_index('visit_id'), lsuffix='_1', rsuffix='_2')
dataset.shape
list(dataset.columns.values)<jupyter_output><empty_output><jupyter_text>Let's get the visit_id back<jupyter_code>dataset = dataset.reset_index()
list(dataset.columns.values)<jupyter_output><empty_output><jupyter_text>Let's tokenize inner_text first:<jupyter_code>stemmer = PorterStemmer()

def tokenize(line):
    if (line is None):
        line = ''
    printable = set(string.printable)
    line = ''.join(filter(lambda x: x in printable, line)) 
    
    tokens = nltk.word_tokenize(line)
    
    tokens = [f for f in tokens if f != '']
    tokens = [stemmer.stem(f) for f in tokens]

    return tokens
countVec = CountVectorizer(tokenizer=tokenize, binary=True).fit(dataset['inner_text_processed'])<jupyter_output><empty_output><jupyter_text>What is the length of the vocabulary?<jupyter_code>len(countVec.vocabulary_)<jupyter_output><empty_output><jupyter_text>Let's create the bag of words representation.<jupyter_code>lineVec = countVec.transform(dataset['inner_text_processed'])
lineVec.shape<jupyter_output><empty_output><jupyter_text>First, scale the remaining columns:<jupyter_code>scaler = MinMaxScaler()
cols = dataset[['num_buttons', 'num_imgs', 'num_anchors', 'top', 'left']]
cols = scaler.fit_transform(cols)
cols<jupyter_output><empty_output><jupyter_text>Add these columns in<jupyter_code>lineVec = hstack((lineVec, cols))<jupyter_output><empty_output><jupyter_text>Shape of the vector?<jupyter_code>lineVec.shape<jupyter_output><empty_output><jupyter_text>Let's compute the euclidean distance<jupyter_code>dist = euclidean_distances(lineVec)<jupyter_output><empty_output><jupyter_text>Next, let's convert this to vector format. This is necessary as the linkage method below requires it in this format.<jupyter_code>distVec = squareform(dist, checks = False)
res = fastcluster.linkage(distVec, method = 'ward', preserve_input = False)
dataset['cluster'] = fcluster(res, 5, criterion='distance')
dataset.to_csv('clusters.csv', encoding='utf-8', columns=['site_url', 'cluster', 'inner_text', 'top', 'left', 'width', 'height', 'time_stamp'], index=False)
dataset.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13275726630007856, 'fraction_numerical': 0.0054988216810683424, 'mean_word_length': 9.324324324324325, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '581736', 'license': 'Apache-2.0', 'n_tokens_mistral': 1353, 'n_tokens_neox': 1298, 'n_words': 218, 'path': 'analysis/clustering-initial-attempts/pilot-clustering/Analysis-2.ipynb', 'repo_name': 'BogDAAAMN/dark-patterns', 'size': '3819'}