<jupyter_start><jupyter_text>BERT ClusteringUnsupervised clustering of forum posts using the text body. A simple approach to try and find similar posts without strict classification.Right now the model doesn't do anything, which is very sad. References1. [Topic Modeling with BERT](https://towardsdatascience.com/topic-modeling-with-bert-779f7db187e6)2. [Multi-label Text Classification using BERT - The Mighty Transformer](https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d)3. [BERT Fine-Tuning Tutorial with PyTorch](https://mccormickml.com/2019/07/22/BERT-fine-tuning)4. [Usering BERT For Classifying Documents with Long Texts](https://medium.com/@armandj.olivares/using-bert-for-classifying-documents-with-long-texts-5c3e7b04573d)<jupyter_code>!pip install transformers -q

import re
import math
import torch
import numpy as np
import pandas as pd
from tqdm import tqdm
import seaborn as sns
import matplotlib.pyplot as plt
from transformers import AutoTokenizer, AutoModel
from tokenizers.normalizers import BertNormalizer
np.random.RandomState(123)
tqdm.pandas()

if torch.cuda.is_available():
    device = torch.device('cuda')
    print(f'using {torch.cuda.get_device_name(0)}')
else:
    print('no GPU avaiable')<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>df_ori = pd.read_json('https://github.com/charlotte-zhuang/forum-recommender/blob/master/data/sitepoint.json?raw=true')
df_ori.head()<jupyter_output><empty_output><jupyter_text>Preprocessing<jupyter_code>def pre_tokenize_post(text_l: list) -> str:
    '''Converts a list of text into a single string that
        can be tokenized.
    '''
    
    text = ' '.join(text_l)
    
    # remove links
    text = re.sub(r'https?:\/\/[^\s]*', r'', text)
    
    # remove closed single quotes
    text = re.sub(r"‘([^‘’]*)’", r'\1', text)

    # remove unwanted characters
    text = re.sub(r"[^\w\s’]+", r' ', text)
    
    # turn whitespace into a space
    text = re.sub(r'\s+', r' ', text)

    # lowercase
    text = text.lower()
    
    return text
df = df_ori.drop(labels=['title', 'tags'], axis=1)
df['post'] = df['post'].apply(pre_tokenize_post)

for i in range(5):
    print(df.iat[i, 1], end='\n\n')
# optionally drop data for convenience
# df = df[:1000]<jupyter_output><empty_output><jupyter_text>Tokenize<jupyter_code>tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
tokenizer.normalizer = BertNormalizer()<jupyter_output><empty_output><jupyter_text>Split samples up to avoid truncation, then tokenize each sample.<jupyter_code>def tokenize_text(text: str) -> list:
    '''Tokenizes text using chunks of up to 300 words. This does the exact
        same thing as encode_text(), but step by step.

    Returns:
        numpy.ndarray: 2D array of IDs: axis0=chunks, axis1=tokens.
        numpy.ndarray: 2D array of attention masks: axis0=chunks, axis1=mask.
    '''

    # break up sentences into chunks of 300 words with 50 overlapping
    res = []
    text = text.split()
    n = int(math.ceil((len(text) - 100) / 200))

    res.append(' '.join(text[:300]))

    for i in range(1, n):
        res.append(' '.join(text[i * 250 : i * 250 + 300]))

    # tokenize
    res = list(map(lambda x: ['[CLS]'] + tokenizer.tokenize(x), res))

    # convert to ids
    res = list(map(tokenizer.convert_tokens_to_ids, res))

    # pad sentences to be 512 characters
    res = np.array(list(map(lambda x: x + [0] * (512 - len(x)), res)))

    # attention mask
    attention_mask = np.where(res != 0, 1, 0)

    return res , attention_mask
def encode_text(text):
    '''Tokenizes text using chunks of up to 300 words. This does the exact
        same thing as tokenize_text(), using the encode() method instead.

    Returns:
        numpy.ndarray: 2D array of IDs: axis0=chunks, axis1=tokens.
        numpy.ndarray: 2D array of attention masks: axis0=chunks, axis1=mask.
    '''
    

    # break up sentences into chunks of 300 words with 50 overlapping
    res = []
    text = text.split()
    n = int(math.ceil((len(text) - 100) / 200))

    res.append(' '.join(text[:300]))

    for i in range(1, n):
        res.append(' '.join(text[i * 250 : i * 250 + 300]))
    
    res = np.array(res)

    # tokenize
    res = list(map(
        lambda x: tokenizer.encode(x, add_special_tokens=True),
        res))

    # pad sentences to be 512 characters
    res = np.array(list(map(lambda x: x + [0] * (512 - len(x)), res)))
    
    # mask
    attention_mask = np.where(res != 0, 1, 0)
    
    return res , attention_mask
tokens = list(map(encode_text, df['post']))
df['tokens'] = [x[0] for x in tokens]
df['mask'] = [x[1] for x in tokens]
df.head()<jupyter_output><empty_output><jupyter_text>ClassificationI'm actually not using the BERT classifier anymore.<jupyter_code>bert_model = AutoModel.from_pretrained('distilbert-base-uncased')<jupyter_output><empty_output><jupyter_text>Get class embeddings. Each chunk from a sample gets a vector of 768 embeddings (from the 768 hidden layers).<jupyter_code>def embed_tokens(sample):

    tokens = torch.tensor(sample['tokens'])
    attention_mask = torch.tensor(sample['mask'])

    with torch.no_grad():
        last_hidden_states = bert_model(tokens, attention_mask=attention_mask)

    embeddings = last_hidden_states[0][:,0,:].numpy()

    return embeddings<jupyter_output><empty_output><jupyter_text>This step takes a while.<jupyter_code>df['embed'] = df.progress_apply(embed_tokens, axis=1)
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13156469951272334, 'fraction_numerical': 0.02598808879263671, 'mean_word_length': 3.974865350089767, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '657344', 'license': 'MIT', 'n_tokens_mistral': 2008, 'n_tokens_neox': 1862, 'n_words': 471, 'path': 'notebooks/bert_clustering2.ipynb', 'repo_name': 'charlotte-zhuang/forum-recommender', 'size': '5541'}