<jupyter_start><jupyter_text>News ClustererThis notebook will be running an incremental clustering algorithm that groups news sources into events. Based on [this research article](https://www.researchgate.net/publication/258028563_Incremental_Clustering_of_News_Reports).<jupyter_code>import numpy as np
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import json
import string
import math
nltk.download('stopwords')
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\Benedict\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\Benedict\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>Step 1: Gathering dataWe will be retrieving:1. articles that are not clustered into events1. events that were previously clustered (if they exist)1. the global word index list (if it exists)<jupyter_code>with open('articles.json', 'r', encoding='utf-8') as f:
    articles = json.loads(f.read())<jupyter_output><empty_output><jupyter_text>Step 2: PreprocessingThis will stem the words down to their base forms and remove stopwords.<jupyter_code>def preprocess_string(text: str) -> str:
    text = text.translate(str.maketrans('', '', string.punctuation))
    stop_words = set(stopwords.words('english'))
    tokens = word_tokenize(text)
    filtered_tokens = [token for token in tokens if token.lower() not in stop_words]
    return filtered_tokens<jupyter_output><empty_output><jupyter_text>Step 3: VectorizingThis will create a vector based on the term frequency of a word. Full term weights with document frequency will be calculated in a later step to support incremental additions.<jupyter_code>def vectorize(processed_text:list, index:list[str]=[]) -> (np.ndarray, list):
    unique_words = set(processed_text)

    text_freq_dict = {word: 0 for word in unique_words}
    for word in processed_text:
        text_freq_dict[word] += 1

    if index == []:
        index = list(unique_words)
    else:
        index += [word for word in unique_words if word not in index]
    
    text_freq = []
    for word in index:
        text_freq.append(text_freq_dict[word] if word in unique_words else 0)
    vector = np.array(text_freq) / len(processed_text)
    return np.array(text_freq) / len(processed_text), index<jupyter_output><empty_output><jupyter_text>Step 4: ClusteringThis will add the vector to the cluster using the full TF-IDF weighing to compare vectors. Each cluster is a dict that contains an average vector and its articles. Each article item contains a company name, tf vector, title, and description. Only the tf vector is necessary for clustering.<jupyter_code>def similarity(a:np.ndarray, b:np.ndarray) -> float:
    cos_similarity = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
    return cos_similarity

def add_to_cluster(new_article:dict, index:list, clusters:list=[], threshold:float=0.4) -> list:
    new_clusters = []

    
    for cluster in clusters:
        padding_len = len(index) - len(cluster['vector'])
        new_clusters.append({
            'vector': np.pad(cluster['vector'], (0, padding_len)),
            'articles': cluster['articles']
        })
        for article in cluster['articles']:
            article['vector'] = np.pad(article['vector'], (0, padding_len))
            
    document_freq = np.ceil(new_article['vector'])
    for cluster in new_clusters:
        for article in cluster['articles']:
            document_freq += np.ceil(article['vector'])

    inv_document_freq = np.zeros(len(document_freq))
    document_count = sum([sum([1 for article in cluster['articles']]) for cluster in new_clusters]) + 1
    for i in range(len(document_freq)):
        num = document_freq[i]
        if num != 0:
            num = math.log((1 / num) * document_count)
            inv_document_freq[i] = num

    closest_dist = 0
    closest_cluster = None

    for cluster in new_clusters:
        new_dist = similarity(np.multiply(new_article['vector'], inv_document_freq), np.multiply(cluster['vector'], inv_document_freq))
        if new_dist < threshold:
            continue
        if new_dist > closest_dist:
            closest_dist = new_dist
            closest_cluster = cluster
    
    if closest_cluster == None:
        new_clusters.append({
            'vector': new_article['vector'],
            'articles': [new_article]
        })
    else: 
        closest_cluster['articles'].append(new_article)
        closest_cluster['vector'] = sum([closest_article['vector'] for closest_article in closest_cluster['articles']]) / len(closest_cluster['articles'])
    return new_clusters<jupyter_output><empty_output><jupyter_text>Step 5: Putting it togetherWith the previous steps, this will make a news clusterer.<jupyter_code>class NewsClusterer:
    def __init__(self, threshold:float=0.4, current_clusters:list=[], word_index:list=[]):
        self.clusters = current_clusters
        self.word_index = word_index
        self.threshold = threshold

    def add(self, new_articles:list[dict]) -> list:
        for article in new_articles:
            text = article['title'] + ' ' + article['description']
            processed = preprocess_string(text)
            article['vector'], self.word_index = vectorize(processed, self.word_index)
            self.clusters = add_to_cluster(article, self.word_index, self.clusters, self.threshold)
        return self.clusters

article_dicts = []
for article in articles:
    article_dicts.append({
        'company': article['company'],
        'title': article['title'],
        'description': article['description'],
        'vector': None
    })
    
    
clusterer = NewsClusterer(0.2)
clusters = clusterer.add(article_dicts)<jupyter_output><empty_output><jupyter_text>Printing the clusterBecause I want to :)<jupyter_code>for i, cluster in enumerate(sorted(clusters, reverse=True, key=lambda x: len(x['articles']))):
    if len(cluster['articles']) == 1:
        continue
    print('Cluster ', i + 1, ': ')
    for article in cluster['articles']:
        print('\t', article['company'] if 'company' in article else '?', ': ',  article['title'])<jupyter_output>Cluster  1 : 
	 LA Times :  Some 200 California projects may be funded by infrastructure bill. Search your city's projects here
	 LA Times :  Biden talks up bipartisan infrastructure deal in swing-state Wisconsin
	 LA Times :  As Biden reassures moderates on infrastructure, progressives worry
	 LA Times :  'We have a deal': Biden, lawmakers reach tentative bipartisan infrastructure agreement
	 LA Times :  Democrats to seek citizenship pathway for immigrants in infrastructure bill, Sanders says
	 CBS News :  Democrats race to bring bipartisan infrastructure deal to Senate
	 CBS News :  Progressives demand climate action be in infrastructure deal
	 Daily Mail :  'Never a veto threat!' Biden backtracks on tying bipartisan infrastructure deal to reconciliation
	 Daily Mail :  Republican FURY after Biden tied $1.2trillion infrastructure deal to huge reconciliation package
	 Daily Mail :  <NAME>'s infrastructure deal slammed by The Squad AOC as suggests racism is behind $1.2tn plan
	 Dail[...]<jupyter_text>Extra, Part 1This part isn't from the article. I'm a bit worried that I incorrectly implemented the inverse document frequency weights, because that part of the article (and its source) confused me a little. To fix it, maybe I could cluster the initial articles as a batch instead of incremental, and handle future articles incrementally. The actual clustering algorithm will stay the same, but the weights are now consistent. This alternative could possibly be used to start clusters, while the add_to_cluster function can be used to add to existing clusters.<jupyter_code>def create_cluster(articles:list[dict], index:list, threshold:float=0.4) -> list:
    for article in articles:
        padding_len = len(index) - len(article['vector'])
        article['vector'] = np.pad(article['vector'], (0, padding_len))
            
    document_freq = np.zeros(len(index))
    for article in articles:
        document_freq += np.ceil(article['vector'])

    inv_document_freq = np.zeros(len(document_freq))
    document_count = len(articles)
    for i in range(len(document_freq)):
        num = document_freq[i]
        if num != 0:
            num = math.log((1 / num) * document_count)
            inv_document_freq[i] = num

    new_clusters = []

    for article in articles:
        closest_dist = 0
        closest_cluster = None

        for cluster in new_clusters:
            new_dist = similarity(np.multiply(article['vector'], inv_document_freq), np.multiply(cluster['vector'], inv_document_freq))
            if new_dist < threshold:
                continue
            if new_dist > closest_dist:
                closest_dist = new_dist
                closest_cluster = cluster
        
        if closest_cluster == None:
            new_clusters.append({
                'vector': article['vector'],
                'articles': [article]
            })
        else: 
            closest_cluster['articles'].append(article)
            closest_cluster['vector'] = sum([closest_article['vector'] for closest_article in closest_cluster['articles']]) / len(closest_cluster['articles'])
    return new_clusters<jupyter_output><empty_output><jupyter_text>Extra, Part 2This will extend the class made previously with the new create function.<jupyter_code>class NewNewsClusterer(NewsClusterer):
    def create(self, new_articles:list[dict]) -> list:
        for article in new_articles:
            text = article['title'] + ' ' + article['description']
            processed = preprocess_string(text)
            article['vector'], self.word_index = vectorize(processed, self.word_index)
        self.clusters = create_cluster(new_articles, self.word_index, self.threshold)
        return self.clusters

new_clusterer = NewNewsClusterer(0.2)
new_clusters = new_clusterer.create(article_dicts)
for i, cluster in enumerate(sorted(new_clusters, reverse=True, key=lambda x: len(x['articles']))):
    if len(cluster['articles']) == 1:
        continue
    print('Cluster ', i + 1, ': ')
    for article in cluster['articles']:
        print('\t', article['company'] if 'company' in article else '?', ': ',  article['title'])<jupyter_output>Cluster  1 : 
	 CNN :  Biden warns Putin during call that 'we expect him to act' on Russian ransomware attacks
	 HuffPost :  Biden Tells Putin Russia Must Crack Down On Cybercriminals
	 CBS News :  Biden warns Putin to act on Russian cyber-criminals targeting U.S.
	 CBS News :  Biden presses Putin to crack down on cyberattacks
	 CBS News :  Biden speaks to Putin after latest ransomware attacks
	 CBS News :  Biden faces pressure to respond amid latest ransomware attacks
	 Epoch Times :  Biden Warns Putin on Russian-Based Ransomware Attacks
	 Daily Mail :  Biden calls Putin to say US will take necessary action to defend against Russian ransomware attacks
Cluster  2 : 
	 CNN :  Biden defends pulling US out of Afghanistan as Taliban advances: 'We did not go to Afghanistan to nation-build'
	 LA Times :  As Taliban gains, Biden defends U.S. exit from Afghanistan, says it's ahead of schedule
	 CBS News :  President Biden announced that troops will leave Afghanistan by August 31
	 CBS News :  [...]
{'fraction_non_alphanumeric': 0.09553306578258594, 'fraction_numerical': 0.005234688536032106, 'mean_word_length': 3.684511646914589, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '294164', 'license': 'MIT', 'n_tokens_mistral': 3345, 'n_tokens_neox': 3203, 'n_words': 1028, 'path': 'clusterer.ipynb', 'repo_name': 'btaguinod/news-clusterer', 'size': '11465'}