<jupyter_start><jupyter_text>In the first notebook we were able to create a dataset that contains the content for each discipline pulled from wikipedia. Now we will use that data to:1. Get counts of all the words in each and create a sparsematrix representation using the countvectorizer method from sklearn2. Reduce the dimensions from all the disciplines into two. We do this both to see where the data plot in relation to eachother, but also to make a sweet visualization3. Clusters will then be grouped using HDBSCAN algorithm<jupyter_code>pip install umap-learn[plot]
pip install hdbscan
import pandas as pd
import numpy as np
import plotly.express as px
import umap
import umap.plot
import hdbscan
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import adjusted_rand_score, adjusted_mutual_info_score
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import silhouette_score
df = pd.read_csv(r"https://raw.githubusercontent.com/noahkarsky/Geoscience_Disciplines/master/Parsed_wiki_data.csv", names=["index","related_word","discipline"],skiprows=[0])
#this will make each discipline into a row and the new column will be called the article_text
df1 = df.groupby('discipline')['related_word'].apply(list).reset_index(name='article_text')
df1['text'] = df1.article_text.astype(str).str.replace('\[|\]|\'', '').str.replace(',','')
df1.head()

vectorizer = CountVectorizer(min_df=5, stop_words='english') #pretty sure I am double applying stopwords here....oh well now we know FOR SURE there are no stop words gunking up the workflow
word_doc_matrix = vectorizer.fit_transform(df1.text)
mapper = umap.UMAP(n_neighbors=10, metric='cosine', min_dist=0.1, random_state=42).fit_transform(word_doc_matrix)
embedding = pd.DataFrame(mapper, columns = ['x','y'])
#the embedding is the output of the UMAP algorithm, so we will rejoin the discipline names for viewing in the interactive plot below
embedding['title'] = df1.discipline
fig, ax = plt.subplots(figsize = (10,10))
plt.scatter(embedding['x'], embedding['y'],)<jupyter_output><empty_output><jupyter_text>Visually we can see clusters of data, but lets take a look at what HDBSCAN finds and distinguishes as outliers within this 2D mapping of the data<jupyter_code>clusterer = hdbscan.HDBSCAN()
clusterer.fit(mapper)
clusterer.condensed_tree_.plot(select_clusters=True)<jupyter_output><empty_output><jupyter_text>Eight clusters!<jupyter_code>#these two lines allow us to desaturate the color of points based on their probabilities of being within a cluster.
pal = sns.color_palette('Paired', 87)
colors = [sns.desaturate(pal[col], sat) for col, sat in zip(clusterer.labels_,clusterer.probabilities_)]

ns
fig, ax = plt.subplots(figsize = (10,10))
plt.scatter(embedding['x'], embedding['y'], c=colors,)
fig = px.scatter(embedding,x='x',y='y',color=clusterer.labels_,hover_data=['title'],width=800, height=800)
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10250338294993235, 'fraction_numerical': 0.011502029769959404, 'mean_word_length': 6.843501326259947, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '364451', 'license': 'MIT', 'n_tokens_mistral': 953, 'n_tokens_neox': 930, 'n_words': 295, 'path': 'Geologic_branches_2.ipynb', 'repo_name': 'noahkarsky/Geoscience_Disciplines', 'size': '2956'}