<jupyter_start><jupyter_text>Unsupervised Clustering Pipeline:  COVID-19 and Mental HealthWe have gathered posts from a seleted set of subreddits, including several relevant to mental health.  Through supervised classification of posts to their subreddits of origin, we will identify the distinguishing characteristics of each subreddit and develop an approch to track changes over time in prediction certainty and results as the pandemic alters post content.  Through unsupervised clustering of these posts, we aim to:1. Identify natural subgroupings among the posts (e.g., do posts from subreddits related to anxiety disorders cluster separately from posts on subreddits related to mood disorders, or psychotic disorders)2. Track changes in representation of these natural subgroupings over the progression of the pandemicTo accomplish this, we use methods to determine the appropriate cluster number and to define clusters.  Then, we select a final cluster set and identify the features most characteristic of the clusters.  Finally, use these clusters to characterize mental health subreddits in the absence of a pandemic, to investigate changes in cluster representation on mental health subreddits during the pandemic, and to characterize non-mental-health related subreddits during the pandemic (e.g., r/COVID19_support)<jupyter_code># Import necessary packages
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import umap
import os
from datetime import datetime, timedelta
import pickle
from sklearn.cluster import KMeans
from sklearn import preprocessing
from sklearn.metrics.pairwise import cosine_similarity
from scipy.cluster.hierarchy import dendrogram, linkage
import scipy.sparse
import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
from gensim import corpora, models
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
import nltk
import seaborn as sns; sns.set()
# import datetime
nltk.download('wordnet')
stemmer = SnowballStemmer(language='english')
# Set seed for reproducibility
np.random.seed(0) 
# Mount GDrive and attach it to the colab for data I/O
from google.colab import drive
drive.mount('/content/drive')

# # Open Science Framework path
# input_dir = './data/input/'
# output_dir = './data/output/'<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Be sure to run any cells in the final section ("Repository of Useful Function Definitions"), which defines useful functions for basic operations.  This section is relegated to the end of the notebook for organizational clarity. Reload Data Objects, As Needed<jupyter_code># To save/restore intermediate objects. These are outputted throughout the script for you to recreate.
embedding = pd.read_csv('/content/drive/My Drive/ML4HC_Final_Project/data/processed_tfidf_umap50k.csv')
processed_data = pd.read_csv('/content/drive/My Drive/ML4HC_Final_Project/data/processed_posts_preTFIDF.csv')
posts_tfidf_bow = scipy.sparse.load_npz('/content/drive/My Drive/ML4HC_Final_Project/data/processed_tfidf_sparse_matrix.npz')<jupyter_output><empty_output><jupyter_text>Import DataFrom all available pre-pandemic data, select only mental health subreddits.<jupyter_code># Select files to import
data_folder = '/content/drive/My Drive/ML4HC_Final_Project/data/input/feature_extraction/'
all_filenames = os.listdir(data_folder)

# List candidate files to manually select mental health subreddits
current_filenames = current_filenames = [i for i in all_filenames if ".csv" in i]
current_filenames = current_filenames = [i for i in current_filenames if "_pre_" in i]
# print(current_filenames) 

mh_subreddit_filenames = ['suicidewatch_pre_features.csv',
 'depression_pre_features.csv',
 'ptsd_pre_features.csv',
 'anxiety_pre_features.csv',
 'socialanxiety_pre_features.csv',
 'healthanxiety_pre_features.csv',
 'bipolarreddit_pre_features.csv',
 'mentalhealth_pre_features.csv',
 'bpd_pre_features.csv',
 'schizophrenia_pre_features.csv',
 'paranoia_pre_features.csv',
 'EDAnonymous_pre_features.csv',
 'alcoholism_pre_features.csv',
 'addiction_pre_features.csv',
 'adhd_pre_features.csv',
 'psychosis_pre_features.csv',
 'mentalillness_pre_features.csv',
 'autism_pre_features.csv']
current_filenames = mh_subreddit_filenames 
# Import and join files
# Will print filenames to allow progress tracking
reddit_data = pd.read_csv(data_folder +current_filenames[0], index_col=False)
for i in np.arange(1, len(current_filenames)):
  print(current_filenames[i])
  new_data = pd.read_csv(data_folder +current_filenames[i], index_col=False)
  reddit_data = pd.concat([reddit_data, new_data], axis=0)

# Examine all columns
reddit_data.columns<jupyter_output>depression_pre_features.csv
ptsd_pre_features.csv
anxiety_pre_features.csv
socialanxiety_pre_features.csv
healthanxiety_pre_features.csv
bipolarreddit_pre_features.csv
mentalhealth_pre_features.csv
bpd_pre_features.csv
schizophrenia_pre_features.csv
paranoia_pre_features.csv
EDAnonymous_pre_features.csv
alcoholism_pre_features.csv
addiction_pre_features.csv
adhd_pre_features.csv
psychosis_pre_features.csv
mentalillness_pre_features.csv
autism_pre_features.csv<jupyter_text>Process Posts to Extract Text Features<jupyter_code># Preprocessing matches feature development for LDA
# Adapted from: https://towardsdatascience.com/topic-modeling-and-latent-dirichlet-allocation-in-python-9bf156893c24

def lemmatize_stemming(text):
    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v'))
def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            result.append(lemmatize_stemming(token))
    return result
processed_data = reddit_data['post'].map(preprocess)
# Remove posts with fewer than 20 terms after preprocessing (57950 posts removed)
processed_lengths = list(map(lambda x: len(x),processed_data['post']))
newly_short_posts = np.where(np.array(processed_lengths)<20)[0]
reddit_data.index = np.arange(0,len(reddit_data))
reddit_data.drop(list(newly_short_posts), inplace=True)
# Drop short posts from processed data
processed_data.index = np.arange(0,len(processed_data))
processed_data.drop(list(newly_short_posts), inplace=True)
processed_data.shape
# Collapse separate terms into single string for TF-IDF
collapsed_processed_posts = list(map(lambda x: paste(x, collapse=" "),processed_data))
# Add processed data to data frame
reddit_data['processed_post'] = collapsed_processed_posts
# Option: To Explore BOW without TF-IDF
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 3), max_df = 0.75, min_df = 0.001)
post_bow = vectorizer.fit_transform(reddit_data['processed_post'])
print(vectorizer.get_feature_names()[0:10])
print(len(vectorizer.get_feature_names()))<jupyter_output><empty_output><jupyter_text>KMeans ClusteringAdapted from: https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html<jupyter_code># Import required packages
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import metrics
from sklearn.cluster import KMeans, MiniBatchKMeans
# Develop TF-IDF features
vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(1, 3), max_df=0.75, min_df=0.001, use_idf=True)
posts_tfidf_bow = vectorizer.fit_transform(reddit_data['processed_post'])
# Examine TF-IDF features
print(vectorizer.get_feature_names()[0:20])
print(len(vectorizer.get_feature_names()))
# Test different cluster sizes
init_vals = np.random.choice(np.arange(100), size=10, replace=False)
silhouette_vals = []
sse_vals = []
for selected_k in np.array([19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]):
  print("Cluster")
  print(selected_k)
  for i in np.arange(3):
    print("Rep")
    print(i)
    km = MiniBatchKMeans(n_clusters=selected_k, init='k-means++', n_init=1, init_size=1000, 
                     batch_size=1000, random_state = init_vals[i])
    km.fit(posts_tfidf_bow)
    silhouette_vals.append(metrics.silhouette_score(posts_tfidf_bow, km.labels_, sample_size=1000))
    sse_vals.append(km.inertia_)
# Set k values to test
n_clusters = np.repeat(np.array([19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]), 3)
# Plot silhouette scores
to_plot_d = {'Init_Value': np.repeat(init_vals[0:3], 15), 'Silhouette': silhouette_vals, 'Squared_Errors_Sum': sse_vals, 
             'Num_Clusters': n_clusters}
to_plot_df = pd.DataFrame(data=to_plot_d)
ax = sns.lineplot(x="Num_Clusters", y="Silhouette", data=to_plot_df)
# Plot Sum of Squared Errors values
ax = sns.lineplot(x="Num_Clusters", y="Squared_Errors_Sum", data=to_plot_df)
# Generate clusters for a selected k
selected_k = 22
km = MiniBatchKMeans(n_clusters=selected_k, init='k-means++', n_init=3, init_size=1000, 
                     batch_size=1000)
km.fit(posts_tfidf_bow)
print("Silhouette Coefficient: %0.3f"
      % metrics.silhouette_score(posts_tfidf_bow, km.labels_, sample_size=1000))
# Cluster "performance" for subreddit grouping
print("Homogeneity: %0.3f" % metrics.homogeneity_score(reddit_data['subreddit'], km.labels_))
print("Completeness: %0.3f" % metrics.completeness_score(reddit_data['subreddit'], km.labels_))
print("V-measure: %0.3f" % metrics.v_measure_score(reddit_data['subreddit'], km.labels_))
print("Adjusted Rand-Index: %.3f"
      % metrics.adjusted_rand_score(reddit_data['subreddit'], km.labels_))
# Examine representation among the cells selected for the embedding
reddit_data.iloc[selected_cells,:]['subreddit'].value_counts()
# Identify the top terms for each cluster, using the TF-IDF terms with the highest values in the centroid
# Adapted From: https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]

terms = vectorizer.get_feature_names()
for i in range(selected_k):
  print("Cluster %d:" % i, end='')
  for ind in order_centroids[i, :20]:
    print(' %s' % terms[ind], end='')
  print()<jupyter_output>Top terms per cluster:
Cluster 0: drive anxieti driver road licens feel like time know work go think accid want home start year get thing test
Cluster 1: adhd diagnos like medic thing think work test help time focus know diagnosi school diagnos adhd feel symptom peopl year start
Cluster 2: mother year parent famili brother life live abus time know go want father sister like school tell help feel think
Cluster 3: feel like feel like know want peopl think time life thing go friend talk make person year work help depress anymor
Cluster 4: fuck want life hate shit like feel know peopl go kill think live anymor time tire friend thing love care
Cluster 5: mental health mental health https help ill mental ill peopl disord issu know like feel want post survey need https reddit reddit anxieti
Cluster 6: med take work take med feel like medic help start time go adhd know year effect week think need want anxieti
Cluster 7: suicid want thought life kill suicid thought think feel know help attempt [...]<jupyter_text>Cluster 0: "Bipolar/Borderline"- disord bipolar diagnos depress diagnosi anxieti person disord bipolar disord person like know feel think mental year symptom peopl borderlin help anxieti disordCluster 1: "Advertisements?"- hack servic contact issu contact devic credit card twitter spous hangout snapchat network expos credit hire instagram card facebook cheat rule specialCluster 2: "Unknown_2"- like think know feel time help go thing peopl look want need work start get year today say tri anxietiCluster 3: "ADHD"- adhd diagnos like medic thing think help work focus time know diagnosi feel diagnos adhd test peopl symptom year med lifeCluster 4: "Unknown_2"- feel like feel like know want think peopl time thing life go depress make talk help friend start person work yearCluster 5: "Medication"- medic med take effect doctor adderal dose work vyvans feel start prescrib help like experi anxieti week month time yearCluster 6: "Sleeplessness"- sleep night wake asleep hour fall asleep feel like fall time dream go anxieti think awak know work nightmar help wantCluster 7: "Social Anxiety"- social social anxieti anxieti peopl talk like feel think friend know want convers time person go thing awkward interact work startCluster 8: "Angry/Expletive"- fuck want shit like life feel know peopl go think kill hate time fuck fuck live anymor friend thing tell yearCluster 9: "EatingHabits_1"- eat bing calori restrict eat disord like disord feel weight food purg fast know want go fuck think week today mealCluster 10: "SubstanceUse/Alcohol"- drink alcohol sober night beer know feel time year like go want stop week start think help day quit problemCluster 11: "PolarizedEmotions_?"- hate want feel like peopl life know hate hate fuck think love wish live friend time go thing feel like hate life killCluster 12: "WeightChange"- weight gain lose weight lose gain weight eat like pound bodi look feel weigh want restrict weight loss know bing weight gain loss caloriCluster 13: "School"- school work class year time like go feel know colleg start want think high high school grade life thing week needCluster 14: "Relationships"- emot feel like know thing peopl think want time relationship abus life feel like person love help depress control go yearCluster 15: "Resources"- https mental mental health health help peopl ill mental ill youtub issu https youtub reddit https reddit post survey https youtub watch youtub watch like depress feelCluster 16: "EatingHabits_2"- food purg eat bing like calori meal feel want know go restrict time think fast fuck today dinner hungri safe foodCluster 17: "Friends/Support"- friend talk like know feel want peopl best friend best tell think time go year say hang thing person life groupCluster 18: "PanicAttacks"- anxieti attack panic panic attack feel heart like go start think have pain symptom know time breath help chest doctor yearCluster 19: "Philosophical_?"- want know life think feel like time tell go thing year love peopl live help say suicid depress talk friendCluster 20: "Exhaustion"- tire feel want life like know anymor time live tire tire think peopl go tri fuck depress tire feel work feel like betterCluster 21: "WaterConsumption"- water drink drink water like feel eat go know time fast today day food think bottl shower hour help work take<jupyter_code># Define cluster annotations based on results above
cluster_annotations = ["Bipolar/Borderline", "Advertisements?","Unknown_1", "ADHD","Unknown_2", 
                       "Medication", "Sleeplessness", "Social Anxiety", "Angry/Expletive", "EatingHabits_1",
                       "SubstanceUse/Alcohol", "PolarizedEmotions_?", "WeightChange", "School", "Relationships", 
                       "Resources", "EatingHabits_2", "Friends/Support", "PanicAttacks", "Philosophical_?",
                       "Exhaustion", "WaterConsumption"]
# Store cluster numbers and annotations 
reddit_data['cluster_number'] = km.labels_

reddit_data.index = np.arange(len(reddit_data))
reddit_data['cluster_annotations'] = np.ones(len(reddit_data))
for i in np.arange(22):
  selected_rows = np.where(reddit_data['cluster_number']==i)[0]
  reddit_data.loc[selected_rows,'cluster_annotations'] = np.repeat([cluster_annotations[i]], len(selected_rows))

reddit_data['cluster_annotations'].value_counts()
# For manual examination of posts in a selected cluster
list(reddit_data.iloc[np.where(reddit_data['cluster_annotations']=="School")[0], :]['post'][0:10])
# Examine Subreddit Representation by Cluster
reddit_data.iloc[np.where(km.labels_==0)[0],:]['subreddit'].value_counts()<jupyter_output><empty_output><jupyter_text>Test statistical associationsThe hypergeometric distribution models drawing objects from a bin. M is the total number of objects, n is total number of Type I objects. The random variate represents the number of Type I objects in N drawn without replacement from the total population.<jupyter_code>from scipy.stats import hypergeom
### Hypergeometric Test for cluster representation in subreddits

n_tests = len(np.unique(reddit_data['subreddit']))*len(np.unique(reddit_data['cluster_annotations']))
bonf_thresh = 0.05/n_tests
print("Bonferroni threshold is ", bonf_thresh)
n_sig_enrichments = 0

for sub in np.unique(reddit_data['subreddit']):
  for clust in np.unique(reddit_data['cluster_annotations']):

    #M is the population size (previously N):  all posts on all subs
    M = len(reddit_data)

    # n is the number of successes in the population (previously K)
    n= len(np.where(reddit_data['cluster_annotations']==clust)[0])

    # N is the sample size (previously n):  all post in this subreddit
    N = len(np.where(reddit_data['subreddit']==sub)[0])

    # X is the number of drawn “successes”:  posts in this sub from the candidate cluster
    x = len(np.intersect1d(np.where(reddit_data['subreddit']==sub)[0], 
                       np.where(reddit_data['cluster_annotations']==clust)[0]))
    
    p_val = hypergeom.sf(x-1, M, n, N)
    #if p_val < 0.05:
    #  print("Nominal", sub, " ", clust)
    if p_val < bonf_thresh:
      print("MHT significant: ", sub, " ", clust)
      n_sig_enrichments += 1

print("Total Significant Enrichments: ", n_sig_enrichments)
# Number of significant enrichments
n_sig_enrichments
# Show Representation of Topic Across Multiple Subreddits

# Select example subreddit 
cluster_label = "SubstanceUse/Alcohol"

# Plot representation
counts = reddit_data.iloc[np.where(reddit_data['cluster_annotations']==cluster_label)[0],:]['subreddit'].value_counts()
counts
to_plot_df = pd.DataFrame({"Number of Posts in Cluster": counts, "Subreddit": counts.index})
sns.set(style="whitegrid")
ax = sns.barplot(x="Subreddit", y="Number of Posts in Cluster", data=to_plot_df)
ax.set_xticklabels(
    ax.get_xticklabels(), 
    rotation=45, 
    horizontalalignment='right',
    fontweight='light',
    #fontsize='x-large'
)
# Examine alcohol cluster on alcoholism subreddit
# Sixth most common, first most common is Relationships
sub = "alcoholism"
reddit_data.iloc[np.where(reddit_data['subreddit']==sub)[0],:]['cluster_annotations'].value_counts()/len(np.where(reddit_data['subreddit']==sub)[0])
# Save intermediate object
reddit_data.to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/assigned_unsup_clusters.csv', index=False)<jupyter_output><empty_output><jupyter_text>Visualize with UMAP<jupyter_code>import umap
n_subset =50000
selected_cells = np.random.choice(np.arange(posts_tfidf_bow.shape[0]), size = n_subset, replace = False)
mapper = umap.UMAP(metric='hellinger', random_state=42).fit(posts_tfidf_bow[selected_cells,:])
embedding = mapper.transform(posts_tfidf_bow[selected_cells,:]) 
# To save/restore intermediate objects 
df_embedding = pd.DataFrame(embedding)
df_embedding.to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/processed_tfidf_umap50k.csv', index=False) 
# To save/restore intermediate objects 
df_selected_cells = pd.DataFrame(selected_cells)
df_selected_cells.to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/selected_cells.csv', index=False) 
# To save/restore intermediate objects 
# Note:  Saved Before Removal of Short Posts
df_processed_posts = pd.DataFrame(processed_data)
df_processed_posts.to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/processed_posts_preTFIDF.csv', index=False) 
# To save/restore intermediate objects 
reddit_data = pd.read_csv('/content/drive/My Drive/ML4HC_Final_Project/data/assigned_unsup_clusters.csv')
# To save/restore intermediate objects
scipy.sparse.save_npz('/content/drive/My Drive/ML4HC_Final_Project/data/processed_tfidf_sparse_matrix.npz', posts_tfidf_bow)
# Plot Clusters on UMAP
plt.scatter(embedding[:, 0], embedding[:, 1], c=reddit_data.iloc[selected_cells,:]['cluster_number'], cmap='Spectral', s=5)
plt.gca().set_aspect('equal', 'datalim')
num_clust = len(np.unique(reddit_data['cluster_number']))
plt.colorbar(boundaries=np.arange(num_clust+1)-0.5).set_ticks(np.arange(num_clust))
plt.title('UMAP Projection of Pre-Pandemic Posts, TF-IDF', fontsize=14);<jupyter_output><empty_output><jupyter_text>Examine Cluster Representation in Pandemic Data<jupyter_code># Select files to import
data_folder = '/content/drive/My Drive/ML4HC_Final_Project/data/input/feature_extraction/'
all_filenames = os.listdir(data_folder)

# List candidate files to manually select mental health subreddits
current_filenames = current_filenames = [i for i in all_filenames if ".csv" in i]
current_filenames = current_filenames = [i for i in current_filenames if "_post_" in i]
print(current_filenames) 
# To import non-mental health posts with relevance to the pandemic
current_filenames = ['COVID19_support_post_features.csv', 'relationships_post_features.csv', 
                     'personalfinance_post_features.csv', 'legaladvice_post_features.csv']
# Import and join files
# Will print filenames to allow progress tracking
pandemic_data = pd.read_csv(data_folder +current_filenames[0], index_col=False)
for i in np.arange(1, len(current_filenames)):
  print(current_filenames[i])
  new_data = pd.read_csv(data_folder +current_filenames[i], index_col=False)
  pandemic_data = pd.concat([pandemic_data, new_data], axis=0)

# Examine all columns
pandemic_data.columns
# Preprocess posts before TF-IDF
processed_data = pandemic_data['post'].map(preprocess)

# Remove posts with fewer than 20 terms after preprocessing 
processed_lengths = list(map(lambda x: len(x),processed_data))
newly_short_posts = np.where(np.array(processed_lengths)<20)[0]
pandemic_data.index = np.arange(0,len(pandemic_data))
pandemic_data.drop(list(newly_short_posts), inplace=True)

processed_data.index = np.arange(0,len(processed_data))
processed_data.drop(list(newly_short_posts), inplace=True)
processed_data.shape

collapsed_processed_posts = list(map(lambda x: paste(x, collapse=" "),processed_data))
pandemic_data['processed_post'] = collapsed_processed_posts
# Examine number of posts
pandemic_data.shape
# Number of posts removed due to length = 12281
len(newly_short_posts)
# Save intermediate object
pandemic_data.to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/pandemic_data.csv', index=False) 
# Save intermediate object
pandemic_data = pd.read_csv('/content/drive/My Drive/ML4HC_Final_Project/data/pandemic_data.csv')
# Develop TF-IDF features
vectorizer_pandemic = TfidfVectorizer(analyzer='word', ngram_range=(1, 3), max_df=0.75, min_df=0.001, use_idf=True)
pandemic_tfidf_bow = vectorizer_pandemic.fit_transform(pandemic_data['processed_post'])
print(vectorizer_pandemic.get_feature_names()[0:20])
print(len(vectorizer_pandemic.get_feature_names()))
# Save intermediate object
scipy.sparse.save_npz('/content/drive/My Drive/ML4HC_Final_Project/data/pandemic_tfidf_bow.npz', pandemic_tfidf_bow)
# Convert dates to date format
pandemic_data['date'] = [datetime.strptime(x, '%Y/%m/%d') for x in pandemic_data['date']]

# Make date groups by week
min_date = np.min(panemic_data['date'])
max_date = np.max(pandemic_data['date'])
pandemic_data['week'] = np.ones(pandemic_data.shape[0])
start_week = min_date + timedelta(days=7)
n_week = 2
week_col = np.where(pandemic_data.columns=='week')[0]
while start_week <= max_date:
  end_week = start_week + timedelta(days=6)
  loc_posts_after_start = np.where(pandemic_data['date'] >=start_week)[0]
  loc_posts_before_end = np.where(pandemic_data['date'] <=end_week)[0]
  loc_week_dates = np.intersect1d(loc_posts_after_start, loc_posts_before_end)
  pandemic_data.iloc[loc_week_dates, week_col] = np.ones(len(loc_week_dates))*n_week
  n_week = n_week + 1
  start_week = start_week + timedelta(days=7)
# Save intermediate object
pd.DataFrame(vectorizer_pandemic.get_feature_names()).to_csv('/content/drive/My Drive/ML4HC_Final_Project/data/pandemic_tfidf_features.csv', index=False)
# Restore intermediate object
pandemic_tfidf_features = pd.read_csv('/content/drive/My Drive/ML4HC_Final_Project/data/pandemic_tfidf_features.csv')
pandemic_tfidf_features = list(pandemic_tfidf_features.iloc[:,0])<jupyter_output><empty_output><jupyter_text>Determine Representation of Conversation Themes on Covid Support<jupyter_code># Identify relevant posts
covid_support_posts = np.where(pandemic_data['subreddit']=="COVID19_support")[0]
relationship_posts = np.where(pandemic_data['subreddit']=="relationships")[0]
from scipy.spatial import distance
# Develop TF-IDF features
pandemic_tfidf_transform = vectorizer.transform(pandemic_data['processed_post'])
# COVID-19 Support topic representation
assigned_cluster = []
for n_post in np.arange(len(covid_support_posts)):
  distances = np.zeros(22)
  for n_centroid in np.arange(22):
    distances[n_centroid] = distance.euclidean(km.cluster_centers_[n_centroid],pandemic_tfidf_transform[covid_support_posts,:][n_post,:].toarray())
  assigned_cluster.append(cluster_annotations[np.argmin(distances)])

pd.DataFrame(assigned_cluster).iloc[:,0].value_counts()

# Conversation topic representation on Relationships subreddit 
assigned_cluster = []
for n_post in np.arange(len(relationship_posts)):
  distances = np.zeros(22)
  for n_centroid in np.arange(22):
    distances[n_centroid] = distance.euclidean(km.cluster_centers_[n_centroid],pandemic_tfidf_transform[relationship_posts,:][n_post,:].toarray())
  assigned_cluster.append(cluster_annotations[np.argmin(distances)])

pd.DataFrame(assigned_cluster).iloc[:,0].value_counts()

# Show Representation of Topics Within the Subreddit
# (specific to the subreddit used to create assigned_cluster vector)
counts = pd.DataFrame(assigned_cluster).iloc[:,0].value_counts()
to_plot_df = pd.DataFrame({"Number of Posts in Cluster": counts, "Cluster": counts.index})
sns.set(style="whitegrid")
ax = sns.barplot(x="Cluster", y="Number of Posts in Cluster", data=to_plot_df)
ax.set_xticklabels(
    ax.get_xticklabels(), 
    rotation=45, 
    horizontalalignment='right',
    fontweight='light',
    #fontsize='x-large'
)<jupyter_output><empty_output><jupyter_text>Repository of Useful Function Definitions<jupyter_code># Paste Function Equivalent
# Source: https://stackoverflow.com/questions/21292552/equivalent-of-paste-r-to-python

import functools
def reduce_concat(x, sep=""):
    return functools.reduce(lambda x, y: str(x) + sep + str(y), x)

def paste(*lists, sep=" ", collapse=None):
    result = map(lambda x: reduce_concat(x, sep=sep), zip(*lists))
    if collapse is not None:
        return reduce_concat(result, sep=collapse)
    return list(result)

# Example Use
#print(paste([1,2,3], [11,12,13], sep=','))
#print(paste([1,2,3], [11,12,13], sep=',', collapse=";"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09786092270999479, 'fraction_numerical': 0.013639412685399121, 'mean_word_length': 6.362139917695473, 'pattern_counts': {'":': 4, '<': 48, '<?xml version=': 0, '>': 44, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 9, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '96632', 'license': 'Apache-2.0', 'n_tokens_mistral': 8858, 'n_tokens_neox': 8484, 'n_words': 2319, 'path': 'Unsupervised_Clustering_Pipeline.ipynb', 'repo_name': 'SarahEAdams/reddit', 'size': '26834'}