<jupyter_start><jupyter_text>Analysis of Praise by Words (🐙octopus)**Date: June 9, 2011** In this session we will work to gain insight about the way in which different types of activities influence the distribution of Impact Hours.  Loading Packages and Importing the Data<jupyter_code>import pandas as pd
import numpy as np

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.cluster import KMeans

import matplotlib.pyplot as plt
import seaborn

praise_df = pd.read_csv("cleaned-non-quantifier-data.csv")
praise_df.columns
praise_df.head()<jupyter_output><empty_output><jupyter_text>Getting a Feel for the Words  Before attempting a sophisticated algorithms, we want to do some human analysis in the data. We look at a large sample of "Reason for Dishing" to get a feel for the reasons that praise is given.<jupyter_code>praise_df["Reason for dishing"].to_list()<jupyter_output><empty_output><jupyter_text>Doing a Slight Bit of Cleaning  It will help to get consistency in the words. First, we make all of the strings lower case.<jupyter_code>make_lower_case = lambda x: x.lower()
praise_df["Reason for dishing"] = praise_df["Reason for dishing"].map(make_lower_case)<jupyter_output><empty_output><jupyter_text>As we've seen before, the Impact Hours are quite right-skewed.<jupyter_code>praise_df['IH per Praise'].describe()<jupyter_output><empty_output><jupyter_text>Let's check for missing values.<jupyter_code>praise_df['IH per Praise'].isna().sum()<jupyter_output><empty_output><jupyter_text>There appear to be several -- let's verify.<jupyter_code>praise_df[praise_df['IH per Praise'].isna()]<jupyter_output><empty_output><jupyter_text>We drop these 502 missing values since they have no quantitative information.  We then re-set the index.<jupyter_code>praise_df.dropna(subset=["IH per Praise"], inplace = True)
praise_df["IH per Praise"].isna().sum()
praise_df = praise_df.reset_index()<jupyter_output><empty_output><jupyter_text>Many of the columns will not be useful for this particular analysis. We drop all of the following:* To* From* v1 Norm* v2 Norm* v3 Norm* IH per Person* Cred per Praise* Cred per person* Period* Room (since we have Room-NoEmoji, which is cleaner)* v1* v2* v3* Cred per Praise* Cred per person* Unnamed: 12 (a duplicate of "To")* To.1We are focusing on other information for this analysis.<jupyter_code>praise_df.drop(inplace = True, columns = ["To", "From", "v1 norm", "v2 norm", "v3 norm", "IH per person", "Cred per Praise", "Cred per person",
                                         "period", "Room", "v1", "v2", "v3", "Cred per Praise", "Cred per person", "Unnamed: 12", "To.1"])
praise_df.head()<jupyter_output><empty_output><jupyter_text>We can also drop "Avg %" and "Date" (since this is duplicated in other columns.)<jupyter_code>praise_df.drop(columns = ["Avg %", "Date"], inplace = True)
praise_df.head()<jupyter_output><empty_output><jupyter_text>We notice an issue with "Server" name that we perhaps should have caught earlier.<jupyter_code>np.unique(praise_df["Server"])<jupyter_output><empty_output><jupyter_text>Both "TG" and "Telegram" are used for Telegram. It's time to fix this.<jupyter_code>replace_TG = lambda x: "Telegram" if x == "TG" else x
praise_df["Server"] = praise_df["Server"].map(replace_TG)
np.unique(praise_df["Server"])<jupyter_output><empty_output><jupyter_text>Let's see how much each Server contributed.<jupyter_code>praise_df.groupby("Server")["IH per Praise"].count()
praise_df.groupby("Server")["IH per Praise"].count()/len(praise_df["IH per Praise"])
praise_df.groupby("Server")["IH per Praise"].sum()
praise_df.groupby("Server")["IH per Praise"].sum()/praise_df["IH per Praise"].sum()<jupyter_output><empty_output><jupyter_text>Since both "Bot Training Ground" and "TEC template" contributed less than 1%, we will remove them from the data frame.<jupyter_code>praise_df = praise_df[~(praise_df["Server"] == "Bot Training Ground")]
praise_df = praise_df[~(praise_df["Server"] == "TEC template")]  
praise_df = praise_df.reset_index()
praise_df.groupby("Server")["IH per Praise"].count().plot.pie()
praise_df.groupby("Server")["IH per Praise"].count().plot.bar()
praise_df.groupby("Server")["IH per Praise"].sum().plot.pie()
praise_df.groupby("Server")["IH per Praise"].mean().plot.bar()
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Since we originally made "Source" based on Server name and we have now changed Server, we also change Source.<jupyter_code>praise_df["Source"] = praise_df["Server"] + " : " + praise_df["Room-NoEmoji"]
praise_df["Source"]<jupyter_output><empty_output><jupyter_text>There are some activities that, according to the quantiifiers, that are exceptionally great (such as "inventing augmented bonded curves") and thus atypical. We'd like to decide on a cutoff point, to remove these occurrences and analyze them separately.<jupyter_code>list_of_quantiles = np.quantile(praise_df["IH per Praise"], [0.8,0.85,0.9,0.95,0.99])
list_of_quantiles<jupyter_output><empty_output><jupyter_text>The top 5% begins at approximately 4.42. This feels right to us, though we can change the quantile later if we wish. We create a new data frame that holds only exceptional data (top 5%)<jupyter_code>cutoff_quant = 0.95
exceptional_df = praise_df[praise_df["IH per Praise"] >= np.quantile(praise_df["IH per Praise"], cutoff_quant)]
exceptional_df
print("How many exceptional praises are there?")
print("There are {} praises in this exceptional data frame, which begins at the {} percentile.".format(len(exceptional_df), cutoff_quant))
print("\n")
print("What percentage of the Impact Hours awarded come from these praises?")
pct_exceptional_IH =100*(exceptional_df["IH per Praise"].sum()/praise_df["IH per Praise"].sum())
print("{} percent of the Impact Hours awarded come from this group".format(pct_exceptional_IH))
exceptional_df.groupby("Server")["IH per Praise"].count()/len(exceptional_df)
exceptional_df.groupby("Server")["IH per Praise"].sum()/exceptional_df["IH per Praise"].sum()<jupyter_output><empty_output><jupyter_text>Organizing Into Words  Now we are interested in the word-level data, with a hope of recognizing and grouping like activities. We use a basic [CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html), which simply creates a new column for each word that appears in the data set, then gives each observation the number of times that word appeared. Settings:* input: "content" -- since our source is a list* stop_words = "english" -- to avoid common words like "the", "and", "a", etc.Settings We Did Not Use, but Could:* max_df: maximum document frequency -- ignore words which occur too frequently* min_df: minimum_document_frequency - ignore words which are very rare<jupyter_code>vectorizer = CountVectorizer(input = "content", stop_words = 'english', binary = True, analyzer = "word")
words_encoded = vectorizer.fit_transform(praise_df["Reason for dishing"].to_list())

words_df = pd.DataFrame(data = words_encoded.toarray(), columns = vectorizer.get_feature_names())
words_df.columns
words_df.columns.to_list()
words_df.head()<jupyter_output><empty_output><jupyter_text>Now we join this data back to the original data, so we also have context of source, date, and Impact Hours.<jupyter_code>len(words_df)
sum(words_df.index == praise_df.index) == len(praise_df)
final_df = pd.concat([words_df, praise_df], axis = 1)
len(final_df)<jupyter_output><empty_output><jupyter_text>Now we can use his data frame to answer questions related to the textual information in the praise, as well as date, server, and room. To give an example of what this might look like, we focus on social media, finding all messages which contain "retweeting" or "mentioning".<jupyter_code>final_df.groupby("Server")["retweeting"].count()
final_df[final_df["retweeting"] == 1].groupby("Server")["IH per Praise"].mean()
final_df[final_df["mentioning"] == 1].groupby("Server")["IH per Praise"].mean()
final_df[final_df["retweeting"] == 1].groupby("Month")["IH per Praise"].mean().plot.barh()
plt.title("Impact Hours for retweeting by Month")
plt.show()
final_df[final_df["retweeting"] == 1].groupby("Month")["IH per Praise"].count().plot.barh()
final_df[final_df["retweeting"] == 1].groupby("Month")["IH per Praise"].sum().plot.barh()
social_condition = (final_df["retweeting"]== 1) | (final_df["mentioning"] == 1)
social_df = final_df[social_condition]
len(social_df)
social_df["Reason for dishing"].to_list()
month_and_server = pd.pivot_table(data = social_df,  values = "IH per Praise", index = "Month", 
                                  columns = "Server", aggfunc = 'mean')
month_and_server
month_and_server = month_and_server.fillna(0)
fig, ax = plt.subplots(figsize = (12, 7))
sns.heatmap(month_and_server, cmap ='RdYlGn', linewidths = 0.30, 
            annot = True)
ax.set_title("Value of Retweets and Mentions")<jupyter_output><empty_output><jupyter_text>Well, this looks strange -- why would social media activity on Telegram in December be worth twice as much as similar activity in Token Engineering Commons in April?<jupyter_code>dec_TG_df = social_df.query("Server == 'Telegram' & Month == 12")
dec_TG_df["Reason for dishing"].to_list()
dec_TG_df[["Reason for dishing","IH per Praise"]]<jupyter_output><empty_output><jupyter_text>1. One reason for discrepancy is obviously fair: the work of writing an article vs. the work of retweeting. So perhaps "mentioning" needs to be utilized more carefully.    2. There are identical items which receive various values. This is likely a result of "hand-editing" by the quantifiers. In [my conversation with Griff on May 20](https://www.youtube.com/watch?v=XTlfElzjPWg), he said that validators would sometimes "hand-edit" the overall amount of Impact Hours that a contributor received if they felt it was not proportional to actual impact. They did this by adjusting an arbitrary row. I think we are seeing that here.    Using Clustering to Group Like PraiseThis would be a good place for someone with more expertise in clustering to come in:* How many clusters/categories should we make?* What isa good algothm?* How should the data be preprocessed/cleaned/embedded beforehand? (Perhaps td-idr or Doc2Vec, etc.)* How to use human labeling?<jupyter_code>clusterer = KMeans(n_clusters = 4, n_init = 20)
word_clusters = clusterer.fit(words_df)
(clusterer.labels_==0).mean()
(clusterer.labels_==1).mean()
(clusterer.labels_==2).mean()
(clusterer.labels_==3).mean()
words_df.loc[clusterer.labels_==0, :].sum().sort_values(ascending = False)[0:20]
words_df.loc[clusterer.labels_==1, :].sum().sort_values(ascending = False)[0:20]
words_df.loc[clusterer.labels_==2, :].sum().sort_values(ascending = False)[0:20]
words_df.loc[clusterer.labels_==3, :].sum().sort_values(ascending = False)[0:20]
praise_df.loc[clusterer.labels_==0, :]["IH per Praise"].describe()
praise_df.loc[clusterer.labels_==1, :]["IH per Praise"].describe()
praise_df.loc[clusterer.labels_==2, :]["IH per Praise"].describe()
praise_df.loc[clusterer.labels_==3, :]["IH per Praise"].describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12667205314660204, 'fraction_numerical': 0.008977466558937068, 'mean_word_length': 6.506738544474394, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 86, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '402481', 'license': 'MIT', 'n_tokens_mistral': 3733, 'n_tokens_neox': 3596, 'n_words': 1055, 'path': 'word-clustering.ipynb', 'repo_name': 'spacelover92/praiseanalysis', 'size': '11139'}