<jupyter_start><jupyter_text>The first part of this notebook loads the data. It was done by @ygg_anderson<jupyter_code>import panel as pn
pn.extension()
import pandas as pd
import numpy as np
#import hvplot.pandas
import param, random
import datetime as dt

import matplotlib.pyplot as plt

%matplotlib inline
periods = [
    "#17 May 7",
    "#16 Apr 24",
    "#15 Apr 9",
    "#14 Mar 26",
    "#13 Mar 12",
    "#12 Feb 26",
    "#11 Feb 12",
    "#10 Jan 29",
    "#9 Jan 15", 
    "#8 Jan 1",
    "#7 Dec 18",
    "#6 Dec 4",
    "#5 Nov 20", #
    "#4 Nov 6", #
    "#3 Oct 23", #
    "#2 Oct 9",
    "#1 Sept 24", #
    "#0 Sept 7 (historic)", #
]
data = []
for i, period in enumerate(periods):
    
    if i not in (17, 16, 14, 12, 13):
    
        df = pd.read_excel('data/TEC Praise Quantification.xlsx', skiprows=2, sheet_name=period,engine='openpyxl', usecols="A:M")

        df[['v1','v2','v3']] = list(df.columns[6:9])

        df.columns = list(df.columns[:6]) + ['v1 norm', 'v2 norm', 'v3 norm'] + list(df.columns[9:])

        df['period'] = period

        df = df.dropna(thresh=8)

        data.append(df)
combined_data = pd.concat(data)
combined_data<jupyter_output><empty_output><jupyter_text>quantifiers = combined_data[combined_data[['IH per Praise', 'IH per person', 'Unnamed: 12']].isna().all(axis=1)]quantifiers<jupyter_code>receivers = combined_data[~combined_data[['IH per Praise', 'IH per person', 'Unnamed: 12']].isna().all(axis=1)]
receivers<jupyter_output><empty_output><jupyter_text>--- Investigations from octopus🐙 I'm going to do some basic analysis of the parts of the data set that are encoded with words.<jupyter_code>import seaborn as sns #seaborn is my plotting tool of choice
receivers.columns<jupyter_output><empty_output><jupyter_text>Where does praise happen?<jupyter_code>sources = receivers.groupby("Unnamed: 3").count()
sources

ax = sns.barplot(y = sources.index, x = sources["To"],  order = sources.sort_values("To", ascending = False).index)
ax.set(xlabel="Count", ylabel="Source", title = "Where Praise is Given")
room_sources = receivers.groupby("Room").count()
room_sources

plt.figure(figsize=(10,8))
ax = sns.barplot(y = room_sources.index, x = room_sources["To"],  order = room_sources.sort_values("To",                                                                                                 ascending = False).index)
ax.set(xlabel="Count", ylabel="Source", title = "Where Praise is Given")<jupyter_output><empty_output><jupyter_text>Do some basic cleaning and analysis.  Now I'm going to create a new data frame that incorporates how many times each user gave and received praise. The next few cells accomplish this by using pivot_tables to create data frames which are then merged, with missing values replaced by 0.<jupyter_code>praise_to = receivers.pivot_table(index = ["To"], aggfunc = 'size' ).to_frame(name = "to")
praise_to
praise_from = receivers.pivot_table(index = ["From"], aggfunc = 'size').to_frame(name = "from")
praise_from
praise_to_and_from = pd.concat([praise_from, praise_to], axis = 1)
praise_to_and_from = praise_to_and_from.fillna(0)
praise_to_and_from.head(5)<jupyter_output><empty_output><jupyter_text>A naming issue in our data  So now we have **praise_to_and_from** as a data frame where each row is a user, and we see how many times they gave oraise (measured in **from**) and received praise (measured in **to**). There is a **naming** issue that should be addressed at some point. Look below.<jupyter_code>zep_df = praise_to_and_from.filter(like = "zep", axis = 0)
zep_df
ygg_df = praise_to_and_from.filter(like = "ygg", axis = 0)
ygg_df<jupyter_output><empty_output><jupyter_text>The issue is that some users receive praise with variations on their names. It would be good to consoliate these users for this analysis. However, this is an issue that's unlikely to affect most users, so we wll come back to it.   Relationship Between Givers and Receivers Let's make a scatterplot to see if we think there is a relationship betwen these two quantities.<jupyter_code>ax = sns.scatterplot(x = "from", y = "to", data = praise_to_and_from)
ax.set(title = "Praise: To and From")<jupyter_output><empty_output><jupyter_text>There doen't apear to be much of a relationship, primarily because **so many users** have zero values of **from**; this includes many users who are frequent praise recipients but do not give much. This is partially distorted by the naming issues above.<jupyter_code>sns.histplot(data = praise_to_and_from, x = "to")
sns.histplot(data = praise_to_and_from.query("to > 0"), x = "to")
sum(praise_to_and_from["to"] <= 1)/len(praise_to_and_from)
praise_to_and_from["from"] == 0
sum(praise_to_and_from["from"] == 0)/len(praise_to_and_from)
praise_to_and_from[praise_to_and_from["from"] > 0]["from"].mean()<jupyter_output><empty_output><jupyter_text>Almost 80% of users here have not given any praise (this might be skewed *slightly* by the naming issue above, but probably not much). This is probably described by a Pareto distribution -- with 20% of the users giving 100% of the praise. Among users who gave some praise, they gave an average of roughly 90 praises.<jupyter_code>praise_to_and_from.describe()
praise_to_and_from["generosity"] = praise_to_and_from["from"]/(praise_to_and_from["to"] + 1)
praise_to_and_from
praise_to_and_from.query("generosity > 0").query("generosity < 1")<jupyter_output><empty_output><jupyter_text>Investigations from courier<jupyter_code># Run this cell to install the pygraphviz library
#!apt-get install python3-dev graphviz libgraphviz-dev pkg-config
#!pip install pygraphviz
import networkx as nx
import pygraphviz<jupyter_output><empty_output><jupyter_text>We will attempt to make a graph from the dataframe. First we will begin with a simple proof of concept.<jupyter_code>df = pd.read_csv("recievers.csv")
df.head()
df.columns
G = nx.DiGraph()
edges = df[['To','From']]


for ind, tup in edges[1:2].iterrows():
  to = tup[0]
  fro = tup[1]
  G.add_edge(fro, to)

  
G.edges
G = nx.nx_agraph.to_agraph(G)
G.layout(prog="dot")
G.draw("file.png") 
edges[1:2]<jupyter_output><empty_output><jupyter_text>![](https://drive.google.com/uc?export=view&id=1u9YuyqRxjnbXfDNA3ywuxNTI4CPWo54h) Now we create a function that will automate this process.<jupyter_code>def viz_graph(df, file_name):
  """
  Creates a visualization of a graph created from a dataset of praise. 

  :param df: The dataframe that contains who the praise is from and who it is to.
  Note that this dataframe should ONLY contain columns To and From.
  :param file_name: The name of the image that will be saved to your computer. 
  """

  G = nx.DiGraph()

  for ind, tup in df.iterrows():
    to = tup[0]
    fro = tup[1]
    G.add_edge(fro, to)

  G = nx.nx_agraph.to_agraph(G)
  G.layout(prog="dot")
  G.draw(file_name) 
  return G
#viz_graph(edges, "praise_graph.png")<jupyter_output><empty_output><jupyter_text>You can use boolean conditions to get more specific graphs.<jupyter_code>#viz_graph(edges[edges["To"] == "zeptimusQ"], "zeptimus_praise_graph.png")
def to_user_graph(df, user, file_name):
  """
  Creates a visualization of a graph created from a dataset of praise. 

  :param df: The dataframe that contains who the praise is from and who it is to.
  Note that this dataframe should ONLY contain columns To and From.
  :param user: All donations to this user will be graphed. 
  :param file_name: The name of the image that will be saved to your computer. 
  """
  df = pd.read_csv("recievers.csv")
  G = nx.DiGraph()

  for ind, tup in edges[edges["To"] == user].iterrows():
    to = tup[0]
    fro = tup[1]
    G.add_edge(fro, to)

  G = nx.nx_agraph.to_agraph(G)
  G.layout(prog="dot")
  G.draw(file_name) 
  return G
def from_user_graph(df, user, file_name):
  """
  Creates a visualization of a graph created from a dataset of praise. 

  :param df: The dataframe that contains who the praise is from and who it is to.
  Note that this dataframe should ONLY contain columns To and From.
  :param user: All donations from this user will be graphed. 
  :param file_name: The name of the image that will be saved to your computer. 
  """
  df = pd.read_csv("recievers.csv")
  G = nx.DiGraph()

  for ind, tup in edges[edges["To"] == user].iterrows():
    to = tup[0]
    fro = tup[1]
    G.add_edge(to, fro)

  G = nx.nx_agraph.to_agraph(G)
  G.layout(prog="dot")
  G.draw(file_name) 
  return G<jupyter_output><empty_output><jupyter_text>You can utilize these 2 forms to create user graphs. The first one will create a graph of donations from the provided user and the second form will graph donations to the provided user.<jupyter_code>df_str = "recievers.csv" #@param {type:"string"}
user = "zeptimusQ" #@param {type:"string"}
file_name = "zeptimusQ_to_graph.png" #@param {type:"string"}
from_user_graph(df=df_str, user=user,file_name=file_name)
df_str = "recievers.csv" #@param {type:"string"}
user = "recievers.csv" #@param {type:"string"}
file_name = "zeptimusQ_from_graph.png" #@param {type:"string"}
to_user_graph(df=df_str, user=user,file_name=file_name)

df.columns
df.isna().sum()<jupyter_output><empty_output><jupyter_text>We observe many null values in the final 3 columns. We will drop these for now.<jupyter_code>df.drop(['Cred per Praise', 'Cred per person', 'To.1'], inplace=True, axis=1)<jupyter_output><empty_output><jupyter_text>Date inconsistencyIt appears that the format of the dates in the data is inconsistent. Some entries opt for the format Month-Day-Year while others use Year-Month-Day.<jupyter_code>df['Date']
df["Date"].tail()
df["Date"].head()<jupyter_output><empty_output><jupyter_text>We will drop any NA dates.<jupyter_code>df["Date"] = df["Date"].dropna()
from dateutil import parser<jupyter_output><empty_output><jupyter_text>First we begin by turning all of the dates in the column into strings.<jupyter_code>df["Date"] = df["Date"].apply(str)<jupyter_output><empty_output><jupyter_text>Some other dates just have the word "dup" instead of a date. We can get rid of these.<jupyter_code>df[df["Date"] == "dup"]
df = df[df["Date"] != "dup"]
df = df[df["Date"] != "nan"]
def normalize_dates(date_str):
  """
  This function takes in a string that contains a date and converts it to the 
  format Year-Month-Day.
  
  :param date_str: The string containing the date. 
  :return: The date in the format Year-Month-Day.
  """
  d = parser.parse(date_str)
  return d.strftime("%Y-%m-%d")<jupyter_output><empty_output><jupyter_text>A quick sanity check to ensure that we have the correct results after this function was applied.<jupyter_code>df["Date"].apply(normalize_dates)
df["Date"]
df["Date"] = pd.to_datetime(df["Date"])<jupyter_output><empty_output><jupyter_text>Now we add features for the year month and day.<jupyter_code>df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month
df["Day"] = df["Date"].dt.day<jupyter_output><empty_output><jupyter_text>Now we can group the data based on these features.<jupyter_code>days = df.groupby(['Day']).To.nunique().reset_index()
days<jupyter_output><empty_output><jupyter_text>Consider a plot of the number of praises given broken down by day of the month. We see that the most praise is given on the 5th day of the month.<jupyter_code>fig, ax = plt.subplots()
ax.bar(days["Day"], days["To"], color='k')
ax.set_ylim(0, days["To"].max()+5)
fig.autofmt_xdate()
plt.xlabel('Day of Month')
plt.ylabel('Counts')
plt.show()

months = df.groupby(['Month']).To.nunique().reset_index()<jupyter_output><empty_output><jupyter_text>Consider a plot of the number of praises given broken down by the month. We see that the most praise is given in April. Interestingly, no praise at all is given between the months of June-October.<jupyter_code>fig, ax = plt.subplots()
ax.bar(months["Month"], months["To"], color='k')
ax.set_ylim(0, months["To"].max()+5)
fig.autofmt_xdate()
plt.xlabel('Month')
plt.ylabel('Counts')
plt.show()
dates = df.groupby(['Date']).To.nunique().reset_index()


fig, ax = plt.subplots()
ax.plot_date(dates["Date"], dates["To"], color='k', xdate=True)
ax.set_ylim(0, dates["To"].max()+5)
fig.autofmt_xdate()
plt.xlabel('Date')
plt.ylabel('Counts')
plt.show()
sns.lineplot(x="Date", y="To", data=dates)
dates.sort_values(by=['To'], ascending=False)
df.to_csv("processed_praise.csv")<jupyter_output><empty_output><jupyter_text><jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13074005326446614, 'fraction_numerical': 0.011298523121620531, 'mean_word_length': 5.086444007858546, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 104, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '12241', 'license': 'MIT', 'n_tokens_mistral': 4473, 'n_tokens_neox': 4287, 'n_words': 1251, 'path': 'praise_to_and_from.ipynb', 'repo_name': 'jdclifton2/praiseanalysis', 'size': '12391'}