<jupyter_start><jupyter_text>This project explores PhD stipends across universities. Dataset can be found in [on the Kaggle website](https://www.kaggle.com/paultimothymooney/phd-stipends/data). The original data is from http://www.phdstipends.com/csv. A First Draft...<jupyter_code>import pandas as pd
import numpy as np
import io
from scipy.stats import zscore
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="ticks")<jupyter_output><empty_output><jupyter_text><jupyter_code>#read data file
file_URL = "https://drive.google.com/file/d/1lp09EJkJJkhKcU_67PSQ27rdunDB4NWK/view?usp=sharing"
data_path = 'https://drive.google.com/uc?export=download&id='+file_URL.split('/')[-2]

df = pd.read_csv(data_path)<jupyter_output><empty_output><jupyter_text>Get basic characteristics like size and summary statistics<jupyter_code>display(df.head())
display ("Size of Data:", df.shape)
display(df.info())

#not using comments column, so dropping
df.drop(columns = "Comments", inplace=True)
#helper functions

#get total annual pay: by adding summer pay to 9 months pay, where applicable
def calc_total_income(row):
  twelve_mth_pay = row["12 M Gross Pay"]
  if twelve_mth_pay==0:
    return row["9 M Gross Pay"]+ row["3 M Gross Pay"]
  else: return twelve_mth_pay<jupyter_output><empty_output><jupyter_text>From the describe method of the dataframe, we see most columns have many missing values. Also most of the data are in string format. We can address these issues as follows.<jupyter_code>#replace null values
df["University"].fillna("Unspecified", inplace = True)
df["Department"].fillna("Unspecified", inplace = True)

#get list of abbreviated departments
#display(df[(df["Department"].apply(len)<5)]["Department"].value_counts().head(3))
df.dropna(subset=["Overall Pay"], inplace=True)

#convert salaries to numeric 
money_columns = ["Overall Pay", "12 M Gross Pay", "9 M Gross Pay",\
                 "3 M Gross Pay", "Fees"] 

df[money_columns] = df[money_columns].apply(lambda x: x.str.replace(",", "")\
                                            .str.replace("$", "")\
                                            .astype("float")) 



#strip away "st", "nd" etc, assuming that program year is single digit
#if year is NaN replace with zero
df["Program Year"].fillna(0, inplace=True)
df["Program Year"] = df["Program Year"].apply(lambda x: str(x)[0])\
                        .astype("int")

#looking at output we see outliers 
df = df[(np.abs(zscore(df["Overall Pay"])) < 3)]

df["Overall Pay"].fillna(df["Overall Pay"].median(), inplace = True) 

#handle NaNs for academic year:very few, so can drop
df.dropna(subset=["Academic Year"], inplace=True)

#consider numeric columns
numeric_columns = df.select_dtypes('number').columns
#Most NaN are in non-applicable fields, so replacing with zero 
df[numeric_columns] = df[numeric_columns].fillna(0) 

#create a column with total 12 month gross income
df["Total Income"] = df.apply(calc_total_income, axis=1)

#set as categorical
cat_columns = df.select_dtypes('object').columns
df[cat_columns] = df[cat_columns].astype('category')

display(df.info())
#df.head() 
#get summary statistics for numeric columns
display(df.describe())
#Visualisations

plot_title = "Distribution of Overall Pay for All PhDs"
ax = sns.distplot(df["Overall Pay"], bins=10)
ax.set_title(plot_title) 
plt.show() 
display("------------------------------------------\n")
plot_title = "Box Plots of Pay Parameters for All PhDs"
ax=sns.boxplot(data=df) 
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
plt.tight_layout()
ax.set_title(plot_title) 
plt.show()

#ignore program year of zero (NaNs)

#top 10 universities for overall pay
display("top 10 universities for median overall pay")
grp_data = df.groupby("University")["Overall Pay"].agg([min, np.median, max])\
        .sort_values(by="median", ascending=False).reset_index().head(10)
display(grp_data)

display(" bottom 10 universities for median overall pay")
grp_data = df.groupby("University")["Overall Pay"].agg([min, np.median, max])\
        .sort_values(by="median", ascending=False).reset_index().tail(10)
display(grp_data)


display("top 10 departments for median overall pay")
grp_data = df.groupby("Department")["Overall Pay"].agg([min, np.median, max])\
        .sort_values(by="median", ascending=False).reset_index().head(10)
display(grp_data)

display("#bottom 10 departments for median overall pay")
grp_data = df.groupby("Department")["Overall Pay"].agg([min, np.median, max])\
        .sort_values(by="median", ascending=False).reset_index().tail(10)
display(grp_data)

plot_title = "Highest pay by Program Year"
grp_data = df.groupby("Program Year")[money_columns].max().reset_index()
ax = sns.scatterplot(x="Program Year", y="Overall Pay", data=grp_data)
#ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
ax.set_title(plot_title) 
plt.show()
display(grp_data)

plot_title = "Highest pay by Academic Year"
grp_data = df.groupby("Academic Year")[money_columns].max().reset_index()
ax = sns.scatterplot(x="Academic Year", y="Overall Pay", data=grp_data)
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
ax.set_title(plot_title) 
plt.show()
#TODO: some overall pay are negative (mostly because fees exceed pay)
#TODO: clustering<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14833615341229556, 'fraction_numerical': 0.01015228426395939, 'mean_word_length': 5.136101499423298, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 17, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '345304', 'license': 'MIT', 'n_tokens_mistral': 1830, 'n_tokens_neox': 1720, 'n_words': 448, 'path': 'phd_stipends.ipynb', 'repo_name': 'smayo2/BHousePricePredictor', 'size': '5319'}