<jupyter_start><jupyter_text>Blog post on Stackoverflow Data This shows step by step the analysis that was conducted to address several questions on the 2017 stackoverflow survey data. Focus was on social and behavioral aspects if you will:What makes people update their resumes? Are StackOverflow users more inclined to be the "challenge myself" type or "compete with peers" type?How's the distribution of people who like/dislike working with other people's code and can we predict wether people like/dislike working with other people's code?<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import OneHotEncoder
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read CSV Files<jupyter_code>df=pd.read_csv("so-survey-2017/survey_results_public.csv")
df_schema=pd.read_csv("so-survey-2017/survey_results_schema.csv")<jupyter_output><empty_output><jupyter_text>Data understanding<jupyter_code>df.info()
df.head()<jupyter_output><empty_output><jupyter_text>Check Columns that we want to use for further analysis: OtherPeoplesCode, ChallengeMyself, CompetePeers, ResumePrompted, SelfTaughtType, EnjoyDebugging, DifficultCommunication, ShipIt, ProgramHobby, Professional, FormalEducation, HomeRemote<jupyter_code>df.ResumePrompted.unique()<jupyter_output><empty_output><jupyter_text>define function for calculating the percentage of missing values<jupyter_code>def missing_values(column):
    """returns the percentage of missing values for the provided column"""
    return (df[column].isnull().sum()/df.shape[0])*100
#percentage of missing values
print(missing_values("ResumePrompted"))<jupyter_output>56.294754047322535<jupyter_text>We have about 56% missing values in this column define function for building bar chart of value distribution<jupyter_code>def value_dist(column):
    """produces a bar chart with value distribution of the provided column"""
    (df[column].value_counts()/df.shape[0]).plot(kind="bar")
    s_title=f"Distribution of values in {column} column"
    plt.title(s_title)
    return

#check distribution of values
value_dist("ResumePrompted")<jupyter_output><empty_output><jupyter_text>We can see that the single most selected option (about 13%) was that people update their resume on a regular basis. Less than 1% update their resume due to receiving negative feedback.<jupyter_code>df.OtherPeoplesCode.unique()
#percentage of missing values
print(missing_values("OtherPeoplesCode"))<jupyter_output><empty_output><jupyter_text>We have about 42% missing values in this column<jupyter_code>#check distribution of values
value_dist("OtherPeoplesCode")<jupyter_output><empty_output><jupyter_text>In this case, the question was if it is torture to work with other peoples code<jupyter_code>df.ChallengeMyself.unique()
#percentage of missing values
print(missing_values("ChallengeMyself"))<jupyter_output>39.73575653798257<jupyter_text>We have about 40% missing values in this column<jupyter_code>#check distribution of values
value_dist("ChallengeMyself")
df.CompetePeers.unique()
#percentage of missing values
print(missing_values("CompetePeers"))<jupyter_output><empty_output><jupyter_text>We have about 40% missing values in this columnLet's look at the distribution.<jupyter_code>#check distribution of values
value_dist("CompetePeers")
print(missing_values("SelfTaughtTypes"))
df.SelfTaughtTypes.unique()
df.SelfTaughtTypes.nunique()<jupyter_output><empty_output><jupyter_text>=> Will split into different columns of individual attributes later. It is hard to work with it like this. Examine EnjoyDebugging column, percentage of missing values first, then distribution of values<jupyter_code>print(missing_values("EnjoyDebugging"))
#check distribution of values
value_dist("EnjoyDebugging")<jupyter_output><empty_output><jupyter_text>Examine DifficultCommunication column, percentage of missing values first<jupyter_code>print(missing_values("DifficultCommunication"))
#check distribution of values
value_dist("DifficultCommunication")<jupyter_output><empty_output><jupyter_text>Examine ShipIt column, percentage of missing values first<jupyter_code>print(missing_values("ShipIt"))
#check distribution of values
value_dist("ShipIt")<jupyter_output><empty_output><jupyter_text>Examine ProgramHobby column, percentage of missing values first<jupyter_code>print(missing_values("ProgramHobby"))
#check distribution of values
value_dist("ProgramHobby")<jupyter_output><empty_output><jupyter_text>Examine Professional column, percentage of missing values first<jupyter_code>print(missing_values("Professional"))
#check distribution of values
value_dist("Professional")<jupyter_output><empty_output><jupyter_text>Examine FormalEducation column, percentage of missing values first<jupyter_code>print(missing_values("FormalEducation"))
#check distribution of values
value_dist("FormalEducation")<jupyter_output><empty_output><jupyter_text>Examine HomeRemote column, percentage of missing values first<jupyter_code>print(missing_values("HomeRemote"))
#check distribution of values
value_dist("HomeRemote")<jupyter_output><empty_output><jupyter_text>This concludes the examination of the columns we want to use for prediction later. None of them seems to have a very high number of missing values. So I will try to drop rows with null values instead of imputing values. Will impute if we end up with too few rows! However we will have to work on the selftaughttypes column.  Data preperation Splitting selftaughttypes into individual strings, based on ; separator<jupyter_code>df.SelfTaughtTypeslist=df.SelfTaughtTypes.str.split(";")
df.SelfTaughtTypeslist.apply(pd.Series).stack().value_counts()<jupyter_output><empty_output><jupyter_text>introduce new columns instead of selftaughttypes - depending on the individual values<jupyter_code>#put all possible values in array, not considering spaces
possible_vals = ["Official documentation", "Stack Overflow Q&A", 
                 "Other", "Trade book", "Textbook", "Stack Overflow Docs",
                 "Non-Stack online communities", "Company internal community", "Friends network",
                 "Built-in help", "Tutoring/mentoring"]
#dropping na and resetting index
df=df.dropna(subset=["SelfTaughtTypes"]).reset_index()
df=df.reindex(df.columns.tolist() + possible_vals, axis=1)
for val in possible_vals:
    df[val]=df["SelfTaughtTypes"].apply(lambda x: 1 if val in x else 0)<jupyter_output><empty_output><jupyter_text>See if it worked<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>define dataframe for predicting and see how many rows are left after dropping NA<jupyter_code>#define columns that will be taken into account for predicting otherpeoplescode column
df_pred=df[["HomeRemote","FormalEducation","Professional","ProgramHobby","OtherPeoplesCode","EnjoyDebugging","ShipIt","DifficultCommunication","Official documentation", "Stack Overflow Q&A", 
                 "Other", "Trade book", "Textbook", "Stack Overflow Docs",
                 "Non-Stack online communities", "Company internal community", "Friends network",
                 "Built-in help", "Tutoring/mentoring"]]

#columns do not seem to be terribly well suited for imputing: drop na rows
df_pred=df_pred.dropna()
df_pred.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 15712 entries, 0 to 26731
Data columns (total 19 columns):
HomeRemote                      15712 non-null object
FormalEducation                 15712 non-null object
Professional                    15712 non-null object
ProgramHobby                    15712 non-null object
OtherPeoplesCode                15712 non-null object
EnjoyDebugging                  15712 non-null object
ShipIt                          15712 non-null object
DifficultCommunication          15712 non-null object
Official documentation          15712 non-null int64
Stack Overflow Q&A              15712 non-null int64
Other                           15712 non-null int64
Trade book                      15712 non-null int64
Textbook                        15712 non-null int64
Stack Overflow Docs             15712 non-null int64
Non-Stack online communities    15712 non-null int64
Company internal community      15712 non-null int64
Friends network                 157[...]<jupyter_text>With 15,7k rows to work with, we can try RF and don't need to impute values<jupyter_code>y=df_pred.OtherPeoplesCode
X=df_pred.drop(labels="OtherPeoplesCode",axis=1)
#getdummy columns for categorical columns
for col in ["HomeRemote","FormalEducation","Professional","ProgramHobby","EnjoyDebugging","ShipIt","DifficultCommunication"]:
        X=pd.concat([X.drop(col,axis=1),pd.get_dummies(X[col],prefix=col,prefix_sep="_",drop_first=True)],axis=1)
X.head()<jupyter_output><empty_output><jupyter_text>Modeling<jupyter_code>#split into train and test data
X_train, X_test, y_train, y_test = train_test_split(X, y , test_size=.30, random_state=42)

#def classifier
rf_model=RandomForestClassifier(max_depth=10, n_estimators=100,min_samples_split=2, random_state=42)
#fit model
rf_model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>#check testscore
print(rf_model.score(X_test,y_test))
#check trainscore
rf_model.score(X_train, y_train)
#check what features/columns are most important
df_feature_importance=pd.DataFrame(rf_model.feature_importances_, index=X_train.columns,
                                    columns=['importance']).sort_values('importance',ascending=False)
print(df_feature_importance)<jupyter_output>importance
EnjoyDebugging_Strongly disagree                      0.037834
DifficultCommunication_Disagree                       0.036802
EnjoyDebugging_Disagree                               0.035819
Built-in help                                         0.035664
Trade book                                            0.035037
DifficultCommunication_Strongly disagree              0.032838
Non-Stack online communities                          0.032579
Stack Overflow Docs                                   0.032444
Textbook                                              0.032351
ShipIt_Disagree                                       0.030802
Friends network                                       0.029695
Stack Overflow Q&A                                    0.028885
Official documentation                                0.028815
ProgramHobby_Yes, I program as a hobby                0.028663
ProgramHobby_Yes, both                                0[...]<jupyter_text>recheck distribution of otherpeoplescode column after deleting rows with na columns<jupyter_code>(df_pred.OtherPeoplesCode.value_counts()/df_pred.shape[0]).plot(kind="bar")
plt.title("Working with other people's code is torture, overview");
df_pred.OtherPeoplesCode.value_counts()/df_pred.shape[0]<jupyter_output><empty_output><jupyter_text>It turns out the random forest model is only marginally better than just predicting "somewhat agree" for everything.  How do "Competing against peers" and "challenge myself" attirbutes correlate?Creating Heatmap of CompetePeers and ChallengeMyself data to see if there are generally more competetive and less competitive people (towards themselves and other people)<jupyter_code>df_heatmap=df[["CompetePeers","ChallengeMyself"]]
df_heatmap=df_heatmap.dropna(axis=0,how="any")
piv=df.pivot_table(index=df_heatmap["CompetePeers"],columns=df_heatmap["ChallengeMyself"],aggfunc='size',fill_value=0)
df.pivot_table(index=df_heatmap["CompetePeers"],columns=df_heatmap["ChallengeMyself"],aggfunc='size',fill_value=0)
piv=piv[["Strongly agree","Agree","Somewhat agree","Disagree","Strongly disagree"]]
piv=piv.reindex(["Strongly disagree","Disagree","Somewhat agree","Agree","Strongly agree"])

piv=piv/np.sum(piv.sum())
sns.heatmap(data=piv, annot=True, fmt=".2g")
plt.title("Percentage of people for each combination");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0869673148531237, 'fraction_numerical': 0.024906909391808025, 'mean_word_length': 5.076420311714429, 'pattern_counts': {'":': 0, '<': 106, '<?xml version=': 0, '>': 107, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '574935', 'license': 'CNRI-Python', 'n_tokens_mistral': 3780, 'n_tokens_neox': 3475, 'n_words': 912, 'path': 'Stackoverflow_blogpost.ipynb', 'repo_name': 'a-boerd/uda_stackoverflow', 'size': '12085'}