<jupyter_start><jupyter_text>Problem StatementWhen it comes to income distribution and earnings across races and social backgrounds, most people will agree that there is a huge disparity and imbalance. In the tech industry, globalisation of work has removed market and location barriers to income and as such, background and racial identity should matter very little to earnings in our current society. In this notebook, I'm going to examine the relationship between race and family background of developers and their current level of income in the tech industry. In order to get useful insight into this relationship, I set out to answer the following questions: 1.  How does the annual income of developers vary between ethnic groups?2. Do student developers of different backgrounds also have widely different salary expectations from their peers?3. Is a developer likely to earn more if either of the parents graduated from college?  The DataI am making use of the    2017 Stack Overflow developer survey results because of the varying background of its participants from around the world and its focus wide range of standardised questions.<jupyter_code>#Now let's take a look at the survey results and the questions that generated them. 

#Import libraries
import numpy as np
import pandas as pd
import seaborn as sns
from collections import defaultdict
import matplotlib.pyplot as plt
import Diversity_RepresentationInTech as d
%matplotlib inline

df = pd.read_csv('survey_results_public.csv') #loading survey results
df.head()
#The data has 154 columns of different data types that could be useful in our analysis.
#But first, let's  get a bit more understanding of our data.
print(df.shape); #number of rows and columns
df.describe()<jupyter_output>(51392, 154)<jupyter_text>We have 51392 respondents answering the 154 questions. From the description above, it looks like we have 7 numeric columns  which include the Salary and Expected Salary. All the numeric columns above have missing values, but our analysis is simply a check of the average, no ML models are needed.<jupyter_code>df.hist(figsize = (9,5))
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Salary, ExpectedSalary and HoursPerWeek columns have non-symmetric right-skewed distribution (majority of participants are on the lower range of respective variables) while the other numerical columns are left-skewed (majority on the upper range of respective variables)<jupyter_code>#Next, let's  load and have a quick look at the schema that contains 
#the questions posed to respondents

df2 = pd.read_csv('survey_results_schema.csv')
df2.head()
#We can check for specific question that participants answered for confirmation in the 
#survey schema as follows
df2[df2['Column'] =='Race']['Question']<jupyter_output><empty_output><jupyter_text>Now that we have an understanding of our data, we can proceed to our first question and prepare the data for each analysis requirement.   1. How does the annual income of developers vary between ethnic groups?To answer this question, let's see the answers provided by participants to the question of Race.<jupyter_code>#Count the unique values of race entered by respondents
race_count_val = df['Race'].value_counts(); 
race_count_val 
#A plot of the race_count_val will quickly tell us that there's need to clean the results
(race_count_val/df.shape[0]).plot(kind = 'bar')<jupyter_output><empty_output><jupyter_text>The plot above gives no useful information. Hence, there is a need to clean up this data so we can have a better perspective of the race distribution of the survey participants on stackoverflow.<jupyter_code>possible_race = ['White or of European descent', 'South Asian', 'Hispanic or Latino/Latina', 
                 'East Asian', 'Middle Eastern','Native American, Pacific Islander, or Indigenous Australian',
                 'I prefer not to say','Black or of African descent','I don\'t know']

def clean_and_plot(df, title='Race of Respondents', plot=True):
    '''
    INPUT 
        df - a dataframe holding the Race column
        title - string the title of your plot
        plot - bool providing whether or not you want a plot back
        
    OUTPUT
        study_df - a dataframe with the count of how many individuals
        Displays a plot of pretty things related to the Race column.
    '''
    race_group = df['Race'].value_counts().reset_index()
    race_group.rename(columns={'index': 'method', 'Race': 'count'}, inplace=True)
    race_group_df = d.total_count(race_group, 'method', 'count', possible_race)

    race_group_df.set_index('method', inplace=True)
    if plot:
        (race_group_df/race_group_df.sum()).plot(kind='bar', legend=None,);
        plt.title(title);
        plt.show()
        plt.tight_layout()
    props_race_group = race_group_df/race_group_df.sum()
    return props_race_group
    
props_df = clean_and_plot(df)<jupyter_output><empty_output><jupyter_text>As seen, more than 70% of respondents to the Stackoverflow survey are of `White or of European descent`, about 13% of `Asian descent`, 2% of `Black or of African descent` and the lowest being from `Native American, Pacific Islander, or Indigenous Australian` at just 1% of the total. It is important we establish that the survey results is very likely not a true representation of the ethnic distribution in the tech industry as 80% of the survey respondents  are visitors to a stack exchange site. Also,only frequent visitors to stack overflow during the survey period were more likely to take the poll.  Thus, this analysis is simply taking a dive and presenting the results.  - Race vs SalaryThe participants provided data on their current or expected annual salary which Stack Overflow has helped to convert to US Dollars. If we're curious enough, how exactly does this annual income vary amongst the ethnic group represented? Is there a pattern we can immediately observe?<jupyter_code>#What question requested participants' salaries?
list(df2[df2['Column'] == 'Salary']['Question']) 
#The function below is also included in the python script
#It cleans up the race to give the items in possible_race list
def clean_race(race_item):
    possible_race = ['White or of European descent', 'South Asian', 'Hispanic or Latino/Latina', 
                 'East Asian', 'Middle Eastern','Native American, Pacific Islander, or Indigenous Australian',
                 'I prefer not to say','Black or of African descent','I don\'t know']
    '''
    INPUT
    race_item - line item of the dataframe column
    
    OUTPUT
    val - cleaned and streamlined data
    
    '''
    race_item = str(race_item)
    for val in possible_race:
        if val in race_item:
            return val 
        else:
            pass
df['Race'] = df['Race'].apply(d.clean_race)
#Let's group our df into races and then find average salary for each race
race_salary_grp = df.groupby('Race')['Salary'].mean().sort_values(ascending= False).round()
race_salary_grp.plot(kind = 'barh', figsize = (12,8), fontsize = 13).grid(axis = 'x');
plt.tight_layout() 
#Display values in each bar
for index, value in enumerate(race_salary_grp):
    plt.text(value/race_salary_grp[index], index-.10, str(value), color = 'white', fontweight = 'bold')
#Save plot
plt.savefig('Race_Salary.jpg')<jupyter_output><empty_output><jupyter_text>- Even though the Native American, Pacific Islander, or Indigenous Australian have the lowest number of respondents, they are the only group earning above $70,000 on average - if you're feeling lost like I am, you can also consider data like the Years of Experience, Developer Type, and some geographical factors to get more insight into why this is the case. - Another curious case is that of the South Asian respondents. Even though they are the second highest respondents in the survey, we see that they earn the least at just about $30, 000 annually. Does this tell us something about the general income level in South Asia or can we say that South Asians charge lower wages than everyone else in the global industry? - Whichever is the case, these findings are drawn from data collated from just a sample of the developer population in these Ethnic groups. The reality could be very different.   2. Do student developers of different backgrounds also have widely different salary expectations from their peers?Since we also have post-graduation Expected Salary for students that were learning how to code during the survey period, we can get a better perspective of how students of each ethnic group percieve the future worth of their skills.<jupyter_code>list(df2[df2['Column'] == 'ExpectedSalary']['Question'])
race_exp_salary_grp = df.groupby('Race')['ExpectedSalary'].mean().sort_values(ascending = False).round()
race_exp_salary_grp.plot(kind = 'barh', figsize = (12,8), fontsize = 13).grid(axis = 'x');
plt.tight_layout()
#Display values in each bar
for index, value in enumerate(race_exp_salary_grp):
    plt.text(value/race_exp_salary_grp[index], index-.10, str(value), color = 'white', fontweight = 'bold')
#Save plot
plt.savefig('Race_Expected_Salary.jpg')<jupyter_output><empty_output><jupyter_text>The student developers of White or of European descent and East Asian descents place a higher price on their skills relative to the rest of the population. These groups probably have a higher perception of their worth on the market, but it could be that they understand the going rate for developers in their locations. Surprisingly, the surveyed students with Native American, Pacific Islander, or Indigenous Australian descent projected the lowest annual income of all ethnic groups. Our earlier findings about Professional developers from the same ethnic group show that they earn the most compared to other ethnic group. _Could there be a disconnect between the two groups or the data provided by respondents is the problem?_Coming second-bottom on the expected salary list are the South Asians. The result is not surprising as it gives credence to the possibility of a generally lower income for South Asian developers compared with their counterparts.   3. Highest Education of Parents vs Earnings - Is a developer likely to earn more if either of the parents graduated from college?The common social perception is that people with parents with at least a Bachelor's degree tend to do better in life and have higher income levels compared to everyone else. Let's check if developers with parents with at least a bachelors degree tend to earn higher than their counterparts in the same ethnic group that are first generation students.<jupyter_code>list(df2[df2['Column'] == 'HighestEducationParents']['Question'])
df['HighestEducationParents'].unique() #Check the list of highest parents education available
#Encode the data into first-generation degree holders and those with degree-holder parents

def higher_ed(parents_ed_str):
    '''
    INPUT
        parents_ed_str - a string of one of the values from the Highest Education of parents column
    
    OUTPUT
        return 1 if the string is  in ("A bachelor's degree", "A professional degree", "A doctoral degree",
        "A master's degree")
        return 0 otherwise
    '''
    if parents_ed_str in ("A bachelor's degree", "A professional degree", "A doctoral degree", "A master's degree"):
        return 0
    else:
        return 1

#Test your function to assure it provides 1 and 0 values for a copy of our df
df_new = df.copy()
df_new["HighestEducationParents"].apply(higher_ed)[:5]
#Apply function to encode the column
df_new['HighestEducationParents'] = df_new["HighestEducationParents"].apply(higher_ed)
par_ed_1  = df_new[df_new['HighestEducationParents'] == 1] #Subset of dataframe that are first generation degree holders
par_ed_0 = df_new[df_new['HighestEducationParents'] == 0] #Subset of dataframe that are not first-generation degree holders

ed_1_mean = par_ed_1.groupby('Race')['Salary'].mean() #Average Salary of first_generation degree holders
ed_0_mean = par_ed_0.groupby('Race')['Salary'].mean() #Average Salary of non-first_generation degree holders

differ_ed_mean = ed_0_mean - ed_1_mean #Earning gap/Difference in salary of first-generation degree holders and non-first generation degree holders
perc_pay_gap = (differ_ed_mean/ed_1_mean) # Earning gap proportion
perc_pay_gap.plot(kind = 'barh', figsize = (14,10), fontsize = 13, grid = True);
plt.tight_layout()

plt.savefig('Race_Par_Salary_prop.jpg')
#plot title = 'Earning Gap if Parent\'s have a high degree and if they don\'t'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06851822377844978, 'fraction_numerical': 0.007878402037243356, 'mean_word_length': 4.751487414187643, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '119107', 'license': 'Unlicense', 'n_tokens_mistral': 3438, 'n_tokens_neox': 3222, 'n_words': 1569, 'path': 'EthnicDiversityRepresentationInTech.ipynb', 'repo_name': 'joshasgard/Diversity_RepresentationInTech', 'size': '12566'}