<jupyter_start><jupyter_text>PROJECT - CRISP_DM Cross-Industry Standard Process for Data MiningFirst Udacity Project* This project is divided in 3 process: *Business Understanding, Data Understanding and Data Preparation.*  Business UnderstandingThe goal of this project is to analyse and explore the dataset in order to answer the following questions: Questions 1- Which countries have better salaries ? 2- Which gender has the highest salary ? 3- Do those who work from home earn more than those who don't ? 4- Do the larger companies have more employees who earn more ? 5- What race has the highest salary ?   Data Understanding<jupyter_code># Imports

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
# Load the data

df = pd.read_csv('data.csv')
# Visualizing the first 5 lines

df.head()
# The shape of the data

print(f'Dataframe has {df.shape[0]} lines and {df.shape[1]} columns')
# Amount of NA's for each column
df.isna().sum()
# Which column has more than 90% of missing values?

cols = df.loc[:,df.isna().mean() > .90].columns
df.loc[:,df.isna().mean() > .90].columns<jupyter_output><empty_output><jupyter_text>Dropping the columns* I'm gonna drop the columns, because there's just a little of information for each column, and if I impute 90% of the missing values I would affect too much the dataset with values the are probably wrong. However, dropping the columns also affect the dataset, but I have to make a decision, and I'll drop them.* If I had chosen to impute the values with the mean, I'd have 90% of the same values and therefore no variance in the column.* The first approach is to find out why we have missing values, maybe the people who in the business section could help us with this information* I'm choosing the threshold of 90% because in my opinion if the column has less than this, and if its information is useful, I consider impute the values or create a dummy binary variable representing 0 for Non NAN's and 1 for NAN's values for that column, which can be great for modeling the data (it's not my case).<jupyter_code># Let's drop these columns

# Subsetting the cols to be removed
cols = df.loc[:,df.isna().mean() > .90].columns

# Dropping them
df.drop(columns=cols, inplace=True)
# Visualize

df.head()
# Let's use only the necessary columns to answer the questions

# Subsetting the columns
cols = ['Country', 'Salary', 'Gender', 'CompanySize', 'HomeRemote', 'Race', 'ProgramHobby']

# Creating a new dataframe with only the columns above
new_df = df[cols]
# View the first 5 line of our new dataset

new_df.head()
# Types of features
new_df.dtypes
# Checking the amount of missing values

new_df.isna().sum()<jupyter_output><empty_output><jupyter_text>Filling missing values:Filling missing values with the mean for float type columns, in this case we only have one: **salary**.Another approach when we have missing data, is to impute them with some statistical value, for instance, here I have one numeric column (salary), and because it has less than 90% of the values missing, and it's an important variable, in my opinion the best approach is to fill the values with mean, and then I don't lose the rest of the data.<jupyter_code># Filling the missing values with the mean for salary column.

new_df['Salary'].fillna(np.mean(new_df.Salary), inplace=True)
# Visualize the data

new_df.head()<jupyter_output><empty_output><jupyter_text>Now, let's fill the missing values for categorical variables, in this case with the mode, because it's almost the only statistical measure that can be taken from categorical variable, we can not take the mean ou median for example, and I'm not gonna drop the columns because I already dropped those that had 90% or more missing values<jupyter_code># filling the missing values

# Function to get the mode and fill missing value
mode = lambda x: x.fillna(pd.Series.mode(x)[0])

# Filtering only categorical columns
cat_cols = new_df.select_dtypes('object').columns

# Using method apply to use our function in each column and visualizing it
new_df[cat_cols].apply(mode)
# Assign the new imputed values to dataframe
new_df.loc[:,cat_cols] = new_df.loc[:,cat_cols].apply(mode)
# Visualizing again
new_df.head()
# Checking the shape
new_df.shape
# Checking the amount of NA's
new_df.isna().sum()<jupyter_output><empty_output><jupyter_text>Now we have a clean dataframe, therefore we are able to analysing the data and answer the questions asked in the beggining of the project   Data Preparation  First Question: Which countries have better salaries ?<jupyter_code># First Question: Better country have better salaries ?
df1 = new_df.groupby('Country')['Salary'].agg(['mean', 'std', 'min', 'max'])\
                                         .sort_values(ascending=False, by='mean').reset_index().head(10)
df1
# Creating new column with the number of time that each country showed up in the dataset
df1 = df1.sort_values(by='Country')
df1['Qtd'] = new_df.Country[new_df.Country.isin(df1['Country'].values)].value_counts().sort_index().values
# Visualizing, high values have strong colors
df1.style.background_gradient(cmap='GnBu')<jupyter_output><empty_output><jupyter_text>We can notice from the frame above that EUA has appeared most time in the dataset, and it has the max salary. Although Bermuda has the highest Salary mean, it has only observations in the dataset, so we can not conlude so much about this country.<jupyter_code># Let's take a look for only those country that appeared 100 or more times
# Visualizing the max values for each variable

df1[df1.Qtd >= 100].style.highlight_max()
# A bar graph for the mean

df1[df1.Qtd >= 100].style.bar(subset='mean', align='mid', color='#5fba7d')<jupyter_output><empty_output><jupyter_text>Those are the countries with highest salaries.  Second Question: Which gender has the highest salary ?<jupyter_code># Visualizing the amount of each differente gender
new_df.Gender.value_counts()
# Amount of unique genders
new_df.Gender.nunique()
# Let's focus on the gender Male, Female, Transgender, the most common ones.

genders = ['Male', 'Female', 'Transgender']
# Subsetting dataset
df2 = new_df[new_df.Gender.isin(genders)]
df2.head()
# Informations about salary for each gender
df2.groupby('Gender')['Salary'].agg(['mean','std','max']).style.highlight_max(subset='mean')<jupyter_output><empty_output><jupyter_text>Besides we have only 20 transgenders in our dataset, they have the highest mean salary.Let's check more information about transgenders.<jupyter_code># Subsetting dataframe with only transgenders
trans = new_df[new_df.Gender == 'Transgender']
trans.head()
# We can see that the mean imputation wasn't a good ideia in this case
# we have only 2 differents values from that salary mean that we imputed
# so we can not conclude that transgenders have the highest salary mean
trans.Salary.value_counts()
# Let's check the mean for the genders male and female
new_df.loc[new_df.Gender.isin(['Male','Female'])].groupby('Gender')['Salary'].mean()\
                        .reset_index().style.highlight_max(color='lightgreen')<jupyter_output><empty_output><jupyter_text>There's a slight difference between the salary mean for men and women  Third Question: Do those who work from home earn more than those who don't ?<jupyter_code># Visualize
new_df.head()
# Filtering the dataset
subset = new_df[['HomeRemote', 'Salary']]
subset.head()
# Frequency table
print("How often people work from home\n")
print(subset.HomeRemote.value_counts(),'\n')
# Frequency table
print("How often people work from home %\n")
print(subset.HomeRemote.value_counts(normalize=True),'\n')<jupyter_output>How often people work from home %

A few days each month                                      0.445346
Never                                                      0.276830
All or almost all the time (I'm full-time remote)          0.100618
Less than half the time, but at least one day each week    0.076641
More than half, but not all, the time                      0.035389
It's complicated                                           0.033138
About half the time                                        0.032039
Name: HomeRemote, dtype: float64<jupyter_text>Creating a function to plot a bar graph<jupyter_code># Function
def bar_plot(series, rotation = 90, figure_size = (10,6)):
    '''
    Description: This function can be used to plot a bar graph

    Arguments:
        series: a series object type; 
        rotation: rotation of xlabels, default = 90;
        figure_size: a tuple representing the size of the graph, default = (10,6);

    Returns:
        A bar graph 
    '''
   
    # Graph size
    plt.figure(figsize=(figure_size))
    
    # Plotting the graph
    series.plot.bar()
    
    # The rotation of Xlabels
    plt.xticks(rotation = rotation)
# Let's check the mean salary for each kind of value in HomeRemote column, using the function created above

bar_plot(series = subset.groupby("HomeRemote")['Salary'].mean().sort_values(ascending=False), rotation = 60)<jupyter_output><empty_output><jupyter_text>The tables don't seem to show any evidence that most of people who work remotely has a higher salary.  Fourth Question: Do the larger companies have more employees who earn more ?<jupyter_code># Visualize
new_df.head()
# Frequency table for Company size
new_df.CompanySize.value_counts()<jupyter_output><empty_output><jupyter_text>Now I'll apply a function to classify if the company is big or small, I'm doing this because I want to see the differences between big and small comapanies in a more general scenario, therefore I don't wanna analyze a company with a speficic number of employees, instead I'm putting a threshold of 500 employees, more than that it's big, less it's small.<jupyter_code># Creating a dictionary, I'll use it later to change the names of categorical data
mapper = {'20 to 99 employees':'small', '100 to 499 employees':'small', '10,000 or more employees':'big',
          '10 to 19 employees':'small', '1,000 to 4,999 employees':'big', 'Fewer than 10 employees':'small',
          '500 to 999 employees':'big', '5,000 to 9,999 employees':'big'}
# Dropping answers that I'll not use, because they have no information: I don't Know and I prefer not to answer.

idx = new_df[(new_df.CompanySize == "I don't know") | (new_df.CompanySize == "I prefer not to answer")].index
subset = new_df.drop(index=idx)
# Apply the mapper
subset.CompanySize.replace(mapper, inplace=True)
# View
subset.head()
# Amount of employees who work at small and big companies
subset.CompanySize.value_counts()
# Average salary for big and small companies
bar_plot(subset.groupby('CompanySize')['Salary'].mean(), rotation = 60)<jupyter_output><empty_output><jupyter_text>There's just a little difference between the people working from home from big companies and small companies  Fifth Question: What race has the highest salary ?<jupyter_code># View

new_df.head()
# Frequency table for race
new_df.Race.value_counts()
# Amount of unique races

new_df.Race.nunique()
# Let's work with only the first 3 races

races = list(new_df.Race.value_counts().index[:3])
# Subsetting

subset = new_df.loc[new_df.Race.isin(races),:]
# Subset dataframe

subset.head()
# Frequency table again

subset.Race.value_counts()
# Salary mean for each race in our subset

bar_plot(subset.groupby('Race')['Salary'].mean(), rotation=45)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08962264150943396, 'fraction_numerical': 0.015373864430468204, 'mean_word_length': 4.538945331398161, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '100026', 'license': 'MIT', 'n_tokens_mistral': 3475, 'n_tokens_neox': 3238, 'n_words': 1374, 'path': 'Project-CRISP.ipynb', 'repo_name': 'DenerBEM/Udacity-CRISP', 'size': '11448'}