<jupyter_start><jupyter_text>The world of developers   Section 1: Business Understanding  Stackoverflow’s 2017 Annual Developer Survey aims to understand various aspects of jobs that relate to data analysis and software development. It has around 64,000 records spread over about 213 countries and regions. Based on the survey, we want to predict job satisfaction through a sample, in addition to answering the following business questions:  Question 1:  How is job satisfaction related to earned salary and hours worked per week for professional developers?  Question 2: On average, how much do professional developers earn and what size of company do they work most?   Question 3: On average, how is job satisfaction and salary according to the importance of each person’s education?   Question 4: How is the job satisfaction ratio for professional developers who can work at home?   Import libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Section 2: Data Understanding   Import Data - Stack Overflow Data - 2017 Survey<jupyter_code>df = pd.read_csv('C://Users//<NAME>//Desktop//survey_results_public.csv') 
df.head()
## To know the number of records that the base has 

df.shape[0]<jupyter_output><empty_output><jupyter_text>Base for modeling<jupyter_code>df = df[['CareerSatisfaction', 'JobSatisfaction', 'HoursPerWeek', 'StackOverflowSatisfaction', 'Salary', 
               'ExpectedSalary', 'Professional', 'University', 'FormalEducation', 'EmploymentStatus', 'DiversityImportant',
               'HomeRemote', 'EducationImportant', 'CompanySize']]

# Describe numerical variables

df.describe()
## Histogram to analize numerical variables

df.hist()<jupyter_output><empty_output><jupyter_text>Section 3: Prepare Data<jupyter_code># Missing values of the target variable are cleared because null variables cannot be predicted

df = df.dropna(axis = 0, subset = ['JobSatisfaction'])
df.shape[0]
# Fill numeric columns with the mean, to ensure consistency of the data and that model works well. Data with missing values
# can't be processing by machine learning algorithns, so is necessary to replace missing values by a approximation. In this
# case I used the mean because is the best approximation.

def add_mean(df):
        

    ''' 
    INPUT:
    df - Sample of the survey data

    OUTPUT:
    df - Sample survey data with mean instead of missing values

    '''

    vn = df.select_dtypes(include = ['float', 'int']).columns

    for i in vn:
        df[i].fillna((df[i].mean()), inplace = True)
        
    return df
    

# The function add_mean repalce missing values by mean

df = add_mean(df)
# Dummy the categorical variables, so that categorical variables can be consumed by the model
# Matrix with categorical variables can't be processed by certain algorithns as lineal regression, so is necessary to encode
# the categorical variables to dummies

def dummies(df):

    ''' 
    INPUT:
    df - Sample of the survey data
    
    OUTPUT:
    df - Sample survey data with with dymmies of categorical variables instead of categorical variables

    '''

    vc = df.select_dtypes(include = ['object']).copy().columns

    for ct in vc:
        df = pd.concat([df.drop(ct, axis = 1), pd.get_dummies(df[ct], prefix = ct, prefix_sep = '_', drop_first = True)], axis = 1)

    return df

# The function dummies replace categorical vairbles to dummies for each of them

df = dummies(df)<jupyter_output><empty_output><jupyter_text>Section 4: Modeling<jupyter_code># Split into explanatory and response variables

def split_data(df):
    
    ''' 
    INPUT:
    df - Sample of the survey data
    
    OUTPUT:
    X - Explanatory variables
    Y - Target Variable

    '''

    Y = df['JobSatisfaction']

    X = df.drop(['JobSatisfaction'], axis = 1)

    return X, Y
# Function split_data split the data in explanatory and target variable

X, Y = split_data(df)
# Split into train and test - Fit model

X_Train, X_Test, Y_Train, Y_Test = train_test_split(X, Y, test_size = 0.3, random_state = 48)

mco_model = LinearRegression(normalize=True)
mco_model.fit(X_Train, Y_Train)<jupyter_output><empty_output><jupyter_text>Section 5: Evaluation<jupyter_code># Score using the model

pred_test = mco_model.predict(X_Test)
r2 = r2_score(Y_Test, pred_test)
r2
# Is overfitting generated?

pred_train = mco_model.predict(X_Train)

test_sc = r2_score(Y_Test, pred_test)
train_sc = r2_score(Y_Train, pred_train)
train_sc<jupyter_output><empty_output><jupyter_text>Overfitting is not generated because the metrics evaluated in train and test give similar.  Section 6: Evaluate the Results  1. How does job satisfaction relate to earned salary and hours worked per week for professional developers?<jupyter_code>## A correlation matrix is done

df[df['Professional'] == 'Professional developer'][['JobSatisfaction', 'Salary', 'HoursPerWeek']].corr()
# A Heat Map is done to analize the correlation between interest variables

sns.heatmap(df[df['Professional'] == 'Professional developer'][['JobSatisfaction', 'Salary', 'HoursPerWeek']].corr(),
            annot = True)<jupyter_output><empty_output><jupyter_text>According to the previous results for the year 2017, the job satisfaction of professional developers would have a positive relationship with salary, which means that with a salary increase there would eventually be greater job satisfaction. However, as expected in the face of an increase in working hours, job satisfaction would tend to decrease, this means that they have a negative correlation.  2. On average, how much do professional developers earn and what size of company do they work most?<jupyter_code># Only preffesional developer

PD = pd.DataFrame(df[df['Professional'] == 'Professional developer'][['Professional','Salary', 'CompanySize']])
PD = PD.dropna(axis = 0, subset = ['Salary'], how = 'all')
PD['Salary'].mean()
# Histogram is done to analize the wage distribution

plt.hist(PD['Salary'], color = 'purple', alpha = 0.5)
plt.title('Wage Distribution')
# Proportion of proffesional developer workers by size company

((PD['CompanySize'].value_counts())/PD.shape[0]).plot(kind='bar', color= 'red', 
                                                      title='Proportion of proffesional developer workers by size company',
                                                      alpha = 0.7)<jupyter_output><empty_output><jupyter_text>Professional developers have an average salary of 56,298, and around 43% are employed in companies with fewer than 500 employees.  3. On average, how is job satisfaction and salary according to the importance of each person's education?<jupyter_code># Filter Data Frame

IE = pd.DataFrame(df[['JobSatisfaction', 'EducationImportant', 'Salary']])
# Comparison in a table

A = IE.groupby(['EducationImportant']).mean()['JobSatisfaction'].reset_index()
B = IE.groupby(['EducationImportant']).mean()['Salary'].reset_index()
C = A.merge(B, how = 'left', left_on = 'EducationImportant', right_on = 'EducationImportant')
C<jupyter_output><empty_output><jupyter_text>On average, people who place the highest importance on education have a higher salary and greater job satisfaction than people who place less importance on education.   4. How is the job satisfaction ratio for professional developers who can work at home?<jupyter_code># New filter data frame

HO = df[df['Professional'] == 'Professional developer'][['JobSatisfaction', 'HomeRemote']]
HO.groupby(['HomeRemote']).mean()['JobSatisfaction'].plot(kind='bar', color= 'grey', 
                                                      title='Job satisfaction by type of job',
                                                      alpha = 0.7)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08971306853777722, 'fraction_numerical': 0.008520235559453703, 'mean_word_length': 4.3355614973262036, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '62583', 'license': 'CNRI-Python', 'n_tokens_mistral': 2358, 'n_tokens_neox': 2163, 'n_words': 832, 'path': 'Proyecto.ipynb', 'repo_name': 'pcmorenov/Stackoverflow_Survey', 'size': '7993'}