<jupyter_start><jupyter_text>Important Skills on Getting HiredIn this notebook, I have explored the important skills to get hired in the tech industry using the StackOverFlow survey data.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
%matplotlib inline
pd.set_option('display.max_rows', 500)

'''Reading the Data'''
df = pd.read_csv('./survey_results_public.csv')
schema = pd.read_csv('./survey_results_schema.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Data Understanding and Data Preparation: Cheking data type (object or numeric) in next 2 cells<jupyter_code>'''Checking columns with object type'''
df.select_dtypes(include=['object'])
'''Checking columns with numeric type'''
df.select_dtypes(include=['float','int'])<jupyter_output><empty_output><jupyter_text>Data Understanding and Data Preparation: Checking the content of probable variable to be used in next 4 cells<jupyter_code>'''Checking the content of column value'''
list(schema[schema['Column']=='Professional']['Question'])[0]
list(schema[schema['Column']=='FormalEducation']['Question'])[0]
list(schema[schema['Column']=='ExpectedSalary']['Question'])[0]
list(schema[schema['Column']=='ImportantHiringEducation']['Question'])[0]

list(schema[schema['Column']=='ImportantHiringCommunication']['Question'])[0]
list(schema[schema['Column']=='ImportantHiringTechExp']['Question'])[0]
'''Basic Checkings of values'''
df['Professional'].value_counts()
df['FormalEducation'].value_counts()
df['ExpectedSalary'].value_counts()
df['StackOverflowSatisfaction'].value_counts()<jupyter_output><empty_output><jupyter_text>Business understanding: Questions we want to answer through data Question1: What is the trend in expected salary reported by people with different formal education, professioanl background and people working on different company size? This is the most common question many people encounters and sometimes it is tricky to report appropriate amount for the question. I think it might be of interest for the business firm to see the trend in the expected salary reported by the people from various backgounds.  Analysis:To answer this question, we have calculated the average Expected salary reported by the people from different backgrounds in three parts (Part I, II and III).  Question 1: Part IThis section provides information on Expected Salary based on Formal Education<jupyter_code>'''This section provides information on Expected Salary based on formal education'''

df.groupby(['FormalEducation']).mean()['ExpectedSalary'].sort_values().dropna()<jupyter_output><empty_output><jupyter_text>Question 1: Part IIThis section provides information on Expected Salary based on Professional<jupyter_code>'''Question 1, part 2'''
'''This section provides information on Expected Salary based on Professional'''
df.groupby(['Professional']).mean()['ExpectedSalary'].sort_values()<jupyter_output><empty_output><jupyter_text>Question 1: Part IIIThis section provides information on Expected Salary based on formal education<jupyter_code>'''This section provides information on Expected Salary based on formal education'''
df_cs=df.groupby(['CompanySize']).mean()['ExpectedSalary'].sort_values()
df_cs<jupyter_output><empty_output><jupyter_text>Question 2: What people consider IMPORTANT for hiring?This question is of utmost important when it comes to hiring. It will be useful from the perspective of business to explore what people think about this very imporant question of interest.Checked 3 Categorical variables (Higher Education, Tech Experience, Communication) from bar diagram AnalysisTo answer this question, we did the analysis through the bar plot comparisons for the 3 variables. Initially, the separate bar plots are created with the combined one in the last part to see the relative effect of 3 variables.  Qeustion 2: Part IDefining a function to plot a bar diagram from DataFrameChecking the distribution of Imporant Hiring with Higher Education<jupyter_code>def plotbar(df,title, savefig= True):
    """
    Description: This function can be used to plot a bar diagram from a dataframe.
    
    Arguments: 
    df: Dataframe
    title: String for the title
    savefig: Boolean to save the figure.
             If True saves by the title.
             
    Returns: None (plots a bar plot)
    
    """
    df.plot(kind='bar')
    plt.title(title)
    plt.tight_layout()
    if savefig:
        plt.savefig(title + '.png')
        
'''Making a dataframe of all Important Hiring'''

ImpHiring=['ImportantHiringAlgorithms','ImportantHiringTechExp','ImportantHiringCommunication',
           'ImportantHiringOpenSource','ImportantHiringPMExp','ImportantHiringCompanies','ImportantHiringTitles',
           'ImportantHiringEducation','ImportantHiringRep','ImportantHiringGettingThingsDone']
list(schema[schema['Column']=='ImportantHiringPMExp']['Question'])[0]
df_ImpHiringClean=df[ImpHiring].dropna()
df_ImpHireHE=df_ImpHiringClean['ImportantHiringEducation']
df_ImpHireHE_avg= df_ImpHireHE.value_counts()/df_ImpHireHE.shape[0]

'''Calling function to plot dar diagram'''

plotbar(df_ImpHireHE_avg,"Higher Education", savefig=False)<jupyter_output><empty_output><jupyter_text>Question 2: Part IIChecking the distribution of important hiring with Tech Skills<jupyter_code>df_ImpHireTExp=df_ImpHiringClean['ImportantHiringTechExp']
df_ImpHireTExp_avg= df_ImpHireTExp.value_counts()/df_ImpHireTExp.shape[0]

'''Calling function to plot dar diagram'''

plotbar(df_ImpHireTExp_avg,"Tech Experience", savefig=False)<jupyter_output><empty_output><jupyter_text>Question 2: Part IIIChecking the distribution of Import Hiring with Communication Skill.<jupyter_code>df_ImpHireCom=df_ImpHiringClean['ImportantHiringCommunication']
df_ImpHireCom_avg= df_ImpHireCom.value_counts()/df_ImpHireCom.shape[0]

'''Calling function to plot dar diagram'''

plotbar(df_ImpHireCom_avg,"Communication", savefig=False)<jupyter_output><empty_output><jupyter_text>Question 2: Part IVWe want to explore the effect of three skills on the Hiring process. We want to combine the distributions. Initially, a new datafrme with three skills are created.<jupyter_code>'''Create a sub dataframe with 3 Skills'''

df2=df[['ImportantHiringEducation','ImportantHiringTechExp','ImportantHiringCommunication']]
df3=df2.apply(lambda x: x.value_counts()/x.shape[0])
df3=df3.rename(columns={"ImportantHiringEducation": "Education", "ImportantHiringTechExp": "TechExp", 
                    "ImportantHiringCommunication": "Communication"}, errors="raise")
df3
'''Calling function to plot dar diagram'''

plotbar(df3,"ImportantHiringAll", savefig=True)<jupyter_output><empty_output><jupyter_text>Question 3: What is the correlation among three numerical variables  (Salary, HoursPerWeek and JobSatisfaction) and can a statistical model predict the missing salary?It is important to know about the job statisfaction of the employee for any successful business. Exploring the parameters in with this important variable relies is of utmost importance. From the data sceince point of view, it is also useful to know if the data sample is adequate in predicting the missing information. In order to test this, we have trained a linear regression model and carried out the analysis. Analysis:To answer this question, at first we plotted a heatmap for correlation among three variables. Then we trained two linear regression models.  Question 3: Part ICreate a dataframe with 3 numerical variables.Draw a heatmap to check the correlation between the variables.Create 2 datafranes 1) Dropping all NANs 2) Imputing the misising values by Mean<jupyter_code>''' Data frame with 3 numerical values'''

df_sub=df[['Salary','HoursPerWeek','JobSatisfaction']]

'''Dropping ALL NANs'''

drop_df=df_sub.dropna()

''' Imputing the sub data set with mean '''

fill_mean = lambda col: col.fillna(col.mean())
fill_df = df_sub.apply(fill_mean, axis=0)

'''heatmap to see the correlation'''

sns.heatmap(df_sub.corr(), annot=True, fmt=".2f");
plt.tight_layout()
plt.savefig('heatmap.png')<jupyter_output><empty_output><jupyter_text>Question 3: Part IIBuilding linear regression model based 1) After dropping NA's and 2) Imputing with Mean  Data ModelingFrom the data sceince point of view, it is also useful to know if the data sample is adequate in predicting the missing information. In order to test this, we have trained a linear regression model and carried out the analysis.TWO linear regresssion models are created. One by dropping NAN's and other by imputing the missing values by Mean. We want to see the effect of both in the model predictability.   1) Linear Regression model prediction after dropping NAN<jupyter_code>'''Split into explanatory and response variables'''

X=drop_df[['Salary','HoursPerWeek']]
y=drop_df[['JobSatisfaction']]

'''Split into train and test'''

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state=42) 

lm_model = LinearRegression(normalize=True) # Instantiate
lm_model.fit(X_train, y_train) #Fit
        
''' Predict and score the model '''
y_test_preds = lm_model.predict(X_test) 

'''Rsquared and y_test'''
rsquared_score = r2_score(y_test, y_test_preds)
length_y_test = len(y_test)
xx=np.arange(-1,11,1)
yy=np.arange(-1,11,1)
plt.plot(y_test,y_test_preds,'r.')
plt.plot(xx,yy,'b-')
plt.xlim([-1,11])
plt.ylim([-1,11])
plt.xlabel('Ground truth (JobSatisfaction)')
plt.ylabel('Prediction')
plt.title('Dropping NAN')
plt.tight_layout()
#plt.show()
plt.savefig('model_pred_NoNan.png')
"The r-squared score for your model was {} on {} values.".format(rsquared_score, length_y_test)<jupyter_output><empty_output><jupyter_text>2 ) Linear Regression model prediction after imputing<jupyter_code>'''Split into explanatory and response variables'''
X=fill_df[['Salary','HoursPerWeek']]
y=fill_df[['JobSatisfaction']]

'''Split into train and test'''

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state=42) 

lm_model = LinearRegression(normalize=True) # Instantiate
lm_model.fit(X_train, y_train) #Fit
        
'''Predict and score the model'''

y_test_preds = lm_model.predict(X_test) 

'''Rsquared and y_test'''

rsquared_score = r2_score(y_test, y_test_preds)
length_y_test = len(y_test)
xx=np.arange(-1,11,1)
yy=np.arange(-1,11,1)
plt.plot(y_test,y_test_preds,'r.')
plt.plot(xx,yy,'b-')
plt.xlim([-1,11])
plt.ylim([-1,11])
plt.xlabel('Ground truth (JobSatisfaction)')
plt.ylabel('Prediction')
plt.title('Imputing by Mean')
plt.tight_layout()
#plt.show()
plt.savefig('model_pred_ImputingMean')

"The r-squared score for your model was {} on {} values.".format(rsquared_score, length_y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10410732334834145, 'fraction_numerical': 0.008361664982082146, 'mean_word_length': 6.875542691751085, 'pattern_counts': {'":': 3, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '579114', 'license': 'Python-2.0', 'n_tokens_mistral': 3467, 'n_tokens_neox': 3255, 'n_words': 1007, 'path': 'ImportantHiring.ipynb', 'repo_name': 'ap439111/StackOverFlow', 'size': '10883'}