<jupyter_start><jupyter_text>EXPLORATORY DATA ANALYSIS ON SALARIES This project analyses bank salary information of people with of different educational backgrounds, gender, minority backgrounds and job categories. It tend to explain how the various columns relate to each other.   PROCESSMy appraoch will include Data exploration, Data cleaning, Model building, and  Presenting results respectively.  Data ExplorationI will start preparing the data by importing needed libraries and loading the data<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import statsmodels.api as sms
import statsmodels.api as sm
import statsmodels.formula.api as smf
from string import ascii_letters


%matplotlib inline
salary = pd.read_csv("/Users/raymondboateng/neuefische/EDA/data/us_bank_wages.csv") 
del salary['Unnamed: 0']
salary.head(10)
salary.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 474 entries, 0 to 473
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   SALARY    474 non-null    int64
 1   EDUC      474 non-null    int64
 2   SALBEGIN  474 non-null    int64
 3   GENDER    474 non-null    int64
 4   MINORITY  474 non-null    int64
 5   JOBCAT    474 non-null    int64
dtypes: int64(6)
memory usage: 22.3 KB<jupyter_text>We will study the the distribution of the salary and an independent variable<jupyter_code>print(salary['SALARY'].describe())
#print(salary['SALBEGIN'].describe())
# A view of the salary distribution
plt.figure(figsize=(9, 8))
sns.distplot(salary['SALARY'], color='g', bins=100, hist_kws={'alpha': 0.4});<jupyter_output>/Users/raymondboateng/.pyenv/versions/3.8.5/lib/python3.8/site-packages/seaborn/distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>The salary distribution is skewed towards right with outliers above ~ 80000. It can be normalised using the log function.   Analyses of the Numerical data distributionThe initial analyses proves that the given is rife for furtheranaylses.Since all the data are numerical forms, we could plot them and look at the distribution<jupyter_code>salary.hist(figsize=(16, 20), bins=50, xlabelsize=8, ylabelsize=8); # ; avoid having the matplotlib verbose informations<jupyter_output><empty_output><jupyter_text>Salary and SALBEGIN have similar features, whiles minority and gender seem similar. We could analyse further by looking for correlations  Based on the initial analyses, to adequately advise the stakeholder, I propose the following hypothesis  Hypothesis:1. Educated population will earn more beginner´s salary and high annual salary 2. Strong correlation between  salary and Job categories3. Majority of people should cluster around the median salary   Hypothesis 1Test Hypothesis 1 by plotting to view relationship between education level and annual salary<jupyter_code>"salary.plot(x='EDUC', y='SALARY', kind='hexbin', gridsize=15, figsize=(11, 9) );"

salary.plot(x='EDUC', 
          y='SALARY', 
          kind='scatter',figsize=(11, 9) );

#reviews[reviews['price'] < 100].plot.hexbin(x='price', y='points', gridsize=15)<jupyter_output>*c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.<jupyter_text>Regression Analyses is carried on the relation between Education and Salary<jupyter_code>def get_line_equation(p1, p2):
    """
    Solve the system of equations:
    y1 = m*x1 + b
    y2 = m*x2 + b
    
    This translates to:
    m = (y2 - y1) / (x2 - x1)
    b = y1 - m*x1
    
    Input:
    p1: first point [x1, y1]
    p2: second point [x2, y2]
    
    returns: slope, intercept
    """
    m = (p2[1] - p1[1]) / (p2[0] - p1[0]) # Slope
    b = p1[1] - m * p1[0] # Intercept
    return  m, b
# Calculate the intercept and slope of a line between two points
Line1 = [salary.SALARY[1],salary.EDUC[1]]
Line2 = [salary.SALARY[2],salary.EDUC[2]]
slope, intercept = get_line_equation(p1=Line1, p2=Line2) 
print('intercept: ', intercept)
print('slope: ', slope)
"# Plot line fitted on all of the pointsfig = salary.plot(x='EDUC', y='SALARY', kind='scatter', figsize=(11, 9) # plot the linear regression lineplt.plot(np.unique(salary.EDUC), np.poly1d(np.polyfit(salary.EDUC, salary.SALARY, 1))(np.unique(salary.EDUC)));# plot the line between two points#plt.plot([car1[0], car2[0]], [car1[1], car2[1]], color='red', linewidth=4)plt.title('Linear Regression Line');"


# Plotting the results of our model
fig, ax = plt.subplots(figsize=(11, 9))
x = salary.EDUC
# add data points
ax.scatter(x, y, alpha=0.5, color='orchid')
fig.suptitle('Relationship between SALARY and EDUCATION')
# plotting regression line
ax.plot(x, x*slope +intercept, '-', color='darkorchid', linewidth=2);
ax.set_ylabel("salary");
ax.set_xlabel("EDUC");<jupyter_output><empty_output><jupyter_text>The line shows correlation between annual salary and education. The strength of the correlation is tested as follows:<jupyter_code># Prepare data for modeling
# Choose the predictor and add a constant term
X = salary[['EDUC']]
y = salary.SALARY
# Our model needs an intercept so we add a column of 1s:
X = sms.add_constant(X)
display(X.head())
display(y.head())
# Create an OLS model
model = sms.OLS(y, X)
# use the data to calculate the intercept and slope
results = model.fit()
# return the output of the model
results.summary()<jupyter_output><empty_output><jupyter_text>EDUC Coef shows that for each increase in number of finished years of education, there is an increase in annual salary of 3909.9067.The R2 value however is too weak to back the claim of a strong correlation<jupyter_code>intercept, slope = results.params
intercept<jupyter_output><empty_output><jupyter_text>The same analyses is carried on salary and beginner´s salary to study the correlation between them.<jupyter_code>#Linear Regression

smf.ols(formula='SALARY ~ SALBEGIN', data=salary).fit().summary()
# Plotting the results of our model
sns.lmplot(x="SALBEGIN", y="SALARY", data=salary, x_estimator=np.mean);



fig = plt.gcf()
fig.set_size_inches(11, 9)<jupyter_output><empty_output><jupyter_text>This analysis proves strong correlation between the annual salary and the beginner´s salary. The R2 is high and the linear regression also proves the correlation.    Hypothesis 2The correlation between salary and job categories is tested. Below the R squared for each independent variable to salary is calculated and the corresponding data is also plotted.<jupyter_code># simple linear regression with each explanatory variable
print('R squared for each possible explanatory variable:' )
for explanatory_var in explanatory_vars:
    model = 'SALARY ~ {:}'.format(explanatory_var)
    rs = smf.ols(formula=model, data=salary).fit().rsquared
    print('{:7s} \t \t= {:.3}'.format(explanatory_var, rs))<jupyter_output>R squared for each possible explanatory variable:
SALARY  	 	= 1.0
EDUC    	 	= 0.436
SALBEGIN 	 	= 0.775
GENDER  	 	= 0.202
MINORITY 	 	= 0.0314
JOBCAT  	 	= 0.609<jupyter_text>It can be observed here that both Gender and Minority have very small R2 values signifying that the correlation between them and annual salary is weak. The plot below also confirms no correlation between them, But job categories has a high R2.<jupyter_code># geting a feeling of the data
sns.pairplot(salary);
sns.lmplot(x="SALARY", y="JOBCAT", hue = "JOBCAT", data=salary);
#sns.lmplot(x="SALARY", y="MINORITY", hue="MINORITY", data=salary);
sns.pairplot(salary, x_vars=["GENDER", "JOBCAT"], y_vars=["SALARY"],
             hue="MINORITY", height=5, aspect=.8, kind="reg");

sns.set_theme(style="white")

# Compute the correlation matrix
corr = salary.corr()

# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(corr, dtype=bool))

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(230, 20, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
# use all explanatory variables
model = 'SALARY ~ EDUC + SALBEGIN + GENDER + MINORITY + JOBCAT'
smf.ols(formula=model, data=salary).fit().summary()
#salary[['EDUC', 'GENDER', 'MINORITY','JOBCAT']].plot(kind='box', figsize = (20,25))
plt.figure(figsize = (10, 6))
ax = sns.boxplot(y='SALARY','SALBEGIN', data=salary)
plt.setp(ax.artists, alpha=.5, linewidth=2, edgecolor="k")
plt.xticks(rotation=45)<jupyter_output><empty_output><jupyter_text>plt.figure(figsize = (10, 6))ax = sns.boxplot(x='SALARY', y='SALBEGIN', data=salary)plt.setp(ax.artists, alpha=.5, linewidth=2, edgecolor="k")plt.xticks(rotation=45)<jupyter_code>salary[['SALARY', 'SALBEGIN']].plot(kind='box', figsize = (20,25))
#salary['GENDER'].value_counts().plot(kind='bar',figsize = (20,25))

fig, ax = plt.subplots(figsize = (20,10))    
chart = sns.countplot(x = 'GENDER', hue = 'MINORITY', data = salary) #estimator = median
chart.set_xticklabels(chart.get_xticklabels(),rotation = 45);
salary.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11508844953173777, 'fraction_numerical': 0.026118626430801248, 'mean_word_length': 5.067550505050505, 'pattern_counts': {'":': 1, '<': 43, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '549724', 'license': 'MIT', 'n_tokens_mistral': 3395, 'n_tokens_neox': 3143, 'n_words': 1005, 'path': 'notebooks/EDA Salary.ipynb', 'repo_name': 'RayKwame/EDA', 'size': '9610'}