<jupyter_start><jupyter_text>GENDER WAGE GAPHere we ask and answer the following question:**What is the difference in predicted wages between men and women with the same job-relevant characteristics?**Our data comes from the March Supplement of the U.S. Current Population Survey, year 2012. We focus on the single (never married) workers with education levels equal to high-school, some college, or college graduates. The sample size is about 4,000.Our outcome variable Y is hourly wage, and our X ’s are various characteristics of workers such as gender, experience, education, and geographical indicators.The dataset contains the following variables:1. wage   : weekly wage2. female : female dummy3. cg     : college Graduate Dummy4. sc     : some college dummy5. hsg    : High School graduate dummy6. mw     : mid-west dummy7. so     : south dummy8. we     : west dummy9. ne     : northeast dummy10. exp1   : experience(year)11. exp2   : experience squared (taken as experience squared/100)12. exp3   : experience cubed (taken as experience cubed/1000)<jupyter_code>#importing necessary libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
#to ignore warnings
import warnings
warnings.filterwarnings('ignore')
# Load data
df = pd.read_csv('./data/predicting_wages.csv')
# See variables in the dataset
df.head()<jupyter_output><empty_output><jupyter_text>Checking the info of the dataset<jupyter_code># Checking info of the dataset
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3835 entries, 0 to 3834
Data columns (total 12 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   female  3835 non-null   int64  
 1   cg      3835 non-null   int64  
 2   sc      3835 non-null   int64  
 3   hsg     3835 non-null   int64  
 4   mw      3835 non-null   int64  
 5   so      3835 non-null   int64  
 6   we      3835 non-null   int64  
 7   ne      3835 non-null   int64  
 8   exp1    3835 non-null   float64
 9   exp2    3835 non-null   float64
 10  exp3    3835 non-null   float64
 11  wage    3835 non-null   float64
dtypes: float64(4), int64(8)
memory usage: 359.7 KB<jupyter_text>- The dataset has 3835 observations and 12 different variables.- There is no missing value in the dataset. - The type of all the dummy variables( cg, sc, hsg... etc) is int, but the are binary variables 0 and 1.   Checking the summary statistics of the dataset<jupyter_code># Printing the summary statistics for the dataset
df.describe().T<jupyter_output><empty_output><jupyter_text>- The average wage is about 15 dollars per hour while the maximum wage is 348, which is very high.- 42% of workers are women- Average experience is 13 years, while min and max 2 too 35 years, which shows the data is diverse and is taken from different experience groups. - 38% of the people in the data are college graduates. - 32% have gone to some college, and 30% hold only high school diploma.- You can also see geographical distribution of workers across major geographical regions of the states, and seem to be nearly same between 22-28%, which again shows maybe data is collect from different regions in a uniform manner.<jupyter_code>df[['exp1','exp2','exp3','wage']].boxplot(figsize=(20,10))
plt.show()<jupyter_output><empty_output><jupyter_text>- There are outliers in the wage, which make sense also, as some people have high wages as compared to others.   As this objective of this case study to analyze the gender wage gap, let's look at the statistics of all columns for female and male.<jupyter_code># Mean value of all females
df[df['female'] == 1].mean()
# Mean value of all males
df[df['female'] == 0].mean()<jupyter_output><empty_output><jupyter_text>- We first take a look at the following descriptive statistics for the subsamples of single men and single women, with educational attainment equal to high-school, some college, or college.- The mean hourly wage is 16 dollars for men and about 15 dollars for women, so the difference is 1, without controlling for job-relevant characteristics.- If we take a look at some of these characteristics, we see that on average men have more experience, but women are more likely to have college degrees or some college education. - Geographical distribution or men and women is similar.  Basic Model<jupyter_code>####################  Linear and Quadratic specifications ##############################

# Wage linear regression
import statsmodels.api as sm
Y = df['wage'] # target variable
X = df[['female' , 'sc', 'cg', 'mw' , 'so' , 'we' , 'exp1' , 'exp2' , 'exp3']] #regressors
X = sm.add_constant(X) # adding constant for intercept
model = sm.OLS(Y, X)
results = model.fit() # train the model
print(results.summary()) # summary of the model<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                   wage   R-squared:                       0.095
Model:                            OLS   Adj. R-squared:                  0.093
Method:                 Least Squares   F-statistic:                     44.87
Date:                Sat, 12 Feb 2022   Prob (F-statistic):           3.17e-77
Time:                        11:22:44   Log-Likelihood:                -15235.
No. Observations:                3835   AIC:                         3.049e+04
Df Residuals:                    3825   BIC:                         3.055e+04
Df Model:                           9                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>- Model performance is very poor as the r squared is very low. - We can see that cg graduate have high coefficient which signifies that graduated people are getting high wages.- On the other hand exp2 has a negative coefficient which means with exp2 the wages are less. - Coefficient of the female indicator is negative, signifies that females are getting lower wages.   Flexible Model<jupyter_code># import PolynomialFeatures library to create polynomial features
from sklearn.preprocessing import PolynomialFeatures 

# create an object of PolynomialFeatures with only the interaction terms
poly = PolynomialFeatures(interaction_only=True,include_bias = False) 

#Dropping constant and female as we dont want to create interaction features for them
X.drop(['const','female'],axis = 1,inplace = True)
print(X.shape)

# transform the data to add the polynomial features too
X_poly = poly.fit_transform(X) 

# convert to dataframe
X_poly = pd.DataFrame(X_poly,columns= poly.get_feature_names(X.columns)) 

print(X_poly.shape)
X_poly['female'] = df['female']
X['female'] = df['female']
X_poly = sm.add_constant(X_poly)

#defining the model
model = sm.OLS(Y, X_poly) # Linear regression/OLS object

#fitting the model
results = model.fit() # train the model

# summary of the model
print(results.summary())<jupyter_output>(3835, 8)
(3835, 36)
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                   wage   R-squared:                       0.104
Model:                            OLS   Adj. R-squared:                  0.096
Method:                 Least Squares   F-statistic:                     13.79
Date:                Sat, 12 Feb 2022   Prob (F-statistic):           5.53e-69
Time:                        11:22:44   Log-Likelihood:                -15217.
No. Observations:                3835   AIC:                         3.050e+04
Df Residuals:                    3802   BIC:                         3.071e+04
Df Model:                          32                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std er[...]<jupyter_text>- R2 have improved very slightly. - Coefficient of cg*exp is quite high, represents Experience variable times the indicator of having a college degree will have a high wage,or people with good education and good experience are paid good.- coefficient of cg*mw is negative, implies College graduate from mid west are not getting paid good, which is not a good sign as for the people living in mid west as from the basic model we know that the college graduates are getting paid good.- Coefficient of cg * sc, so * we,mw * we,mw * so, is almost 0 and doesn't not make any contribution to the model.  We estimate the linear regression model:Y = $β_1D$ + $β^r_2W$ + ε.1. D is the indicator of being a female (1 if female and 0 otherwise). W ’s arecontrols.2. Basic model: W ’s consist of education and regional indicators, experience, experience squared, and experience cubed.3. Flexible model: W ’s consist of controls in the basic model plus all of their two-way interactions. || Estimate | Standard Error | Confidence Interval|| --- | --- | --- | --- || basic reg | -1.8264 | 0.425 | [-2.659  -0.994] || flexi reg | -1.8800 | 0.425 | [-2.713  -1.047] |- In the above table we see the estimated regression coefficient, its standard error, and the 95% confidence interval, for both the basic and flexible regression model.- The results for basic and flexible regression models are in a very close agreement:- The estimated gender gap in hourly wage is about −2 dollars. This means that women get paid 2 dollars less per hour on average than men, controlling for experience, education, and geographical region.- The 95% confidence interval ranges from −2.7 to −1 dollars and -2 is lying within the region of the confidence interval therefore We conclude that the difference in the hourly wage for men and women, who have the same recorded characteristics, is both statistically and economically significant.  Illustration of Partialling Out: (Linear Specification - Basic Model)  **“Partialling-out” is an important tool that provides a conceptual understanding of the regression coefficient β1.  The Steps for partialling out are:**1. We predict Y using W only and find its residuals. i.e. removing the dependence of W on Y. 2. We predict D using W and find its residuals, i.e. removing the dependence of W on D. 3. Then we model residuals from step 1 and step 2, and this will give us how Y is dependent on D only.   Step 1: We predict Y using W only and find its residuals. i.e. removing the dependence of W on Y.<jupyter_code># target variable
Y = df['wage'] 

#regressors W
W = df[['sc', 'cg', 'mw' , 'so' , 'we' , 'exp1' , 'exp2' , 'exp3']] 
W = sm.add_constant(W)

# Linear regression/OLS object
model = sm.OLS(Y, W) 

# train the model
results = model.fit() 

# get the residuals
t_Y = results.resid 
t_Y<jupyter_output><empty_output><jupyter_text>Step 2: We predict D using W and find its residuals, i.e. removing the dependence of W on D.<jupyter_code># target regressor D, i.e. female in our predicitng wage example
D = df['female'] 

#regressors W
W = df[['sc', 'cg', 'mw' , 'so' , 'we' , 'exp1' , 'exp2' , 'exp3']] #regressors
W = sm.add_constant(W)

# Linear regression/OLS object
model = sm.OLS(D, W) 

# train the model
results = model.fit() 

# get the residuals
t_D = results.resid
t_D<jupyter_output><empty_output><jupyter_text>Step 3: Then we model residuals from step 1 and step 2, and this will give us how Y is dependent on D only.<jupyter_code># Run OLS coefficient get coefficients and 95% confidence intervals

# target variable
Y = t_Y 

X = t_D
X = sm.add_constant(X)

# Linear regression/OLS object
model = sm.OLS(Y, X) 

# train the model
results = model.fit() 

# print the summary of the trained model
print(results.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.005
Model:                            OLS   Adj. R-squared:                  0.005
Method:                 Least Squares   F-statistic:                     18.55
Date:                Sat, 12 Feb 2022   Prob (F-statistic):           1.70e-05
Time:                        11:22:44   Log-Likelihood:                -15235.
No. Observations:                3835   AIC:                         3.047e+04
Df Residuals:                    3833   BIC:                         3.049e+04
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>Illustration of Partialling Out: (Quadratic Specification - Flexible model)<jupyter_code># target variable
Y = df['wage'] 

#regressors W
W = df[['sc', 'cg', 'mw' , 'so' , 'we' , 'exp1' , 'exp2' , 'exp3']] 

# create an object of PolynomialFeatures with only the interaction terms
poly = PolynomialFeatures(interaction_only=True,include_bias = False)

# transform the data to add the polynomial features too
X_poly = poly.fit_transform(W)
X_poly = sm.add_constant(X_poly)

#defining the model
model = sm.OLS(Y, X_poly) # Linear regression/OLS object

 # train the model
results = model.fit()

# get the residuals
t_Y = results.resid
t_Y
# target regressor
D = df['female'] 

#Regressors w
W = df[['sc', 'cg', 'mw' , 'so' , 'we' , 'exp1' , 'exp2' , 'exp3']] 

# create an object of PolynomialFeatures with only the interaction terms
poly = PolynomialFeatures(interaction_only=True,include_bias = False)
X_poly = poly.fit_transform(W)
X_poly = sm.add_constant(X_poly)


#defining the model
model = sm.OLS(D, X_poly)

# train the model
results = model.fit() 

# get the residuals
t_D = results.resid
t_D
# Run OLS coefficient get coefficients and 95% confidence intervals
Y = t_Y # target variable
X = t_D
X = sm.add_constant(X)
model = sm.OLS(Y, X)
results = model.fit() # train the model
print(results.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.005
Model:                            OLS   Adj. R-squared:                  0.005
Method:                 Least Squares   F-statistic:                     19.75
Date:                Sat, 12 Feb 2022   Prob (F-statistic):           9.07e-06
Time:                        11:22:44   Log-Likelihood:                -15217.
No. Observations:                3835   AIC:                         3.044e+04
Df Residuals:                    3833   BIC:                         3.045e+04
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]
{'fraction_non_alphanumeric': 0.1203210315110848, 'fraction_numerical': 0.03322149858561937, 'mean_word_length': 2.536171202605257, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '26224', 'license': 'MIT', 'n_tokens_mistral': 4535, 'n_tokens_neox': 4020, 'n_words': 1687, 'path': 'genderWageGap.ipynb', 'repo_name': 'AlexHamn/us-gender-wage-gap-case-study', 'size': '15201'}