<jupyter_start><jupyter_text>Introduction to the NLSY79 dataset We will see:1. How to perform simple regressions2. How to generate density plot3. How to generate a heatmap Background: Investigation of the wage dynamics in [The career decisions of young men](http://www.journals.uchicago.edu/doi/10.1086/262080) by <NAME>. and <NAME>. (1997).$\rightarrow$ How persistent are the wage shocks?* Perform wage regressions* Use wage residuals where the effect of observable characteristics and common aggregate time trends have been eliminated* Investigate persistence by     * Density plots    * Coveariance matrix in a heatmaps Sample selection in this analysis:* White males aged 16 or less as of October 1, 1977* Time dimension is measured in periods, i.e. period 0 begins once an individual has turned 16 by October* 10 years of follow-up  1) Import packagesImporting packages in a notebook once is sufficient.<jupyter_code>import pandas as pd
import statsmodels.api as sm
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from patsy import dmatrices, dmatrix
import math
% matplotlib inline
# We ensure a proper formatting of the variables.
pd.options.display.float_format = '{:,.2f}'.format
# Adjust the default options s.t. the full dataset can be viewed.
pd.set_option('display.max_columns', 100)<jupyter_output><empty_output><jupyter_text>2) Import the dataset<jupyter_code>df = pd.read_pickle('Data/nlsy_intro_data')
# show dataframe

df
# Seperate the dataset by years

df_container = []
for i in range(0,11):
    df_container.append(df.loc[df['Period'] == i])
    df_container[i] = df_container[i].set_index('Identifier')

# show data_container for different years

df_container[8]<jupyter_output><empty_output><jupyter_text>3) Explore the dataset<jupyter_code># number of individuals:

len(df.Identifier.unique())
# describe the dataset

df.describe()<jupyter_output><empty_output><jupyter_text>3.1.) Explore the wage variable<jupyter_code>#  Generate a table with the average wage by occupation and wage

pd.crosstab(index = df['Age'], columns = df['Choice'], values = df['Wage'], aggfunc = 'mean', margins =True)<jupyter_output><empty_output><jupyter_text>4) Run a simple regressions for period 9 $\text{log}(\text{wage}_{i,t}) = $ $\beta_{t,0}$ $+ \beta_{t,1} \cdot \text{schooling}_{i,t} $ $+ \beta_{t,2} \cdot \text{AFQT}_{i,t} $ $+  \epsilon_{i,t}$where $t = 9$<jupyter_code># create matrices
y, x = dmatrices('Log_wage ~ Schooling + AFQT_1', data = df_container[9])

# show dependent variable
y
# show regressor matrix
x
# choose the model (here OLS)

model_fit = sm.OLS(y,x)
# fit the model and store results

results = model_fit.fit()
# print results

print(results.summary())
# Predict fitted values

y_hat = results.predict()
y_hat
# compute the residuals

u = y - y_hat
u

# Access the parameters

results.params
# t statistic

results.tvalues<jupyter_output><empty_output><jupyter_text>Please check [Statsmodels's](https://www.statsmodels.org/dev/regression.html) documentation website for further information and examples.   4) Describe the regressionsFor each year run the regression: $\text{log}(\text{wage}_{i,t}) = $$\beta_{t,0}$ $+ \beta_{t,1} \cdot \text{schooling}_{i,t} $$+ \beta_{t,2} \cdot \text{exper_blue}_{i,t} $$+ \beta_{t,3} \cdot \text{exper_blue}_{i,t}^2 $$+ \beta_{t,4} \cdot \text{exper_white}_{i,t} $$+ \beta_{t,5} \cdot \text{exper_white}_{i,t}^2 $$+ \beta_{t,6} \cdot \text{exper_military}_{i,t} $$+ \beta_{t,7} \cdot \text{exper_military}_{i,t}^2 $$+ \beta_{t,8} \cdot \text{AFQT}_{i,t} $$+ \beta_{t,9} \cdot \text{rotter_score}_{i,t} $$+ \beta_{t,10} \cdot \text{rosenberg_score}_{i,t} $$+ \beta_{t, 11} \cdot \text{mother_schooling}_{i,t} $$+ \text{year_dummies} $$+  \epsilon_{i,t}$  5) Run the regressions for period 9<jupyter_code>y, x = dmatrices('Log_wage ~ Schooling + exper_blue + np.power(exper_blue,2)+ exper_white + \
                np.power(exper_white,2) + exper_military + np.power(exper_military,2) + AFQT_1 + \
                ROTTER_SCORE + ROSENBERG_SCORE + Mother_edu + d_1985 + d_1986  + d_1987 \
                + d_1988', data = df_container[9])

# choose the model (here OLS)

model_fit = sm.OLS(y,x)
# fit the model and store results

results = model_fit.fit()
# print results

print(results.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:               Log_wage   R-squared:                       0.168
Model:                            OLS   Adj. R-squared:                  0.140
Method:                 Least Squares   F-statistic:                     6.076
Date:                Sun, 25 Nov 2018   Prob (F-statistic):           5.23e-11
Time:                        15:06:47   Log-Likelihood:                -321.46
No. Observations:                 436   AIC:                             672.9
Df Residuals:                     421   BIC:                             734.1
Df Model:                          14                                         
Covariance Type:            nonrobust                                         
===============================================================================================
                                  c[...]<jupyter_text>5) Run the regressions for all periods and store the residuals as a variable<jupyter_code>model_string = 'Log_wage ~ Schooling + exper_blue + np.power(exper_blue,2)+ exper_white + \
                np.power(exper_white,2) + exper_military + np.power(exper_military,2) + AFQT_1 + \
                ROTTER_SCORE + ROSENBERG_SCORE + Mother_edu + d_1978 + d_1979 + d_1980 + d_1981 + \
                d_1982 + d_1983 + d_1984 + d_1985 + d_1986  + d_1987 + d_1988'
for i in df_container:
    y,x = dmatrices(model_string, i)
    i['resid'] = sm.OLS(y, x).fit().resid<jupyter_output><empty_output><jupyter_text>6) Density Plots of Residuals  6.1) Density between residuals of period 2 and period 9<jupyter_code># Intersection of residuals in period 2 and period 9 (may not be the case due to attrition)

matched_identifier = list(set(df_container[2].index) & set(df_container[9].index)) 
# Define values used for the x and y axis of the density plot
x = df_container[2].loc[matched_identifier]['resid'].values
y = df_container[9].loc[matched_identifier]['resid'].values
# Create density plot

graph = sns.jointplot(x,y, kind = 'kde')

x_axis = 'Resid of period 2'
y_axis = 'Residual of period 9'

graph.set_axis_labels(x_axis,y_axis, fontsize =12)<jupyter_output><empty_output><jupyter_text>6.1) Density between residuals of period 8 and period 9<jupyter_code># Intersection of residuals in period 8 and period 9 (may not be the case due to attrition)

matched_identifier = list(set(df_container[8].index) & set(df_container[9].index)) 
# Define values used for the x and y axis
x = df_container[8].loc[matched_identifier]['resid'].values
y = df_container[9].loc[matched_identifier]['resid'].values
graph = sns.jointplot(x, y, kind = "kde")
x_axis = 'Residual of Period 8'
y_axis = 'Residual of Period 9'
graph.set_axis_labels(x_axis, y_axis, fontsize=12)
sns.plt.title('Observations: ' + str(len(matched_identifier)))<jupyter_output><empty_output><jupyter_text>More information on seaborn jointplots may be found [here](https://seaborn.pydata.org/generated/seaborn.jointplot.html)  7) Heatmap  7.1) Covariance matrix<jupyter_code>cov = np.empty((11,11))
cov[:] = np.nan

column = -1
for period in df_container:
    row = -1
    column += 1
    for lag in df_container:
        row += 1
        matched_identifier = list(set(period.index) & set(lag.index))  
        
        # more than 30 observations:
        if len(matched_identifier) >= 30:
            cov[(row,column)] = round((np.cov(period.loc[matched_identifier]['resid'].values, 
                                 lag.loc[matched_identifier]['resid'].values, rowvar = False, bias = True)[(0,1)]),2)
        else:
            pass
# show coariance
cov<jupyter_output><empty_output><jupyter_text>7.2) Covariance matrix as a heatmap  Result<jupyter_code># Heatmap of covariance matrix of residuals

mask = np.zeros_like(cov[1:11,1:11])
mask[np.triu_indices_from(mask, k = 1)] = True
with sns.axes_style("white"):
    ax = sns.heatmap(np.round(cov[1:11,1:11],2),mask=mask,annot=True)

plt.xlabel('Residuals of Period')
plt.ylabel('Residuals of Period')
ax.set_yticklabels(reversed(range(1, 11)))
ax.set_xticklabels(range(1, 11))
plt.title('Covariance matrix of residuals')
plt.savefig('Figures/cov_heatmap.png')<jupyter_output><empty_output><jupyter_text>Generate the covariance matrix heatmap step by step1. Plot the covariance matrix in a heatmap2. Select relevant covariance matrix rows and columns3. Create a mask to show only covariance triangular matrix below the diagonal4. Add covariance values to the heatmap5. Add x axis and y axis names. Add a title.6. Adjust the counter of the axes7. Save the figure8. Change the color to blue<jupyter_code># create the heatmap step by step
mask = np.zeros_like(cov[1:11, 1:11])
mask[np.triu_indices_from(mask, k = 1)] = True
with sns.axes_style('white'):
        ax = sns.heatmap(cov[1:11, 1:11], mask = mask, annot = True, cmap="YlGnBu")
        
plt.title('Covariance matrix')
plt.xlabel('Resid of period')
plt.ylabel('Resid of period')
ax.set_yticklabels(reversed(range(1, 11)))
ax.set_xticklabels(range(1, 11))

plt.savefig('Figures/cov_matrix.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13791649093209615, 'fraction_numerical': 0.030577815267819487, 'mean_word_length': 3.80253164556962, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '127019', 'license': 'MIT', 'n_tokens_mistral': 3353, 'n_tokens_neox': 3073, 'n_words': 827, 'path': 'guest_lectures/material/nlsy_introduction_notebook.ipynb', 'repo_name': 'HumanCapitalAnalysis/research-module', 'size': '9495'}