<jupyter_start><jupyter_text>* Python code replication of:" https://www.kaggle.com/janniskueck/pm2-notebook-jannis "* Created by: <NAME> and <NAME>  This notebook contains an example for teaching.  Testing the Convergence Hypothesis  Introduction We provide an additional empirical example of partialling-out with Lasso to estimate the regression coefficient $\beta_1$ in the high-dimensional linear regression model:  $$  Y = \beta_1 D +  \beta_2'W + \epsilon.  $$  Specifically, we are interested in how the rates  at which economies of different countries grow ($Y$) are related to the initial wealth levels in each country ($D$) controlling for country's institutional, educational, and other similar characteristics ($W$).  The relationship is captured by $\beta_1$, the *speed of convergence/divergence*, which measures the speed at which poor countries catch up $(\beta_1 0)$ rich countries, after controlling for $W$. Our inference question here is: do poor countries grow faster than rich countries, controlling for educational and other characteristics? In other words, is the speed of convergence negative: $ \beta_1 <0?$ This is the Convergence Hypothesis predicted by the Solow Growth Model. This is a structural economic model. Under some strong assumptions, that we won't state here, the predictive exercise we are doing here can be given causal interpretation. The outcome $Y$ is the realized annual growth rate of a country's wealth  (Gross Domestic Product per capita). The target regressor ($D$) is the initial level of the country's wealth. The target parameter $\beta_1$ is the speed of convergence, which measures the speed at which poor countries catch up with rich countries. The controls ($W$) include measures of education levels, quality of institutions, trade openness, and political stability in the country.  Data analysis We consider the data set GrowthData which is included in the package *hdm*. First, let us load the data set to get familiar with the data.<jupyter_code>import hdmpy
import pandas as pd
import numpy as np
import pyreadr
import math
import matplotlib.pyplot as plt
import random
# I downloaded the data that the author used
growth_read = pyreadr.read_r("../data/GrowthData.RData")

# Extracting the data frame from rdata_read
growth = growth_read[ 'GrowthData' ]
list(growth)<jupyter_output><empty_output><jupyter_text>We determine the dimension of our data set.<jupyter_code>growth.shape<jupyter_output><empty_output><jupyter_text>The sample contains $90$ countries and $63$ controls. Thus $p \approx 60$, $n=90$ and $p/n$ is not small. We expect the least squares method to provide a poor estimate of $\beta_1$.  We expect the method based on partialling-out with Lasso to provide a high quality estimate of $\beta_1$. To check this hypothesis, we analyze the relation between the output variable $Y$ and the other country's characteristics by running a linear regression in the first step.<jupyter_code>import statsmodels.api as sm
import statsmodels.formula.api as smf
# We create the main variables
y = growth['Outcome']
X = growth.drop('Outcome', 1)
# OLS regression
reg_ols  = sm.OLS(y, X).fit()
print(reg_ols.summary())
# output: estimated regression coefficient corresponding to the target regressor
est_ols = reg_ols.summary2().tables[1]['Coef.']['gdpsh465']

# output: std. error
std_ols = reg_ols.summary2().tables[1]['Std.Err.']['gdpsh465']

# output: 95% confidence interval
lower_ci = reg_ols.summary2().tables[1]['[0.025']['gdpsh465']
upper_ci = reg_ols.summary2().tables[1]['0.975]']['gdpsh465']<jupyter_output><empty_output><jupyter_text>Summarize OLS results<jupyter_code>table_1 = np.zeros( (1, 4) )

table_1[0,0] = est_ols  
table_1[0,1] = std_ols   
table_1[0,2] = lower_ci
table_1[0,3] = upper_ci    


table_1_pandas = pd.DataFrame( table_1, columns = [ "Estimator","Std. Error", "lower bound CI", "upper bound CI"  ])
table_1_pandas.index = [ "OLS" ]
table_1_html = table_1_pandas.to_html()
table_1_html<jupyter_output><empty_output><jupyter_text>estimator   standard error   lower bound CI   upper bound CI       OLS   -0.009   0.030   -0.071   0.052      Least squares provides a rather noisy estimate (high standard error) of thespeed of convergence, and does not allow us to answer the questionabout the convergence hypothesis since the confidence interval includes zero.In contrast, we can use the partialling-out approach based on lasso regression ("Double Lasso").<jupyter_code># Create main variables
Y = growth['Outcome']
W = growth.drop(['Outcome','intercept', 'gdpsh465'], 1 )
D = growth['gdpsh465']<jupyter_output><empty_output><jupyter_text>Method 1 - Using Sklearn<jupyter_code>from sklearn import linear_model

# Seat values for Lasso
lasso_model = linear_model.Lasso( alpha = 0.00077 )
r_Y = Y - lasso_model.fit( W, Y ).predict( W )
r_Y = r_Y.rename('r_Y')

# Part. out d
r_D = D - lasso_model.fit( W, D ).predict( W )
r_D = r_D.rename('r_D')

# ols 
partial_lasso_fit = sm.OLS(r_Y, r_D).fit()

est_lasso = partial_lasso_fit.summary2().tables[1]['Coef.']['r_D']
std_lasso = partial_lasso_fit.summary2().tables[1]['Std.Err.']['r_D']
lower_ci_lasso = partial_lasso_fit.summary2().tables[1]['[0.025']['r_D']
upper_ci_lasso = partial_lasso_fit.summary2().tables[1]['0.975]']['r_D']
# from sklearn import linear_model

# # Seat values for Lasso
# lasso_model = linear_model.Lasso( alpha = 0.00077 )
# r_Y = Y - lasso_model.fit( W, Y ).predict( W )
# r_Y = r_Y.rename('r_Y')

# # Part. out d
# r_D = D - lasso_model.fit( W, D ).predict( W )
# r_D = r_D.rename('r_D')

# # Regress residuales
# partial_lasso_fit = sm.OLS(r_Y, r_D).fit()

# # output: estimated regression coefficient corresponding to the target regressor
# est_lasso = partial_lasso_fit.summary2().tables[1]['Coef.']['r_D']

# # output: std. error
# std_lasso = partial_lasso_fit.summary2().tables[1]['Std.Err.']['r_D']

# # output: 95% confidence interval
# lower_ci_lasso = partial_lasso_fit.summary2().tables[1]['[0.025']['r_D']
# upper_ci_lasso = partial_lasso_fit.summary2().tables[1]['0.975]']['r_D']
# Regress residuales
partial_lasso_fit = sm.OLS(r_Y, r_D).fit()
partial_lasso_est = partial_lasso_fit.summary2().tables[1]['Coef.']['r_D']

print( f"Coefficient for D via partialling-out using lasso {partial_lasso_est}" )
# output: estimated regression coefficient corresponding to the target regressor
est_lasso = partial_lasso_fit.summary2().tables[1]['Coef.']['r_D']

# output: std. error
std_lasso = partial_lasso_fit.summary2().tables[1]['Std.Err.']['r_D']

# output: 95% confidence interval
lower_ci_lasso = partial_lasso_fit.summary2().tables[1]['[0.025']['r_D']
upper_ci_lasso = partial_lasso_fit.summary2().tables[1]['0.975]']['r_D']<jupyter_output><empty_output><jupyter_text>Summary LASSO results Finally, let us have a look at the results.<jupyter_code>table_2 = np.zeros( (1, 4) )

table_2[0,0] = est_lasso  
table_2[0,1] = std_lasso   
table_2[0,2] = lower_ci_lasso
table_2[0,3] = upper_ci_lasso    


table_2_pandas = pd.DataFrame( table_2, columns = [ "Estimator","Std. Error", "lower bound CI", "upper bound CI"  ])
table_2_pandas.index = [ "LASSO" ]
table_2_pandas
table_3 = table_1_pandas.append(table_2_pandas)
table_3
table_3_html = table_3.to_html()
print(table_3_html)<jupyter_output><table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Estimator</th>
      <th>Std. Error</th>
      <th>lower bound CI</th>
      <th>upper bound CI</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>OLS</th>
      <td>-0.009378</td>
      <td>0.029888</td>
      <td>-0.070600</td>
      <td>0.051844</td>
    </tr>
    <tr>
      <th>LASSO</th>
      <td>-0.047747</td>
      <td>0.017705</td>
      <td>-0.082926</td>
      <td>-0.012567</td>
    </tr>
  </tbody>
</table><jupyter_text>The least square method provides a rather noisy estimate of the speed of convergence. We can not answer the question if poor countries grow faster than rich countries. The least square method does not work when the ratio $p/n$ is large.In sharp contrast, partialling-out via Lasso provides a more precise estimate. The Lasso based point estimate is $-5\%$ and the $95\%$ confidence interval for the (annual) rate of convergence $[-7.8\%,-2.2\%]$ only includes negative numbers. This empirical evidence does support the convergence hypothesis.                   Estimator      Std. Error      lower bound CI      upper bound CI                  OLS      -0.009378      0.029888      -0.070600      0.051844              LASSO      -0.047747      0.017705      -0.082926      -0.012567        Method 2 - HDMPY<jupyter_code>res_Y = hdmpy.rlasso( W, Y, post=True ).est['residuals']
res_D = hdmpy.rlasso( W, D, post=True ).est['residuals']

r_Y = pd.DataFrame(res_Y, columns=['r_Y'])
r_D = pd.DataFrame(res_D, columns=['r_D'])
# OLS regression
reg_ols  = sm.OLS(r_Y, r_D).fit()
print(reg_ols.summary())
# output: estimated regression coefficient corresponding to the target regressor
est_lasso = reg_ols.summary2().tables[1]['Coef.']['r_D']

# output: std. error
std_lasso = reg_ols.summary2().tables[1]['Std.Err.']['r_D']

# output: 95% confidence interval
lower_ci_lasso = reg_ols.summary2().tables[1]['[0.025']['r_D']
upper_ci_lasso = reg_ols.summary2().tables[1]['0.975]']['r_D']
table_3 = np.zeros( (1, 4) )

table_3[0,0] = est_lasso   
table_3[0,1] = std_lasso    
table_3[0,2] = lower_ci_lasso 
table_3[0,3] = upper_ci_lasso     


table_3_pandas = pd.DataFrame( table_3, columns = [ "Estimator","Std. Error", "lower bound CI", "upper bound CI"  ]) 
table_3_pandas.index = [ "LASSO" ]
table_3_pandas<jupyter_output><empty_output><jupyter_text>Method 3 - HDMPY Direct<jupyter_code>lasso_direct = hdmpy.rlassoEffect(x=W, y=Y, d=D, method="partialling out")
lasso_direct
est_lasso = lasso_direct["coefficients"]
std_lasso = lasso_direct["se"]
lower_ci_lasso = est_lasso - 1.96*std_lasso
upper_ci_lasso = est_lasso + 1.96*std_lasso
table_4 = np.zeros( (1, 4) )

table_4[0,0] = est_lasso   
table_4[0,1] = std_lasso    
table_4[0,2] = lower_ci_lasso 
table_4[0,3] = upper_ci_lasso     


table_4_pandas = pd.DataFrame( table_4, columns = [ "Estimator","Std. Error", "lower bound CI", "upper bound CI"  ]) 
table_4_pandas.index = [ "LASSO_direct" ]
table_4_pandas<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12734155100456176, 'fraction_numerical': 0.03610598854702514, 'mean_word_length': 4.727626459143969, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 80, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '132707', 'license': 'MIT', 'n_tokens_mistral': 3792, 'n_tokens_neox': 3575, 'n_words': 998, 'path': 'Jupyter_Notebooks/.ipynb_checkpoints/pm2_notebook_jannis-checkpoint.ipynb', 'repo_name': 'JannisKueck/14.38_Causal_ML', 'size': '10320'}