<jupyter_start><jupyter_text>__Q1: Descriptive analysis__ __Q1.1: 1.1 Summary statistics__<jupyter_code>#Read the data
data = pd.read_csv('progresa-sample.csv.bz2')
#Checking all the columns of the data
data.columns
#Validating the data
data.head()
#Verifying the shape
data.shape
#Recoding the progresa variable
data['progresa'] = np.where(data['progresa'] == 'basal' ,1.0,0.0) 
#Checking the validity of the variable
data['progresa'].nunique()
#Dropping the null variables
data = data.dropna()
data['progresa'].nunique()
#Shape after dropping the null variables.
data.shape<jupyter_output><empty_output><jupyter_text>_There were totally 10128 rows affected by null values_<jupyter_code>#Putting all the necessary variables in the data to check the mean, median etc
data2 = pd.DataFrame(data[['sex', 'indig', 'dist_sec','sc','grc','fam_n', 'min_dist',
       'dist_cap', 'poor', 'progresa', 'hohedu', 'hohwag', 'welfare_index',
       'hohsex', 'hohage', 'age', 'grc97', 'sc97']].describe())
#Validating the data
data2.head()
#Resetting index
data2 = data2.reset_index()
#Validating the data
data2.head()
#Pivoting the table
data2.pivot_table(columns=['index'])<jupyter_output><empty_output><jupyter_text>_All the variables are displayed in a neat tabular format__The variables appear in alphabetic order_ __1.2 : Differences at baseline?__<jupyter_code>from scipy import stats
from scipy.stats import t<jupyter_output><empty_output><jupyter_text>__Q1.2.1 and Q1.2.2__<jupyter_code>#Reading the data again to make changes to the data and not copying it to the original
data2 = pd.read_csv('progresa-sample.csv.bz2')

#Mapping poor and progresa
data2['poor'] = data2['poor'].map({'pobre': 1, 'no pobre': 0})
data2['progresa'] = data2['progresa'].map({'basal' : 1, '0':0})
# Segregating into treatment and Control data
treatment_97 = pd.DataFrame(data2[(data2.year == 97) & (data2.poor == 1) & (data2.progresa == 1)])
control_97 = pd.DataFrame(data2[(data2.year == 97) & (data2.poor == 1) & (data2.progresa == 0)])
# Selecting rows where poor=1 and the year=97, and then grouping by 'progresa' column
new_data = data2[(data2.year == 97) & (data2.poor == 1)].groupby('progresa').mean()
new_data.drop(new_data.columns[[0,9,16,17]], axis =1,inplace=True)

new_data = new_data.transpose()
# swapping columns to match the structure of the required table
new_data = new_data[[1.0,0.0]]

# Resetting Index
new_data.reset_index(level=0, inplace=True)
new_data.rename(columns={'index' : 'Variable name', 0: 'Average value (Control villages)', 1: 'Average value (Treatment villages)'}, inplace=True)

# List of all Variables
var_list = list(new_data['Variable name'])

# Calculating T test for the Treatment, Control
tt = list(stats.ttest_ind(treatment_97[var_list], control_97[var_list], nan_policy='omit'))

# Adding the remaining two columns.
new_data['Difference (Treat - Control)'] = tt[0]
new_data['p-value'] = tt[1]

# for a better look at the insignificant data with respect to the value of p
new_data['p<0.05'] = new_data['p-value'] < 0.05

new_data.sort_values('Variable name')<jupyter_output><empty_output><jupyter_text>_Displayed above are all the means differences and p-values with Variable name in ascending order_ __Q1.2.3, Q1.2.4, Q1.2.5__<jupyter_code>new_data[new_data['p<0.05']==True].sort_values('Variable name')<jupyter_output><empty_output><jupyter_text>_Q1.2.3 There are 8 variables shown above which are statistically significant between the control and the treatment variables__They are namely: dist_cap, dist_sec, hohage, hohwag, min_dist, sex, welfare_index__Q1.2.4 It matters that there are baseline differences because if the differences are too large, we can say that the data is biased thus making the causality of the progresa program weak__However, the differences are not too large for the statistically significant variables__Q1.2.5 The measurement of the impact cannot be acurately measured by the baseline differences alone, we need to explore the linear relationship between the control and treatments variables to correctly identify the impact of the measurement._ __Q2: Measuring Impact__<jupyter_code>#Checking the shape
data.shape
#Taking only the poor data and creating a variable called after
newdata = data[data['poor'] == 'pobre']
newdata['after'] = np.where(newdata['year']==98, True, False)
#Validating the data
newdata.head()
#Checking the shape
newdata.shape
#Dropping NAs
newdata = newdata.dropna()
#Describing the data
newdata.after.describe()<jupyter_output><empty_output><jupyter_text>__Q2.1.1 compute the estimator by just comparing the average schooling rates for these villages.__<jupyter_code>#Average schooling rates
newdata[newdata.progresa==1.0].groupby('after').sc.mean()<jupyter_output><empty_output><jupyter_text>_Displayed the average schooling rates:__The average schooling rate before 1998 was 82.2%  and the average schooling rate after 1998 was 84.92%_ __Q2.1.2 now re-compute the estimator using linear regression, and individual schooling rates. Do not include other regressors.__<jupyter_code>#Linear model on sc and after
m = smf.ols(formula = 'sc~after', data=newdata[newdata.progresa == 1.0])
m.fit().summary()<jupyter_output><empty_output><jupyter_text>_Above displayed is the linear regression model of the school rates with the after variable__We can see that the estimate increases by 0.0266 when the after is true. The value for After=True is statistically significant__Therefore, The average schooling rate before 1998 was 82.27%  and the average schooling rate after 1998 was 84.87% which is comparable to the mean model_ __Q2.1.3 finally, estimate a multiple regression model that includes other covariates.__<jupyter_code>#Multiple regression model on sc and other covariates
m = smf.ols(formula = 'sc~after + dist_sec + sex + min_dist + dist_cap + hohedu', data=newdata[newdata.progresa == 1.0])
m.fit().summary()
#r.summary()<jupyter_output><empty_output><jupyter_text>_In the multiple regression model we can see that after=True is	0.0255 which means that the individual schooling rates have increased by 0.0255 after 1998. This variable is slightly smaller as compared to the linear model without covariates.__All the variables are statistically significant in the above model with p-value = 0.00. The t-value is the highest for the progresa variable. It seems to be much more significant compared to others__Therefore, The average schooling rate before 1998 was 71.47%  and the average schooling rate after 1998 was 73.97% which is comparable to the mean model_ __compare all the estimators. Are your estimates statistically significant? What do they suggestabout the efficacy of the progresa__ _The average enrollment rate before and after 1998 compares as follows:_                    Before               After                       Simple mean     82.2%               84.92%    Linear model     82.27%              84.87%    Multiple model   71.47%              73.97%_In all the cases, we see that the enrollment rate has increased with time having the progresa effect. The simple mean model and the linear model are almost comparable. The multiple model is quite lesser because of the effect of all the variables._ _In all three cases, the values for After are statistically significant._ __2.2: Cross-sectional estimator__<jupyter_code>#Validating the data
data.shape
#Selecting only the poor households
newdata = data[data['poor'] == 'pobre']
#Dropping NAs
newdata = newdata.dropna()
#Making an after variable
newdata['after'] = np.where(newdata['year']==98, True, False)
#Describing the after variable
newdata.after.describe()<jupyter_output><empty_output><jupyter_text>__Begin by estimating the impact of Progresa by compring the average enrollment rate amongpoor households in the treatment villages and the average enrollment rate among poor householdsin the control villages. What do you find?__<jupyter_code>#Getting the average enrollment rate for treatment and control villages
newdata[newdata['after']==True].groupby('progresa').sc.mean()<jupyter_output><empty_output><jupyter_text>_We find that the average enrollment rate for the treatment villages(Progresa=1) is 84% whereas the average enrollment rate for control villages(Progresa=0) is 81%_ __Now repeat the estimator using simple regression.__<jupyter_code>#Making a linear model
m = smf.ols(formula = 'sc ~ progresa', data=newdata)
r = m.fit()
r.summary()<jupyter_output><empty_output><jupyter_text>_From the above results, we can see that with the effect of progresa the average enrollment rate increased by 0.0218 that is 2.1%__Without the effect of progresa, the enrollment rate is 81.32%. Therefore with the effect of progresa, the enrollment rate is 83.42%_ __Third, use multiple regression to get the same estimate.__<jupyter_code>#Using multiple regression model
m = smf.ols(formula = 'sc ~ progresa + dist_sec + sex + min_dist + dist_cap + hohedu', data=newdata)
r = m.fit()
r.summary()<jupyter_output><empty_output><jupyter_text>_In the multiple regression model, we can see that the impact of progresa is slightly lesser as compared to the linear model. The enrollment rate is increasing by 1.9% here compared to the 2.1% in the linear model.__Without the effect of progresa or any other factors, the average enrollment according to the model above is 70.41%. Therefore, the enrollment rate for treatment villages is 72.34% with progresa__Many factors are afftecting this model there by diminishing the effect of progresa_ __Finally, as above, compare your three estimators. What do you find? Are the effects statisti-cally significant?__ _The average enrollment rate for the treatment and control villages compares as follows:_                  Treatment               Control                       Simple mean     84%                   81%    Linear model    83.5%                 81.32%    Multiple model  72.34%                70.41%_In all the cases, we see that the enrollment rate has increased with progresa. The simple mean model and the linear model are almost comparable. The multiple model is slightly lesser because of the effect of all the variables._ _In all three cases, the values for progresa are statistically significant_ __2.3: Differences-in-differences estimator__<jupyter_code>#Validating the data
data.shape
#Selecting only the poor households
newdata = data[data['poor'] == 'pobre']
#Dropping NAs
newdata = newdata.dropna()
#Making an after variable
newdata['after'] = np.where(newdata['year']==98, True, False)
#Describing the after variable
newdata.after.describe()<jupyter_output><empty_output><jupyter_text>__Start with the simple table. However, DiD requires 4-way comparison. So compare the averageenrollment rate among poor households in the treatment villages and the average enrollment rateamong poor households in the control villages, both 1997 and 1998. What do you find?__<jupyter_code>#Displaying the diff table
newdata.groupby(['progresa', 'after'], as_index = False).sc.mean()<jupyter_output><empty_output><jupyter_text>_We found that:__The difference for the Control Sample(Progresa = 0) is:_0.815066 - 0.810923 = 0.004143_The difference for the Treatment Sameple(Progresa = 1) is:_0.822697 - 0.849257 = -0.02656_The estimate of the impact(Diff-in-Diff) is:_0.004143 - (-0.02656) = 0.030703_The difference for 97 is:_0.815066 - 0.822697 = -0.007631_The difference for 98 is:_0.810923 - 0.849257 = -0.038334_The estimate of the impact(Diff-in-Diff) is:_-0.007631 - (-0.038334) = 0.030703 _This indicates that the increase in the enrollment rate can be credited to the progresa treatment__Without the progresa and time in place, the enrollment rate is 81.5% and with both of them it is, 84.57%_In both cases, the estimate of the impact is 0.030703_ __2.3.2:Now repeat the estimator using simple regression.__<jupyter_code>#Cross linear model of progresa and after
m = smf.ols(formula = 'sc ~ progresa * after', data=newdata)
r = m.fit()
r.summary()<jupyter_output><empty_output><jupyter_text>_The area of interest in the above regression model is progresa:after[T:True] whose coefficient is 0.0307 which is ths same as compared to the diff-in-diff calculated in the tabular format.__From the above table:__The enrollment rate without the effect of progresa or time: 81.51%__The enrollment rate with progresa, without the effect of time: 82.27%__The enrollment rate in 1998, without the effect of progresa: 80.75__The enrollment rate with the effect of progressa and in 1998: 84.58%__Therefore, the enrollment is the highest for effect of progresa and 1998. These values are comparable to the tabular diff-in-diff values__We can say that estimate through difference in difference approach are more accurate than simple difference because control and treatment variables are considered together with the introduction of interaction term where as they are considered in silos for simple difference method._ __And as above, use multiple regression to get the same estimate.__<jupyter_code>#Using multiple regression model
m = smf.ols(formula = 'sc ~ progresa * after + dist_sec + sex + min_dist + dist_cap + hohedu', data=newdata)
r = m.fit()
r.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08737201365187713, 'fraction_numerical': 0.03670838073568449, 'mean_word_length': 5.246328754144955, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '30147', 'license': 'MIT', 'n_tokens_mistral': 4254, 'n_tokens_neox': 3796, 'n_words': 1493, 'path': 'Causality and Diff-in-Diff with Progresa Data - Python.ipynb', 'repo_name': 'sp2609/Machine-Learning-with-Python', 'size': '13185'}