<jupyter_start><jupyter_text>--- Lecture 11: Regression discontinuity--- --- Lee (2008)--- The author studies the "incumbency advantage", i.e. the overall causal impact of being the current incumbent party in a district on the votes obtained in the district's election.* Lee, <NAME>. (2008). Randomized experiments from non-random selection in U.S. House elections. Journal of Econometrics.<jupyter_code>df_base = pd.read_csv("../../datasets/processed/msc/house.csv")
df_base.head()<jupyter_output><empty_output><jupyter_text>--- What are the basic characteristics of the dataset?---<jupyter_code>df_base.plot.scatter(x=0, y=1)<jupyter_output><empty_output><jupyter_text>What is the re-election rate?<jupyter_code>pd.crosstab(
    df_base.vote_last > 0.0,
    df_base.vote_next > 0.5,
    margins=True,
    normalize="columns",
)<jupyter_output><empty_output><jupyter_text>--- Regression discontinuity design--- How does the average vote in the next election look like as we move along last year's election.<jupyter_code>df_base["bin"] = pd.cut(df_base.vote_last, 200, labels=False)
df_base.groupby("bin").vote_next.mean().plot()<jupyter_output><empty_output><jupyter_text>Now we turn to an explicit model of the conditional mean.<jupyter_code>def fit_regression(incumbent, level=4):
    assert incumbent in ["republican", "democratic"]

    if incumbent == "republican":
        df_incumbent = df_base.loc[df_base.vote_last < 0.0, :]
    else:
        df_incumbent = df_base.loc[df_base.vote_last > 0.0, :]

    for level in range(2, level + 1):
        label = "vote_last_{:}".format(level)
        df_incumbent.loc[:, label] = df_incumbent["vote_last"] ** level

    formula = "vote_next ~ vote_last + vote_last_2 + vote_last_3 + vote_last_4"
    rslt = smf.ols(formula=formula, data=df_incumbent).fit()

    return rslt


rslt = dict()
for incumbent in ["republican", "democratic"]:
    rslt = fit_regression(incumbent, level=4)
    title = "\n\n {:}\n".format(incumbent.capitalize())
    print(title, rslt.summary())<jupyter_output>Republican
                             OLS Regression Results                            
==============================================================================
Dep. Variable:              vote_next   R-squared:                       0.277
Model:                            OLS   Adj. R-squared:                  0.276
Method:                 Least Squares   F-statistic:                     262.5
Date:                Tue, 28 Apr 2020   Prob (F-statistic):          4.33e-191
Time:                        13:15:10   Log-Likelihood:                 1761.3
No. Observations:                2740   AIC:                            -3513.
Df Residuals:                    2735   BIC:                            -3483.
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err   [...]<jupyter_text>How does the predictions look like?<jupyter_code>for incumbent in ["republican", "democratic"]:

    rslt = fit_regression(incumbent, level=4)

    # For our predictions, we need to set up a grid for the evaluation.
    if incumbent == "republican":
        grid = np.linspace(-0.5, 0.0, 100)
    else:
        grid = np.linspace(+0.0, 0.5, 100)

    df_grid = pd.DataFrame(grid, columns=["vote_last"])
    for level in range(2, 5):
        label = "vote_last_{:}".format(level)
        df_grid.loc[:, label] = df_grid["vote_last"] ** level

    ax = rslt.predict(df_grid).plot(title=incumbent.capitalize())
    plt.show()<jupyter_output><empty_output><jupyter_text>We can now compute the difference at the cutoffs to get an estimate for the treatment effect.<jupyter_code>before_cutoff = df_base.groupby("bin").vote_next.mean()[99]
after_cutoff = df_base.groupby("bin").vote_next.mean()[100]

effect = after_cutoff - before_cutoff
print("Treatment Effect: {:5.3f}%".format(effect * 100))<jupyter_output>Treatment Effect: 8.823%<jupyter_text>--- How does the estimated treatment effect depend on the choice of the bin width?---<jupyter_code>for num_bins in [100, 200]:
    df = df_base.copy(deep=True)
    df["bin"] = pd.cut(df_base.vote_last, num_bins, labels=False)
    info = df.groupby("bin").vote_next.mean()
    lower = (num_bins / 2) - 1
    effect = info[lower + 1] - info[lower]
    print(
        " Number of bins: {:}, Width {:>5}, Effect {:5.2f}%".format(
            num_bins, 1.0 / num_bins, effect * 100
        )
    )<jupyter_output>Number of bins: 100, Width  0.01, Effect  8.12%
 Number of bins: 200, Width 0.005, Effect  8.82%<jupyter_text>--- Regression --- There are several alternatives to estimate the conditional mean functions.* pooled regressions* local linear regressions<jupyter_code># It will be useful to split the sample by the cutoff value
# for easier access going forward.
df_base["D"] = df_base.vote_last > 0<jupyter_output><empty_output><jupyter_text>Pooled regressionWe estimate the conditinal mean using the whole function.\begin{align*}Y = \alpha_r + \tau D + \beta X + \epsilon\end{align*}This allows for a difference in levels but not slope.<jupyter_code>smf.ols(formula="vote_next ~ vote_last + D", data=df_base).fit().summary()<jupyter_output><empty_output><jupyter_text>Local linear regressionWe now turn to local regressions by restricting the estimation to observations close to the cutoff.\begin{align*}Y = \alpha_r + \tau D + \beta X + \gamma X D + \epsilon,\end{align*}where $-h \geq X \geq h$. This allows for a difference in levels and slope.<jupyter_code>for h in [0.3, 0.2, 0.1, 0.05, 0.01]:
    # We restrict the sample to observations close
    # to the cutoff.
    df = df_base[df_base.vote_last.between(-h, h)]

    formula = "vote_next ~ D + vote_last + D * vote_last"
    rslt = smf.ols(formula=formula, data=df).fit()
    info = [h, rslt.params[1] * 100, rslt.pvalues[1]]
    print(
        " Bandwidth: {:>4}   Effect {:5.3f}%    pvalue {:5.3f}".format(*info)
    )<jupyter_output>Bandwidth:  0.3   Effect 8.318%    pvalue 0.000
 Bandwidth:  0.2   Effect 7.818%    pvalue 0.000
 Bandwidth:  0.1   Effect 6.058%    pvalue 0.000
 Bandwidth: 0.05   Effect 4.870%    pvalue 0.010
 Bandwidth: 0.01   Effect 9.585%    pvalue 0.001
{'fraction_non_alphanumeric': 0.14544880896040385, 'fraction_numerical': 0.035336803912289, 'mean_word_length': 2.947696139476961, 'pattern_counts': {'":': 2, '<': 43, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '5995', 'license': 'MIT', 'n_tokens_mistral': 2263, 'n_tokens_neox': 2020, 'n_words': 531, 'path': 'lectures/11-regression-discontinuity/notebook.ipynb', 'repo_name': 'milakis/microeconometrics', 'size': '6340'}