<jupyter_start><jupyter_text>Find the best bandwidth around the threshold (cutoff point)<jupyter_code>bandwidth_opt = rdd.optimal_bandwidth(data['y'], data['x'], cut=threshold)
print("Optimal bandwidth:", bandwidth_opt)<jupyter_output>Optimal bandwidth: 0.7448859965965812<jupyter_text>Preserve the data with 'x' within bandwidth of the cutoff point<jupyter_code>data_rdd = rdd.truncated_data(data, 'x', bandwidth_opt, cut=threshold)
# directly plot the datapoints

plt.figure(figsize=(12, 8))
plt.scatter(data_rdd['x'], data_rdd['y'], facecolors='none', edgecolors='r')
plt.xlabel('x')
plt.ylabel('y')
plt.axvline(x=threshold, color='b')
plt.show()
plt.close()
data_rdd.head()<jupyter_output><empty_output><jupyter_text>To better visualize, put data into bins (here 100 bins in total) and compute the mean<jupyter_code>data_binned = rdd.bin_data(data_rdd, 'y', 'x', 100)
data_binned.head()
plt.figure(figsize=(12, 8))
plt.scatter(data_binned['x'], data_binned['y'],
    s = data_binned['n_obs'], facecolors='none', edgecolors='r')
plt.axvline(x=threshold, color='b')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>Estimation Here the effect of the 'cutoff' event can be estimated using different formulas. The formulas can be specified in the class 'rdd'. In the following case, we use the regression y ~ TREATED + x. Here TREATED=1 if x >= cut, and 0 otherwise.<jupyter_code># estimate the model directly
# treatment = 1 if x>=threshold
model = rdd.rdd(data_rdd, 'x', 'y', cut=threshold)
print(model.fit().summary())<jupyter_output>Estimation Equation:	 y ~ TREATED + x
                            WLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.508
Model:                            WLS   Adj. R-squared:                  0.508
Method:                 Least Squares   F-statistic:                     2811.
Date:                Mon, 20 Jan 2020   Prob (F-statistic):               0.00
Time:                        23:20:31   Log-Likelihood:                -7794.0
No. Observations:                5442   AIC:                         1.559e+04
Df Residuals:                    5439   BIC:                         1.561e+04
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
==============================================================================
              [...]<jupyter_text>This example includes other variables and interaction w1*w2<jupyter_code>model = rdd.rdd(data_rdd, 'x', cut=threshold, equation='y ~ TREATED + x + w1*w2')
print(model.fit().summary())<jupyter_output>Estimation Equation:	 y ~ TREATED + x + w1*w2
                            WLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.523
Model:                            WLS   Adj. R-squared:                  0.523
Method:                 Least Squares   F-statistic:                     1194.
Date:                Mon, 20 Jan 2020   Prob (F-statistic):               0.00
Time:                        23:21:35   Log-Likelihood:                -7709.6
No. Observations:                5442   AIC:                         1.543e+04
Df Residuals:                    5436   BIC:                         1.547e+04
Df Model:                           5                                         
Covariance Type:            nonrobust                                         
==============================================================================
      [...]<jupyter_text>This example involves interaction x*TREATED and allows different coefficients of x before and after the event.<jupyter_code>model = rdd.rdd(data_rdd, 'x', cut=threshold, equation='y ~ TREATED + x + x*TREATED')
print(model.fit().summary())<jupyter_output>Estimation Equation:	 y ~ TREATED + x + x*TREATED
                            WLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.508
Model:                            WLS   Adj. R-squared:                  0.508
Method:                 Least Squares   F-statistic:                     1874.
Date:                Mon, 20 Jan 2020   Prob (F-statistic):               0.00
Time:                        23:22:25   Log-Likelihood:                -7793.7
No. Observations:                5442   AIC:                         1.560e+04
Df Residuals:                    5438   BIC:                         1.562e+04
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
  [...]
{'fraction_non_alphanumeric': 0.17988584924227513, 'fraction_numerical': 0.038771895296201536, 'mean_word_length': 1.7220139260846277, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '301781', 'license': 'MIT', 'n_tokens_mistral': 1534, 'n_tokens_neox': 1317, 'n_words': 329, 'path': 'RDD_interface.ipynb', 'repo_name': 'raym96/personal', 'size': '5081'}