<jupyter_start><jupyter_text>Estimate the required bandwidthWe use global bandwidth parameters for the kde that builds PBjam's prior.  These parameters (hyperparameters) have to be estimated.  We use the cross validated maximum likelihood method detailed in statsmodels. Read in the current data<jupyter_code>import pandas as pd
df_full = pd.read_csv('prior_data.csv')
print(df_full.columns)
df_full = df_full.drop(columns='Unnamed: 0')
df = df_full[::10]
df.head()
print(df.values.shape)<jupyter_output>(1217, 10)<jupyter_text>Set up the kde<jupyter_code>import statsmodels.api as sm
kde = sm.nonparametric.KDEMultivariate(data=df.values, 
                                      var_type='cccccccccc', 
                                      bw='cv_ml')<jupyter_output>/home/davies/.local/lib/python3.6/site-packages/statsmodels/nonparametric/kernel_density.py:161: RuntimeWarning: invalid value encountered in log
  L += func(f_i)
/home/davies/.local/lib/python3.6/site-packages/statsmodels/nonparametric/kernel_density.py:161: RuntimeWarning: divide by zero encountered in log
  L += func(f_i)<jupyter_text>Display the optimal bandwidth<jupyter_code>print(kde.bw)<jupyter_output>[0.0104277  0.0087742  0.03554314 0.02627906 0.11038797 0.63698899
 0.03219839 0.06540209 0.0051097  0.08510194]
{'fraction_non_alphanumeric': 0.10061919504643962, 'fraction_numerical': 0.08204334365325078, 'mean_word_length': 4.455696202531645, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '234529', 'license': 'MIT', 'n_tokens_mistral': 515, 'n_tokens_neox': 424, 'n_words': 92, 'path': 'pbjam/data/EstimateBandwidth.ipynb', 'repo_name': 'nielsenmb/PBjam', 'size': '1292'}