<jupyter_start><jupyter_text>Data GatherWe need lots of data to provide a meaningful prior - at a minimum we need:[ID, epsilon, epsilon_err, numax, numax_err, dnu, dnu_err, BP_RP, BP_RP_err, Teff, Teff_err]First we need a set of star catalogues to work from.  The obvious red giant catalogue is Yu.  The obvious dwarf (& SG) catalogue is Serenelli.  Where we get the epsilons from is harder.  The the Legacy & Kages stars we can do this easily.   For the SG's we might need to use TWhite's paper but these can probably be improved.Broadly speaking then, we 1) Read in Yu2) Read in Serenelli3) Concatenate Yu and Serenelli4) Use the concatenated sample to go and get BP_RP.5) The select down to a list of stars where we have either epsilon from a published source, or can calculate epsilon from the individual mode frequencies we have already.6) To calculate our own epsilons, for all peak bagged red giants, Legacy, and Kages, read in the mode frequencies and fit a simple 2nd order asymptotic expansion to get epsilon and its uncertainty.7) Merge the published epsilon and calculated epsilons (how='inner') back into the the main DataFrame.8) Save the sample as a csv file to the data dir than cab read in as the Prior sample.  Step 1 - read in Yu.Data source: (18/4/19) arXiv - table1.txt & table2.txtI have edited the files to update the header<jupyter_code>df_yu1 = pd.read_csv('Table1.txt', sep='|')
df_yu1.head()
df_yu2 = pd.read_csv('Table2.txt', sep='|')
df_yu2.head()
df_yu = df_yu1.merge(df_yu2, on='KICID', how='inner')
print(len(df_yu))
fig, ax = plt.subplots()
cs=ax.scatter(df_yu.numax, df_yu.dnu, c=df_yu.Teff)
ax.set_xscale('log')
ax.set_yscale('log')
cbar = fig.colorbar(cs)
cbar.ax.set_ylabel('Teff')
ax.set_xlabel('Numax')
ax.set_ylabel('Dnu')<jupyter_output><empty_output><jupyter_text>Step 2 - Read in SerenelliData source: (18/4/19) arXiv - table3.txt, table4.txt, table5.txtIn the end I wont use table4 for now (the SDSS temps) but with use table5 (the ASCAP temps).<jupyter_code>df_aldo1 = pd.read_csv('table3.txt', skiprows=22, sep='\s+')
df_aldo1.head()
df_aldo2 = pd.read_csv('table5.txt', skiprows=42
                       , sep='\s+')
df_aldo2.head()
df_aldo = df_aldo1.merge(df_aldo2, how='inner', on='KIC')
print(len(df_aldo))
df_aldo.columns
fig, ax = plt.subplots(figsize=[16,9])
cs=ax.scatter(df_yu.numax, df_yu.dnu, c=df_yu.Teff, cmap='Reds')
cs2=ax.scatter(df_aldo.numax, df_aldo.Dnu, c=df_aldo.Teff, cmap='Blues')
ax.set_xscale('log')
ax.set_yscale('log')
cbar = fig.colorbar(cs)
cbar.ax.set_ylabel('Teff')
cbar2 = fig.colorbar(cs2)
cbar2.ax.set_ylabel('Teff')
ax.set_xlabel('Numax')
ax.set_ylabel('Dnu')<jupyter_output><empty_output><jupyter_text>Step 3 - Concat Yu and SerenelliTake only what you need - MGMT - and the concatenate.<jupyter_code>df_yu_ = df_yu[['KICID', 'numax', 'numax_err', 'dnu', 'dnu_err', 'Teff', 'Teff_err', 'EvoPhase']]
df_aldo_ = df_aldo[['KIC', 'numax', 'enumax', 'Dnu', 'eDnu', 'Teff', 'eTeff']]
df_aldo_.rename(columns={'enumax': 'numax_err', 'Dnu': 'dnu', 'eDnu': 'dnu_err', 'eTeff': 'Teff_err'}, inplace=True)
df_yu_.rename(columns={'KICID': 'KIC'}, inplace=True)
df = pd.concat([df_yu_, df_aldo_])
df.head()
fig, ax = plt.subplots(figsize=[16,9])
cs=ax.scatter(df.numax, df.dnu, c=df.Teff, cmap='viridis')
ax.set_xscale('log')
ax.set_yscale('log')
cbar = fig.colorbar(cs)
cbar.ax.set_ylabel('Teff')
ax.set_xlabel('Numax')
ax.set_ylabel('Dnu')<jupyter_output><empty_output><jupyter_text>Step 4 - Get the *Gaia* BP-RP dataWe'll get this from <NAME>'s Kepler-Gaia crossmatch found here: https://gaia-kepler.fun/<jupyter_code>from astropy.table import Table
data = Table.read('kepler_dr2_1arcsec.fits',format='fits')
kdf = data.to_pandas()
print(list(kdf))
kdf = kdf[['kepid', 'bp_rp', 'kepler_gaia_ang_dist']]
kdf.rename(columns={'kepid':'KIC'},inplace=True)
def kill_duplicates(df,ID, sep):
    df = df.copy(deep=True)
    sel = df.duplicated(ID, keep=False)
    s = df[sel]
    for idx in range(s[ID].nunique()):
        subset = s[s[ID] == s[ID].values[idx]]
        lock = True  #Iterate and remove the target of largest separation
        while lock:
            if len(subset) > 1.:
                drop = subset[sep].idxmax()
                subset.drop(drop, inplace=True)
                s.drop(drop,inplace=True)
                df.drop(drop, inplace=True)
            elif len(subset == 1.):
                lock = False
            else:
                print('The code shouldnt be here?')
    return df

import numpy as np
bprpdf = pd.merge(df, kdf, on='KIC', how='left')
#bprpdf = kill_duplicates(bprpdf, 'KIC','kepler_gaia_ang_dist')
bprpdf = bprpdf[np.isfinite(bprpdf.bp_rp)]
bprpdf.head()
bprpdf = bprpdf.loc[bprpdf.bp_rp < 2.0]
df = bprpdf
fig, ax = plt.subplots(figsize=[16,9])
cs=ax.scatter(df.numax, df.dnu, c=df.bp_rp, cmap='viridis', vmax=2.0)
ax.set_xscale('log')
ax.set_yscale('log')
cbar = fig.colorbar(cs)
cbar.ax.set_ylabel('BP_RP')
ax.set_xlabel('Numax')
ax.set_ylabel('Dnu')<jupyter_output><empty_output><jupyter_text>Step 5 - Make a list of stars where we can easily get epsilonWe are going to take all the RG stars I have peak bagged, all Legacy and all Kages.  We will add the Tim White sample too.I actually already have an EpsilonFit repo that has the epsilons for a bunch of stars (1000 Rgs, Legacy, Kages).  I will read in the data from there.And then I'll add it to the Tim White Data set. (White et al. 2012).   This data set comes from TimW via Warrick (I think that's enough for coauthorship).  There are three files ... Gstars_temp_gamma_alt.txt, Fstars_avoid_temp.txt, Fstars_ScAB_teff_gamma_alt_pc.txt<jupyter_code>df_g = pd.read_csv('Gstars_temp_gamma_alt.txt', sep='\s+', error_bad_lines=False)
df_favoid = pd.read_csv('Fstars_avoid_temp.txt', sep='\s+', error_bad_lines=False)
df_f = pd.read_csv('Fstars_ScAB_teff_gamma_alt_pc.txt', sep='\s+', error_bad_lines=False)
df_f.rename({'epsA': 'eps', 'epsA_err': 'eps_err'})
df_epsilon = pd.concat([df_g, df_favoid, df_f])
df_epsilon.head()
df_mine = pd.read_csv('my_epsilons.csv')
df_eps = pd.concat([df_mine, df_epsilon])
df_eps = df_eps[['KIC', 'eps', 'eps_err']]
print(len(df_eps))
df_comb = pd.merge(df, df_eps, how='inner', on='KIC')
epm = df_comb.eps
epm[(epm < 0.5)] += 1.0
epm[(epm > 1.5)] -= 1.0
df_comb['eps_mod'] = epm
print(len(df_comb))
fig, ax = plt.subplots(figsize=[16,9])
ax.scatter(df_comb.dnu, df_comb.eps_mod, c=df_comb.bp_rp)
ax.set_xscale('log')
cols = ['KIC','Teff','Teff_err','dnu','dnu_err','numax','numax_err','bp_rp','eps','eps_err','eps_mod']
df_comb[cols].to_csv('prior_data.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1491453637876267, 'fraction_numerical': 0.013764937225835728, 'mean_word_length': 4.99456029011786, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '72683', 'license': 'MIT', 'n_tokens_mistral': 2712, 'n_tokens_neox': 2609, 'n_words': 639, 'path': 'pbjam/data/Data Gather.ipynb', 'repo_name': 'nielsenmb/PBjam', 'size': '6617'}