<jupyter_start><jupyter_text>Final ProcessingThe final processing stage requires:1. Quick validation of catalogues and Bayesian Pvalue maps2. Skewness level3. Adding flag to catalogue<jupyter_code>import seaborn as sns
from astropy.table import Table
%matplotlib inline
import numpy as np
import pylab as plt<jupyter_output><empty_output><jupyter_text>Read tables<jupyter_code>SERVS_cat=Table.read('../data/ELAIS_N1/MIPS/dmu26_XID+MIPS_ELAIS-N1_SERVS_cat_20170710.fits')
SWIRE_cat=Table.read('../data/ELAIS_N1/MIPS/dmu26_XID+MIPS_ELAIS-N1_SWIRE_cat_20170725.fits')
SERVS_cat[0:10]
SWIRE_cat[0:10]<jupyter_output><empty_output><jupyter_text>Look at Symmetry of PDFs to determine depth level of catalogue<jupyter_code>skew=(SERVS_cat['FErr_MIPS_24_u']-SERVS_cat['F_MIPS_24'])/(SERVS_cat['F_MIPS_24']-SERVS_cat['FErr_MIPS_24_l'])
skew.name='(84th-50th)/(50th-16th) percentile'
g=sns.jointplot(x=np.log10(SERVS_cat['F_MIPS_24']),y=skew, kind='hex')
skew=(SWIRE_cat['FErr_MIPS_24_u']-SWIRE_cat['F_MIPS_24'])/(SWIRE_cat['F_MIPS_24']-SWIRE_cat['FErr_MIPS_24_l'])
skew.name='(84th-50th)/(50th-16th) percentile'
g=sns.jointplot(x=np.log10(SWIRE_cat['F_MIPS_24']),y=skew, kind='hex')<jupyter_output><empty_output><jupyter_text>Both seem to have flux pdfs that become Gaussian at ~20$\mathrm{\mu Jy}$   Add flag to catalogue<jupyter_code>from astropy.table import Column
SERVS_cat.add_column(Column(np.zeros(len(SERVS_cat), dtype=bool),name='flag_mips_24'))
SWIRE_cat.add_column(Column(np.zeros(len(SWIRE_cat), dtype=bool),name='flag_mips_24'))

ind_SERVS=(SERVS_cat['Pval_res_24']>0.5) | (SERVS_cat['F_MIPS_24'] < 20.0)
ind_SWIRE=(SWIRE_cat['Pval_res_24']>0.5) | (SWIRE_cat['F_MIPS_24'] < 20.0)
SERVS_cat['flag_mips_24'][ind_SERVS]=True
SWIRE_cat['flag_mips_24'][ind_SWIRE]=True

SERVS_cat.write('../data/ELAIS_N1/MIPS/dmu26_XID+MIPS_ELAIS-N1_SERVS_cat_20170725.fits', format='fits',overwrite=True)
SWIRE_cat.write('../data/ELAIS_N1/MIPS/dmu26_XID+MIPS_ELAIS-N1_SWIRE_cat_20170725.fits', format='fits',overwrite=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15316642120765833, 'fraction_numerical': 0.06136475208640157, 'mean_word_length': 10.91812865497076, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '319456', 'license': 'MIT', 'n_tokens_mistral': 996, 'n_tokens_neox': 902, 'n_words': 94, 'path': 'dmu26/dmu26_XID+MIPS_ELAIS-N1/XID+MIPS_ELAIS-N1_final_processing.ipynb', 'repo_name': 'djbsmith/dmu_products', 'size': '2037'}