<jupyter_start><jupyter_text>Herschel Stripe 82 master catalogue Preparation of Spitzer SHELA dataThe Spitzer/HETDEX Exploratory Large-Area (SHELA) survey covers ~24 sq. deg at 3.6 and 4.5 microns. The Spitzer/SHELA catalogues are available in `dmu0_SHELA`.In the catalouge, we keep:- The internal identifier;- The position;- The fluxes in 4 arcsecond apertures;- The “auto” flux;<jupyter_code>from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

from collections import OrderedDict
import os

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, flux_to_mag
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "shela_ra"
DEC_COL = "shela_dec"<jupyter_output><empty_output><jupyter_text>I - Column selection<jupyter_code>#IRAC1 = 3.6 um
#IRAC2 = 4.5 um
#No stelarity 

imported_columns = OrderedDict({
        'ID': "shela_intid",
        'RA': "shela_ra",
        'DEC': "shela_dec",
        'F3P6_AUTO': "f_shela_irac1",
        'F3P6ERR_AUTO': "ferr_shela_irac1",
        'F3P6_4ARCS': "f_ap_shela_irac1",
        'F3P6ERR_4ARCS': "ferr_ap_shela_irac1",
        'F4P5_AUTO': "f_shela_irac2",
        'F4P5ERR_AUTO': "ferr_shela_irac2",
        'F4P5_4ARCS': "f_ap_shela_irac2",
        'F4P5ERR_4ARCS': "ferr_ap_shela_irac2"
    })


catalogue = Table.read("../../dmu0/dmu0_SHELA/data/shela_irac_v1.3_flux_cat.fits")[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2009

# Clean table metadata
catalogue.meta = None
# Adding magnitude and band-flag columns
for col in catalogue.colnames:
    if col.startswith('f_'):
        errcol = "ferr{}".format(col[1:])
        
        # Note that some fluxes are 0.
        catalogue[col][np.isclose(catalogue[col], 0.)] = np.nan
        catalogue[errcol][np.isclose(catalogue[errcol], 0.)] = np.nan
        
        magnitude, error = flux_to_mag(
            np.array(catalogue[col])/1.e6, np.array(catalogue[errcol])/1.e6)
        
        
        catalogue.add_column(Column(magnitude, name="m{}".format(col[1:])))
        catalogue.add_column(Column(error, name="m{}".format(errcol[1:])))
        
        # Band-flag column
        if "ap" not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))

catalogue[:10].show_in_notebook()<jupyter_output><empty_output><jupyter_text>II - Removal of duplicated sources We remove duplicated objects from the input catalogues.<jupyter_code>SORT_COLS = ['ferr_ap_shela_irac1', 'ferr_ap_shela_irac2']
FLAG_NAME = "shela_flag_cleaned"

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL, sort_col=SORT_COLS,flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))<jupyter_output>The initial catalogue had 2294786 sources.
The cleaned catalogue has 2294786 sources (0 removed).
The cleaned catalogue has 0 sources flagged as having been cleaned<jupyter_text>III - Astrometry correctionWe match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.<jupyter_code>gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_Herschel-Stripe-82.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec, near_ra0=True)
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords, near_ra0=True
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
catalogue[RA_COL] = catalogue[RA_COL]  +  delta_ra.to(u.deg)
catalogue[DEC_COL] = catalogue[DEC_COL] + delta_dec.to(u.deg)
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec, near_ra0=True)<jupyter_output><empty_output><jupyter_text>IV - Flagging Gaia objects<jupyter_code>catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
GAIA_FLAG_NAME = "shela_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))<jupyter_output>68747 sources flagged.<jupyter_text>V - Flagging objects near bright stars  VI - Saving to disk<jupyter_code>catalogue.write("{}/SHELA.fits".format(OUT_DIR), overwrite=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11681980123757735, 'fraction_numerical': 0.019313707106694167, 'mean_word_length': 5.027118644067796, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '547927', 'license': 'MIT', 'n_tokens_mistral': 1984, 'n_tokens_neox': 1922, 'n_words': 389, 'path': 'dmu1/dmu1_ml_Herschel-Stripe-82/1.6_SHELA.ipynb', 'repo_name': 'djbsmith/dmu_products', 'size': '5333'}