<jupyter_start><jupyter_text>COSMOS master catalogue Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) dataThis catalogue comes from `dmu0_HSC`.In the catalogue, we keep:- The `object_id` as unique object identifier;- The position;- The g, r, i, z, y (no N921) aperture magnitude in 2” that we aperture correct;- The g, r, i, z, y (no N921) kron fluxes and magnitudes.- The extended flag that we convert to a stellariy.**Note**: On ELAIS-N1 the HSC-SSP catalogue does not contain any N816 magnitudes.We use 2016 as the epoch.<jupyter_code>from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
import datetime
print("This notebook was executed on: \n{}".format(datetime.datetime.now()))
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

from collections import OrderedDict
import os

from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
    nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "hsc-deep_ra"
DEC_COL = "hsc-deep_dec"
# Pristine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_deep_COSMOS.fits")<jupyter_output><empty_output><jupyter_text>I - Aperture correctionTo compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.**Target aperture**: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:- The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.- The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.**Magnitude range**: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.<jupyter_code>bands = ["g", "r", "i", "z", "y", "n921"]
apertures = ["10", "15", "20", "30", "40", "57", "84", "118", "168", "235"]

magnitudes = {}
magnitude_errors ={}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
    )
    # Some sources have an infinite magnitude
    mask = np.isinf(magnitudes[band])
    magnitudes[band][mask] = np.nan
    try:
        magnitude_errors[band] = np.array(
        [orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
        )
        magnitude_errors[band][mask] = np.nan
    except KeyError:
        print("No error column for a " + band + " band aperture magnitude.")
        
    stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
    
mag_corr = {}<jupyter_output><empty_output><jupyter_text>I.a - g band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], stellarities['g'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We will use magnitudes between 18.5 and 20.8<jupyter_code># Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], 
    stellarities['g'],
    mag_min=20.8, mag_max=22.0)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for g band:
Correction: -0.20549774169921875
Number of source used: 3768
RMS: 0.017911572957685837<jupyter_text>I.b - r band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], stellarities['r'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We use magnitudes between 17.6 and 19.7.<jupyter_code># Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], 
    stellarities['r'],
    mag_min=20.0, mag_max=21.0)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for r band:
Correction: -0.08256912231445312
Number of source used: 3891
RMS: 0.011581382492618938<jupyter_text>I.c - i band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], stellarities['i'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We use magnitudes between 18.5 and 19.8.<jupyter_code># Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], 
    stellarities['i'],
    mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for i band:
Correction: -0.10167789459228516
Number of source used: 3474
RMS: 0.01759486506775622<jupyter_text>I.d - z band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], stellarities['z'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We use magnitudes between 17.5 and 19.8.<jupyter_code># Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], 
    stellarities['z'],
    mag_min=17.5, mag_max=19.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for z band:
Correction: -0.06690597534179688
Number of source used: 5966
RMS: 0.008133258602144438<jupyter_text>I.e - y band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], stellarities['y'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We use magnitudes between 17 and 18.7.<jupyter_code># Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], 
    stellarities['y'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for y band:
Correction: -0.13036632537841797
Number of source used: 1008
RMS: 0.008730060428994603<jupyter_text>I.f - n921 band<jupyter_code>nb_plot_mag_ap_evol(magnitudes['n921'], stellarities['n921'], labels=apertures)<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()<jupyter_text>We will use aperture 40 as target.<jupyter_code>nb_plot_mag_vs_apcor(orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], stellarities['n921'])<jupyter_output>/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)<jupyter_text>We use magnitudes between 17 and 18.7.<jupyter_code># Aperture correction
mag_corr['n921'], num, std = aperture_correction(
    orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], 
    stellarities['n921'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for n921 band:")
print("Correction: {}".format(mag_corr['n921']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))<jupyter_output>Aperture correction for n921 band:
Correction: -0.11112499237060547
Number of source used: 419
RMS: 0.0083674661823173<jupyter_text>II - StellarityHSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS ([cf this page](http://wsa.roe.ac.uk/www/gloss_p.htmldxssource_pstar)) to compute a stellarity based on the class in each band:\begin{equation*}P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i }\end{equation*}where $i$ is the band, and with using the same probabilities as UKDISS:| HSC flag | UKIDSS flag | Meaning         | P(star) | P(galaxy) | P(noise) | P(saturated) ||:--------:|:-----------:|:----------------|--------:|----------:|---------:|-------------:||          |      -9     | Saturated       |     0.0 |       0.0 |      5.0 |         95.0 ||          |      -3     | Probable galaxy |    25.0 |      70.0 |      5.0 |          0.0 ||          |      -2     | Probable star   |    70.0 |      25.0 |      5.0 |          0.0 ||     0    |      -1     | Star            |    90.0 |       5.0 |      5.0 |          0.0 ||          |      0      | Noise           |     5.0 |       5.0 |     90.0 |          0.0 ||     1    |      +1     | Galaxy          |     5.0 |      90.0 |      5.0 |          0.0 |<jupyter_code># We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
    orig_hsc[colname] for colname in 
    ['gclassification_extendedness',
     'rclassification_extendedness',
     'iclassification_extendedness',
     'zclassification_extendedness',
     'yclassification_extendedness',
     'n921classification_extendedness']
])
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)

# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
    [np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)

stellarity = np.round(stellarity, 3)
vis.hist(stellarity, bins='scott');
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))<jupyter_output><empty_output><jupyter_text>II - Column selection<jupyter_code>imported_columns = OrderedDict({
        "object_id": "hsc-deep_id",
        "ra": "hsc-deep_ra",
        "dec": "hsc-deep_dec",
        "gmag_aperture20": "m_ap_hsc-deep_g",
        "gmag_aperture20_err": "merr_ap_hsc-deep_g",
        "gmag_kron": "m_hsc-deep_g",
        "gmag_kron_err": "merr_hsc-deep_g",
        "rmag_aperture20": "m_ap_hsc-deep_r",
        "rmag_aperture20_err": "merr_ap_hsc-deep_r",
        "rmag_kron": "m_hsc-deep_r",
        "rmag_kron_err": "merr_hsc-deep_r",
        "imag_aperture20": "m_ap_hsc-deep_i",
        "imag_aperture20_err": "merr_ap_hsc-deep_i",
        "imag_kron": "m_hsc-deep_i",
        "imag_kron_err": "merr_hsc-deep_i",
        "zmag_aperture20": "m_ap_hsc-deep_z",
        "zmag_aperture20_err": "merr_ap_hsc-deep_z",
        "zmag_kron": "m_hsc-deep_z",
        "zmag_kron_err": "merr_hsc-deep_z",
        "ymag_aperture20": "m_ap_hsc-deep_y",
        "ymag_aperture20_err": "merr_ap_hsc-deep_y",
        "ymag_kron": "m_hsc-deep_y",
        "ymag_kron_err": "merr_hsc-deep_y",
            "n921mag_aperture20": "m_ap_hsc-deep_n921",
        "n921mag_aperture20_err": "merr_ap_hsc-deep_n921",
        "n921mag_kron": "m_hsc-deep_n921",
        "n921mag_kron_err": "merr_hsc-deep_n921",
        "stellarity": "hsc-deep_stellarity"
    })


catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
# Aperture correction
for band in bands:
    catalogue["m_ap_hsc-deep_{}".format(band)] += mag_corr[band]
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        
        #Some inf and 99. mags
        mask = (catalogue[col] > 90.) | (catalogue[errcol] > 90.)
        catalogue[col][mask] = np.nan
        catalogue[errcol][mask] = np.nan
        
        
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
catalogue[:10].show_in_notebook()<jupyter_output><empty_output><jupyter_text>III - Removal of duplicated sources We remove duplicated objects from the input catalogues.<jupyter_code>SORT_COLS = [
        'merr_ap_hsc-deep_i', 'merr_ap_hsc-deep_r', 'merr_ap_hsc-deep_z',
        'merr_ap_hsc-deep_y', 'merr_ap_hsc-deep_g', 'merr_ap_hsc-deep_n921']
FLAG_NAME = 'hsc-deep_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(
    catalogue, RA_COL, DEC_COL, 
    sort_col= SORT_COLS,
    flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))<jupyter_output>/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)<jupyter_text>III - Astrometry correctionWe match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.<jupyter_code>gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_COSMOS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
catalogue[RA_COL] +=  delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)<jupyter_output><empty_output><jupyter_text>IV - Flagging Gaia objects<jupyter_code>catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
GAIA_FLAG_NAME = "hsc-deep_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))<jupyter_output>5103 sources flagged.<jupyter_text>V - Flagging objects near bright stars  VI - Saving to disk<jupyter_code>catalogue.write("{}/HSC-DEEP.fits".format(OUT_DIR), overwrite=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11910131852879945, 'fraction_numerical': 0.03504510756419153, 'mean_word_length': 5.015392642838508, 'pattern_counts': {'":': 28, '<': 91, '<?xml version=': 0, '>': 112, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '218288', 'license': 'MIT', 'n_tokens_mistral': 8482, 'n_tokens_neox': 7857, 'n_words': 1815, 'path': 'dmu1/dmu1_ml_COSMOS/1.4.1_HSC-DEEP.ipynb', 'repo_name': 'djbsmith/dmu_products', 'size': '23056'}