<jupyter_start><jupyter_text>HELP Versions of Bootes Matched Aperture CatalogsThe Bootes multi-wavelength catalog comes in two versions, an I-band selected version ('Idet') and an IRAC Ch2-selected ('ch2det') version. In almost all respects the catalogs are the same, with identical photometry methods applied to both versions and the same raw data and columns.There are however slight formatting differences in the catalogs as well a few columns that differ due to the nature of the selection band.The key differences are:1. The Idet catalog contains a flag column 'FLAG_DEEP' which incorporate both information from a mask and information about duplicate object in regions where the I-band frames overlap. When using the catalog, it is usually best to require (FLAG_DEEP == 1) but for the purposes of joining the two versions and completeness it is essential to keep all sources in the 'raw' catalogs.2. The Idet catalog contains a 'CLASS_STAR' column based on the I-band. However no 'CLASS_STAR' column is available for the ch2 det catalog.3. For convenience in filesize, the Idet catalog is separated into subsections in Declination.4. Aperture column naming convention differs slightly.The output catalogs contain aperture-corrected 2" fluxes (uJy) and AB magnitudes. Aperture corrections and zeropoint conversions are done based on the curve-of-growth files and supporting information provided by M.Brown.Before merging, catalogs should have astrometry corrected to the GAIA frame as is done for all raw HELP dataproducts.When merging the two catalogs, if a source is detected in both catalogs then it is a reasonable assumption that all common values are the same. The fluxes/magnitudes can therefore be taken from the primary catalog (suggest 'Idet'), with unique entries in the secondary catalog ('ch2det') being appended.<jupyter_code>%matplotlib inline
import os
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import griddata

from astropy.table import Table, Column, vstack
from astropy.coordinates import SkyCoord
import astropy.units as u

date = '07112017'
BANDS = ['u', 'Bw', 'R',  'I',  
         'z', 'z_Subaru', 'y', 'J', 'H', 'K', 'Ks',
         'ch1', 'ch2', 'ch3', 'ch4']

opt_nir_bands = ['u','Bw','R','I','z','z_Subaru','y','J','H', 'K', 'Ks']

# 'FUV', 'NUV',

help_col_names = ['lbc_u', 
                  'mosaic_bw', 'mosaic_r', 'mosaic_i',
                  'prime90_z', 'suprime_z', 'lbc_y', 
                  'newfirm_j', 'newfirm_h', 'onis_k', 'newfirm_k',
                  'irac_ch1', 'irac_ch2', 'irac_ch3', 'irac_ch4']

# 'galex_fuv', 'galex_nuv',<jupyter_output><empty_output><jupyter_text>Catalog meta information, including aperture sizes and zeropoints for all bands<jupyter_code>aper_sizes = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 15.0, 20.0]

zeropoints_table = Table.read('bandinfo.dat',format='ascii.commented_header')
zeropoints_dict = dict(zip(zeropoints_table['brownname'], zeropoints_table['brown_jyzero']))<jupyter_output><empty_output><jupyter_text>Aperture column for I selected catalog (format = 'APER_01', 'APER_02',...')<jupyter_code>aper1 = 2
aper_irac = 2
aper_mips = 10<jupyter_output><empty_output><jupyter_text>Aperture column for IRAC Ch2 selected catalog (format = 'APER', 'APER_1', ...')<jupyter_code>aper_id_1 = 1
aper_id_irac = 1
aper_id_mips = 7<jupyter_output><empty_output><jupyter_text>Load in aperture correction information:<jupyter_code>galex_aper = np.array([1.5, 2.3, 3.8, 6.0, 9.0, 12.8, 17.3])
galex_fuv_corr = np.array([1.65, 0.77, 0.20, 0.1, 0.07, 0.05, 0.04])
galex_nuv_corr = np.array([1.33, 0.62, 0.21, 0.12, 0.08, 0.06, 0.04])

aper_corr_fuv = 1/10**griddata(galex_aper, galex_fuv_corr, aper1)
aper_corr_nuv = 1/10**griddata(galex_aper, galex_nuv_corr, aper1)

dat = np.loadtxt("moffat_0.68_2.50.dat")
arg = np.argmin(np.abs(dat[:,0]-aper1))
aper_corr_068 = dat[arg,1]

dat = np.loadtxt("moffat_1.35_2.50.dat")
arg = np.argmin(np.abs(dat[:,0]-aper1))
aper_corr_135 = dat[arg,1]

dat = np.loadtxt("moffat_1.60_2.50.dat")
arg = np.argmin(np.abs(dat[:,0]-aper1))
aper_corr_160 = dat[arg,1]<jupyter_output><empty_output><jupyter_text>I-band Selected Catalog<jupyter_code>subcat_names = np.array(['32_33', '33_34', '34_35', '35_36'])

for ixs, subcat in enumerate(subcat_names): 
    print('DEC Range: {0} {1}'.format(subcat, '\n'))
    outcat = Table()

    cat = Table.read('/data2/ken/bootes/mbrown_cat/Bootes2014a/Bootes_I_2014a_Idet_{0}_phot.fits'.format(subcat),format='fits')
    index = np.arange(1, len(cat)+1)
    outcat['id'] = index

    outcat.add_column(cat['ALPHA_J2000'])
    outcat.add_column(cat['DELTA_J2000'])
    outcat.add_column(cat['FLAG_DEEP'])
    outcat.add_column(cat['CLASS_STAR'])

    outcat.add_column(cat['IMAFLAGS_APER_0{0}'.format(aper1)], name='IMAFLAGS')
    outcat.add_column(cat['SEGFLAGS_APER_0{0}'.format(aper1)], name='SEGFLAGS')

    for idx, band in enumerate(BANDS):
        cat = Table.read('/data2/ken/bootes/mbrown_cat/Bootes2014a/Bootes_{0}_2014a_Idet_{1}_phot.fits'.format(band, subcat),format='fits')

        if band in ['FUV']:
            mag = cat['MAG_APER_{0:02d}'.format(aper1)]
            magerr = cat['MAGERR_APER_{0:02d}'.format(aper1)]
            aper_corr = aper_corr_fuv

        if band in ['NUV']:
            mag = cat['MAG_APER_{0:02d}'.format(aper1)]
            magerr = cat['MAGERR_APER_{0:02d}'.format(aper1)]
            aper_corr = aper_corr_nuv
                
        if band in opt_nir_bands:
            mag = cat['MAG_APER_{0:02d}'.format(aper1)]
            magerr = cat['MAGERR_APER_{0:02d}'.format(aper1)]

            if band in ['u', 'z', 'J']:
                aper_corr = aper_corr_160
            elif band == "z_Subaru":
                aper_corr = aper_corr_068
            else:
                aper_corr = aper_corr_135

        elif band in ['ch1','ch2','ch3', 'ch4']:
            mag = cat['MAG_APER_{0:02d}'.format(aper_irac)]
            magerr = cat['MAGERR_APER_{0:02d}'.format(aper_irac)]

            dat = np.loadtxt("{0}_psf_flux.dat".format(band))
            arg = np.argmin(np.abs(dat[:,0]-aper1))
            aper_corr = dat[arg,1]

        elif band == '24':
            mag = cat['MAG_APER_{0:02d}'.format(aper_mips)]
            magerr = cat['MAGERR_APER_{0:02d}'.format(aper_mips)]
            dat = np.loadtxt("mips24_prf_flux.dat".format(band))
            arg = np.argmin(np.abs(dat[:,0]-aper_mips))
            aper_corr = dat[arg,1]

        zp = zeropoints_dict[band]
            
        mag_copy = np.copy(mag)
        notobs = np.logical_or(mag_copy == 0., mag_copy == 99.9)

        flux = ((zp*10.0**(-0.4*np.abs(mag_copy)))*u.Jy).to(u.uJy)
        
        isnegative = (mag_copy < 0)        
        flux[isnegative] *= -1.
        
        fluxerr = np.abs(flux) * (magerr/(2.5*np.log10(np.e)))
        
        flux[notobs] = -99.*u.uJy
        fluxerr[notobs] = -99.*u.uJy

        # Aperture correction
        flux /= aper_corr
        
        # Correct Flux error to maintain measured S/N
        fluxerr /= aper_corr 

        
        lim = (flux/fluxerr < 2.)

        # Convert Jansky fluxes back to AB magnitudes
        mag = 23.9 - 2.5*np.log10(flux.value)
        #magerr = 2.5*np.log10(np.e)*(fluxerr/flux).value
        
        magerr[lim] = -99.
        mag[lim] = 23.9 - 2.5*np.log10((flux + 2*fluxerr).value)[lim]
        
        flux[fluxerr <= 0.0] = -99.*u.uJy
        fluxerr[flux == -99.*u.uJy] = -99*u.uJy

        mag[np.logical_or(flux < -90*u.uJy, fluxerr < -90.*u.uJy)] = -99.
        magerr[np.logical_or(flux < -90.*u.uJy, fluxerr < -90*u.uJy)] = -99.
           
        outcat.add_column(Column(name='m_ap_{0}'.format(help_col_names[idx]),
                                 data=mag))
        outcat.add_column(Column(name='merr_ap_{0}'.format(help_col_names[idx]),
                                 data=magerr))

        outcat.add_column(Column(name='f_ap_{0}'.format(help_col_names[idx]),
                                 data=flux))
        outcat.add_column(Column(name='ferr_ap_{0}'.format(help_col_names[idx]),
                                 data=fluxerr))

        print('Band: {0}  Zeropoint: {1}  Aperture Correction: {2}'.format(band, zp, aper_corr))
        
    outname = 'bootes_merged_Icorr_2014a_{0}_ap{1}_{2}.fits'.format(subcat, aper1, date)
    if os.path.isfile(outname):
        os.remove(outname)
    outcat.write(outname,
                 format='fits')

    if ixs == 0:
        combined_cat = outcat.copy()
    else:
        combined_cat = vstack([combined_cat, outcat])
new_ids = np.arange(1, len(combined_cat)+1)

combined_cat['id'] = new_ids
outname = 'Bootes_merged_Icorr_2014a_{0}_ap{1}_{2}.fits'.format('all', aper1, date)
if os.path.isfile(outname):
    os.remove(outname)
combined_cat.write(outname, format='fits')<jupyter_output><empty_output><jupyter_text>IRAC Ch2-selected Catalog<jupyter_code>outcat = Table()

cat = Table.read('/data2/ken/bootes/mbrown_cat/Bootes2014a_ch2/ch2v34_sdwfs_{0}_2014a.fits'.format('ch2'),format='fits')
index = np.arange(1, len(cat)+1)
outcat['id'] = index

outcat.add_column(cat['ALPHA_J2000'])
outcat.add_column(cat['DELTA_J2000'])
#outcat.add_column(cat['FLAG_DEEP'])
#outcat.add_column(cat['CLASS_STAR'])

outcat.add_column(cat['IMAFLAGS_APER_{0}'.format(aper_id_1)], name='IMAFLAGS')
outcat.add_column(cat['SEGFLAGS_APER_{0}'.format(aper_id_1)], name='SEGFLAGS')


for idx, band in enumerate(BANDS):
    cat = Table.read('/data2/ken/bootes/mbrown_cat/Bootes2014a_ch2/ch2v34_sdwfs_{0}_2014a.fits'.format(band),format='fits')

    if band in ['NUV']:
        mag = cat['MAG_APER_{0}'.format(aper_id_1)]
        magerr = cat['MAGERR_APER_{0}'.format(aper_id_1)]
        aper_corr = aper_corr_nuv
    
    if band in opt_nir_bands:
        mag = cat['MAG_APER_{0}'.format(aper_id_1)]
        magerr = cat['MAGERR_APER_{0}'.format(aper_id_1)]

        if band in ["u", "z", "J"]:
            aper_corr = aper_corr_160
        elif band == "z_Subaru":
            aper_corr = aper_corr_068
        else:
            aper_corr = aper_corr_135

    elif band in ['ch1','ch2','ch3', 'ch4']:
        mag = cat['MAG_APER_{0}'.format(aper_id_irac)]
        magerr = cat['MAGERR_APER_{0}'.format(aper_id_irac)]

        dat = np.loadtxt("{0}_psf_flux.dat".format(band))
        arg = np.argmin(np.abs(dat[:,0]-aper1))
        aper_corr = dat[arg,1]

    elif band == '24':
        mag = cat['MAG_APER_{0}'.format(aper_id_mips)]
        magerr = cat['MAGERR_APER_{0}'.format(aper_id_mips)]
        dat = np.loadtxt("mips24_prf_flux.dat".format(band))
        arg = np.argmin(np.abs(dat[:,0]-aper_mips))
        aper_corr = dat[arg,1]

    zp = zeropoints_dict[band]

    mag_copy = np.copy(mag)
    notobs = np.logical_or(mag_copy == 0., mag_copy == 99.9)

    flux = ((zp*10.0**(-0.4*np.abs(mag_copy)))*u.Jy).to(u.uJy)

    isnegative = (mag_copy < 0)        
    flux[isnegative] *= -1.

    fluxerr = np.abs(flux) * (magerr.data/(2.5*np.log10(np.e)))

    flux[notobs] = -99.*u.uJy
    fluxerr[notobs] = -99.*u.uJy

    # Aperture correction
    flux /= aper_corr
    # Correct Flux error to maintain measured S/N
    fluxerr /= aper_corr 
    
    lim = (flux/fluxerr < 2.)

    # Convert Jansky fluxes back to AB magnitudes
    mag = 23.9 - 2.5*np.log10(flux.value)
    #magerr = 2.5*np.log10(np.e)*(fluxerr/flux).value

    magerr[lim] = -99.
    mag[lim] = 23.9 - 2.5*np.log10((flux + 2*fluxerr).value)[lim]

    flux[fluxerr <= 0.0] = -99.*u.uJy
    fluxerr[flux == -99.*u.uJy] = -99*u.uJy

    mag[np.logical_or(flux < -90*u.uJy, fluxerr < -90.*u.uJy)] = -99.
    magerr[np.logical_or(flux < -90.*u.uJy, fluxerr < -90*u.uJy)] = -99.

    outcat.add_column(Column(name='m_ap_{0}'.format(help_col_names[idx]),
                             data=mag))
    outcat.add_column(Column(name='merr_ap_{0}'.format(help_col_names[idx]),
                             data=magerr))

    outcat.add_column(Column(name='f_ap_{0}'.format(help_col_names[idx]),
                             data=flux))
    outcat.add_column(Column(name='ferr_ap_{0}'.format(help_col_names[idx]),
                             data=fluxerr))

    print('Band: {0}  Zeropoint: {1}  Aperture Correction: {2}'.format(band, zp, aper_corr))

outname = 'bootes_merged_ch2corr_2014a_all_ap{0}_{1}.fits'.format(aper1, date)

if os.path.isfile(outname):
    os.remove(outname)

outcat.write(outname, format='fits')<jupyter_output>/software/local/lib64/python2.7/site-packages/astropy/units/quantity.py:641: RuntimeWarning: divide by zero encountered in true_divide
  *arrays, **kwargs)
/software/local/lib64/python2.7/site-packages/ipykernel/__main__.py:73: RuntimeWarning: invalid value encountered in log10
/software/local/lib64/python2.7/site-packages/ipykernel/__main__.py:77: RuntimeWarning: invalid value encountered in log10
{'fraction_non_alphanumeric': 0.15036246005144593, 'fraction_numerical': 0.04287161898822979, 'mean_word_length': 3.239920687376074, 'pattern_counts': {'":': 2, '<': 42, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '545041', 'license': 'MIT', 'n_tokens_mistral': 5271, 'n_tokens_neox': 4799, 'n_words': 892, 'path': 'dmu0/dmu0_Bootes_Brown/HELP-Bootes.ipynb', 'repo_name': 'djbsmith/dmu_products', 'size': '12829'}