<jupyter_start><jupyter_text>Produce input catalogs for lensed system postage stamp generation_Last Updated: <NAME>, December 2018_This is a notebook to produce the host AGN and SN catalogs that are used to created FITS postage stamps of the lensed systems. The goal here is to associate the lens and host information into a single catalog so that Nan Li can use it properly model the host galaxy shapes and we include them as fits files with the instance catalogs. PhoSim is able to take these fits files as input for the given line in the instance catalog so that we have properly shaped host galaxies in our lensed systems.  Load in galaxy catalogs as dataframesThese come from unsprinkled instance catalogs of the DDF region.<jupyter_code>import pandas as pd
from astropy.io import fits
import numpy as np
from desc.sims.GCRCatSimInterface import InstanceCatalogWriter
from lsst.sims.utils import SpecMap
import matplotlib.pyplot as plt
from lsst.utils import getPackageDir
from lsst.sims.photUtils import Sed, BandpassDict, Bandpass
from lsst.sims.catUtils.matchSED import matchBase
import os
%matplotlib inline
base_columns = ['prefix', 'uniqueId', 'raPhoSim', 'decPhoSim',
                        'phosimMagNorm', 'sedFilepath', 'redshift',
                        'shear1', 'shear2', 'kappa', 'raOffset', 'decOffset',
                        'spatialmodel']
df_galaxy = pd.read_csv(os.path.join(os.environ['SCRATCH'],
                                     'bulge_gal_cat_197356.txt.gz'),
                                delimiter=' ', header=None,
                                names=base_columns+['majorAxis', 'minorAxis',
                                                    'positionAngle', 'sindex',
                                                    'internalExtinctionModel',
                                                    'internalAv', 'internalRv',
                                                    'galacticExtinctionModel',
                                                    'galacticAv', 'galacticRv'])
df_disk = pd.read_csv(os.path.join(os.environ['SCRATCH'],
                                     'disk_gal_cat_197356.txt.gz'),
                                delimiter=' ', header=None,
                                names=base_columns+['majorAxis', 'minorAxis',
                                                    'positionAngle', 'sindex',
                                                    'internalExtinctionModel',
                                                    'internalAv', 'internalRv',
                                                    'galacticExtinctionModel',
                                                    'galacticAv', 'galacticRv'])
df_agn = pd.read_csv(os.path.join(os.environ['SCRATCH'],
                                     'agn_gal_cat_197356.txt.gz'),
                                 delimiter=' ', header=None,
                                 names=base_columns+['internalExtinctionModel',
                                                     'galacticExtinctionModel',
                                                     'galacticAv', 'galacticRv'])
df_agn['galaxy_id'] = np.right_shift(df_agn['uniqueId'], 10)
df_agn.head()
df_galaxy['galaxy_id'] = np.right_shift(df_galaxy['uniqueId'], 10)
df_galaxy.head()
df_disk['galaxy_id'] = np.right_shift(df_disk['uniqueId'], 10)
df_disk.head()<jupyter_output><empty_output><jupyter_text>Make the AGN catalogsWhat we are going to do is take the information on the unlensed bulge and disks from the instance catalog above and combine it with information on the lens that will be sprinkled in front of it so that we can accurately model the appearance of the lensed hosts in a fits postage stamp to include with the instance catalogs and give us properly modeled host galaxy images. We first do it for the AGN here and then repeat for the SNe below. Load in the AGN lensed systems and cache fileThe cache file associates the `galaxy_id` values with the `twinklesId` values in the lensed system catalog. The file `cosmoDC2_vX.X_matched_AGN.fits` contains the information on the lensed systems.<jupyter_code>catalog_version = 'cosmoDC2_v1.1.4'
hdulist = fits.open('../../data/%s_matched_AGN.fits' % catalog_version)
twinkles_lenses = hdulist[1].data
agn_cache_df = pd.read_csv('../../data/%s_agn_cache.csv' % catalog_version)<jupyter_output><empty_output><jupyter_text>Next we add in the lens properties to go with each of the images.<jupyter_code>#TwinklesId can range from 0 - 1099
ra_images = []
dec_images = []
num_images = []
twinkles_system_images = []
unique_id_lens = []

unique_id_bulge = []
unique_id_disk = []

bulge_index = []
disk_index = []

disk_valid = []

i = 0
for twinkles_system in range(len(twinkles_lenses)):
    
    if twinkles_system % 100 == 0:
        print(twinkles_system)
    
    cache_info = agn_cache_df.query('twinkles_system == %i' % twinkles_system)
    gal_id = cache_info['galtileid'].values[0]
    lens_info = twinkles_lenses[np.where(twinkles_lenses['twinklesId'] == twinkles_system)]
    bulge_info = df_galaxy.query('galaxy_id == %i' % gal_id)
    disk_info = df_disk.query('galaxy_id == %i' % gal_id)

    
    #print(bulge_info, disk_info)
    
    for lens_image in range(lens_info[0]['NIMG']):
        ra_shift = np.radians(lens_info[0]['XIMG'][lens_image] / 3600.) / np.cos(np.radians(bulge_info['decPhoSim']))
        dec_shift = np.radians(lens_info[0]['YIMG'][lens_image] / 3600.)
        
        image_ra = np.radians(bulge_info['raPhoSim']) + ra_shift
        image_dec = np.radians(bulge_info['decPhoSim'] + dec_shift)
        
        ra_images.append(np.degrees(image_ra).values[0])
        dec_images.append(np.degrees(image_dec).values[0])
        num_images.append(lens_image)
        unique_id_lens.append(bulge_info['uniqueId'].values[0])
        twinkles_system_images.append(twinkles_system)
        base_id = (gal_id + int(1.5e10))*10000 + twinkles_system*4 + lens_image
        unique_id_bulge.append(np.left_shift(base_id, 10) + 97)
        unique_id_disk.append(np.left_shift(base_id, 10) + 107)
        
        bulge_index.append(bulge_info.index[0])
        
        if len(disk_info) == 0:
            print(twinkles_system)
        else:
            disk_index.append(disk_info.index[0])
            disk_valid.append(i)
            
        i += 1
host_bulge = df_galaxy.iloc[bulge_index]
host_bulge = host_bulge.reset_index(drop=True)
host_bulge['twinkles_system'] = twinkles_system_images
host_bulge['image_number'] = num_images
host_bulge['uniqueId_lens'] = unique_id_lens
host_bulge['uniqueId'] = unique_id_bulge
host_bulge['raPhoSim_lens'] = host_bulge['raPhoSim']
host_bulge['decPhoSim_lens'] = host_bulge['decPhoSim']
host_bulge['raPhoSim'] = ra_images
host_bulge['decPhoSim'] = dec_images
host_bulge.head()
host_disk = df_disk.iloc[disk_index]
host_disk = host_disk.reset_index(drop=True)
host_disk['twinkles_system'] = np.array(twinkles_system_images)[disk_valid]
host_disk['image_number'] = np.array(num_images)[disk_valid]
host_disk['uniqueId_lens'] = np.array(unique_id_lens)[disk_valid]
host_disk['uniqueId'] = np.array(unique_id_disk)[disk_valid]
host_disk['raPhoSim_lens'] = host_disk['raPhoSim']
host_disk['decPhoSim_lens'] = host_disk['decPhoSim']
host_disk['raPhoSim'] = np.array(ra_images)[disk_valid]
host_disk['decPhoSim'] = np.array(dec_images)[disk_valid]
host_disk.head()
host_bulge.to_csv('../../data/%s_bulge_agn_host.csv' % catalog_version)
host_disk.to_csv('../../data/%s_disk_agn_host.csv' % catalog_version)<jupyter_output><empty_output><jupyter_text>Make the SNe catalogsWe do the same as we did for the lensed AGN systems with the lensed SNe systems below. Load in the SNE lensed systems and cache fileThe cache file associates the `galaxy_id` values with the `twinklesId` values in the lensed system catalog. The file `cosmoDC2_vX.X_sne_cat.csv` contains the information on the images.<jupyter_code>sne_df = pd.read_csv('../../data/%s_sne_cat.csv' % catalog_version)
sne_cache_df = pd.read_csv('../../data/%s_sne_cache.csv' % catalog_version)
sne_df.head()<jupyter_output><empty_output><jupyter_text>Next we add in the lens properties to go with each of the images.<jupyter_code>#TwinklesId ranges from 1100 - 1100+number of cached SNe
ra_images = []
dec_images = []
num_images = []
twinkles_system_images = []
unique_id_lens = []

unique_id_bulge = []
unique_id_disk = []

bulge_index = []
disk_index = []

disk_valid = []

i = 0
for twinkles_system in range(1100, 1100+len(sne_cache_df)):
    
    if twinkles_system % 100 == 0:
        print(twinkles_system)
    
    cache_info = sne_cache_df.query('twinkles_system == %i' % twinkles_system)
    gal_id = cache_info['galtileid'].values[0]
    lens_info = sne_df.iloc[np.where(sne_df['twinkles_sysno'] == twinkles_system)]
    bulge_info = df_galaxy.query('galaxy_id == %i' % gal_id)
    disk_info = df_disk.query('galaxy_id == %i' % gal_id)

    
    #print(bulge_info, disk_info)
    
    for lens_image in lens_info['imno'].values:
        ra_shift = np.radians(lens_info['x'].iloc[lens_image] / 3600.) / np.cos(np.radians(bulge_info['decPhoSim']))
        dec_shift = np.radians(lens_info['y'].iloc[lens_image] / 3600.)
        
        image_ra = np.radians(bulge_info['raPhoSim']) + ra_shift
        image_dec = np.radians(bulge_info['decPhoSim'] + dec_shift)
        
        ra_images.append(np.degrees(image_ra).values[0])
        dec_images.append(np.degrees(image_dec).values[0])
        num_images.append(lens_image)
        unique_id_lens.append(bulge_info['uniqueId'].values[0])
        twinkles_system_images.append(twinkles_system)
        base_id = (gal_id + int(1.5e10))*10000 + twinkles_system*4 + lens_image
        unique_id_bulge.append(np.left_shift(base_id, 10) + 97)
        unique_id_disk.append(np.left_shift(base_id, 10) + 107)
        
        bulge_index.append(bulge_info.index[0])
        
        if len(disk_info) == 0:
            print(twinkles_system)
        else:
            disk_index.append(disk_info.index[0])
            disk_valid.append(i)
            
        i += 1
host_bulge = df_galaxy.iloc[bulge_index]
host_bulge = host_bulge.reset_index(drop=True)
host_bulge['twinkles_system'] = twinkles_system_images
host_bulge['image_number'] = num_images
host_bulge['uniqueId_lens'] = unique_id_lens
host_bulge['uniqueId'] = unique_id_bulge
host_bulge['raPhoSim_lens'] = host_bulge['raPhoSim']
host_bulge['decPhoSim_lens'] = host_bulge['decPhoSim']
host_bulge['raPhoSim'] = ra_images
host_bulge['decPhoSim'] = dec_images
host_bulge.head()
host_disk = df_disk.iloc[disk_index]
host_disk = host_disk.reset_index(drop=True)
host_disk['twinkles_system'] = np.array(twinkles_system_images)[disk_valid]
host_disk['image_number'] = np.array(num_images)[disk_valid]
host_disk['uniqueId_lens'] = np.array(unique_id_lens)[disk_valid]
host_disk['uniqueId'] = np.array(unique_id_disk)[disk_valid]
host_disk['raPhoSim_lens'] = host_disk['raPhoSim']
host_disk['decPhoSim_lens'] = host_disk['decPhoSim']
host_disk['raPhoSim'] = np.array(ra_images)[disk_valid]
host_disk['decPhoSim'] = np.array(dec_images)[disk_valid]
host_disk.head()
host_bulge.to_csv('../../data/%s_bulge_sne_host.csv' % catalog_version)
host_disk.to_csv('../../data/%s_disk_sne_host.csv' % catalog_version)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11238938053097346, 'fraction_numerical': 0.01247787610619469, 'mean_word_length': 3.230999625608386, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '545456', 'license': 'MIT', 'n_tokens_mistral': 3866, 'n_tokens_neox': 3665, 'n_words': 710, 'path': 'examples/notebooks/postage_stamp_generation_inputs.ipynb', 'repo_name': 'LSSTDESC/Twinkles', 'size': '11308'}