<jupyter_start><jupyter_text>GRANDMA/Kilonova-catcher --- KN-MangroveThe purpose of this notebook is to inspect the ZTF alerts that were selected by the Fink KN-Mangrove filter as potential Kilonova candidates in the period 2021/04/01 to 2021/08/31, and forwarded to the GRANDMA/Kilonova-catcher project for follow-up observations.With the other filter (KN-LC), we need at least two days to identify a candidate. It may seem like a short amount of time, but if the object is a kilonova, it will already be fading or even too faint to be observed. The second filter aims to tackle younger detections. An alert will be considered as a candidate if, on top of the other cuts, one can identify a suitable host and the resulting absolute magnitude is compatible with the kilonovae models.To identify possible hosts, we used the MANGROVE catalog [1]. It is an inventory of 800,000 galaxies. At this point, we are only interested in their position in the sky: right ascension, declination, and luminosity distance. We only considered the galaxies in a 230 Mpc range, as it is the current observation range of the gravitational waves interferometers.This filter uses the following cuts:- Point-like object: the star/galaxy extractor score must be above 0.4. This could be justified by saying that the alert should be a point-like object. Actually, few objects score below 0.4 given the current implementation, and objects that do are most likely boguses.- Non-artefact: the deep real/bogus score must be above 0.5.- Object non referenced in SIMBAD catalog (galactic ones).- Young detection: less than 6 hours.- Galaxy association: The alert should within 10 kpc of a galaxy from the Mangrove catalog.- Absolute magnitude: the absolute magnitude of the alert should be −16 ± 1.According to [2], we expect a kilonova event to display an absolute magnitude of −16 ± 1. We don’t know the distance of the alerts in general, so we will compute the absolute magnitude of an alert as if it were in a given galaxy. This threshold is given in the band g but it was implemented and band g and r without distinction. This hypothesis is due to the lack of observations.The galaxy association method is also not perfect: it can lead to the mis-association of an event that is in the foreground or the background of a galaxy. But this is necessary as the luminosity distance between the earth and the alert is usually unknown.[1] <NAME> et al. “Optimizing gravitational waves follow-up using galaxies stellar mass”. In: Monthly Notices of the Royal Astronomical Society 492.4 (Jan. 2020), pp. 4768–4779. issn: 1365-2966. doi: 10.1093/mnras/staa114. url: http://dx.doi.org/10.1093/ mnras/staa114.[2] <NAME> et al. “Kilonova Luminosity Function Constraints Based on Zwicky Transient Facility Searches for 13 Neutron Star Merger Triggers during O3”. In: The Astrophysical Journal 905.2 (Dec. 2020), p. 145. issn: 1538-4357. doi: 10.3847/1538- 4357/abc335. url: http://dx.doi.org/10.3847/1538-4357/abc335.<jupyter_code>import os
import requests

import pandas as pd
import numpy as np

from astropy.coordinates import SkyCoord
from astropy import units as u

# pip install fink_filters
from fink_filters import __file__ as fink_filters_location

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('talk')

APIURL = 'https://fink-portal.org'<jupyter_output><empty_output><jupyter_text>KN-Mangrove dataLet's load the alert data from this filter:<jupyter_code>pdf_kn_ma = pd.read_parquet('data/0104_3009_kn_filter2_class.parquet')

nalerts_kn_ma = len(pdf_kn_ma)
nunique_alerts_kn_ma = len(np.unique(pdf_kn_ma['objectId']))

print(
    '{} alerts loaded ({} unique objects)'.format(
        nalerts_kn_ma, 
        nunique_alerts_kn_ma
    )
)<jupyter_output>68 alerts loaded (59 unique objects)<jupyter_text>Visualising the candidatesFinally, let's inspect one lightcurve:<jupyter_code>oid = pdf_kn_ma['objectId'].values[2]
tns_class = pdf_kn_ma['TNS'].values[2]
kn_trigger = pdf_kn_ma['candidate'].apply(lambda x: x['jd']).values[2]

r = requests.post(
  '{}/api/v1/objects'.format(APIURL),
  json={
    'objectId': oid,
    'withupperlim': 'True'
  }
)

# Format output in a DataFrame
pdf = pd.read_json(r.content)

fig = plt.figure(figsize=(15, 6))

colordic = {1: 'C0', 2: 'C1'}

for filt in np.unique(pdf['i:fid']):
    maskFilt = pdf['i:fid'] == filt

    # The column `d:tag` is used to check data type
    maskValid = pdf['d:tag'] == 'valid'
    plt.errorbar(
        pdf[maskValid & maskFilt]['i:jd'].apply(lambda x: x - 2400000.5),
        pdf[maskValid & maskFilt]['i:magpsf'],
        pdf[maskValid & maskFilt]['i:sigmapsf'],
        ls = '', marker='o', color=colordic[filt]
    )

    maskUpper = pdf['d:tag'] == 'upperlim'
    plt.plot(
        pdf[maskUpper & maskFilt]['i:jd'].apply(lambda x: x - 2400000.5),
        pdf[maskUpper & maskFilt]['i:diffmaglim'],
        ls='', marker='^', color=colordic[filt], markerfacecolor='none'
    )

    maskBadquality = pdf['d:tag'] == 'badquality'
    plt.errorbar(
        pdf[maskBadquality & maskFilt]['i:jd'].apply(lambda x: x - 2400000.5),
        pdf[maskBadquality & maskFilt]['i:magpsf'],
        pdf[maskBadquality & maskFilt]['i:sigmapsf'],
        ls='', marker='v', color=colordic[filt]
    )

plt.axvline(kn_trigger - 2400000.5, ls='--', color='grey')
plt.gca().invert_yaxis()
plt.xlabel('Modified Julian Date')
plt.ylabel('Magnitude')
plt.title('{}'.format(oid))
plt.show()
print('{}/{}'.format(APIURL, oid))<jupyter_output><empty_output><jupyter_text>Circles (●) with error bars show valid alerts that pass the Fink quality cuts. Upper triangles with errors (▲), representing alert measurements that do not satisfy Fink quality cuts, but are nevetheless contained in the history of valid alerts and used by classifiers. Lower triangles (▽), representing 5-sigma mag limit in difference image based on PSF-fit photometry contained in the history of valid alerts. The vertical line shows the KN trigger by Fink.  Evolution of the classification Each alert was triggered because the Fink pipelines favoured the KN flavor at the time of emission. But the underlying object on the sky might have generated further alerts after, and the classification could evolve. For a handful of alerts, let see what they became. For this, we will use the Fink REST API, and query all the data for the underlying object:<jupyter_code>NALERTS = 3
oids = pdf_kn_ma['objectId'].values[0: NALERTS]
kn_triggers = pdf_kn_ma['candidate'].apply(lambda x: x['jd']).values[0: NALERTS]

for oid, kn_trigger in zip(oids, kn_triggers):
    r = requests.post(
      '{}/api/v1/objects'.format(APIURL),
      json={
        'objectId': oid,
        'output-format': 'json'
      }
    )

    # Format output in a DataFrame
    pdf_ = pd.read_json(r.content)
    times, classes = np.transpose(pdf_[['i:jd','v:classification']].values)
    
    fig = plt.figure(figsize=(12, 5))
    
    plt.plot(times, classes, ls='', marker='o')
    plt.axvline(kn_trigger, ls='--', color='C1')
    
    plt.title(oid)
    plt.xlabel('Time (Julian Date)')
    plt.ylabel('Fink inferred classification')
    plt.show()
oids<jupyter_output><empty_output><jupyter_text>Note that Kilonova classification does not appear here as this label is reserved to the KN-LC filter. We are working on giving a new label. One can see that alert classification for a given object can change over time. With time, we collect more data, and have a clearer view on the nature of the object. Let's make an histogram of the final classification for each object (~1min to run)<jupyter_code>final_classes = []
oids = np.unique(pdf_kn_ma['objectId'].values)
for oid in oids:
    r = requests.post(
      '{}/api/v1/objects'.format(APIURL),
      json={
        'objectId': oid,
        'output-format': 'json'
      }
    )
    pdf_ = pd.read_json(r.content)
    if not pdf_.empty:
        final_classes.append(pdf_['v:classification'].values[0])
fig = plt.figure(figsize=(12, 5))

plt.hist(final_classes)

plt.xticks(rotation=15.)
plt.title('Final Fink classification of KN candidates');<jupyter_output><empty_output><jupyter_text>Most of the objects are still unknown according to Fink.  Follow-up of candidates by other instruments Some of the alerts benefited from follow-up by other instruments to determine their nature. Usually this information can be found on the TNS server (although this is highly biased towards Supernovae). We attached this information to the alerts (if it exists):<jupyter_code>pdf_kn_ma.groupby('TNS').count().sort_values('objectId', ascending=False)['objectId']<jupyter_output><empty_output><jupyter_text>We can see that among all 53 alerts forwarded by Fink, 39 have no known counterpart in TNS (i.e. no follow-up result was reported).   Retrieving  Mangrove data<jupyter_code>catalog_path = os.path.join(os.path.dirname(fink_filters_location), 'data/mangrove_filtered.csv')
pdf_mangrove = pd.read_csv(catalog_path)
pdf_mangrove.head(2)
# ZTF
ra1 = pdf_kn_ma['candidate'].apply(lambda x: x['ra'])
dec1 = pdf_kn_ma['candidate'].apply(lambda x: x['dec'])

# Mangrove
cols = ['internal_names', 'ra', 'declination', 'discoverydate', 'type']
ra2, dec2, name, lum_dist, ang_dist = pdf_mangrove['ra'], pdf_mangrove['dec'], pdf_mangrove['2MASS_name'], pdf_mangrove['lum_dist'], pdf_mangrove['ang_dist']

# create catalogs
catalog_ztf = SkyCoord(ra=ra1.values*u.degree, dec=dec1.values*u.degree)
catalog_tns = SkyCoord(ra=np.array(ra2, dtype=np.float)*u.degree, dec=np.array(dec2, dtype=np.float)*u.degree)

# cross-match
idx, d2d, d3d = catalog_ztf.match_to_catalog_sky(catalog_tns)

pdf_kn_ma['2MASS_name'] = name.values[idx]

pdf_kn_ma['separation (Kpc)'] = d2d.radian * ang_dist.values[idx] * 1000

pdf_kn_ma['lum_dist (Mpc)'] = lum_dist.values[idx]
pdf_kn_ma[['objectId', '2MASS_name', 'separation (Kpc)', 'lum_dist (Mpc)']].head(5)
fig = plt.figure(figsize=(12, 6))

plt.hist(pdf_kn_ma['lum_dist (Mpc)'], bins=20)

plt.xlabel('Luminosity distance of matching galaxies (Mpc)');
np.max(pdf_kn_ma['lum_dist (Mpc)'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11153081510934394, 'fraction_numerical': 0.024254473161033796, 'mean_word_length': 4.661789532920653, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '4443', 'license': 'Apache-2.0', 'n_tokens_mistral': 3456, 'n_tokens_neox': 3209, 'n_words': 1089, 'path': 'KN-Mangrove_filter.ipynb', 'repo_name': 'JulienPeloton/fink_grandma_kn', 'size': '10075'}