<jupyter_start><jupyter_text>Anomaly Detection Filter explained & example<jupyter_code># Download any missing packages 
#!pip install antares_client
import warnings
warnings.filterwarnings('ignore')  #suppress annoying deprecation warnings

import antares_client
from antares_client.search import get_by_id

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

plt.rc('font', size=15)
%config InlineBackend.figure_format = 'retina'  #for MacOS, make plots crisp<jupyter_output><empty_output><jupyter_text>Running this notebookThis notebook has been replicated to run on Google Colab here: https://colab.research.google.com/drive/12Vf8tL_HqejskVDb3uIlgzADLDOADUhm?usp=sharing You will have to:1. sign into your Google account2. acknowledge that the notebook was not written by Google and click "Run Anyway"3. select Runtime --> Run allOr, if you already have a datalab account at https://datalab.noirlab.edu, you can run this notebook under quickstart --> launch a jupyter notebook   Filter backgroundWe can search for loci by their associated tags, where a tag is produced by a single Filter. We are interested in the newly implemented (April 5, 2021) anomaly detection filter, written by <NAME> and <NAME> on behalf of the ANTARES team. The anomaly detection filter aims to tag transient events like supernovae, cataclysmic variables, and weird events/objects in the night sky. It is an isolation forest-based algorithm trained on 1,000,000 ANTARES alerts on 106 total light curve features (i.e. periodogram, stetson K, linear trend, etc.-- for full list see https://antares.noirlab.edu/properties), 53 for both R and g bands. Out of 1 million loci drawn from all over the sky, the overwhelming majority will be rather common non-variable stars and objects. But a small subsect of loci will be highly variable and short-lived transient events. In the streaming alerts-era, we want to catch, prioritize quick followup of, and study exotic objects/events in the midst of their interesting behavior instead of weeks or months later, which is often times too late. Thus, a Filter which looks for anomalous objects every night as new alerts come in is incredibly useful, particularly for short-lived, time sensitive events like supernovae.Each loci which has the "lc_feature_extractor" tag (currently 1.8 million) and all 106 features calculated can pass through to be assigned an anomaly score by the anomaly detection filter. Here is the performance of the Filter on a testing set of ~500,000 alerts: Currently a loci with an anomaly score > 0.53, does not live in the galactic plane (abs(b) > 10) and is not extremely red and does not have its strongest period between 100 and 1000 days (i.e. to weed out Mira stars) are tagged an "anomaly/outlier".  Searching for anomalous/outlier loci To find all the anomalous/outlier loci with the "iso_forest_anomaly_detection" tag, we can use the ANTARES elastic search query:<jupyter_code>%%time

query = {
    "query": {
        "term": {
            "tags": "iso_forest_anomaly_detection"
        }
    }
}
loci = antares_client.search.search(query)
locus_ids = [l.locus_id for l in loci]<jupyter_output>CPU times: user 50.3 ms, sys: 7.65 ms, total: 58 ms
Wall time: 274 ms<jupyter_text>All the ANTARES loci which successfully passes the conditions for the anomaly detection filter are collected here in a list. Because this filter is new, the amount of anomalous loci are small (24 objects):<jupyter_code>locus_ids
# get locus by ANTARES ID
locus = get_by_id("ANT2021ipbqa")<jupyter_output><empty_output><jupyter_text>With this locus we can also find its ZTF ID if we wish.<jupyter_code>locus.properties['ztf_object_id']<jupyter_output><empty_output><jupyter_text>> Note: With this ZTF ID, we can lookup this object's page in the ANTARES API with one quick click: https://antares.noirlab.edu/loci/lookup/ZTF21aaqjmnn.> And clicking on the ANTARES ID hyperlink will bring you to the object page.   Using anomalous loci information Via this webpage or in our notebook, we can see the other tags associated with this particular locus:<jupyter_code>locus.tags<jupyter_output><empty_output><jupyter_text>It seems like this may be an extragalactic transient. Let's take a closer look at the light curve information for this locus and plot.<jupyter_code>locus.lightcurve
df_lc = pd.DataFrame(locus.lightcurve)
# Origin or type of the Alert. 1 = ZTF Candidate, 2 = ZTF Upper Limit.
df_candidates = df_lc[df_lc['ant_survey'] == 1]
df_ulim = df_lc[df_lc['ant_survey'] == 2]
fig = plt.figure(figsize=(10,6))

x = df_candidates.ant_mjd
y = df_candidates.ant_mag
yerr = df_candidates.ant_magerr
c = df_candidates.ant_passband # green is g-band, red is R-band

plt.gca().invert_yaxis()
plt.scatter(x, y, c=c, marker='o', s=30, label='observation')
plt.errorbar(x, y, yerr=yerr, ecolor=c, fmt='', ls='')
plt.scatter(df_ulim.ant_mjd, df_ulim.ant_maglim, c=df_ulim.ant_passband, 
            marker='v', s=30, alpha=0.2, label='upper limit')
plt.xlabel('MJD')
plt.ylabel('Magnitude')
plt.legend(loc='upper left')
plt.title("ANT2021ipbqa")
plt.show()<jupyter_output><empty_output><jupyter_text>In ~10 days, this locus rose ~ 2 mags in g-band, and is currently flattening out as of April 6, 2021.According to the TNS page of this object: https://www.wis-tns.org/object/2021gyr, this object is a Supernova candidate as reported by the ALeRCE team. > Note: the TNS link can be found using the TNS External Services Link attached to this locus' webpage on the ANTARES API.  Because this object is interesting, we should request a TOM toolkit observation to collect more data.To do so, we should connect to TOM with our object as shown here: https://github.com/lchjoel1031/ANTARES/blob/main/ANTARES-TOM-AEON.ipynb  Looking at a non-anomalous light curve for comparison Here is the light curve of the loci 'ANT2020pyjuu', found to have an anomaly score at the low-end: anom_score = 0.40125.<jupyter_code># get locus by ANTARES ID
locus_inlier = get_by_id("ANT2020pyjuu")
locus_inlier.tags
df_lc_inlier = pd.DataFrame(locus_inlier.lightcurve)
# Origin or type of the Alert. 1 = ZTF Candidate, 2 = ZTF Upper Limit.
df_candidates_inlier = df_lc_inlier[df_lc_inlier['ant_survey'] == 1]
df_ulim_inlier = df_lc_inlier[df_lc_inlier['ant_survey'] == 2]
fig = plt.figure(figsize=(10,6))

x = df_candidates_inlier.ant_mjd
y = df_candidates_inlier.ant_mag_corrected
yerr = df_candidates_inlier.ant_magerr_corrected
c = df_candidates_inlier.ant_passband # green is g-band, red is R-band

plt.gca().invert_yaxis()
plt.scatter(x, y, c=c, marker='o', s=30, label='observation')
plt.errorbar(x, y, yerr=yerr, ecolor=c, fmt='', ls='')
plt.scatter(df_ulim_inlier.ant_mjd, df_ulim_inlier.ant_maglim, c=df_ulim_inlier.ant_passband, 
            marker='v', s=30, alpha=0.2, label='upper limit')
plt.xlabel('MJD')
plt.ylabel('Magnitude')
plt.legend(loc='upper left')
plt.title('ANT2020pyjuu')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08532374100719424, 'fraction_numerical': 0.019136690647482014, 'mean_word_length': 5.2734657039711195, 'pattern_counts': {'":': 3, '<': 30, '<?xml version=': 0, '>': 37, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '303163', 'license': 'MIT', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2210, 'n_words': 823, 'path': 'ANTARES/IF_AD_broker_workshop_final.ipynb', 'repo_name': 'deepchatterjeeligo/tutorials', 'size': '6970'}