<jupyter_start><jupyter_text>TDE Candidates <NAME>This notebook was originally written by <NAME>.  It has been slightly modified and annotated by <NAME> for tutorial purposes. It bypasses the Lasair Client code for the time being.  The code demonstrates use of the Lasair API to:* Grab your token* Make Queries* Grab lightcurvesThe code also demonstrates how to interact with the TNS API and also the PS1 postage stamp server at MAST. Requirements pip installastropy, matplotlib, requests shell tools:curl API Tokens (must be obtained beforehand - temporary ones used here)Lasair API token, TNS API token  Do the imports<jupyter_code>import requests, json
import matplotlib.pyplot as plt
from astropy.time import Time
from collections import OrderedDict
import os
import csv
import io as StringIO
import matplotlib.image as mpimg<jupyter_output><empty_output><jupyter_text>Grab my Lasair API tokenGet your token before you start.  Change the cell below to enter your own credentials, or better still, set the last line of this cell to be your token ID.  You can also collect your token this way```bashcurl --data "username=tutorial&password=***********" https://lasair-iris.roe.ac.uk/api/auth-token/```You should get the following response:```json{"token":"acf14c2788ec5d7abb351f06e9ce2624a810b2ab"}```We will use this token temporarily, but note that it will be disabled after the workshop!<jupyter_code># What is my Lasair API Key?
# THIS ACCOUNT IS TEMPORARY AND WILL BE DELETED AFTER THE WORKSHOP

headers = { 'Authorization': 'Token acf14c2788ec5d7abb351f06e9ce2624a810b2ab' }<jupyter_output><empty_output><jupyter_text>Large SQL querySee https://lasair-iris.roe.ac.uk/schema to see what metadata is available in the Objects table.<jupyter_code># Matt's query
selected = """
objects.objectId,
    ROUND(objects.maggmean - objects.magrmean, 2) AS g_minus_r_mean,
    ROUND(objects.gmag - objects.rmag, 2) AS g_minus_r_latest,
    objects.gmag AS g,
    objects.rmag AS r,
    FROM_UNIXTIME((objects.jdmax - 1721059.5) * (3600 * 24) - TO_SECONDS('1970-01-01 00:00:00')
       + TO_SECONDS(UTC_TIMESTAMP()) - TO_SECONDS(CURRENT_TIMESTAMP())) AS latest_utc,
    objects.ncandgp AS N_obs,
    objects.ncandgp_14 as N_obs_14_day,
    IF(ABS(objects.gmag - objects.maggmin) < 0.2
            AND objects.gmag <= objects.maggmean
            AND ABS(objects.rmag - objects.magrmin) < 0.2
            AND objects.rmag <= objects.magrmean,
        'rising/peak',
        '') rising,
    IF(objects.gmag > objects.maggmin + 0.2
            OR objects.rmag > objects.magrmin + 0.2,
        'fading',
        '') fading,
    sherlock_classifications.classification AS sherlock,
    ROUND(sherlock_classifications.separationArcsec,
            2) AS offset,
    objects.maggmin AS g_max,
    objects.magrmin AS r_max,
    objects.sgmag1 AS g_host,
    objects.srmag1 AS r_host,
    ROUND(objects.sgmag1 - objects.srmag1, 2) as g_minus_r_host,
    objects.sgscore1 AS sg_score,
    ROUND(objects.jdmin - 2400000.5, 1) AS disc_mjd,
    ROUND(objects.jdmax - 2400000.5, 1) AS latest_mjd,
    objects.ramean AS ra,
    objects.decmean AS decl
"""
# Matt's WHERE clause
conditions = """
sherlock_classifications.separationArcsec < 1.0
        AND objects.ncand > 2
        AND objects.ncand - objects.ncandgp < 3
        AND sherlock_classifications.classification in ('NT','SN')
        AND (objects.maggmean - objects.magrmean < 0.05
        OR objects.gmag - objects.rmag < 0.05)
        AND (objects.rmag < 20.0
        OR objects.gmag < 20.0)
        AND (objects.glatmean > 10
        OR objects.glatmean < - 10)
        AND objects.sgmag1 - objects.srmag1 > 0
        AND objects.jdmax > JDNOW()-14
        group by objects.objectId HAVING N_obs_14_day > 1
        order by objects.jdmax DESC
"""<jupyter_output><empty_output><jupyter_text>Send the query request to the Lasair API<jupyter_code>url = 'https://lasair-iris.roe.ac.uk/api/query/'

# KWS: Temporarily added limit to reduce the number of results returned
data = {
'selected'  : selected,
'tables'    : 'objects, sherlock_classifications',
'conditions': conditions,
'limit'     : 10
}

r = requests.post(url, data, headers=headers) 
if r.status_code == 200:
    response = r.json()
else:
    print('Oops, status code is', r.status_code)
    print(r.text)<jupyter_output><empty_output><jupyter_text>Check for TNS classifications using TNS APINote that TNS crossmatching is now done directly on Lasair - so technically this step is not required! The TNS information can be interrogated as part of the /objects/ API.<jupyter_code># Query TNS for classifications

tns_url = "https://www.wis-tns.org/api/get/"

# Lasair test bot2 key.
# THIS TNS TOKEN IS TEMPORARY AND WILL BE DELETED AFTER THE WORKSHOP
api_key="6fa8e1f751ae24f9246c57e14b597291584be093"

print('Searching TNS...')
print ("\nZTF objectId    TNS id    Type ")

for obj in response:

    # start with cone search around ZTF coordinates
    search_obj = OrderedDict([
        ("ra",obj['ra']), 
        ("dec",obj['decl']), 
        ("radius","2.5"), 
        ("units","arcsec"), 
        ("objname",""), 
        ("internal_name","")]) 
    search_url = tns_url+'search'
    search_data = [('api_key',(None, api_key)), ('data',(None,json.dumps(search_obj)))]

    r = requests.post(search_url, files=search_data)

    # If transient is known, will have an IAU name (AT/SN 20XXyy), add such names to our objects
    if r.json()['data']['reply']:
        iau_name = r.json()['data']['reply'][0]['prefix'] + r.json()['data']['reply'][0]['objname']
    else:
        iau_name = ''
        
    obj['iau_name'] = iau_name

    if r.json()['data']['reply']:

        # Now check if object has a classification attached
        # Always will for 'SN' names, but could also happen for 'AT', e.g. if TDE rather than SN
        get_obj = OrderedDict([("objname",iau_name[2:]), ("photometry","0"), ("spectra","0")])
        get_url = tns_url+'object'
        get_data = [('api_key',(None, api_key)), ('data',(None,json.dumps(get_obj)))]
  
        r2 = requests.post(get_url, files=get_data)
     
        # If classified, add type to dictionary
        if r2.json()['data']['reply']['object_type']['name']:
            obj['class'] = r2.json()['data']['reply']['object_type']['name']
        else:
            obj['class'] = ''
    else:
        obj['class'] = ''

            
#     print ("%s   %s   %s  " % (obj['objectId'], obj['iau_name'], obj['class']))

print('\nDone.')<jupyter_output>Searching TNS...

ZTF objectId    TNS id    Type 

Done.<jupyter_text>Grab PS1 Postage StampsGet stamps for the objects we just selected. These are stored in /tmp by default.<jupyter_code># Get PS1 image of field

ps1_url = 'http://ps1images.stsci.edu/cgi-bin/ps1filenames.py?&filters=gri'

cutout_url = 'http://ps1images.stsci.edu/cgi-bin/fitscut.cgi?&filetypes=stack&size=240'

print('Downloading PS1 cutouts...')

for obj in response:
    i = obj['objectId']

    dest_file = '/tmp/' + i + '.jpg'
    obj['cutout'] = dest_file

    if not os.path.exists(dest_file):

        # first step searches for the ps1 images matching target coordinates
        ps1_search_url = ps1_url+'&ra='+str(obj['ra']) + '&dec='+str(obj['decl'])
        print(i,obj['ra'],obj['decl'])
        print(ps1_search_url)
        ps1_im = requests.get(ps1_search_url)
        #print("PS1 im.text = ")
        #print(ps1_im.text)
        
        # 2019-06-16 KWS Modified this to use a csv and StringIO object and present a dictionary.
        response2 = csv.DictReader(StringIO.StringIO(ps1_im.text), delimiter=' ', skipinitialspace = True)
        respList = []
        
        for row in response2:
            respList.append(row)
            
        ps1_g = respList[0]['filename']
        ps1_r = respList[2]['filename']
        ps1_i = respList[1]['filename']
        #print(ps1_g,ps1_r,ps1_i)
        
        cutout_search_url = cutout_url + '&blue='+ps1_g + '&green='+ps1_r + '&red='+ps1_i
        cutout_search_url +=  '&ra='+str(obj['ra']) + '&dec='+str(obj['decl'])

        cmd = 'curl -o %s "%s"' % (dest_file, cutout_search_url)
        os.system(cmd)
    
print('Done.')<jupyter_output>Downloading PS1 cutouts...
Done.<jupyter_text>Get Lightcurves with the Lasair /lightcurves/ APIFor each object we returned go back to Lasair and get the full lightcuve. This can't be done in a single query anymore because the detections ("candidates") are currently not stored in the database.<jupyter_code>lc_url = 'https://lasair-iris.roe.ac.uk/api/lightcurves/'

objectIds = ','.join([obj['objectId'] for obj in response])
data = {'objectIds': objectIds}

r = requests.post(lc_url, data, headers=headers)

# KWS: Matt's code thinks lcs is a dict. But API returns a list.
lcs = r.json()

# KWS: So let's rebuild the dict

lcsDict = {}
for lc in lcs:        
    lcsDict[lc['objectId']] = {'candidates': lc['candidates']}<jupyter_output><empty_output><jupyter_text>Plot the lightcurves and stamps<jupyter_code>cols = {1:'g',2:'r'}

now = Time.now().jd

for obj in response:
    if obj['class'] in ['','TDE']:
        plt.figure()
        plt.subplot(121)
        for i in lcsDict[obj['objectId']]['candidates']:
            if 'isdiffpos' in i:
                if i['isdiffpos']=='t':
                    plt.errorbar(i['jd']-now,i['magpsf'],i['sigmapsf'],fmt='o',color=cols[i['fid']])
        plt.gca().invert_yaxis()
        plt.ylabel('Magnitude')
        plt.xlabel('Days ago')
        plt.title(obj['objectId'])
        
            # show cutout
        if 'cutout' in obj:
            if os.path.exists(obj['cutout']):
                plt.subplot(122)
                file = obj['cutout']
                try:
                    img = mpimg.imread(file)
                    plt.imshow(img)
                except:
                    print("cannot find file ", file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13002124860872205, 'fraction_numerical': 0.023879388849539614, 'mean_word_length': 3.1775147928994083, 'pattern_counts': {'":': 1, '<': 45, '<?xml version=': 0, '>': 41, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '300560', 'license': 'MIT', 'n_tokens_mistral': 3517, 'n_tokens_neox': 3306, 'n_words': 859, 'path': 'Lasair/TDE-candidates-kws.ipynb', 'repo_name': 'deepchatterjeeligo/tutorials', 'size': '9898'}