<jupyter_start><jupyter_text>ZTF alerts demoIn this demo, we will:- Set up `Docker`- Using `Docker`, deploy a `MongoDB` database, fetch a night worth of (pre-filtered to reduce size) public ZTF alerts, ingest it into the database, and create indices in the database for faster queries- Set up `Robo3T` and use it to look at the database and query it- Query the database using `python`- Inspect the contents of an alert packet- Construct and plot a light curve- Plot the cutout images from an alert packet Download and install `Docker` and `Robo3T`- Download and install the appropriate version of `Docker` for your platform from [here](https://www.docker.com/community-edition). You will need to create an account on their website.- Download and install `Robo3T` from [here](https://robomongo.org/download).  We will use it to connect to the database. Fetch, build, and run the code to deploy a `MongoDB` database, fetch a night worth of (pre-filtered to reduce size) public ZTF alerts, ingest it into the database, and create indices in the database for faster queriesThis is a lot of stuff! Sounds scary, however with the help of `Docker`, we will only have to run a few simple commands to do all that.Clone the repo and `cd` into the directory:```bashgit clone https://github.com/dmitryduev/ztf-alerts-demo.gitcd ztf-alerts-demo```The inside/contents of a `Docker` container get destroyed when it is removed, so we need to tell `Docker` to keep the useful data in a "persistent" storage.Create a so-called persistent `Docker` volume for `MongoDB`:```bashdocker volume create alert-fetcher-mongo-volume```Launch the `MongoDB` container. (Feel free to change u/p for the db admin)```bashdocker run -d --restart always --name alert-fetcher-mongo -p 27018:27017 \       -v alert-fetcher-mongo-volume:/data/db \       -e MONGO_INITDB_ROOT_USERNAME=mongoadmin \       -e MONGO_INITDB_ROOT_PASSWORD=<PASSWORD> \       mongo:latest```That's it for the database set-up! You can now connect to it on `localhost` on port `27018`.Finally, build and launch the alert-fetcher container. We will bind-mount a directory on your host machine to store the alerts:```bashcd alert-fetcherdocker build -t alert-fetcher -f Dockerfile . make sure path /path/to/alerts (or whatever path you specify) existsdocker run --rm -v /path/to/alerts:/alerts \           --name alert-fetcher --link alert-fetcher-mongo:mongo -it alert-fetcher```This will launch a program that will fetch a pre-filtered (demo) set of public ZTF alerts from July 13, 2018, ingest that into the MongoDB database, and create indicies to accelerate queries. The script will add a `coordinates` field to each that is not part of the original alert packets, to allow 2d indexation on the sphere for fast positional/cone searches. Additionally, a unique string `candid_objectId` is used a the alert identifier in the database. ---We can now use `Robo3T` to connect to the database and take a look at the alerts. Launch `Robo3T` and click "create" connection. - Name the connection, for example, `local_docker_alerts_demo`- Use `localhost` : `27018` as the address- On the `Authentication` tab, check "perform authentication", and use `admin` for `Database`, `mongoadmin` for `User Name`, `mongoadminsecret` for 'Password'(if you did not choose other u/p), and `SCRAM-SHA-1` for `Auth Mechanism`- Click `Save` and then `Connect``Robo3T` will connect to the database. To see the ingested alerts, click `ztf_alerts` -> `Collections` -> `alerts`. --- Notes [optional]To check the running/stopped containers, type:```bashdocker ps -a```To stop and remove a container, run:```bashdocker stop container_namedocker rm -f container_name```To get all public alerts from the ZTF archive for a given night, remove `--demo` from the last line of the file `alert-fetcher/Dockerfile` and change the date string. Then re-build and re-run the alert-fetcher container.Every time when you change the code that is used inside a container, the latter must be re-build and restarted.Alerts are stored in the database in a serialized binary format that resembles the Apache Avro format used in the packets in that it can be easily converted into `json` or a `python` dictionary.--- We will now explore the contents of an avro packet. We will also learn how to construct and display a light curve for an alert and plot the cutout images._NOTE:_ You may want to look at Eric Bellm's Jupyter [notebook](https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Working_with_avro_files.ipynb) with some more examples. Let's create a `python` virtual environment with `conda` and install the required python modules into it:```bashconda create --name ztfalerts python=3.6 type y to proceed install pip and jupyter:conda install -n ztfalerts pipconda install -n ztfalerts jupyter activate the environment:source activate ztfalerts download requirements.txt:wget https://raw.githubusercontent.com/dmitryduev/ztf-alerts-demo/master/alert-fetcher/code/requirements.txt install required modules into the environment:pip install -r requirements.txt```<jupyter_code>import os
import io
import gzip
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import fastavro

from astropy.time import Time
from astropy.io import fits
import aplpy

%matplotlib inline<jupyter_output><empty_output><jupyter_text>** Contents of an avro packet **Each avro packet contains many different pieces of information about an alert. You can see a description of all of the contents here:https://zwickytransientfacility.github.io/ztf-avro-alert/schema.htmlWe recommend keeping this description open in a separate tab for reference during these exercises. Now, let's look at a particular avro packet:<jupyter_code># You'll have to change the path to match wherever you put the alerts
avro_packet = './ztf-alerts-demo/alert-fetcher/alerts/20180713/558359715915010021.avro'
with open(avro_packet, 'rb') as fa:
    freader = fastavro.reader(fa)
    schema = freader.schema

    for packet in freader:
        print(packet.keys())<jupyter_output>dict_keys(['objectId', 'candidate', 'candid', 'prv_candidates', 'cutoutDifference', 'cutoutScience', 'cutoutTemplate'])<jupyter_text>The schema is stored in the packet:<jupyter_code>schema<jupyter_output><empty_output><jupyter_text>The payload, once in memory, is a python dictionary, so the attributes are easy to access<jupyter_code>packet
print('JD: {} Filter: {} Mag: {:.2f}+/-{:.2f}'.format(
    packet['candidate']['jd'], packet['candidate']['fid'],
    packet['candidate']['magpsf'], packet['candidate']['sigmapsf']))<jupyter_output>JD: 2458312.8597106 Filter: 1 Mag: 18.83+/-0.09<jupyter_text>**NOTE ESPECIALLY:** the magnitudes here do not include the magnitude of the underlying reference source (if present), so if this is a variable star further adjustment is needed.<jupyter_code>### Exercise: Using the schema guide linked above, answer the following questions about this candidate:
# 1) What is the real-bogus score?

print( # Finish this line

# 2) What is the distance to the nearest source in Pan-STARRS? 
    
print( # Finish this line
        
# 3) What is the magnitude and color of the nearest source in Pan-STARRS?
        
print( # Finish this line        

# 4) Do you think that this candidate is a star? 

print( # Add code here<jupyter_output><empty_output><jupyter_text>Record access like the above is a little verbose; let's wrap things up in a pandas dataframe for ease of access (and faster loading). As an example, we will extract the lightcurves. The alert packet formats are nested, so the historical detections (if present) have the same structure as the candidate triggering the alert (minus a couple fields).<jupyter_code>def make_dataframe(packet):
    df = pd.DataFrame(packet['candidate'], index=[0]) # the current alert
    df_prv = pd.DataFrame(packet['prv_candidates']) # obtaining all previous alerts at this location
    return pd.concat([df, df_prv], ignore_index=True) # we put the current alert and previous ones in the same table
dflc = make_dataframe(packet)
dflc<jupyter_output><empty_output><jupyter_text>Each row in the table is one alert. The first row is the most recent alert, and the remaining rows are previous alerts at the same location. Looking at the table, you can see that some of the historical detections are upper limits, signified by the NaNs. Note that the most recent candidate has a few fields that are not present for the prv_candidates.<jupyter_code>### The current candidate is the first line in this table, df[0]
current_cand = df[0]

### Exercise, repeat the exercises above, but this time using the df structure

# 1) What is the JD, mag, filter, and uncertainty on the magnitude?
print(df[0][ # finish this line

# 2) What is the real-bogus score?

print(df[0][ # Finish this line

# 3) What is the distance to the nearest source in Pan-STARRS? 
    
print(df[0][ # Finish this line
        
# 4) What is the magnitude and color of the nearest source in Pan-STARRS?
        
print(df[0][ # Finish this line

# 5) Do you think that this candidate is a star? 

print(df[0][ # Finish this line<jupyter_output><empty_output><jupyter_text>Now, let's move on to plotting the light curve!<jupyter_code>def plot_lightcurve(dflc, days_ago=True):
    
    filter_color = {1: 'green', 2: 'red', 3: 'pink'}
    if days_ago:
        now = Time.now().jd
        t = dflc.jd - now
        xlabel = 'Days Ago'
    else:
        t = dflc.jd
        xlabel = 'Time (JD)'
    
    plt.figure()
    for fid, color in filter_color.items():
        # plot detections in this filter:
        w = (dflc.fid == fid) & ~dflc.magpsf.isnull()
        if np.sum(w):
            plt.errorbar(t[w], dflc.loc[w, 'magpsf'], dflc.loc[w, 'sigmapsf'],
                         fmt='.', color=color)
        wnodet = (dflc.fid == fid) & dflc.magpsf.isnull()
        if np.sum(wnodet):
            plt.scatter(t[wnodet], dflc.loc[wnodet, 'diffmaglim'], 
                        marker='v', color=color, alpha=0.25)
    
    plt.gca().invert_yaxis()
    plt.xlabel(xlabel)
    plt.ylabel('Magnitude')
plot_lightcurve(dflc)<jupyter_output><empty_output><jupyter_text>Now let's figure out how to display the cutout images. Note that these are gzip-compressed fits files stored as bytes.<jupyter_code>def plot_cutout(stamp, fig=None, subplot=None, **kwargs):
    with gzip.open(io.BytesIO(stamp), 'rb') as f:
        with fits.open(io.BytesIO(f.read())) as hdul:
            if fig is None:
                fig = plt.figure(figsize=(4, 4))
            if subplot is None:
                subplot = (1, 1, 1)
            ffig = aplpy.FITSFigure(hdul[0], figure=fig, subplot=subplot, **kwargs)
            ffig.show_grayscale(stretch='arcsinh')
    return ffig


def show_stamps(packet):
    fig = plt.figure(figsize=(12, 4))
    for i, cutout in enumerate(['Science', 'Template', 'Difference']):
        stamp = packet['cutout{}'.format(cutout)]['stampData']
        ffig = plot_cutout(stamp, fig=fig, subplot=(1, 3, i+1))
        ffig.set_title(cutout)
show_stamps(packet)<jupyter_output>WARNING: No WCS information found in header - using pixel coordinates [aplpy.header]
WARNING:astropy:No WCS information found in header - using pixel coordinates
INFO:astropy:Auto-setting vmin to  9.311e+01
INFO:astropy:Auto-setting vmax to  7.263e+02
WARNING: No WCS information found in header - using pixel coordinates [aplpy.header]
WARNING:astropy:No WCS information found in header - using pixel coordinates<jupyter_text>---Let us now look at a few examples of how to query the database.<jupyter_code>import pymongo


def connect_to_db(host='localhost', port=27018, 
                  user='ztf_user', password='<PASSWORD>'):
    _client = pymongo.MongoClient(host=host, port=port)
    # grab main database:
    _db = _client['ztf_alerts']
    # authenticate
    _db.authenticate(user, password)
    
    return _db
db = connect_to_db()<jupyter_output><empty_output><jupyter_text>Find all distinct transient `objectId`'s present in the database:<jupyter_code>objectIds = db['alerts'].distinct('objectId')
# print ten
print(objectIds[:10])<jupyter_output>['ZTF17aaaaamy', 'ZTF17aaadzas', 'ZTF17aaadzci', 'ZTF17aaaebhk', 'ZTF17aaaedhh', 'ZTF17aaaenwn', 'ZTF17aaafyje', 'ZTF17aaajjpk', 'ZTF17aaajjvp', 'ZTF17aaapvru']<jupyter_text>Count the number of alerts in the database:<jupyter_code>num_alerts = db['alerts'].count()
print(num_alerts)<jupyter_output>834<jupyter_text>Count the number of alerts in the database with an rb score > 0.5:<jupyter_code>num_alerts_rb_gt_05 = db['alerts'].count({'candidate.rb': {'$gt': 0.5}})
print(num_alerts_rb_gt_05)<jupyter_output>451<jupyter_text>Get all transient `objectId`'s detected more than once:<jupyter_code>cursor = db['alerts'].aggregate([{'$group': {'_id': '$objectId', 'count': {'$sum': 1}}}, 
                                 {'$match': {'count': {'$gt': 1}}}, 
                                 {'$project': {'objectId': '$_id', '_id': 0}}], 
                                allowDiskUse=True)
for alert in cursor:
    print(alert)<jupyter_output>{'objectId': 'ZTF18abgladq'}
{'objectId': 'ZTF18abesqnb'}<jupyter_text>Get all alert `objectId`'s for transients with more than one detection in R and i bands, each with an rb score of >= 0.3<jupyter_code>cursor = db['alerts'].aggregate([{'$group': {'_id': '$objectId',
                                             'count': {'$sum': {'$cond': [{'$and': [
                                                 {'$in': ['$candidate.fid', [2, 3]]},
                                                 {'$gt': ['$candidate.rb', 0.3]}
                                             ]
                                             }, 1, 0]
                                             }
                                             }}},
                                 {'$match': {'count': {'$gt': 1}}},
                                 {'$project': {'objectId': '$_id', '_id': 0}}], allowDiskUse=True)
for alert in cursor:
    print(alert)<jupyter_output>{'objectId': 'ZTF18abgladq'}<jupyter_text>Get alerts with `objectId` `ZTF18abgladq`. Plot the light curve of the second one and display the cutout images for it.<jupyter_code>cursor = db['alerts'].find({'objectId': {'$eq': 'ZTF18abgladq'}})

alerts = [alert for alert in cursor]

plot_lightcurve(make_dataframe(alerts[1]))
show_stamps(alerts[1])<jupyter_output>WARNING: No WCS information found in header - using pixel coordinates [aplpy.header]
WARNING:astropy:No WCS information found in header - using pixel coordinates
INFO:astropy:Auto-setting vmin to  1.049e+02
INFO:astropy:Auto-setting vmax to  4.320e+02
WARNING: No WCS information found in header - using pixel coordinates [aplpy.header]
WARNING:astropy:No WCS information found in header - using pixel coordinates<jupyter_text>---The `coordinates.radec_geojson` field defined for every object in the database has an associated spherical 2D index, which allows for extremely fast positional queries. `MongoDB` supports many query operators, see [here](https://docs.mongodb.com/manual/reference/operator/query-geospatial/) for more details. The caveat to keep in mind is the following: `MongoDB` uses `GeoJSON` objects to represent `2D` positions on the sphere. Both the longitude (RA) and latitude (Dec) must be expressed in decimal degrees, and the valid longitude values are between `-180` and `180`, both inclusive, so you must subtract 180.0 degrees from your RA value.Let's define a helper function for cone searches:<jupyter_code>def cone_search(_db, collection='alerts',
                ra=0.0, dec=0.0, radius=1.0, unit='arcsec',
                projection=None):
    # cone search radius:
    cone_search_radius = float(radius)
    # convert to rad:
    if unit == 'arcsec':
        cone_search_radius *= np.pi / 180.0 / 3600.
    elif unit == 'arcmin':
        cone_search_radius *= np.pi / 180.0 / 60.
    elif unit == 'deg':
        cone_search_radius *= np.pi / 180.0
    elif unit == 'rad':
        cone_search_radius *= 1
    else:
        raise Exception('Unknown cone search unit. Must be in [deg, rad, arcsec, arcmin]')
    
    # fields to return: everything by default
    if projection is None:
        projection = dict()
    
    cursor = _db[collection].find({'coordinates.radec_geojson': {
        '$geoWithin': {'$centerSphere':
                           [[ra - 180.0, dec],
                            cone_search_radius]
                       }
    }}, projection)
    
    return cursor<jupyter_output><empty_output><jupyter_text>Now let's look for alerts within 30 arcsec from RA, Dec: (239.71 deg, 14.97 deg) and get their `objectId`'s and `rb` scores:<jupyter_code>alerts = list(cone_search(db, ra=239.71, dec=14.97, radius=30.0, unit='arcsec',
                          projection={'_id': 0, 'objectId': 1, 'candidate.rb': 1}))
print(alerts)<jupyter_output>[{'objectId': 'ZTF18abgladq', 'candidate': {'rb': 0.47333332896232605}}, {'objectId': 'ZTF18abgladq', 'candidate': {'rb': 0.5266666412353516}}, {'objectId': 'ZTF18abgladq', 'candidate': {'rb': 0.5133333206176758}}]
{'fraction_non_alphanumeric': 0.10788211905040346, 'fraction_numerical': 0.019646824932756404, 'mean_word_length': 3.745560488346282, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 69, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '300411', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5508, 'n_tokens_neox': 5151, 'n_words': 1840, 'path': 'notebooks/ztf-alerts-demo.ipynb', 'repo_name': 'rnikutta/ztfss18', 'size': '17116'}