<jupyter_start><jupyter_text>Dask pipeline  Example: Tracking the International Space Station with DaskIn this notebook we will be using two APIs:1. [Google Maps Geocoder](https://developers.google.com/maps/documentation/geocoding/overview)2. [Open Notify API for ISS location](http://api.open-notify.org/)We will use them to keep track of the ISS location and next lead time in relation to a list of cities. To create our diagrams and intelligently parallelise data, we use Dask, especially [Dask Delayed](../refactoring/performance/dask.htmlDask-Delayed).  1. Imports<jupyter_code>import requests
import logging
import sys
import numpy as np
from time import sleep
from datetime import datetime
from math import radians
from dask import delayed
from operator import itemgetter
from sklearn.neighbors import DistanceMetric<jupyter_output><empty_output><jupyter_text>2. Logger<jupyter_code>logger = logging.getLogger()
logger.setLevel(logging.INFO)<jupyter_output><empty_output><jupyter_text>3. Latitude and longitude pairs from a list of citiessee also [Location APIs](https://locationiq.com/)<jupyter_code>def get_lat_long(address):
    resp = requests.get(
        'https://eu1.locationiq.org/v1/search.php',
        params={'key': '<KEY>',
                'q': address,
                'format': 'json'}
    )
    if resp.status_code != 200:
        print('There was a problem with your request!')
        print(resp.content)
        return
    data = resp.json()[0]
    return {
        'name': data.get('display_name'),
        'lat': float(data.get('lat')),
        'long': float(data.get('lon')),
    }
get_lat_long('Berlin, Germany')
locations = []
for city in ['Seattle, Washington',
             'Miami, Florida', 
             'Berlin, Germany',
             'Singapore', 
             'Wellington, New Zealand',
             'Beirut, Lebanon',
             'Beijing, China',
             'Nairobi, Kenya',
             'Cape Town, South Africa',
             'Buenos Aires, Argentina']:
    locations.append(get_lat_long(city))
    sleep(2)
locations<jupyter_output><empty_output><jupyter_text>4. Retrieve ISS data and determine lead times for cities<jupyter_code>def get_spaceship_location():
    resp = requests.get('http://api.open-notify.org/iss-now.json')
    location = resp.json()['iss_position']
    return {'lat': float(location.get('latitude')),
            'long': float(location.get('longitude'))}
def great_circle_dist(lon1, lat1, lon2, lat2):
    "Found on SO: http://stackoverflow.com/a/41858332/380442"
    dist = DistanceMetric.get_metric('haversine')
    lon1, lat1, lon2, lat2 = map(np.radians, [lon1, lat1, lon2, lat2])

    X = [[lat1, lon1], [lat2, lon2]]
    kms = 6367
    return (kms * dist.pairwise(X)).max()
def iss_dist_from_loc(issloc, loc):
    distance = great_circle_dist(issloc.get('long'), 
                                 issloc.get('lat'), 
                                 loc.get('long'), loc.get('lat'))
    logging.info('ISS is ~%dkm from %s', int(distance), loc.get('name'))
    return distance
def iss_pass_near_loc(loc):
    resp = requests.get('http://api.open-notify.org/iss-pass.json',
                        params={'lat': loc.get('lat'), 
                                'lon': loc.get('long')})
    data = resp.json().get('response')[0]
    td = datetime.fromtimestamp(data.get('risetime')) - datetime.now()
    m, s = divmod(int(td.total_seconds()), 60)
    h, m = divmod(m, 60)
    logging.info('ISS will pass near %s in %02d:%02d:%02d',loc.get('name'), h, m, s)
    return td.total_seconds()
iss_dist_from_loc(get_spaceship_location(), locations[4])
iss_pass_near_loc(locations[4])<jupyter_output>INFO:root:ISS will pass near Wellington, Wellington City, Wellington, 6011, New Zealand in 02:32:28<jupyter_text>5. Create a `delayed` pipeline<jupyter_code>output = []

for loc in locations:
    issloc = delayed(get_spaceship_location)()
    dist = delayed(iss_dist_from_loc)(issloc, loc)
    output.append((loc.get('name'), dist))

closest = delayed(lambda x: sorted(x, 
                                   key=itemgetter(1))[0])(output)
closest<jupyter_output><empty_output><jupyter_text>6. Show DAG<jupyter_code>closest.visualize()<jupyter_output><empty_output><jupyter_text>7. `compute()`<jupyter_code>closest.compute()<jupyter_output>INFO:root:ISS is ~8893km from Singapore, Singapore, Central, Singapore
INFO:root:ISS is ~9226km from Autonomous City of Buenos Aires, Comuna 6, Autonomous City of Buenos Aires, Argentina
INFO:root:ISS is ~14460km from Wellington, Wellington City, Wellington, 6011, New Zealand
INFO:root:ISS is ~3694km from Cape Town, City of Cape Town, Western Cape, 8001, South Africa
INFO:root:ISS is ~4071km from Beirut, Ras Beirut, Beirut Governorate, Lebanon
INFO:root:ISS is ~1443km from Nairobi, Kenya
INFO:root:ISS is ~13891km from Seattle, King County, Washington, USA
INFO:root:ISS is ~6033km from Berlin, 10117, Germany
INFO:root:ISS is ~11454km from Miami, Miami-Dade County, Florida, USA
INFO:root:ISS is ~10292km from Beijing, Dongcheng District, Beijing, 100010, China
{'fraction_non_alphanumeric': 0.12840236686390533, 'fraction_numerical': 0.027416173570019724, 'mean_word_length': 3.7659774436090228, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '12990', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1745, 'n_tokens_neox': 1609, 'n_words': 414, 'path': 'docs/clean-prep/dask-pipeline.ipynb', 'repo_name': 'veit/jupyter-tutorial', 'size': '5079'}