<jupyter_start><jupyter_text>An introduction to S3, Boto and Nexrad on S3 Adapted from and thank to the first tutorial by <NAME>, formerly at Climate Corp now at Google.https://eng.climate.com/2015/10/27/how-to-read-and-display-nexrad-on-aws-using-python/  Amazon Simple Storage Service (Amazon S3)  is object storage with a simple web service interface to store and retrieve any amount of data from anywhere on the web. It is designed to deliver 99.999999999% durability, and scale past trillions of objects worldwide.Boto is a Python package that provides interfaces to Amazon Web Services.<jupyter_code>#Lets import some stuff!
import boto
from boto.s3.connection import S3Connection
from datetime import timedelta, datetime
import os
import pyart
from matplotlib import pyplot as plt
import tempfile
import numpy as np

%matplotlib inline<jupyter_output><empty_output><jupyter_text>From https://aws.amazon.com/noaa-big-data/nexrad/ :The NEXRAD Level II archive data is hosted in the “noaa-nexrad-level2” Amazon S3 bucket in S3’s US East region. The address for the public bucket is:http://noaa-nexrad-level2.s3.amazonaws.comhttps://noaa-nexrad-level2.s3.amazonaws.comEach volume scan file is its own object in Amazon S3.  The basic data format is the following:`/////`Where: is the year the data was collected is the month of the year the data was collected is the day of the month the data was collected is the NEXRAD ground station (map of ground stations) is the name of the file containing the data. These are compressed files (compressed with gzip). The file name has more precise timestamp information.All files in the archive use the same compressed format (.gz). The data file names are, for example, KAKQ20010101_080138.gz. The file naming convention is:GGGGYYYYMMDD_TTTTTTWhere:GGGG = Ground station ID (map of ground stations)YYYY = yearMM = monthDD = dayTTTTTT = time when data started to be collected (GMT)Note that the 2015 files have an additional field on the file name. It adds “_V06” to the end of the file name. An example is KABX20150303_001050_V06.gz.<jupyter_code>#first lets connect to the bucket
conn = S3Connection(anon = True)
bucket = conn.get_bucket('noaa-nexrad-level2')
#as we can see there is a LOT we can do with a bucket!!!
dir(bucket)<jupyter_output><empty_output><jupyter_text>The contents of the bucket are in bucket.list<jupyter_code>my_list = bucket.list()
help(my_list)<jupyter_output>Help on BucketListResultSet in module boto.s3.bucketlistresultset object:

class BucketListResultSet(builtins.object)
 |  A resultset for listing keys within a bucket.  Uses the bucket_lister
 |  generator function and implements the iterator interface.  This
 |  transparently handles the results paging from S3 so even if you have
 |  many thousands of keys within the bucket you can iterate over all
 |  keys in a reasonably efficient manner.
 |  
 |  Methods defined here:
 |  
 |  __init__(self, bucket=None, prefix='', delimiter='', marker='', headers=None, encoding_type=None)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |  
 |  __iter__(self)
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)<jupyter_text>We can see this is an iterator.. Printing the whole list would be **YUUUUGE!** so we want to subset it.. we can do this via the prefix keyword. We are then going to cast it to a list<jupyter_code>my_pref = '2011/05/20/KVNX/'
bucket_list = list(bucket.list(prefix = my_pref))
print(bucket_list[0:10])<jupyter_output>[<Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_000023_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_000442_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_000901_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_001320_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_001740_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_002201_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_002620_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_003040_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_003459_V06.gz>, <Key: noaa-nexrad-level2,2011/05/20/KVNX/KVNX20110520_003918_V06.gz>]<jupyter_text>So we have a list of key (objects) in an S3 bucket. We can directly access the item and download it to a file using the contents_to_file method<jupyter_code>home_dir = os.path.expanduser('~')
bucket_list[0].get_contents_to_filename(os.path.join(home_dir,'nexrad_tempfile'))<jupyter_output><empty_output><jupyter_text>OK!! That was easy.. lets just take a quick look<jupyter_code>radar = pyart.io.read(os.path.join(home_dir,'nexrad_tempfile'))
print(radar.info())
my_figure = plt.figure(figsize = [10,8])
my_display = pyart.graph.RadarDisplay(radar)
my_display.plot_ppi('reflectivity', 0, vmin = -12, vmax = 64)<jupyter_output><empty_output><jupyter_text>Ok! How do I search for the volume **I** want? And make it open easily in Py-ART? Here is a little documented script<jupyter_code>#Helper function for the search
def _nearestDate(dates, pivot):
    return min(dates, key=lambda x: abs(x - pivot))


def get_radar_from_aws(site, datetime_t):
    """
    Get the closest volume of NEXRAD data to a particular datetime.
    Parameters
    ----------
    site : string
        four letter radar designation
    datetime_t : datetime
        desired date time
    Returns
    -------
    radar : Py-ART Radar Object
        Radar closest to the queried datetime
    """

    #First create the query string for the bucket knowing
    #how NOAA and AWS store the data

    my_pref = datetime_t.strftime('%Y/%m/%d/') + site

    #Connect to the bucket

    conn = S3Connection(anon = True)
    bucket = conn.get_bucket('noaa-nexrad-level2')

    #Get a list of files

    bucket_list = list(bucket.list(prefix = my_pref))

    #we are going to create a list of keys and datetimes to allow easy searching

    keys = []
    datetimes = []

    #populate the list

    for i in range(len(bucket_list)):
        this_str = str(bucket_list[i].key)
        if 'gz' in this_str:
            endme = this_str[-22:-4]
            fmt = '%Y%m%d_%H%M%S_V0'
            dt = datetime.strptime(endme, fmt)
            datetimes.append(dt)
            keys.append(bucket_list[i])

        if this_str[-3::] == 'V06':
            endme = this_str[-19::]
            fmt = '%Y%m%d_%H%M%S_V06'
            dt = datetime.strptime(endme, fmt)
            datetimes.append(dt)
            keys.append(bucket_list[i])

    #find the closest available radar to your datetime

    closest_datetime = _nearestDate(datetimes, datetime_t)
    index = datetimes.index(closest_datetime)

    localfile = tempfile.NamedTemporaryFile()
    keys[index].get_contents_to_filename(localfile.name)
    radar = pyart.io.read(localfile.name)
    return radar<jupyter_output><empty_output><jupyter_text>Lets take it for a spin!<jupyter_code>base_date = "20161006_192700"
fmt = '%Y%m%d_%H%M%S' 
b_d = datetime.strptime(base_date, fmt)

my_radar = get_radar_from_aws('KAMX',b_d )
max_lat = 27
min_lat = 24
min_lon = -81
max_lon = -77

lal = np.arange(min_lat, max_lat, .5)
lol = np.arange(min_lon, max_lon, .5)

display = pyart.graph.RadarMapDisplay(my_radar)
fig = plt.figure(figsize = [10,8])
display.plot_ppi_map('reflectivity', sweep = 0, resolution = 'c',
                    vmin = -8, vmax = 64, mask_outside = False,
                    cmap = pyart.graph.cm.NWSRef,
                    min_lat = min_lat, min_lon = min_lon,
                    max_lat = max_lat, max_lon = max_lon,
                    lat_lines = lal, lon_lines = lol)<jupyter_output>/Users/scollis/anaconda/envs/ams-workshop/lib/python3.5/site-packages/mpl_toolkits/basemap/__init__.py:3435: MatplotlibDeprecationWarning: The ishold function was deprecated in version 2.0.
  b = ax.ishold()
/Users/scollis/anaconda/envs/ams-workshop/lib/python3.5/site-packages/mpl_toolkits/basemap/__init__.py:3444: MatplotlibDeprecationWarning: axes.hold is deprecated.
    See the API Changes document (http://matplotlib.org/api/api_changes.html)
    for more details.
  ax.hold(b)
{'fraction_non_alphanumeric': 0.10857280269327883, 'fraction_numerical': 0.04821450042082482, 'mean_word_length': 3.7997691863819965, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '67998', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3091, 'n_tokens_neox': 2816, 'n_words': 788, 'path': 'notebooks/introduction to S3, Boto and Nexrad on S3 with a hurricane chaser.ipynb', 'repo_name': 'GalileanElectric/AMS_radar_in_the_cloud', 'size': '8331'}