<jupyter_start><jupyter_text>Analysis Ready Data Tutorial Part 2: Use Case 1Time-series analysis (e.g. change detection and trend detection) is a powerful application of satellite imagery. However, a great deal of processing is required to prepare imagery for analysis. Analysis Ready Data (ARD), preprocessed time-series stacks of overhead imagery, allow for time-series analysis without any additional processing of the imagery. See [Analysis Data Defined](https://medium.com/planet-stories/analysis-ready-data-defined-5694f6f48815) for an excellent introduction and discussion on ARD.In [Part 1](ard_1_intro_and_best_practices.ipynb) of this tutorial, we introduced ARD and covered the how and whys of using the Data and Orders APIs to create and interpret ARD.This second part of the tutorial focuses on the first of two use cases. The use case addressed in this tutorial is:* As a software engineer at an ag-tech company, I'd like to be able to order Planet imagery programmatically in a way that enables the data scientist at my organization to create time-series algorithms (e.g. monitoring ndvi curves over time) without further data cleaning and processing.Please see the first part of the tutorial for an introduction to the Data and Orders APIs along with best practices. A lot of functionality developed in that tutorial will be copied here in a compact form. IntroductionTwo things are interesting about this use case. First, we are calculating NDVI, and second, we are compositing scenes together. What is NDVI and what is compositing and why do we want to do it?Great questions!First, NDVI stands for normalized difference vegitation index. It is used a **LOT** to find out if vegetation is growing. You can find out more about NDVI at [USGS](https://www.usgs.gov/land-resources/eros/phenology/science/ndvi-foundation-remote-sensing-phenology?qt-science_center_objects=0qt-science_center_objects) and [Wikipedia](https://en.wikipedia.org/wiki/Normalized_difference_vegetation_index). What we care about here is that NDVI uses the red and near-infrared bands of an image and returns one band with values that range from -1 to 1. So, we expect a single-band image for each order.Compositing is a way to stitch multiple scenes together for maximum coverage. We want this because for a time series, we just want one image for each date and we want that one image to have the most coverage to minimize holes in our data. The composite tool takes in multiple scenes and returns one image. If we feed it scenes from a whole timestack, we still just get one image back! So, to avoid that disaster, we group our scenes by date and only composite the scenes that were collected on the same date. ImplementationThe use case we will cover is: *As a software engineer at an ag-tech company, I'd like to be able to order Planet imagery programmatically in a way that enables the data scientist at my organization to create time-series algorithms (e.g. monitoring ndvi curves over time) without further data cleaning and processing.*For this use case, the area of interest and time range are not specified. The need for no further processing indicates we should specify a strict usable pixel data filter. For time-series analysis the daily coverage of PS satellites is ideal. For our time-series analysis, we would like a single image that covers the entire area of interest (AOI). However, it may take multiple scenes to cover the entire AOI. Therefore, we will use the Composite tool to make a composite for each day in the time series analysis. This is a little tricky because the Composite tool just composites all of the scenes associated with the ids ordered. So we need to parse the scene ids we got from the Data API to get scene ids for each day, then submit an order for each day.To summarize, these are the steps:1. [Initialize API client](Step-1:-Initialize-API-client)1. [Search Data API](Step-2:-Search-Data-API)1. [Group IDs by Date](Step-3:-Group-IDs-by-Date)1. [Submit Orders](Step-4:-Submit-Orders)1. [Download Orders](Step-5:-Download-Orders)1. [Unzip and Verify Orders](Step-6:-Unzip-and-Verify-Orders)Note that, due to the processing-intensiveness of visualizing the NDVI images and UDM2s, we will be covering visualization in the next notebook, [Analysis Ready Data Tutorial Part 2: Use Case 1 - Visualization](ard_2_use_case_1_visualize_images.ipynb) Import Dependencies<jupyter_code>from copy import copy
import datetime
import os
from pathlib import Path
from pprint import pprint
import shutil
import time
from zipfile import ZipFile

import numpy as np
from planet import api
from planet.api import downloader, filters<jupyter_output><empty_output><jupyter_text>Step 1: Initialize API client<jupyter_code># if your Planet API Key is not set as an environment variable, you can paste it below
API_KEY = os.environ.get('PL_API_KEY', 'PASTE_YOUR_KEY_HERE')

client = api.ClientV1(api_key=API_KEY)<jupyter_output><empty_output><jupyter_text>Step 2: Search Data APIThe goal of this step is to get the scene ids that meet the search criteria for this use case.<jupyter_code># these functions were developed in the best practices tutorial (part 1)
# create an api request from the search specifications
def build_request(aoi_geom, start_date, stop_date):
    '''build a data api search request for clear PSScene4Band imagery'''
    query = filters.and_filter(
        filters.geom_filter(aoi_geom),
        filters.range_filter('clear_percent', gte=90),
        filters.date_range('acquired', gt=start_date),
        filters.date_range('acquired', lt=stop_date)
    )
    return filters.build_search_request(query, ['PSScene4Band'])

def search_data_api(request, client, limit=500):
    result = client.quick_search(request)
    
    # this returns a generator
    return result.items_iter(limit=limit)
# define test data for the filter
test_start_date = datetime.datetime(year=2019,month=4,day=1)
test_stop_date = datetime.datetime(year=2019,month=5,day=1)

# iowa crops aoi
test_aoi_geom = {
    "type": "Polygon",
    "coordinates": [
        [
            [-93.299129, 42.699599],
            [-93.299674, 42.812757],
            [-93.288436, 42.861921],
            [-93.265332, 42.924817],
            [-92.993873, 42.925124],
            [-92.993888, 42.773637],
            [-92.998396, 42.754529],
            [-93.019154, 42.699988],
            [-93.299129, 42.699599]
        ]
    ]
}
request = build_request(test_aoi_geom, test_start_date, test_stop_date)
print(request)
items = list(search_data_api(request, client))
print(len(items))<jupyter_output>80<jupyter_text>Step 3: Group IDs by Date<jupyter_code># check out an item just for fun
pprint(items[0])
# item = items[0]
# acquired_date = item['properties']['acquired'].split('T')[0]
# acquired_date
def get_acquired_date(item):
    return item['properties']['acquired'].split('T')[0]

acquired_dates = [get_acquired_date(item) for item in items]
unique_acquired_dates = set(acquired_dates)
unique_acquired_dates
def get_date_item_ids(date, all_items):
    return [i['id'] for i in all_items if get_acquired_date(i) == date]

def get_ids_by_date(items):
    acquired_dates = [get_acquired_date(item) for item in items]
    unique_acquired_dates = set(acquired_dates)
    
    ids_by_date = dict((d, get_date_item_ids(d, items))
                       for d in unique_acquired_dates)
    return ids_by_date
    
ids_by_date = get_ids_by_date(items)
pprint(ids_by_date)<jupyter_output>{'2019-04-02': ['20190402_163631_0e16',
                '20190402_163634_0e16',
                '20190402_163633_0e16'],
 '2019-04-08': ['20190408_154008_1020',
                '20190408_154007_1020',
                '20190408_154006_1020',
                '20190408_154005_1020',
                '20190408_154004_1020',
                '20190408_164038_100e',
                '20190408_164037_100e',
                '20190408_164036_100e',
                '20190408_164035_100e',
                '20190408_164034_100e',
                '20190408_163736_1025',
                '20190408_163735_1025',
                '20190408_163738_1025'],
 '2019-04-15': ['20190415_170304_85_1068', '20190415_170302_79_1068'],
 '2019-04-19': ['20190419_164004_1035',
                '20190419_164002_1035',
                '20190419_164001_1035',
                '20190419_164000_1035',
                '20190419_164003_1035'],
 '2019-04-20': ['20190420_163153_0e19',
                '20190420_163150_0e19',
      [...]<jupyter_text>Step 4: Submit OrdersNow that we have the scene ids for each collect date, we can create the orders for each date. The output of each order is a single zip file that contains one composited scene and one composited UDM2.For this step we will just use the python api. See part 1 for a demonstration of how to use the CLI. Step 4.1: Build Order Requests<jupyter_code>def build_order(ids, name, aoi_geom):
    # specify the psscene4band surface reflectance product
    # make sure to get the *_udm2 bundle so you get the udm2 product
    # note: capitalization really matters in item_type when using planet client orders api
    item_type = 'PSScene4Band'
    bundle = 'analytic_sr_udm2'

    orders_request = {
        'name': name,
        'products': [{
            'item_ids': ids,
            'item_type': item_type,
            'product_bundle': bundle
        }],
        'tools': get_tools(aoi_geom),
        'delivery': {
            'single_archive': True,
            'archive_filename':'{{name}}_{{order_id}}.zip',
            'archive_type':'zip'
        },
            'notifications': {
                       'email': False
        },
    }
    return orders_request

def get_tools(aoi_geom):
    # clip to AOI
    clip_tool = {'clip': {'aoi': aoi_geom}}

    # convert to NDVI
    bandmath_tool = {'bandmath': {
        "pixel_type": "32R",
        "b1": "(b4 - b3) / (b4+b3)"
    }}

    # composite into one image
    composite_tool = {  
       "composite":{}
    }

    tools = [clip_tool, bandmath_tool, composite_tool]
    return tools

# uncomment to see what an order request would look like
# pprint(build_order(['id'], 'demo', test_aoi_geom), indent=4)
def get_orders_requests(ids_by_date, aoi_geom):
    order_requests = [build_order(ids, date, aoi_geom)
                      for date, ids in ids_by_date.items()]
    return order_requests
    
order_requests = get_orders_requests(ids_by_date, test_aoi_geom)
print(len(order_requests))
pprint(order_requests[0])<jupyter_output>9
{'delivery': {'archive_filename': '{{name}}_{{order_id}}.zip',
              'archive_type': 'zip',
              'single_archive': True},
 'name': '2019-04-02',
 'notifications': {'email': False},
 'products': [{'item_ids': ['20190402_163631_0e16',
                            '20190402_163634_0e16',
                            '20190402_163633_0e16'],
               'item_type': 'PSScene4Band',
               'product_bundle': 'analytic_sr_udm2'}],
 'tools': [{'clip': {'aoi': {'coordinates': [[[-93.299129, 42.699599],
                                              [-93.299674, 42.812757],
                                              [-93.288436, 42.861921],
                                              [-93.265332, 42.924817],
                                              [-92.993873, 42.925124],
                                              [-92.993888, 42.773637],
                                              [-92.998396, 42.754529],
                                              [[...]<jupyter_text>Step 4.2: Submit OrdersIn this section, for the sake of demonstration, we limit our orders to 2. Feel free to increase this limit if you want!<jupyter_code>def create_orders(order_requests, client):
    orders_info = [client.create_order(r).get()
                   for r in order_requests]
    order_ids = [i['id'] for i in orders_info]
    return order_ids

# testing: lets just create two orders
order_limit = 2
order_ids = create_orders(order_requests[:order_limit], client)
order_ids<jupyter_output><empty_output><jupyter_text>Step 5: Download Orders Step 5.1: Wait Until Orders are SuccessfulBefore we can download the orders, they have to be prepared on the server.<jupyter_code>def poll_for_success(order_ids, client, num_loops=50):
    count = 0
    polling = copy(order_ids)
    completed = []
    while(count < num_loops):
        count += 1
        states = []
        for oid in copy(polling):
            order_info = client.get_individual_order(oid).get()
            state = order_info['state']
            states += state
            print('{}:{}'.format(oid, state))
            success_states = ['success', 'partial']
            if state == 'failed':
                raise Exception(response)
            elif state in success_states:
                polling.remove(oid)
                completed += oid
        if not len(polling):
            print('done')
            break
        print('--')
        time.sleep(30)
        
poll_for_success(order_ids, client)<jupyter_output>e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:queued
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc114bc502d2:running
--
e04576a7-52c0-4a60-a6b0-af43c00fb1ca:running
05d15aa3-6a3d-46cb-a0f1-bc[...]<jupyter_text>Step 5.2: Run DownloadFor this step we will use the planet python orders API because we want to be able to download multiple orders at once, something the CLI does not yet support.<jupyter_code>data_dir = os.path.join('data', 'use_case_1')

# make the download directory if it doesn't exist
Path(data_dir).mkdir(parents=True, exist_ok=True)
def poll_for_download(dest, endswith, num_loops=50):
    count = 0
    while(count < num_loops):
        count += 1        
        matched_files = (f for f in os.listdir(dest)
                         if os.path.isfile(os.path.join(dest, f))
                         and f.endswith(endswith))

        match = next(matched_files, None)
        if match:
            match = os.path.join(dest, match)
            print('downloaded')
            break
        else:
            print('waiting...')
        time.sleep(10)
    return match
def download_orders(order_ids, client, dest='.', limit=None):
    files = []
    for order_id in order_ids:
        print('downloading {}'.format(order_id))
        filename = download_order(order_id, dest, client, limit=limit)
        if filename:
            files.append(filename)
    return files

def download_order(order_id, dest, client, limit=None):
    '''Download an order by given order ID'''
    # this returns download stats but they aren't accurate or informative
    # so we will look for the downloaded file on our own.
    dl = downloader.create(client, order=True)
    urls = client.get_individual_order(order_id).items_iter(limit=limit)
    dl.download(urls, [], dest)
    
    endswith = '{}.zip'.format(order_id)
    filename = poll_for_download(dest, endswith)
    return filename

downloaded_files = download_orders(order_ids, client, data_dir)
downloaded_files<jupyter_output><empty_output><jupyter_text>Step 6: Unzip and Verify OrdersIn this step we will simply unzip the orders and view one of the ordered composite images. 6.1: Unzip OrderIn this section, we will unzip each order into a directory named after the downloaded zip file.<jupyter_code>def unzip(filename, overwrite=False):
    location = Path(filename)
    zipdir = location.parent / location.stem

    if os.path.isdir(zipdir):
        if overwrite:
            print('{} exists. overwriting.'.format(zipdir))
            shutil.rmtree(zipdir)
        else:
            raise Exception('{} already exists'.format(zipdir))
        
    with ZipFile(location) as myzip:
        myzip.extractall(zipdir)
    return zipdir

zipdirs = [unzip(f, overwrite=True) for f in downloaded_files]
pprint(zipdirs)<jupyter_output>[PosixPath('data/use_case_1/2019-04-02_e04576a7-52c0-4a60-a6b0-af43c00fb1ca'),
 PosixPath('data/use_case_1/2019-04-08_05d15aa3-6a3d-46cb-a0f1-bc114bc502d2')]
{'fraction_non_alphanumeric': 0.09385248345358059, 'fraction_numerical': 0.08139049549818138, 'mean_word_length': 3.23321554770318, 'pattern_counts': {'":': 5, '<': 34, '<?xml version=': 0, '>': 32, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 39, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '483734', 'license': 'Apache-2.0', 'n_tokens_mistral': 6135, 'n_tokens_neox': 5296, 'n_words': 1528, 'path': 'jupyter-notebooks/analysis-ready-data/ard_2_use_case_1.ipynb', 'repo_name': 'mapninja/notebooks', 'size': '16771'}