<jupyter_start><jupyter_text>Install all the things.<jupyter_code>!pip install shapely
!conda install --yes --quiet fiona
!pip install fiona
!pip install gbdxtools<jupyter_output>Requirement already satisfied: shapely in /Users/chambbj/miniconda2/lib/python2.7/site-packages
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /Users/chambbj/miniconda2/envs/dg-shoreline-task-notebook:
#
fiona                     1.7.0                    py27_0  
Requirement already satisfied: fiona in /Users/chambbj/miniconda2/lib/python2.7/site-packages
Requirement already satisfied: cligj in /Users/chambbj/miniconda2/lib/python2.7/site-packages (from fiona)
Requirement already satisfied: click-plugins in /Users/chambbj/miniconda2/lib/python2.7/site-packages (from fiona)
Requirement already satisfied: enum34 in /Users/chambbj/miniconda2/lib/python2.7/site-packages (from fiona)
Requirement already satisfied: munch in /Users/chambbj/miniconda2/lib/python2.7/site-packages (from fiona)
Requirement already satisfied: six in /Users/chambbj/miniconda2/lib/python2.7/site-packages (from fiona)
R[...]<jupyter_text>And then all the imports and such.<jupyter_code>from shapely.geometry import shape
from gbdxtools import Interface
from shapely.geometry import MultiPolygon
from shapely.wkt import loads
from shapely.ops import cascaded_union
import dateutil.parser
from datetime import datetime
import uuid
from os.path import join
import time
import sys
from __future__ import print_function
import fiona
import json<jupyter_output><empty_output><jupyter_text>Load the shapefile, get one of the polygons, and convert to shape.<jupyter_code>c = fiona.open('/Users/chambbj/Downloads/(U) Harbors_Beachfront_Apr2017/Harbors_Priority_Beachfront.shp')
pol = next(iter(c))
geom = shape(pol['geometry'])
geom<jupyter_output><empty_output><jupyter_text>Create our GBDX interface.<jupyter_code>gbdx = Interface()<jupyter_output><empty_output><jupyter_text>Create a multipolygon of all the polys, but I don't appear to actually use this anywhere.<jupyter_code>Multi = MultiPolygon([shape(pol['geometry']) for pol in c])<jupyter_output><empty_output><jupyter_text>Show that we can get the WKT of the geometry.<jupyter_code>geom.wkt<jupyter_output><empty_output><jupyter_text>Create a filter (WV2 with cloud cover less than 10%), and search the catalog.<jupyter_code>filters = ["cloudCover<10"]
types = ["LandsatAcquisition"]
results = gbdx.catalog.search(searchAreaWkt=geom.wkt,filters=filters,types=types)
len(results)<jupyter_output><empty_output><jupyter_text>Show that we can get the footprint WKT of the first result.<jupyter_code>footprint = loads(results[1]['properties']['footprintWkt'])
footprint<jupyter_output><empty_output><jupyter_text>Get a sense for how the footprint overlaps the query geometry. A percent overlap would be nice too.<jupyter_code>geom.boundary.union(footprint.boundary)<jupyter_output><empty_output><jupyter_text>And just look at the acquisition metadata for the result.<jupyter_code>results[1]<jupyter_output><empty_output><jupyter_text>Okay, now we actually get into creating and executing workflows. Here we will start by getting the lat/lon of the footprint centroid, and also the timestamp to be used in predicting tides. We continue by creating a workflow that automatically orders the data, applies atmospheric compensation via AOP, and the performs shoreline detection. We make sure to save the outputs of the final task.<jupyter_code>cat_id = results[1]['identifier']
record = gbdx.catalog.get(cat_id)
centroid = loads(record.get('properties').get('footprintWkt')).centroid
lat = centroid.y
lon = centroid.x
print('lat=%f, lon=%f' % (lat, lon))
timestamp = dateutil.parser.parse(record.get('properties').get('timestamp'))
dtg = datetime.strftime(timestamp, '%Y-%m-%d-%H-%M')
print('dtg="%s"' % dtg)
print('lat="%s", lon="%s", dtg="%s"' % (str(lat), str(lon), dtg))
ordertask = gbdx.Task('Auto_Ordering', cat_id=cat_id)
ordertask.impersonation_allowed = True

aoptask = gbdx.Task('AOP_Strip_Processor',
                 data=ordertask.outputs.s3_location.value,
                 bands='MS',
                 enable_dra=False,
                 enable_pansharpen=False,
                 enable_acomp=True,
                 ortho_epsg='UTM')

tide_task = gbdx.Task('ShorelineDetection_beta',
                      lat = str(lat),
                      lon = str(lon),
                      dtg = dtg,
                      meta = json.dumps(results[1]),
                      image = aoptask.outputs.data.value)

workflow = gbdx.Workflow([ordertask, aoptask, tide_task])
random_str = str(uuid.uuid4())
workflow.savedata(tide_task.outputs.results, location=join('some_random_folder', random_str))<jupyter_output>lat=14.737430, lon=-17.268643
dtg="2009-12-11-11-41"
lat="14.7374302272", lon="-17.2686430881", dtg="2009-12-11-11-41"<jupyter_text>Execute the workflow.<jupyter_code>workflow.execute()<jupyter_output><empty_output><jupyter_text>And view the definition for kicks.<jupyter_code>workflow.definition<jupyter_output><empty_output><jupyter_text>We can loop until the workflow shows up as complete.<jupyter_code>while not workflow.status.get('state') == u'complete':
    print(json.dumps(workflow.events), end='\r')
#     print(datetime.now(), workflow.status, end='\r')
    sys.stdout.flush()
    time.sleep(5.0)
workflow.status<jupyter_output><empty_output><jupyter_text>Download the shoreline result.<jupyter_code>gbdx.s3.download(join('some_random_folder', random_str))<jupyter_output><empty_output><jupyter_text>This notebook needs lots of cleaning up. The following chunk will be used to run only ACOMP, such that we can test this workflow locally too.<jupyter_code>cat_id = results[1]['identifier']
record = gbdx.catalog.get(cat_id)
centroid = loads(record.get('properties').get('footprintWkt')).centroid
lat = centroid.y
lon = centroid.x
print('lat=%f, lon=%f' % (lat, lon))
timestamp = dateutil.parser.parse(record.get('properties').get('timestamp'))
dtg = datetime.strftime(timestamp, '%Y-%m-%d-%H-%M')
print('dtg="%s"' % dtg)
print('lat="%s", lon="%s", dtg="%s"' % (str(lat), str(lon), dtg))
ordertask = gbdx.Task('Auto_Ordering', cat_id=cat_id)
ordertask.impersonation_allowed = True

aoptask = gbdx.Task('AOP_Strip_Processor',
                 data=ordertask.outputs.s3_location.value,
                 bands='MS',
                 enable_dra=False,
                 enable_pansharpen=False,
                 enable_acomp=True,
                 ortho_epsg='UTM')

workflow = gbdx.Workflow([ordertask, aoptask])
random_str = str(uuid.uuid4())
print(random_str)
workflow.savedata(aoptask.outputs.data, location=join('acomp-only-testing', random_str))<jupyter_output>lat=14.737430, lon=-17.268643
dtg="2009-12-11-11-41"
lat="14.7374302272", lon="-17.2686430881", dtg="2009-12-11-11-41"
6fe837c1-dd7d-442c-abda-aa9edbe93c8c<jupyter_text>Execute the ACOMP.<jupyter_code>workflow.execute()
while not workflow.status.get('state') == u'complete':
    print(json.dumps(workflow.events), end='\r')
#     print(datetime.now(), workflow.status, end='\r')
    sys.stdout.flush()
    time.sleep(5.0)
workflow.status<jupyter_output>[{"task": "Auto_Ordering_1975dc0b", "task_id": "4648991738082326025", "timestamp": "2017-07-24T14:16:42.264189+00:00", "when": "19 minutes ago", "note": "", "state": "pending", "event": "submitted"}, {"task": "AOP_Strip_Processor_b3ef9867", "task_id": "4648991738090786724", "timestamp": "2017-07-24T14:16:42.268175+00:00", "when": "19 minutes ago", "note": "", "state": "pending", "event": "submitted"}, {"task": "Auto_Ordering_1975dc0b", "task_id": "4648991738082326025", "timestamp": "2017-07-24T14:16:57.940120+00:00", "when": "19 minutes ago", "note": "instance_id: i-c92fa74a", "state": "pending", "event": "scheduled"}, {"task": "Auto_Ordering_1975dc0b", "task_id": "4648991738082326025", "timestamp": "2017-07-24T14:16:58.511701+00:00", "when": "19 minutes ago", "note": "instance_id: i-08463f0c4314727db, domain: default", "state": "running", "event": "started"}, {"task": "Auto_Ordering_1975dc0b", "task_id": "4648991738082326025", "timestamp": "2017-07-24T14:17:34.620864+00:00", "when": "[...]<jupyter_text>Grab the URL of the ACOMP raster output.<jupyter_code>aopraster = 's3://gbd-customer-data/a6308131-a55f-4f07-a436-68502f273a9f/acomp-only-testing/6fe837c1-dd7d-442c-abda-aa9edbe93c8c/'<jupyter_output><empty_output><jupyter_text>Now execute a workflow that cuts out auto order and AOP steps, using precomputed result.<jupyter_code>cat_id = results[1]['identifier']
record = gbdx.catalog.get(cat_id)
centroid = loads(record.get('properties').get('footprintWkt')).centroid
lat = centroid.y
lon = centroid.x
print('lat=%f, lon=%f' % (lat, lon))
timestamp = dateutil.parser.parse(record.get('properties').get('timestamp'))
dtg = datetime.strftime(timestamp, '%Y-%m-%d-%H-%M')
print('dtg="%s"' % dtg)
print('lat="%s", lon="%s", dtg="%s"' % (str(lat), str(lon), dtg))

tide_task = gbdx.Task('ShorelineDetection_beta',
                      lat = str(lat),
                      lon = str(lon),
                      dtg = dtg,
                      meta = json.dumps(results[1]),
                      minsize = '1000.0',
                      smooth = '0.8',
                      image = aopraster)

workflow = gbdx.Workflow([tide_task])
random_str = str(uuid.uuid4())
workflow.savedata(tide_task.outputs.vector, location=join('shoreline-tests', random_str))<jupyter_output>lat=14.737430, lon=-17.268643
dtg="2009-12-11-11-41"
lat="14.7374302272", lon="-17.2686430881", dtg="2009-12-11-11-41"<jupyter_text>Execute.<jupyter_code>workflow.execute()
while not workflow.status.get('state') == u'complete':
    print(json.dumps(workflow.events), end='\r')
#     print(datetime.now(), workflow.status, end='\r')
    sys.stdout.flush()
    time.sleep(5.0)
workflow.status<jupyter_output>[{"task": "ShorelineDetection_beta_8184557c", "task_id": "4649142211615709556", "timestamp": "2017-07-24T19:15:40.082004+00:00", "when": "3 minutes ago", "note": "", "state": "pending", "event": "submitted"}, {"task": "ShorelineDetection_beta_8184557c", "task_id": "4649142211615709556", "timestamp": "2017-07-24T19:15:56.838067+00:00", "when": "3 minutes ago", "note": "instance_id: i-c92fa74a", "state": "pending", "event": "scheduled"}, {"task": "ShorelineDetection_beta_8184557c", "task_id": "4649142211615709556", "timestamp": "2017-07-24T19:15:57.366285+00:00", "when": "3 minutes ago", "note": "instance_id: i-0856243755a9343ba, domain: default", "state": "running", "event": "started"}]<jupyter_text>for AF test<jupyter_code>ordertask = gbdx.Task('Auto_Ordering', cat_id=cat_id)
ordertask.impersonation_allowed = True

aoptask = gbdx.Task('AOP_Strip_Processor',
                 data=ordertask.outputs.s3_location.value,
                 bands='MS',
                 enable_dra=False,
                 enable_pansharpen=False,
                 enable_acomp=True,
                 ortho_epsg='UTM')

tide_task = gbdx.Task('ShorelineDetection_beta',
                      lat = str(lat),
                      lon = str(lon),
                      dtg = dtg,
                      meta = json.dumps(results[1]),
                      image = aoptask.outputs.data.value)

workflow = gbdx.Workflow([ordertask, aoptask, tide_task])
random_str = str(uuid.uuid4())
workflow.savedata(tide_task.outputs.vector, location=join('some_random_folder', random_str))
workflow.execute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14856202356202355, 'fraction_numerical': 0.0647089397089397, 'mean_word_length': 4.792774711490216, 'pattern_counts': {'":': 53, '<': 80, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '591936', 'license': 'Apache-2.0', 'n_tokens_mistral': 4502, 'n_tokens_neox': 4030, 'n_words': 745, 'path': 'notebooks/Catalog-Search-Then-Shoreline-LS8.ipynb', 'repo_name': 'venicegeo/shoreline-task-notebooks', 'size': '11544'}