<jupyter_start><jupyter_text>Combining Landsat data with Crop DataIn this exercise, we will be computing the average NDVI for our countyover all of 2016 for a specific crop. In this way, we hope to detectthe time of the growing season for that crop.There is 1 objective in this exercise:- __Objective 1__: Graph the average NDVI over time for the county<jupyter_code>import geopyspark as gps
from pyspark import SparkContext
import numpy as np
import os
from datetime import datetime
from shapely.geometry import mapping, shape, Point
import pyproj
from shapely.ops import transform
from functools import partial
import urllib.request, json
from geonotebook.wrappers import TMSRasterData
from PIL import Image
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Setup: State data and Spark initializationThe next 2 cells grab the shapes for our state and start up the spark context.<jupyter_code># Grab data for New Mexico
state_name, county_name = "NM", "Colfax"
def get_state_shapes(state, county):
    project = partial(
        pyproj.transform,
        pyproj.Proj(init='epsg:4326'),
        pyproj.Proj(init='epsg:3857'))

    state_url = "https://raw.githubusercontent.com/johan/world.geo.json/master/countries/USA/{}.geo.json".format(state)
    county_url = "https://raw.githubusercontent.com/johan/world.geo.json/master/countries/USA/{}/{}.geo.json".format(state,county)
    read_json = lambda url: json.loads(urllib.request.urlopen(url).read().decode("utf-8"))
    state_ll = shape(read_json(state_url)['features'][0]['geometry'])
    state_wm = transform(project, state_ll)
    county_ll = shape(read_json(county_url)['features'][0]['geometry'])
    county_wm = transform(project, county_ll)
    return (state_ll, state_wm, county_ll, county_wm)

(state_ll, state_wm, county_ll, county_wm) = get_state_shapes(state_name, county_name) 
# Set up our spark context
conf = gps.geopyspark_conf(appName="Exercise 1") \
          .setMaster("local[*]") \
          .set(key='spark.ui.enabled', value='true') \
          .set(key="spark.driver.memory", value="8G") \
          .set("spark.hadoop.yarn.timeline-service.enabled", False)
sc = SparkContext(conf=conf)<jupyter_output><empty_output><jupyter_text>Setup: Band names and color rampThe ingested layers have the RGB, near infrared, and QA bands of landsat 8 data.This dict maps the band names to band index, for more readable code.We also define a the crops to values, so we can choose which crops we want to inspect.<jupyter_code>bands = { "Blue": 0,
          "Green": 1,
          "Red": 2,
          "NIR": 3,
          "QA": 4 }

catalog_uri = "s3://datahub-catalogs-us-east-1"
landsat_layer_name = "landsat-8-continental-us-2016"
cdl_layer_name = "cdl-2016-zoomed"
cdl_colormap_dict = {1: 0xffd300ff,2: 0xff2626ff,3: 0x00a8e5ff,4: 0xff9e0cff,5: 0x267000ff,6: 0xffff00ff,10: 0x70a500ff,11: 0x00af4cff,12: 0xdda50cff,13: 0xdda50cff,14: 0x7fd3ffff,21: 0xe2007cff,22: 0x896354ff,23: 0xd8b56bff,24: 0xa57000ff,25: 0xd69ebcff,26: 0x707000ff,27: 0xad007cff,28: 0xa05989ff,29: 0x700049ff,30: 0xd69ebcff,31: 0xd1ff00ff,32: 0x7f99ffff,33: 0xd6d600ff,34: 0xd1ff00ff,35: 0x00af4cff,36: 0xffa5e2ff,37: 0xa5f28cff,38: 0x00af4cff,39: 0xd69ebcff,41: 0xa800e5ff,42: 0xa50000ff,43: 0x702600ff,44: 0x00af4cff,45: 0xb27fffff,46: 0x702600ff,47: 0xff6666ff,48: 0xff6666ff,49: 0xffcc66ff,50: 0xff6666ff,51: 0x00af4cff,52: 0x00ddafff,53: 0x54ff00ff,54: 0xf2a377ff,55: 0xff6666ff,56: 0x00af4cff,57: 0x7fd3ffff,58: 0xe8bfffff,59: 0xafffddff,60: 0x00af4cff,61: 0xbfbf77ff,63: 0x93cc93ff,64: 0xc6d69eff,65: 0xccbfa3ff,66: 0xff00ffff,67: 0xff8eaaff,68: 0xba004fff,69: 0x704489ff,70: 0x007777ff,71: 0xb29b70ff,72: 0xffff7fff,74: 0xb5705bff,75: 0x00a582ff,76: 0xead6afff,77: 0xb29b70ff,81: 0xf2f2f2ff,82: 0x9b9b9bff,83: 0x4c70a3ff,87: 0x7fb2b2ff,88: 0xe8ffbfff,92: 0x00ffffff,111: 0x4c70a3ff,112: 0xd3e2f9ff,121: 0x9b9b9bff,122: 0x9b9b9bff,123: 0x9b9b9bff,124: 0x9b9b9bff,131: 0xccbfa3ff,141: 0x93cc93ff,142: 0x93cc93ff,143: 0x93cc93ff,152: 0xc6d69eff,176: 0xe8ffbfff,190: 0x7fb2b2ff,195: 0x7fb2b2ff,204: 0x00ff8cff,205: 0xd69ebcff,206: 0xff6666ff,207: 0xff6666ff,208: 0xff6666ff,209: 0xff6666ff,210: 0xff8eaaff,211: 0x334933ff,212: 0xe57026ff,213: 0xff6666ff,214: 0xff6666ff,216: 0xff6666ff,217: 0xb29b70ff,218: 0xff8eaaff,219: 0xff6666ff,220: 0xff8eaaff,221: 0xff6666ff,222: 0xff6666ff,223: 0xff8eaaff,224: 0x00af4cff,225: 0xffd300ff,226: 0xffd300ff,227: 0xff6666ff,229: 0xff6666ff,230: 0x896354ff,231: 0xff6666ff,232: 0xff2626ff,233: 0xe2007cff,234: 0xff9e0cff,235: 0xff9e0cff,236: 0xa57000ff,237: 0xffd300ff,238: 0xa57000ff,239: 0x267000ff,240: 0x267000ff,241: 0xffd300ff,242: 0x000099ff,243: 0xff6666ff,244: 0xff6666ff,245: 0xff6666ff,246: 0xff6666ff,247: 0xff6666ff,248: 0xff6666ff,249: 0xff6666ff,250: 0xff6666ff,251: 0xffd300ff,252: 0x267000ff,253: 0xa57000ff,254: 0x267000ff}
cdl_colormap = gps.ColorMap.from_break_map(cdl_colormap_dict)
values_to_crops = {0: 'Background',1: 'Corn',2: 'Cotton',3: 'Rice',4: 'Sorghum',5: 'Soybeans',6: 'Sunflower',10: 'Peanuts',11: 'Tobacco',12: 'Sweet Corn',13: 'Pop or Orn Corn',14: 'Mint',21: 'Barley',22: 'Durum Wheat',23: 'Spring Wheat',24: 'Winter Wheat',25: 'Other Small Grains',26: 'Dbl Crop WinWht/Soybeans',27: 'Rye',28: 'Oats',29: 'Millet',30: 'Speltz',31: 'Canola',32: 'Flaxseed',33: 'Safflower',34: 'Rape Seed',35: 'Mustard',36: 'Alfalfa',37: 'Other Hay/Non Alfalfa',38: 'Camelina',39: 'Buckwheat',41: 'Sugarbeets',42: 'Dry Beans',43: 'Potatoes',44: 'Other Crops',45: 'Sugarcane',46: 'Sweet Potatoes',47: 'Misc Vegs & Fruits',48: 'Watermelons',49: 'Onions',50: 'Cucumbers',51: 'Chick Peas',52: 'Lentils',53: 'Peas',54: 'Tomatoes',55: 'Caneberries',56: 'Hops',57: 'Herbs',58: 'Clover/Wildflowers',59: 'Sod/Grass Seed',60: 'Switchgrass',61: 'Fallow/Idle Cropland',63: 'Forest',64: 'Shrubland',65: 'Barren',66: 'Cherries',67: 'Peaches',68: 'Apples',69: 'Grapes',70: 'Christmas Trees',71: 'Other Tree Crops',72: 'Citrus',74: 'Pecans',75: 'Almonds',76: 'Walnuts',77: 'Pears',81: 'Clouds/No Data',82: 'Developed',83: 'Water',87: 'Wetlands',88: 'Nonag/Undefined',92: 'Aquaculture',111: 'Open Water',112: 'Perennial Ice/Snow ',121: 'Developed/Open Space',122: 'Developed/Low Intensity',123: 'Developed/Med Intensity',124: 'Developed/High Intensity',131: 'Barren',141: 'Deciduous Forest',142: 'Evergreen Forest',143: 'Mixed Forest',152: 'Shrubland',176: 'Grassland/Pasture',190: 'Woody Wetlands',195: 'Herbaceous Wetlands',204: 'Pistachios',205: 'Triticale',206: 'Carrots',207: 'Asparagus',208: 'Garlic',209: 'Cantaloupes',210: 'Prunes',211: 'Olives',212: 'Oranges',213: 'Honeydew Melons',214: 'Broccoli',216: 'Peppers',217: 'Pomegranates',218: 'Nectarines',219: 'Greens',220: 'Plums',221: 'Strawberries',222: 'Squash',223: 'Apricots',224: 'Vetch',225: 'Dbl Crop WinWht/Corn',226: 'Dbl Crop Oats/Corn',227: 'Lettuce',229: 'Pumpkins',230: 'Dbl Crop Lettuce/Durum Wht',231: 'Dbl Crop Lettuce/Cantaloupe',232: 'Dbl Crop Lettuce/Cotton',233: 'Dbl Crop Lettuce/Barley',234: 'Dbl Crop Durum Wht/Sorghum',235: 'Dbl Crop Barley/Sorghum',236: 'Dbl Crop WinWht/Sorghum',237: 'Dbl Crop Barley/Corn',238: 'Dbl Crop WinWht/Cotton',239: 'Dbl Crop Soybeans/Cotton',240: 'Dbl Crop Soybeans/Oats',241: 'Dbl Crop Corn/Soybeans',242: 'Blueberries',243: 'Cabbage',244: 'Cauliflower',245: 'Celery',246: 'Radishes',247: 'Turnips',248: 'Eggplants',249: 'Gourds',250: 'Cranberries',254: 'Dbl Crop Barley/Soybeans'}
crops_to_values = {v: k for k, v in values_to_crops.items()}
crops = ['Corn','Cotton','Rice','Sorghum','Soybeans','Sunflower','Peanuts','Tobacco','Sweet Corn','Pop or Orn Corn','Mint','Barley','Durum Wheat','Spring Wheat','Winter Wheat','Other Small Grains','Rye','Oats','Millet','Speltz','Canola','Flaxseed','Safflower','Rape Seed','Mustard','Alfalfa','Other Hay/Non Alfalfa','Camelina','Buckwheat','Sugarbeets','Dry Beans','Potatoes','Other Crops','Sugarcane','Sweet Potatoes','Misc Vegs & Fruits','Watermelons','Onions','Cucumbers','Chick Peas','Lentils','Peas','Tomatoes','Caneberries','Hops','Herbs','Clover/Wildflowers','Cherries','Peaches','Apples','Grapes','Pecans','Almonds','Walnuts','Pears','Pistachios','Triticale','Carrots','Asparagus','Garlic','Cantaloupes','Prunes','Olives','Oranges','Honeydew Melons','Broccoli','Peppers','Pomegranates','Nectarines','Greens','Plums','Strawberries','Squash','Apricots','Vetch','Lettuce','Pumpkins','Blueberries','Cabbage','Cauliflower','Celery','Radishes','Turnips','Eggplants','Gourds','Cranberries']<jupyter_output><empty_output><jupyter_text>Objective 1: Graph the average NDVI over time for the countyThe follow steps will be necessary:- Combine the landsat layer and the CDL layer at zoom level 13- Compute the NDVI from cloud-masked landsat data- Mask by the crop of your choice- Compute the average time series- Plot the time series dataIf a distinct pattern emerges, research your crop to understand if the cycle you are seeing in the data represents a true crop cycle.  How does the values you are seeing relate to ground truth data? Does it match up?Check against internet sources, suche as [this USDA guide from 2010](http://usda.mannlib.cornell.edu/usda/current/planting/planting-10-29-2010.pdf)  Extra credit: Run over a hand-drawn areaDisplay the CDL layer on the map across the US, and filter by the target crop as in Exercise 1. Find an area with high density, and draw a larger polygon over that area. See if this improves your results!Hint: Use the method below to reproject the `M.layers.polygons[0]` polygon from LatLng to WebMercator<jupyter_code>def reproject(geom):
    project = partial(
        pyproj.transform,
        pyproj.Proj(init='epsg:4326'),
        pyproj.Proj(init='epsg:3857'))
    return transform(project, geom)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15289892252437148, 'fraction_numerical': 0.13350436121087736, 'mean_word_length': 7.489547038327526, 'pattern_counts': {'":': 5, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '163519', 'license': 'Apache-2.0', 'n_tokens_mistral': 4774, 'n_tokens_neox': 3865, 'n_words': 796, 'path': 'exercises/exercise3/Exercise 3 - Combining Landsat and CDL.ipynb', 'repo_name': 'lossyrob/foss4g-2017-geopyspark-workshop', 'size': '9745'}