<jupyter_start><jupyter_text>Earth Engine and Live Tiles for Web Maps 1. Build a tile export service mixing precalculated tiles with live tiles from Earth EngineThis Section shows an example of how to mix pre-calculated tiles from Earth Engine (hosted on GCS) with live-calculated tiles from Earth Engine (past zoom level 12).Based on code snippets drawn from [here](https://github.com/wri/gfw-api/blob/master/gfw/gee_tiles.py).Note: the tokens for the live Earth Engine tiles expire every 3 days, and so this code will need to be re-run with an authorised EE account to generate valid (live) tiles. 2. Best Sentinel 2 imagery for specific points and timesThis section shows how to select the best (cloud free) images of Sentinel 2, for specific times and locations, and present them on a web map.<jupyter_code>import os
import ee
import json

ee.Initialize()
def tile_url(image, viz_params=None):
    """Create a target url for tiles for an image.
    e.g.
    im = ee.Image("LE7_TOA_1YEAR/" + year).select("B3","B2","B1")
    viz = {'opacity': 1, 'gain':3.5, 'bias':4, 'gamma':1.5}
    url = tile_url(image=im),viz_params=viz)
    """
    if viz_params:
        d = image.getMapId(viz_params)
    else:
        d = image.getMapId()
    base_url = 'https://earthengine.googleapis.com'
    url = (base_url + '/map/' + d['mapid'] + '/{z}/{x}/{y}?token=' + d['token'])
    return url<jupyter_output><empty_output><jupyter_text>EE pansharpened imageBelow calculates a pan-sharpended image from Landsat, and creates a tile url with temporary authentitication.<jupyter_code>collection = ee.ImageCollection('LANDSAT/LC8_L1T').filterDate('2016-01-01T00:00','2017-01-01T00:00')

composite = ee.Algorithms.SimpleLandsatComposite(collection=collection, percentile=50,
                                                 maxDepth=80, cloudScoreRange=1, asFloat=True)

hsv2 = composite.select(['B4', 'B3', 'B2']).rgbToHsv()

sharpened2 = ee.Image.cat([hsv2.select('hue'), hsv2.select('saturation'),
                           composite.select('B8')]).hsvToRgb().visualize(gain=1000, gamma= [1.15, 1.4, 1.15])


ee_tiles = tile_url(sharpened2)
ee_tiles<jupyter_output><empty_output><jupyter_text>Use the vizzuality microservice to retrieve the urlMicroservice based on the preceeding code should give back the tileset<jupyter_code>import requests
r = requests.get('https://staging-api.globalforestwatch.org/v1/landsat-tiles/2015')
print(r.status_code)
r.json()
ee_tiles= r.json().get('data').get('attributes').get('url')
print(ee_tiles)<jupyter_output><empty_output><jupyter_text>Display the results on a map<jupyter_code>pre_calculated_tileset="https://storage.googleapis.com/landsat-cache/2015/{z}/{x}/{y}.png"
import folium
map = folium.Map(location=[28.29, -16.6], zoom_start=2, tiles='Mapbox Bright' )
map.add_tile_layer(tiles=pre_calculated_tileset, max_zoom=11, min_zoom=0, attr='Earth Engine tiles by Vizzuality')
map.add_tile_layer(tiles=ee_tiles, max_zoom=20, min_zoom=13, attr="Live EE tiles")
map<jupyter_output><empty_output><jupyter_text>2. Sentinel<jupyter_code>import folium
import os
import ee
import json
ee.Initialize()<jupyter_output><empty_output><jupyter_text>The below Earth Engine code returns the optimum cloud-free image for a given point and time range.<jupyter_code>def tile_url(image, viz_params=None):
    """Create a target url for tiles for an image.
    e.g.
    im = ee.Image("LE7_TOA_1YEAR/" + year).select("B3","B2","B1")
    viz = {'opacity': 1, 'gain':3.5, 'bias':4, 'gamma':1.5}
    url = tile_url(image=im),viz_params=viz)
    """
    if viz_params:
        d = image.getMapId(viz_params)
    else:
        d = image.getMapId()
    base_url = 'https://earthengine.googleapis.com'
    url = (base_url + '/map/' + d['mapid'] + '/{z}/{x}/{y}?token=' + d['token'])
    return url


def proxy_sentinel(lat, lon, start, end):
    """
    Sentinel covers all continental land surfaces (including inland waters) between latitudes 56° south and 83° north
        all coastal waters up to 20 km from the shore
        all islands greater than 100 km2
        all EU islands
        the Mediterranean Sea
        all closed seas (e.g. Caspian Sea).
    
    Filter by tiles that intersect a lat,lon point, and are within a date range, and have less than
    10% cloud cover, then find the lowest scoring cloud image.
    
    Note, we first filter by cloud less than X% and then pick the top, rather than just directly pick
    the best cloud scoring image, as these operations default to a subset of images. So we want to 
    pick the best image, from a pre-selected subset of good images.
    
    Note the url generated expires after a few days and needs to be refreshed.
    e.g. variables
    lat = -16.66
    lon = 28.24
    start = '2017-01-01'
    end = '2017-03-01'
    """
    #if lat >= 83 or lat <= -56:
    #    return {'status': 'Latitute {0} invalid: Must be between -56 and 83'.format(lat)}
    #else:
    try:
        point = ee.Geometry.Point(lat, lon)
        S2 = ee.ImageCollection('COPERNICUS/S2'
                               ).filterDate(
                                start, end).filterBounds(
                                point).sort('CLOUDY_PIXEL_PERCENTAGE', True).first()
        S2 = ee.Image(S2)
        d = S2.getInfo()       # grab a dictionary of the image metadata
        S2 = S2.divide(10000)  # Convert to Top of the atmosphere reflectance
        S2 = S2.visualize(bands=["B4", "B3", "B2"], min=0, max=0.3, opacity=1.0) # Convert to styled RGB image
        image_tiles = tile_url(S2)
        boundary = ee.Feature(ee.Geometry.LinearRing(d.get('properties').get("system:footprint").get('coordinates')))
        boundary_tiles = tile_url(boundary, {'color': '4eff32'})
        meta = get_image_metadata(d)
        output = {}
        output = {'boundary_tiles': boundary_tiles,
                  'image_tiles': image_tiles,
                  'metadata': meta,
                  'sucsess': True}

        return output
    except:
        return {'sucsess': False}


def get_image_metadata(d):
    """Return a dictionary of metadata"""
    image_name = d.get('id')
    date_info = image_name.split('COPERNICUS/S2/')[1]
    date_time = ''.join([date_info[0:4],'-',date_info[4:6],'-',date_info[6:8],' ',
                      date_info[9:11],':',date_info[11:13],':',date_info[13:15],"Z"])
    product_id = d.get('properties').get('PRODUCT_ID')
    meta = {}
    meta = {'image_name': image_name, 'date_time': date_time, 'product_id': product_id}
    return meta<jupyter_output><empty_output><jupyter_text>Sentinel-2: MultiSpectral Instrument (MSI), Level-1CJun 23, 2015 - Aug 30, 2017Info on [Sentinel Naming conventions](https://earth.esa.int/web/sentinel/user-guides/sentinel-2-msi/naming-convention)First comes YYYYMMDDHHMMSS then _ then 15 characters of ID and then _ then tile number info<jupyter_code>%%time
# london
#lon = 51.64167220085054
#lat = 0.03

# oslo
#lon = 10.745
#lat = 59.922

# Tenerife
lat = -16.644
lon = 28.266

# Fiji
#lat = 177.825
#lon = -17.916

start ='2017-01-01'
end ='2017-01-10'

sentinel = proxy_sentinel(lon=lon, lat=lat, start=start, end=end)


if sentinel.get('sucsess'):
    pass
else:
    raise ValueError('no data')

print(sentinel.get('metadata'))

sentinel_map = folium.Map(location=[lon, lat], zoom_start=9, tiles='Mapbox Bright' )
sentinel_map.add_tile_layer(tiles=sentinel.get('image_tiles'), max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map.add_tile_layer(tiles=sentinel.get('boundary_tiles'), max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map
sentinel<jupyter_output><empty_output><jupyter_text>Sentinel  microservice V 0.1We have hooked-up a version of the above sentinel code to our API ([here](https://github.com/gfw-api/gfw-analysis-gee)). An example of how to call it to get the same results as above is as follows: {'lat':'-16.644','lon':'28.266', 'start':'2017-01-01', 'end': "2017-09-10"} Wall time: 25.6 s{'lat':'-16.644','lon':'28.266', 'start':'2017-01-01', 'end': "2017-01-10"} Wall time: 25.8 s<jupyter_code>import requests
import folium
%%time
url = "https://staging-api.globalforestwatch.org/v1/sentinel-tiles"
params= {'lat':'-16.644','lon':'28.266', 'start':'2016-01-01', 'end': "2017-01-10"}
r = requests.get(url, params=params)
r.status_code

print(r.json())

dt = r.json().get('data').get('attributes').get('date_time')
boundary = r.json().get('data').get('attributes').get('url_boundary')
sentinel_image = r.json().get('data').get('attributes').get('url_image')<jupyter_output><empty_output><jupyter_text>```{'data': {'attributes': {'date_time': '2017-01-01 11:52:12Z', 'product_id': 'S2A_MSIL1C_20170101T115212_N0204_R123_T28RCS_20170101T115212', 'url_boundary': 'https://earthengine.googleapis.com/map/4b1b9c6f82d50796562521502bc4d9a2/{z}/{x}/{y}?token=fb2181f663a8f2895224fecddd8b1ec4', 'url_image': 'https://earthengine.googleapis.com/map/0341ef17ae8e75cf0b53b5de5fd767ff/{z}/{x}/{y}?token=df18de8c707463d20912e8699f7801b8'}, 'id': None, 'type': 'sentinel_tiles_url'}}```<jupyter_code>sentinel_map = folium.Map(location=[float(params['lon']), float(params['lat'])], zoom_start=9, tiles='Mapbox Bright' )
sentinel_map.add_tile_layer(tiles=sentinel_image, max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map.add_tile_layer(tiles=boundary, max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map<jupyter_output><empty_output><jupyter_text>Sentinel version 2We now need to modify the Sentinel service.We will experiment with a variety of improvements:1. Possibly a wider area (composite/mosaic) of tiles returned2. Definitley need to return a time-stack of tiles 3. Metadata to accompany the tiles that would let a user click through on the front end4. Thumbnails for tiles5. Custom visulization properties?6. Possibly NDVI visulization?We should aim to create a microservice so that it emmits something like the following:input request:```url = "https://staging-api.globalforestwatch.org/v1/high-res-tiles"params= {'lat':'-16.644','lon':'28.266', 'start':'2016-01-01', 'end': "2017-01-10"}r = requests.get(url, params=params)```output:```{{source:'sentinel2', cloud_score: 100, date:'2017-01-01 11:52:12Z', metadata:'S2A_MSIL1C_20170101T115212_N0204_R123_T28RCS_20170101T115212', tile_url:'https://earthengine.googleapis.com/map/0341ef17ae8e75cf0b53b5de5fd767ff/{z}/{x}/{y}?token=df18de8c707463d20912e8699f7801b8', thumbnail_url:'https://earthengine.googleapis.com//api/thumb?thumbid=2a7adddc1b2fbd8d3cc773be997b290c&token=<PASSWORD>7445f4', }, {...},   <--- need as many elements as there are unique images in the date range that intersect with the point}```<jupyter_code>import folium
import os
import ee
import json
import requests
import math
import maya
import numpy as np
from scipy import misc
import shutil
ee.Initialize()
def tile_url(image, viz_params=None):
    """Create a target url for tiles for an image.
    e.g.
    im = ee.Image("LE7_TOA_1YEAR/" + year).select("B3","B2","B1")
    viz = {'opacity': 1, 'gain':3.5, 'bias':4, 'gamma':1.5}
    url = tile_url(image=im),viz_params=viz)
    """
    if viz_params:
        d = image.getMapId(viz_params)
    else:
        d = image.getMapId()
    base_url = 'https://earthengine.googleapis.com'
    url = (base_url + '/map/' + d['mapid'] + '/{z}/{x}/{y}?token=' + d['token'])
    return url


def proxy_sentinel(lat, lon, start, end):
    """
    Sentinel covers all continental land surfaces (including inland waters) between latitudes 56° south and 83° north
        all coastal waters up to 20 km from the shore
        all islands greater than 100 km2
        all EU islands
        the Mediterranean Sea
        all closed seas (e.g. Caspian Sea).
    
    Filter by tiles that intersect a lat,lon point, and are within a date range, and have less than
    10% cloud cover, then find the lowest scoring cloud image.
    
    Note, we first filter by cloud less than X% and then pick the top, rather than just directly pick
    the best cloud scoring image, as these operations default to a subset of images. So we want to 
    pick the best image, from a pre-selected subset of good images.
    
    Note the url generated expires after a few days and needs to be refreshed.
    e.g. variables
    lat = -16.66
    lon = 28.24
    start = '2017-01-01'
    end = '2017-03-01'
    """
    #if lat >= 83 or lat <= -56:
    #    return {'status': 'Latitute {0} invalid: Must be between -56 and 83'.format(lat)}
    #else:
    try:
        point = ee.Geometry.Point(lat, lon)
        S2 = ee.ImageCollection('COPERNICUS/S2'
                               ).filterDate(
                                start, end).filterBounds(
                                point).sort('CLOUDY_PIXEL_PERCENTAGE', True).first()
        S2 = ee.Image(S2)
        d = S2.getInfo()       # grab a dictionary of the image metadata
        S2 = S2.divide(10000)  # Convert to Top of the atmosphere reflectance
        S2 = S2.visualize(bands=["B4", "B3", "B2"], min=0, max=0.3, opacity=1.0) # Convert to styled RGB image
        return S2
        image_tiles = tile_url(S2)
        boundary = ee.Feature(ee.Geometry.LinearRing(d.get('properties').get("system:footprint").get('coordinates')))
        boundary_tiles = tile_url(boundary, {'color': '4eff32'})
        meta = get_image_metadata(d)
        output = {}
        output = {'boundary_tiles': boundary_tiles,
                  'image_tiles': image_tiles,
                  'metadata': meta,
                  'sucsess': True}

        return output
    except:
        return {'sucsess': False}


def get_image_metadata(d):
    """Return a dictionary of metadata"""
    image_name = d.get('id')
    date_info = image_name.split('COPERNICUS/S2/')[1]
    date_time = ''.join([date_info[0:4],'-',date_info[4:6],'-',date_info[6:8],' ',
                      date_info[9:11],':',date_info[11:13],':',date_info[13:15],"Z"])
    product_id = d.get('properties').get('PRODUCT_ID')
    meta = {}
    meta = {'image_name': image_name, 'date_time': date_time, 'product_id': product_id}
    return meta
# Tenerife
lat = -16.644
lon = 28.266

start ='2017-01-01'
end ='2017-02-10'

sentinel = proxy_sentinel(lon=lon, lat=lat, start=start, end=end)
sentinel.getThumbURL()
%%time
# - Synchronous version -
# Downloading thumbnail images test
for x in [1, 2, 3, 4, 6]:
    
    start =f'2017-0{x}-01'
    end =f'2017-0{x}-25'
    sentinel = proxy_sentinel(lon=lon, lat=lat, start=start, end=end)
    print(type(sentinel))
    r = requests.get(sentinel.getThumbURL({'dimensions':[250,250]}), stream=True)
    if r.status_code == 200:
        with open(f'./pics/mythumb{x}.png', 'wb') as f:
            r.raw.decode_content = True
            shutil.copyfileobj(r.raw, f)    
tile_url=sentinel.getThumbURL()
im_arrays = misc.imread(requests.get(tile_url, stream=True).raw, mode='RGBA')
im_arrays<jupyter_output><empty_output><jupyter_text>Async code in Python:https://terriblecode.com/blog/asynchronous-http-requests-in-python/Will need to make the requests asyncronously.<jupyter_code>if sentinel.get('sucsess'):
    pass
else:
    raise ValueError('no data')

print(sentinel.get('metadata'))

sentinel_map = folium.Map(location=[lon, lat], zoom_start=9, tiles='Mapbox Bright' )
sentinel_map.add_tile_layer(tiles=sentinel.get('image_tiles'), max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map.add_tile_layer(tiles=sentinel.get('boundary_tiles'), max_zoom=19, min_zoom=6, attr="Live EE tiles")
sentinel_map
point = ee.Geometry.Point(-16.644, 28.266)
S2 = ee.ImageCollection('COPERNICUS/S2'
                       ).filterDate(
                        '2017-01-01', '2017-01-10').filterBounds(
                        point).sort('CLOUDY_PIXEL_PERCENTAGE', True)
S2.getInfo()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13973715954542568, 'fraction_numerical': 0.05440924385753285, 'mean_word_length': 3.768997880714502, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 48, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '636110', 'license': 'MIT', 'n_tokens_mistral': 6070, 'n_tokens_neox': 5427, 'n_words': 1313, 'path': 'work/EE_as_a_tile_server.ipynb', 'repo_name': 'Vizzuality/data_sci_tutorials', 'size': '15767'}