<jupyter_start><jupyter_text>Generate tilesWe are going to use the Hansen et al. data found [here](https://storage.googleapis.com/earthenginepartners-hansen/GFC-2020-v1.8/download.html). For actual products, this would likely be Planet data or some high resolution map that we would need for classification.<jupyter_code># box inputs are minx, miny, maxx, maxy
geometries = [box(-180 + i * 10, 
                  80 - (j + 1) * 10, 
                  -180 + (i + 1) * 10,
                  80 - (j) * 10) for i in range(36) for j in range(16)]

# Upper left corner
ul_corners = [(-180 + (i) * 10, 80 - (j) * 10) for i in range(36) for j in range(16)]
df_hansen_tiles = gpd.GeoDataFrame(geometry=geometries, crs=CRS.from_epsg(4326))<jupyter_output><empty_output><jupyter_text>Let's see the tiles.<jupyter_code>df_hansen_tiles.exterior.plot()
def url_generator(corner: tuple, 
                  layer: str ='last'):
    
    assert(layer in ['first', 'last', 'treecover2000'])
    
    base_url = 'https://storage.googleapis.com/earthenginepartners-hansen/GFC-2020-v1.8'
    lon, lat = corner
    if lon < 0:
        lon_str = f'{abs(lon):03d}W'
    else:
        lon_str = f'{abs(lon):03d}E'
    if lat >= 0:
        lat_str = f'{abs(lat):02d}N'
    else:
        lat_str = f'{abs(lat):02d}S'
    return f'{base_url}/Hansen_GFC-2020-v1.8_last_{lat_str}_{lon_str}.tif'

df_hansen_tiles['source_url'] = list(map(url_generator, ul_corners))
df_hansen_tiles['year'] = 2020
df_hansen_tiles['bucket'] = 'test-prod--hansen-landsat-mosaics-2020'
df_hansen_tiles['key'] = df_hansen_tiles.source_url.map(lambda x: x.split('/')[-1])

df_hansen_tiles.head()<jupyter_output><empty_output><jupyter_text>Let's read the bounds from the URLs and make sure they match.<jupyter_code>url = df_hansen_tiles.source_url[0]
with rasterio.open(url) as ds:
    print(ds.bounds)
    print(url)<jupyter_output>BoundingBox(left=-180.0001388888889, bottom=69.9998611111111, right=-169.99986111111113, top=80.00013888888888)
https://storage.googleapis.com/earthenginepartners-hansen/GFC-2020-v1.8/Hansen_GFC-2020-v1.8_last_80N_180W.tif<jupyter_text>Get Tiles over AOIs<jupyter_code>aois = gpd.read_file('aois.geojson')
aois_geo = aois.geometry.unary_union
fig, ax = plt.subplots()

df_hansen_deltas = df_hansen_tiles[df_hansen_tiles.geometry.intersects(aois_geo)].reset_index(drop=True)

df_hansen_deltas.plot(ax = ax, alpha=.1)
aois.plot(ax=ax)
df_hansen_deltas
df_hansen_deltas.source_url.tolist()<jupyter_output><empty_output><jupyter_text>Upload Tile Index to s3References from Stackoverflow:+ [How to connect to s3](https://stackoverflow.com/a/40336919) - there are two ways+ [Upload json to s3](https://stackoverflow.com/a/53004085)<jupyter_code>geojson_bytes = bytes(df_hansen_deltas.to_json(drop_id=True).encode('UTF-8'))
geojson_bytes
import boto3

session = boto3.session.Session(profile_name='saml-pub')

s3 = session.resource('s3')

geojson_bucket = 'calval-metadata'
geojson_key = f'{df_hansen_deltas.bucket[0]}.geojson'

s3object = s3.Object(geojson_bucket, geojson_key)
s3object.put(Body=geojson_bytes)
df_hansen_deltas.to_file(geojson_key, driver='GeoJSON')<jupyter_output><empty_output><jupyter_text>Read Geojson on S3Show existing buckets<jupyter_code>buckets = list(s3.buckets.all())
buckets<jupyter_output><empty_output><jupyter_text>Let's inspect the bucket contents.<jupyter_code>metadata_bucket = s3.Bucket('calval-metadata')
list(metadata_bucket.objects.all())<jupyter_output><empty_output><jupyter_text>Read the actual dataframe.<jupyter_code>obj = s3.Object(geojson_bucket, geojson_key)
obj.get()
data = obj.get()['Body']
gpd.read_file(data)<jupyter_output><empty_output><jupyter_text>Upload to Mosaics to s3 (this takes a *long* time on a local machine)It's negligible on a fast server.The buckets have to be unique globally - interesting that this is harder than it seems and will throw an [error](https://stackoverflow.com/q/57146311) like this.Also, there are some constraints on bucket names as discussed in [here](https://stackoverflow.com/a/49548452).<jupyter_code>buckets = list(s3.buckets.all())
buckets
bucket_name = df_hansen_deltas.bucket[0]
if bucket_name not in [b.name for b in buckets]:
    s3.Bucket(bucket_name).create(CreateBucketConfiguration={'LocationConstraint': 'us-west-2'})
records = df_hansen_deltas.to_dict('records')
record = records[0]
import requests

r = requests.get(record['source_url'], stream=True)
total_size = int(r.headers['Content-length'])
print( total_size / 1e9, 'GB')<jupyter_output>5.022295203 GB<jupyter_text>You need to use boto3 multipart upload; tqdm has a nice way to track upload as well.Sources: 1. Basic upload from s3 using public url: [link](https://stackoverflow.com/questions/14346065/upload-image-available-at-public-url-to-s3-using-boto)2. Upload large file using TransferConfig and boto3: [link](https://stackoverflow.com/a/50118024) 3. Getting tqdm to work with the above: [link](https://stackoverflow.com/a/70263266)4. Doc on transferconfig: [link](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.htmlboto3.s3.transfer.TransferConfig)<jupyter_code>def upload_one(record):
    
    r = requests.get(record['source_url'], stream=True)
    total_size = int(r.headers['Content-length'])

    bucket_name = record['bucket']
    key = record['key']
    config = TransferConfig(multipart_threshold=1024, 
                            max_concurrency=30,
                            # Commenting this out appears to make upload speed more variable
                            #multipart_chunksize=1024, 
                            use_threads=True)
    with tqdm(total=total_size,  
              unit='B', 
              unit_scale=True, 
              unit_divisor=1024) as pbar:


        bucket = s3.Bucket(bucket_name)
        resp = bucket.upload_fileobj(r.raw, key, Config=config, Callback=pbar.update)
    
    return resp
list(map(upload_one, tqdm(records[:])))<jupyter_output>0%|                                                               | 0/2 [00:00<?, ?it/s]
  0%|                                                         | 0.00/4.68G [00:00<?, ?B/s][A
  0%|                                                | 256k/4.68G [00:00<1:44:03, 804kB/s][A
  0%|                                                | 1.00M/4.68G [00:00<29:16, 2.86MB/s][A
  0%|                                                | 7.00M/4.68G [00:00<04:06, 20.3MB/s][A
  0%|▏                                               | 19.0M/4.68G [00:00<01:38, 51.0MB/s][A
  1%|▍                                               | 39.8M/4.68G [00:00<00:49, 99.9MB/s][A
  1%|▌                                                | 54.5M/4.68G [00:00<00:39, 124MB/s][A
  1%|▌                                                | 54.5M/4.68G [00:00<00:39, 124MB/s][A
  1%|▋                                                | 70.0M/4.68G [00:00<00:37, 134MB/s][A
  1%|▋                                                | 70.2M/4[...]<jupyter_text>Notes+ S3 objects can be overwritten without problem. However, buckets cannot. So you cannot create a bucket that already exists, but you can easily overwrite an existing object  AppendixI initially used rasterio, but it was much slower. Keeping it here for reference. Probably because the memory file writing was unecessary.References:+ [rasterio mem files with a post](https://rasterio.readthedocs.io/en/latest/topics/memory-files.htmlreading-memoryfiles)+ [write s3 using rasterio](https://github.com/rasterio/rasterio/issues/1159issuecomment-532027616)<jupyter_code>from rasterio.io import MemoryFile

def upload_one_slow(record):
    with rasterio.open(record['source_url']) as ds:
        src_profile = ds.profile
        src_array = ds.read()
    with MemoryFile() as memfile:
        with memfile.open(**src_profile) as dataset:
            dataset.write(src_array)

        s3ob = s3.Object(record['bucket'], record['key'])
        s3ob.put(Body=memfile)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12882205513784462, 'fraction_numerical': 0.058771929824561406, 'mean_word_length': 3.2027382833070037, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 43, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '283932', 'license': 'Apache-2.0', 'n_tokens_mistral': 3125, 'n_tokens_neox': 2772, 'n_words': 529, 'path': 'datasets/0_Organize_Hansen_Landsat_Mosaics.ipynb', 'repo_name': 'OPERA-Cal-Val/cal-val-example', 'size': '7980'}