<jupyter_start><jupyter_text>SpaceNet Rio Chip Classification Data PrepThis notebook prepares data for training a chip classification model on the Rio SpaceNet dataset.* Set `raw_uri` to the local or S3 directory containing the raw dataset.* Set `processed_uri` to a local or S3 directory (you can write to), which will store the processed data generated by this notebook.This is all you will need to do in order to run this notebook.<jupyter_code>raw_uri = 's3://spacenet-dataset/'
# processed_uri = 's3://raster-vision-lf-dev/examples/spacenet/rio/processed-data/'
processed_uri = '/opt/data/examples/spacenet/rio/processed-data'<jupyter_output><empty_output><jupyter_text>The steps we'll take to make the data are as follows:- Get the building labels and AOI (most likely from the SpaceNet AWS public dataset bucket)- Use the AOI and the image bounds to determine which images can be used for training and validation- Split the building labels by image, save a label GeoJSON file per image- Split the labeled images into a training and validation set, using the percentage of the AOI each covers, aiming at an 80%/20% split.This process will save the split label files, and `train_scenes.csv` and `val_scenes.csv` files that are used by the experiment at `experiments.spacenet.chip_classification`<jupyter_code>import os
from os.path import join
import json
import tempfile

import boto3
import botocore
import rasterio
from shapely.geometry import (Polygon, shape)

import rastervision as rv
from rastervision.utils.files import (
    download_if_needed, list_paths, file_to_json, json_to_file, 
    get_local_path, make_dir, sync_to_dir, str_to_file)<jupyter_output><empty_output><jupyter_text>Get the label and AOI data from AWS's public dataset of Space Net<jupyter_code>label_uri = join(raw_uri, 'AOI_1_Rio/srcData/buildingLabels/Rio_Buildings_Public_AOI_v2.geojson')
aoi_uri = join(raw_uri, 'AOI_1_Rio/srcData/buildingLabels/Rio_OUTLINE_Public_AOI.geojson')

label_json = file_to_json(label_uri)
aoi_json = file_to_json(aoi_uri)<jupyter_output><empty_output><jupyter_text>Use the AOI to determine what images are inside the training setHere we compare the AOI to the image extents to deteremine which images we can use for training and validation. We're using `rasterio`'s ability to read the metadata from raster data on S3 without downloading the whole image<jupyter_code>aoi = shape(aoi_json['features'][0]['geometry'])
aoi
images_uri = join(raw_uri, 'AOI_1_Rio/srcData/mosaic_3band')
image_paths = list_paths(images_uri)
def bounds_to_shape(bounds):
    return Polygon([[bounds.left, bounds.bottom],
                    [bounds.left, bounds.top],
                    [bounds.right, bounds.top],
                    [bounds.right, bounds.bottom],
                    [bounds.left, bounds.bottom]])

image_to_extents = {}
for img in image_paths:
    with rasterio.open(img, 'r') as ds:
        image_to_extents[img] = bounds_to_shape(ds.bounds)

intersecting_images = []
for img in image_to_extents:
    if image_to_extents[img].intersects(aoi):
        intersecting_images.append(img)
intersecting_images<jupyter_output><empty_output><jupyter_text>Match labels to imagesFind the labels that intersect with the image's bounding box, which will be saved into a labels geojson that matches the image name.<jupyter_code># Add a class_id and class_name to the properties of each feature
for feature in label_json['features']:
    feature['properties']['class_id'] = 1
    feature['properties']['class_name'] = 'building'
image_to_features = {}
for img in intersecting_images:
    image_to_features[img] = []
    bbox = image_to_extents[img]
    for feature in label_json['features']:
        if shape(feature['geometry']).intersects(bbox):
            image_to_features[img].append(feature)
    
processed_labels_uri = join(processed_uri, 'labels')

for img in image_to_features:
    fc = {}
    fc['type'] = 'FeatureCollection'
    fc['crs'] = label_json['crs']
    fc['features'] = image_to_features[img]
    img_id = os.path.splitext(os.path.basename(img))[0]
    label_path = join(processed_labels_uri, '{}.geojson'.format(img_id))
    json_to_file(fc, label_path)<jupyter_output><empty_output><jupyter_text>Split into train and validationSplit up training and validation data. There's an odd shaped AOI and not that many images, so we'll split the train and validation roughly based on how much area each scene covers of the AOI. Create a CSV that our experiments will use to load the training and validation data.<jupyter_code># Split training and validation
ratio = 0.8
aoi_area = aoi.area
images_to_area = {}
for img in intersecting_images:
    area = image_to_extents[img].intersection(aoi).area
    images_to_area[img] = area / aoi_area

train_imgs = []
val_imgs = []
train_area_covered = 0
for img in sorted(intersecting_images, reverse=True, key=lambda img: images_to_area[img]):
    if train_area_covered < ratio:
        train_imgs.append(img)
        train_area_covered += images_to_area[img]
    else:
        val_imgs.append(img)
print("{} training images with {}% area.".format(len(train_imgs), train_area_covered))
print("{} validation images with {} area.".format(len(val_imgs), 1 - train_area_covered))
def save_split_csv(imgs, path):
    csv_rows = []
    for img in imgs:
        img_id = os.path.splitext(os.path.basename(img))[0]
        img_path = join('AOI_1_Rio', 'srcData/mosaic_3band', '{}.tif'.format(img_id))
        label_path = join('labels','{}.geojson'.format(img_id))
        csv_rows.append('"{}","{}"'.format(img_path, label_path))
    str_to_file('\n'.join(csv_rows), path)
        
save_split_csv(train_imgs, join(processed_uri, 'train-scenes.csv'))
save_split_csv(val_imgs, join(processed_uri, 'val-scenes.csv'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09791055085477465, 'fraction_numerical': 0.004317043688482128, 'mean_word_length': 4.934426229508197, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '13176', 'license': 'Apache-2.0', 'n_tokens_mistral': 1913, 'n_tokens_neox': 1838, 'n_words': 526, 'path': 'notebooks/spacenet/spacenet_rio_chip_class_data_prep.ipynb', 'repo_name': 'yoninachmany/raster-vision-examples', 'size': '5791'}