<jupyter_start><jupyter_text>CV4A ICRL Crop Type Classification Challenge A Guide to Access the data on Radiant MLHubThis notebook walks you through the steps to get access to Radiant MLHub and access the data for the crop type classification competition being organized as part of the [CV4A](https://www.cv4gc.org/cv4a2020/) workshop at 2020 ICLR.   Radiant MLHub APIThe Radiant MLHub API gives access to open Earth imagery training data for machine learning applications. You can learn more about the repository at the [Radiant MLHub site](https://mlhub.earth) and about the organization behind it at the [Radiant Earth Foundation site](https://radiant.earth).Full documentation for the API is available at [docs.mlhub.earth](docs.mlhub.earth).Each item in our collection is explained in json format compliant with [STAC](https://stacspec.org/) [label extension](https://github.com/radiantearth/stac-spec/tree/master/extensions/label) definition.<jupyter_code># Required libraries
import requests
from urllib.parse import urlparse
from pathlib import Path
from datetime import datetime
# output path where you want to download the data
output_path = Path("data/")<jupyter_output><empty_output><jupyter_text>AuthenticationAccess to the Radiant MLHub API requires an access token. To get your access token, go to [dashboard.mlhub.earth](https://dashboard.mlhub.earth). If you have not used Radiant MLHub before, you will need to sign up and create a new account. Otherwise, sign in. Under **Usage**, you'll see your access token, which you will need. *Do not share* your access token with others: your usage may be limited and sharing your access token is a security risk.Copy the access token, and paste it in the box bellow. This header block will work for all API calls.<jupyter_code># copy your access token from dashboard.mlhub.earth and paste it in the following
ACCESS_TOKEN = 'PASTE_YOUR_ACCESS_TOKEN_HERE'

# these headers will be used in each request
headers = {
    'Authorization': f'Bearer {ACCESS_TOKEN}',
    'Accept':'application/json'
}<jupyter_output><empty_output><jupyter_text>Retrieving the competition datasetDatasets are stored as collections on Radiant MLHub catalog. A collection represents the top-most data level. Typically this means the data comes from the same source for the same geography. It might include different years or sub-geographies.The two collections for this competition are:- `ref_african_crops_kenya_02_source`: includes the multi-temporal bands of Sentinel-2- `ref_african_crops_kenya_02_labels`: includes the labels and field IDs<jupyter_code>def get_download_url(item, asset_key, headers):
    asset = item.get('assets', {}).get(asset_key, None)
    if asset is None:
        print(f'Asset "{asset_key}" does not exist in this item')
        return None
    r = requests.get(asset.get('href'), headers=headers, allow_redirects=False)
    return r.headers.get('Location')

def download_label(url, output_path, tileid):
    filename = urlparse(url).path.split('/')[-1]
    outpath = output_path/tileid
    outpath.mkdir(parents=True, exist_ok=True)
    
    r = requests.get(url)
    f = open(outpath/filename, 'wb')
    for chunk in r.iter_content(chunk_size=512 * 1024): 
        if chunk:
            f.write(chunk)
    f.close()
    print(f'Downloaded {filename}')
    return 

def download_imagery(url, output_path, tileid, date):
    filename = urlparse(url).path.split('/')[-1]
    outpath = output_path/tileid/date
    outpath.mkdir(parents=True, exist_ok=True)
    
    r = requests.get(url)
    f = open(outpath/filename, 'wb')
    for chunk in r.iter_content(chunk_size=512 * 1024): 
        if chunk:
            f.write(chunk)
    f.close()
    print(f'Downloaded {filename}')
    return<jupyter_output><empty_output><jupyter_text>Downloading LabelsThe `assets` property of the items in a collection contains all the assets associated with that item and links to download them. The labels for the item will always be the asset with the key `labels`. The following code will go through every item in the collection and download the labels and field_ids raster feature.<jupyter_code># paste the id of the labels collection:
collectionId = 'ref_african_crops_kenya_02_labels'

# these optional parameters can be used to control what items are returned. 
# Here, we want to download all the items so:
limit = 100 
bounding_box = []
date_time = []

# retrieves the items and their metadata in the collection
r = requests.get(f'https://api.radiant.earth/mlhub/v1/collections/{collectionId}/items', params={'limit':limit, 'bbox':bounding_box,'datetime':date_time}, headers=headers)
collection = r.json()
# retrieve list of features (in this case tiles) in the collection
for feature in collection.get('features', []):
    assets = feature.get('assets').keys()
    print("Feature", feature.get('id'), 'with the following assets', list(assets))
for feature in collection.get('features', []):
    
    tileid = feature.get('id').split('tile_')[-1][:2]

    # download labels
    download_url = get_download_url(feature, 'labels', headers)
    download_label(download_url, output_path, tileid)
    
    #download field_ids
    download_url = get_download_url(feature, 'field_ids', headers)
    download_label(download_url, output_path, tileid)<jupyter_output>Downloaded 2_label.tif
Downloaded 2_field_id.tif
Downloaded 3_label.tif
Downloaded 3_field_id.tif
Downloaded 1_label.tif
Downloaded 1_field_id.tif
Downloaded 0_label.tif
Downloaded 0_field_id.tif<jupyter_text>Downloading ImageryThe imagery items associated with the tiles are linked within the links array of the tile metadata. Links which have a rel type of "source" are links to imagery items. By requesting the metadata for the imagery item you can retrieve download URLs for each band of the imagery.<jupyter_code># This cell downloads all the multi-spectral images throughout the growing season for this competition.
# The size of data is about 1.5 GB, and download time depends on your internet connection. 
# Note that you only need to run this cell and download the data once.
for feature in collection.get('features', []):
    for link in feature.get('links', []):
        if link.get('rel') != 'source':
            continue
            
        r = requests.get(link['href'], headers=headers)
        feature = r.json()
        assets = feature.get('assets').keys()
        tileid = feature.get('id').split('tile_')[-1][:2]
        date = datetime.strftime(datetime.strptime(feature.get('properties')['datetime'], "%Y-%m-%dT%H:%M:%SZ"), "%Y%m%d")
        for asset in assets:
            download_url = get_download_url(feature, asset, headers)
            download_imagery(download_url, output_path, tileid, date)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09374537926955494, 'fraction_numerical': 0.00783675883483661, 'mean_word_length': 4.526143790849673, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '148252', 'license': 'Apache-2.0', 'n_tokens_mistral': 1992, 'n_tokens_neox': 1911, 'n_words': 698, 'path': 'notebooks/2020 CV4A Crop Type Challenge/cv4a-crop-challenge-download-data.ipynb', 'repo_name': 'tamara-glazer/mlhub-tutorials', 'size': '6763'}