<jupyter_start><jupyter_text>HLS - DEVSteps:* scene selection - TODO* convert HDF to TIF* creat clear-sky band from QA<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline

import geopandas as gpd
import pandas as pd
from pathlib import Path
import rasterio

import nasa_hls

from src import configs
prjconf = configs.ProjectConfigParser()
tilenames = prjconf.get("Params", "tiles").split(" ")<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Scene selection - TODO**TODO: Find a solution for the following problem. By now we just process all downloaded files.***Here we first downloaded all the data to get the cloud cover and spatial coverage.Now we only want to process scenes which have a cloud cover lower than and a spatial coverage higher than a specific thresold.To keep this easy with snakemake, i.e. to be able to work with simple wildcards, we first create a rule which creates the output directories of the datasets we want to create.Then we can use the output directories as input to the actual processing rule.(**Note:** There is a section about [data-dependent conditional execution](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.htmldata-dependent-conditional-execution) which might (?) also be an option.)<jupyter_code>tile = "32UNU"

basedir__hls_tif = prjconf.get_path("Interim", "hls") / tile

prjconf = configs.ProjectConfigParser()

df_scenes = pd.read_csv(prjconf.get_path("Raw", "hls_tile_lut", tile=tile))

max_cloud_cover = prjconf.get("Params", "max_cloud_cover")
min_spatial_cover = prjconf.get("Params", "min_spatial_cover")


df_scenes_sel = df_scenes[(df_scenes["cloud_cover"] <= float(max_cloud_cover)) & \
                          (df_scenes["cloud_cover"] >= float(min_spatial_cover))]
for sid in df_scenes_sel["sceneid"]:
    basedir_hls_scene = basedir__hls_tif / sid
    if not (basedir_hls_scene).exists():
        basedir_hls_scene.mkdir()<jupyter_output><empty_output><jupyter_text>Convert HDF to TIF An HDF file is a stack of bands. Each HDF will be converted to several single layer TIF files.Input file with wildcards and example:    data/raw/hls/{tile}/{sceneid}.hdf        data/raw/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4.hdfOutput files with wildcards and examples:    data/interim/hls/{tile}/{sceneid}/{sceneid}__{band}.tif        data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__Red.tif    data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__NIR.tif    data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__SWIR1.tif    data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__SWIR2.tif    data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__QA.tifHowever it is easier to specify just the output folder as ouput.    directory("data/interim/hls/{tile}/{sceneid}")        data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4**TODO**: Change ouput to specify all files. How to do this correctly? Rule developement.<jupyter_code>from pathlib import Path
from nasa_hls import convert_hdf2tiffs

hdf_path = prjconf.get_path("Raw", "hls") / "32UNU" / "HLS.L30.T32UNU.2018003.v1.4.hdf" 
# => in the script: snakemake.input[0]
bands = prjconf.get("Params", "bands").split(" ") 
# => in the script: snakemake.params.bands
dir__hls_tif = prjconf.get_path("Interim", "hls") / "32UNU" / "HLS.L30.T32UNU.2018003.v1.4" 
# => in the script: snakemake.output[0]

convert_hdf2tiffs(hdf_path=hdf_path, 
                  dstdir=Path(dir__hls_tif).parent, 
                  bands=bands, 
                  max_cloud_coverage=100)<jupyter_output><empty_output><jupyter_text>Creat clear-sky bandMore details on the conversion from QA-layer to clear sky masks can be found in [nasa_hls’s documentation - Create a clear-sky mask from the QA layer](https://benmack.github.io/nasa_hls/build/html/tutorials/Working_with_HLS_datasets_and_nasa_hls.htmlCreate-a-clear-sky-mask-from-the-QA-layer).Input file with wildcards and example:    data/interim/hls/{tile}/{sceneid}/{sceneid}__QA.tif        data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__QA.tifOutput files with wildcards and examples:    data/interim/hls/{tile}/{sceneid}/{sceneid}__CLEAR.tif        data/interim/hls/32UNU/HLS.L30.T32UNU.2018003.v1.4/HLS.L30.T32UNU.2018003.v1.4__CLEAR.tif    Rule developement.<jupyter_code>from nasa_hls import hls_qa_layer_to_mask

path__qa = prjconf.get_path("Interim", "hls") / "32UNU" / "HLS.L30.T32UNU.2018003.v1.4" / "HLS.L30.T32UNU.2018003.v1.4__QA.tif"
# => in the script: snakemake.input[0]
path__clear = prjconf.get_path("Interim", "hls") / "32UNU" / "HLS.L30.T32UNU.2018003.v1.4" / "HLS.L30.T32UNU.2018003.v1.4__CLEAR.tif"
# => in the script: snakemake.output[0]

# valid ids from the link above
valid = [  0,   4,  16,  20,  32,  36,  48,  52,  64,  68,  80,  84,  96,
         100, 112, 116, 128, 132, 144, 148, 160, 164, 176, 180, 192, 196,
         208, 212, 224, 228, 240, 244]

clear = hls_qa_layer_to_mask(qa_path=path__qa,
                             qa_valid=valid,
                             keep_255=True,
                             mask_path=path__clear,
                             overwrite=False)
assert clear == 0<jupyter_output>Processing skipped. File exists.
{'fraction_non_alphanumeric': 0.11982855013045099, 'fraction_numerical': 0.07659336563548266, 'mean_word_length': 3.937442502299908, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '545344', 'license': 'MIT', 'n_tokens_mistral': 2331, 'n_tokens_neox': 2041, 'n_words': 418, 'path': 'notebooks/01_raw/04a-r2i-dev_bm_hls-dev.ipynb', 'repo_name': 'benmack/classify-hls', 'size': '5366'}