<jupyter_start><jupyter_text>Creating Database FileYou can use cli tool `slurpy` to export a set of products to a file db. But if you need more control over what goes into the cache see below. Create new file dbCreate new database, deleting any previous files that might have existed.<jupyter_code>db_name = 'sample.db'
cache = dscache.create_cache(db_name, truncate=True)<jupyter_output><empty_output><jupyter_text>Get some datasets from DatacubeWe are limiting to 200 for example purposes, also there is an outstanding [issue ( 542)](https://github.com/opendatacube/datacube-core/issues/542) with `find_datasets_lazy`, it's not actually "lazy" the whole SQL query is processed as a whole.<jupyter_code>dc = datacube.Datacube(env='s2')
dss = dc.find_datasets_lazy(product='s2a_nrt_granule', limit=200)<jupyter_output><empty_output><jupyter_text>Write them to file dbDataset cache provides a convenience method `.tee` that accepts dataset stream on input and generates same stream on output, but also saves datasets to the file db.<jupyter_code>dss = cache.tee(dss)<jupyter_output><empty_output><jupyter_text>Then you can just iterate over all datasets doing whatever other thing you needed to do.<jupyter_code>%%time
for i, ds in enumerate(dss):
    if (i % 10) == 0:
        print('.', end='', flush=True)
print()
print('done')<jupyter_output>....................
done
CPU times: user 169 ms, sys: 5.22 ms, total: 174 ms
Wall time: 200 ms<jupyter_text>**NOTE:**`.tee` assumes that all datasets will be consumed, internally it breaks up dataset stream into transactions, it's not super clear what happens if you just stop half way through a transaction and never continue. Eventually transaction will be garbage collected and data written to disk, but in the meantime any writes will be blocked. So if you do exit early without consuming whole stream you should probably call `del dss` as soon as practical.  Alternative way<jupyter_code>%%time
dss = dc.find_datasets_lazy(product='s2b_nrt_granule', limit=200)
cache.bulk_save(dss) # blocks until all are written (in one single transaction)
cache.count, cache.products, cache.metadata
cache.close()<jupyter_output><empty_output><jupyter_text>Bin datasets into Albers TileThere is a cli tool `dstiler` that will go through all datasets in the file and bin them into various tiling regimes. Default regime is 100k side Albers tiles (same as on NCI). But there is also "native" for landsat scenes and "web".<jupyter_code>!dstiler sample.db<jupyter_output>Total bins: 421
[?25lSaving  [####################################]  100%[?25h<jupyter_text>Reading from file db<jupyter_code>cache_ro = dscache.open_ro(db_name)
cache_ro.count, cache_ro.metadata, cache_ro.products<jupyter_output><empty_output><jupyter_text>Stream datasets into RAM: `.get_all()`<jupyter_code>dss = list(islice(cache_ro.get_all(), 10))
type(dss[0]), type(dss[0].type), type(dss[0].type.metadata_type)<jupyter_output><empty_output><jupyter_text>Access individual dataset by UUID<jupyter_code>cache_ro.get('0146ea8e-8462-4fca-a880-7f4a311441bf')<jupyter_output><empty_output><jupyter_text>Working with Groups<jupyter_code>from odc.dscache.tools.tiling import parse_group_name

groups = cache_ro.groups()
len(groups), groups[:3]
group_name, count = groups[1]
group_name, parse_group_name(group_name)<jupyter_output><empty_output><jupyter_text>Read all UUIDs for a given group<jupyter_code>cache_ro.get_group(group_name)<jupyter_output><empty_output><jupyter_text>Load Dataset documents for a given group<jupyter_code>list(cache_ro.stream_group(group_name))<jupyter_output><empty_output><jupyter_text>Interfacing with Datacube's `GridWorkflow.load(..)`There is a helper class that can construct `datacube.mode.Tile`.<jupyter_code>from odc.dscache.tools import DcTileExtract

tiles = DcTileExtract(cache_ro)
tile_id,_ = parse_group_name(group_name)

tile = tiles(tile_id)

tile_id, tile<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10882278159166031, 'fraction_numerical': 0.018815153826595474, 'mean_word_length': 6.947474747474748, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '230974', 'license': 'Apache-2.0', 'n_tokens_mistral': 1360, 'n_tokens_neox': 1285, 'n_words': 374, 'path': 'notebooks/dscache-example.ipynb', 'repo_name': 'zidingz/odc-tools', 'size': '3933'}