<jupyter_start><jupyter_text>Asset Scanner Example Notebook  Asset ExtractorThe Asset Extractor can be used to extract metadata about assets.<jupyter_code>!asset_scanner conf/extract-assets.yaml<jupyter_output>2021-11-29 15:02:13,577 asset_scanner.plugins.input_plugins.intake_esm_input INFO Opening catalog
2021-11-29 15:02:21,092 asset_scanner.plugins.input_plugins.intake_esm_input INFO Found 2 items
2021-11-29 15:02:21,236 asset_generator.media_handlers.object_store_handler INFO Extracting metadata for: https://cmip6-zarr-o.s3-ext.jc.rl.ac.uk/CMIP6.CMIP.MOHC.UKESM1-0-LL/historical.r12i1p1f2.Amon.tas.gn.v20191210.zarr with checksum: None
{'id': '0b747f6213efc1b5cd404e7050ce2809', 'body': {'media_type': 'OBJECT_STORE', 'location': 'https://cmip6-zarr-o.s3-ext.jc.rl.ac.uk/CMIP6.CMIP.MOHC.UKESM1-0-LL/historical.r12i1p1f2.Amon.tas.gn.v20191210.zarr', 'filename': 'historical.r12i1p1f2.Amon.tas.gn.v20191210.zarr', 'extension': '.zarr', 'categories': ['data']}}
2021-11-29 15:02:21,413 asset_generator.media_handlers.object_store_handler INFO Extracting metadata for: https://cmip6-zarr-o.s3-ext.jc.rl.ac.uk/CMIP6.CMIP.MOHC.UKESM1-0-LL/historical.r4i1p1f2.Amon.tas.gn.v20190502.zarr with checksum: N[...]<jupyter_text>Item GeneratorThe item generator is used to extract facets from assets. This can then be used to construct a STAC Item.<jupyter_code>!asset_scanner conf/extract-items.yaml<jupyter_output>2021-11-29 15:02:22,379 asset_scanner.plugins.input_plugins.intake_esm_input INFO Opening catalog
2021-11-29 15:02:29,277 asset_scanner.plugins.input_plugins.intake_esm_input INFO Found 2 items
2021-11-29 15:02:29,278 item_generator.core.facet_extractor INFO Processing: https://cmip6-zarr-o.s3-ext.jc.rl.ac.uk/CMIP6.CMIP.MOHC.UKESM1-0-LL/historical.r12i1p1f2.Amon.tas.gn.v20191210.zarr
2021-11-29 15:02:29,296 item_generator.extraction_methods.regex_extract INFO Found matches for regex extract
{'id': '0b747f6213efc1b5cd404e7050ce2809', 'body': {'item_id': '0b747f6213efc1b5cd404e7050ce2809', 'type': 'item', 'properties': {'mip_era': 'CMIP6', 'activity_id': 'CMIP', 'institution_id': 'MOHC', 'source_id': 'UKESM1-0-LL', 'experiment_id': 'historical', 'member_id': 'r12i1p1f2', 'table_id': 'Amon', 'variable_id': 'tas', 'grid_label': 'gn', 'version': 'v20191210'}}}
{'id': '<KEY>', 'body': {'item_id': '0b747f6213efc1b5cd404e7050ce2809'}}
2021-11-29 15:02:29,297 it[...]
{'fraction_non_alphanumeric': 0.15162907268170425, 'fraction_numerical': 0.14160401002506265, 'mean_word_length': 6.651757188498403, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '194067', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1198, 'n_tokens_neox': 1009, 'n_words': 148, 'path': 'example_notebook.ipynb', 'repo_name': 'matprov/asset-scanner-example', 'size': '2421'}