<commit_before>
..
  We avoid using bigger heading types as readthedocs annoyingly collapses all table-of-contents
  otherwise, which is extremely annoying for small projects.
  (there's a mountain of empty vertical space on almost all projects! why collapse?)

.. testsetup :: *

   integration_data_path = Path('../tests/integration/data').resolve()

   blue_geotiff_path = integration_data_path / 'LC08_L1TP_090084_20160121_20170405_01_T1/LC08_L1TP_090084_20160121_20170405_01_T1_B2.TIF'

   tmp_path = Path(tempfile.mkdtemp())

   collection = tmp_path / 'collection'
   collection.mkdir()

.. testcleanup :: *

   import shutil
   shutil.rmtree(tmp_path)


EO Datasets 3
-------------


EO Datasets aims to be the easiest way to write, validate and convert dataset imagery
and metadata for the `Open Data Cube`_

.. _Open Data Cube: https://github.com/opendatacube/datacube-core

There are two major tools for creating datasets:

 #. :ref:`DatasetAssembler<assembling_metadata>`, for writing a package: including writing the metadata
    document, `COG`_ imagery, thumbnails, checksum files etc.

 #. :ref:`DatasetPrepare<preparing_metadata>`, for preparing a metadata document,
    referencing existing imagery and files.

Their APIs are the same, except the assembler adds methods named ``write_*`` for writing new files.

.. note ::

   methods named ``note_*`` will note an existing file in the metadata, while ``write_*`` will write
   the file into the package too.

.. _assembling_metadata:

Assemble a Dataset Package
--------------------------

Here's a simple example of creating a dataset package with one measurement (called "blue") from an existing image.

The measurement is converted to a `COG`_ image when written to the package:

.. testcode ::

   from eodatasets3 import DatasetAssembler

   with DatasetAssembler(collection, naming_conventions='default') as p:
      p.product_family = "blues"

      # Date of acquisition (UTC if no timezone).
      p.datetime = datetime(2019, 7, 4, 13, 7, 5)
      # When the data was processed/created.
      p.processed_now() # Right now!
      # (If not newly created, set the date on the field: `p.processed = ...`)

      # Write our measurement from the given path, calling it 'blue'.
      p.write_measurement("blue", blue_geotiff_path)

      # Add a jpg thumbnail using our only measurement for the r/g/b bands.
      p.write_thumbnail("blue", "blue", "blue")

      # Complete the dataset.
      p.done()

.. note ::

   Note that until you call ``done()``, nothing will exist in the dataset's final output location. It is stored in a hidden temporary
   folder in the output directory, and renamed by ``done()`` once complete and valid.


.. _preparing_metadata:

Writing only a metadata doc
---------------------------

(ie. "I already have appropriate imagery files!")

Example of generating a metadata document with :class:`DatasetPrepare <eodatasets3.DatasetPrepare>`:

.. testcode ::

   collection_path = integration_data_path

.. testcode ::

   from eodatasets3 import DatasetPrepare

   usgs_level1 = collection_path / 'LC08_L1TP_090084_20160121_20170405_01_T1'
   metadata_path = usgs_level1 / 'odc-metadata.yaml'

   with DatasetPrepare(
       metadata_path=metadata_path,
   ) as p:
       p.product_family = "level1"
       p.datetime = datetime(2019, 7, 4, 13, 7, 5)
       p.processed_now()

       # Note the measurement in the metadata. (instead of ``write``)
       p.note_measurement('red',
          usgs_level1 / 'LC08_L1TP_090084_20160121_20170405_01_T1_B4.TIF'
       )

       # Or give the path relative to the dataset location
       # (eg. This will work unchanged on non-filesystem locations, such as ``s3://`` or tar files)
       p.note_measurement('blue',
          'LC08_L1TP_090084_20160121_20170405_01_T1_B2.TIF',
          relative_to_dataset_location=True
       )

       # Add links to other files included in the package ("accessories"), such as
       # alternative metadata files.
       [mtl_path] = usgs_level1.glob('*_MTL.txt')
       p.note_accessory_file('metadata:mtl', mtl_path)

       # Add whatever else you want.
       ...

       # Validate and write our metadata document!
       p.done()

   # We created a metadata file!
   assert metadata_path.exists()

..  testcleanup ::

   if metadata_path.exists():
       metadata_path.unlink()


Custom properties can also be set directly on ``.properties``::

      p.properties['fmask:cloud_cover'] = 34.0

And known properties are automatically normalised::

      p.platform = "LANDSAT_8"  # to: 'landsat-8'
      p.processed = "2016-03-04 14:23:30Z"  # into a date.
      p.maturity = "FINAL"  # lowercased
      p.properties["eo:off_nadir"] = "34"  # into a number


.. note ::

   By default, a validation error will be thrown if a referenced file lives outside the dataset (location),
   as this will require absolute paths. Relative paths are considered best-practice for Open Data Cube
   metadata.

   You can allow absolute paths in your metadata document using a field on construction
   (:meth:`DatasetPrepare() <eodatasets3.DatasetPrepare.__init__>`)::

       with DatasetPrepare(
          dataset_location=usgs_level1,
          allow_absolute_paths=True,
        ):
            ...

.. _COG: https://www.cogeo.org/


Including provenance
--------------------
Most datasets are processed from an existing input dataset and have the same spatial information as the input.
We can record them as source datasets, and the assembler can optionally copy any common metadata automatically::

   collection = Path('/some/output/collection/path')
   with DatasetAssembler(collection) as p:
      # We add a source dataset, asking to inherit the common properties
      # (eg. platform, instrument, datetime)
      p.add_source_path(level1_ls8_dataset_path, auto_inherit_properties=True)

      # Set our product information.
      # It's a GA product of "numerus-unus" ("the number one").
      p.producer = "ga.gov.au"
      p.product_family = "numerus-unus"
      p.dataset_version = "3.0.0"

      ...


In these situations, we often write our new pixels as a numpy array, inheriting the existing
:class:`grid spatial information <eodatasets3.GridSpec>` of our input dataset::

      # Write a measurement from a numpy array, using the source dataset's grid spec.
      p.write_measurement_numpy(
         "water",
         my_computed_numpy_array,
         GridSpec.from_dataset_doc(source_dataset),
         nodata=-999,
      )

Other ways to reference your source datasets:

- As an in-memory :class:`DatasetDoc <eodatasets3.DatasetDoc>` using :func:`p.add_source_dataset() <eodatasets3.DatasetPrepare.add_source_dataset>`
- Or as raw uuids, using :func:`p.note_source_datasets() <eodatasets3.DatasetPrepare.note_source_datasets>` (without property inheritance)


Creating documents in-memory
----------------------------

You may want to assemble metadata entirely in memory without touching the filesystem.

To do this, prepare a dataset as normal. You still need to give a dataset location, as paths
in the document will be relative to this location:

.. testsetup:: inmem

   from eodatasets3 import GridSpec

   import tempfile

   tmp_path = Path(tempfile.mkdtemp())

   grid_spec = GridSpec(shape=(7721, 7621),
      transform=Affine(30.0, 0.0, 241485.0, 0.0, -30.0, -2281485.0),
      crs=CRS.from_epsg(32656)
   )

   dataset_location = (tmp_path / 'test_dataset')
   measurement_path = dataset_location / "our_image_dont_read_it.tif"<commit_msg>Use readthedocs-compatible reSt syntax of bullets<commit_after>
..
  We avoid using bigger heading types as readthedocs annoyingly collapses all table-of-contents
  otherwise, which is extremely annoying for small projects.
  (there's a mountain of empty vertical space on almost all projects! why collapse?)

.. testsetup :: *

   integration_data_path = Path('../tests/integration/data').resolve()

   blue_geotiff_path = integration_data_path / 'LC08_L1TP_090084_20160121_20170405_01_T1/LC08_L1TP_090084_20160121_20170405_01_T1_B2.TIF'

   tmp_path = Path(tempfile.mkdtemp())

   collection = tmp_path / 'collection'
   collection.mkdir()

.. testcleanup :: *

   import shutil
   shutil.rmtree(tmp_path)


EO Datasets 3
-------------


EO Datasets aims to be the easiest way to write, validate and convert dataset imagery
and metadata for the `Open Data Cube`_

.. _Open Data Cube: https://github.com/opendatacube/datacube-core

There are two major tools for creating datasets:

 #. :ref:`DatasetAssembler<assembling_metadata>`, for writing a package: including writing the metadata
    document, `COG`_ imagery, thumbnails, checksum files etc.

 #. :ref:`DatasetPrepare<preparing_metadata>`, for preparing a metadata document,
    referencing existing imagery and files.

Their APIs are the same, except the assembler adds methods named ``write_*`` for writing new files.

.. note ::

   methods named ``note_*`` will note an existing file in the metadata, while ``write_*`` will write
   the file into the package too.

.. _assembling_metadata:

Assemble a Dataset Package
--------------------------

Here's a simple example of creating a dataset package with one measurement (called "blue") from an existing image.

The measurement is converted to a `COG`_ image when written to the package:

.. testcode ::

   from eodatasets3 import DatasetAssembler

   with DatasetAssembler(collection, naming_conventions='default') as p:
      p.product_family = "blues"

      # Date of acquisition (UTC if no timezone).
      p.datetime = datetime(2019, 7, 4, 13, 7, 5)
      # When the data was processed/created.
      p.processed_now() # Right now!
      # (If not newly created, set the date on the field: `p.processed = ...`)

      # Write our measurement from the given path, calling it 'blue'.
      p.write_measurement("blue", blue_geotiff_path)

      # Add a jpg thumbnail using our only measurement for the r/g/b bands.
      p.write_thumbnail("blue", "blue", "blue")

      # Complete the dataset.
      p.done()

.. note ::

   Note that until you call ``done()``, nothing will exist in the dataset's final output location. It is stored in a hidden temporary
   folder in the output directory, and renamed by ``done()`` once complete and valid.


.. _preparing_metadata:

Writing only a metadata doc
---------------------------

(ie. "I already have appropriate imagery files!")

Example of generating a metadata document with :class:`DatasetPrepare <eodatasets3.DatasetPrepare>`:

.. testcode ::

   collection_path = integration_data_path

.. testcode ::

   from eodatasets3 import DatasetPrepare

   usgs_level1 = collection_path / 'LC08_L1TP_090084_20160121_20170405_01_T1'
   metadata_path = usgs_level1 / 'odc-metadata.yaml'

   with DatasetPrepare(
       metadata_path=metadata_path,
   ) as p:
       p.product_family = "level1"
       p.datetime = datetime(2019, 7, 4, 13, 7, 5)
       p.processed_now()

       # Note the measurement in the metadata. (instead of ``write``)
       p.note_measurement('red',
          usgs_level1 / 'LC08_L1TP_090084_20160121_20170405_01_T1_B4.TIF'
       )

       # Or give the path relative to the dataset location
       # (eg. This will work unchanged on non-filesystem locations, such as ``s3://`` or tar files)
       p.note_measurement('blue',
          'LC08_L1TP_090084_20160121_20170405_01_T1_B2.TIF',
          relative_to_dataset_location=True
       )

       # Add links to other files included in the package ("accessories"), such as
       # alternative metadata files.
       [mtl_path] = usgs_level1.glob('*_MTL.txt')
       p.note_accessory_file('metadata:mtl', mtl_path)

       # Add whatever else you want.
       ...

       # Validate and write our metadata document!
       p.done()

   # We created a metadata file!
   assert metadata_path.exists()

..  testcleanup ::

   if metadata_path.exists():
       metadata_path.unlink()


Custom properties can also be set directly on ``.properties``::

      p.properties['fmask:cloud_cover'] = 34.0

And known properties are automatically normalised::

      p.platform = "LANDSAT_8"  # to: 'landsat-8'
      p.processed = "2016-03-04 14:23:30Z"  # into a date.
      p.maturity = "FINAL"  # lowercased
      p.properties["eo:off_nadir"] = "34"  # into a number


.. note ::

   By default, a validation error will be thrown if a referenced file lives outside the dataset (location),
   as this will require absolute paths. Relative paths are considered best-practice for Open Data Cube
   metadata.

   You can allow absolute paths in your metadata document using a field on construction
   (:meth:`DatasetPrepare() <eodatasets3.DatasetPrepare.__init__>`)::

       with DatasetPrepare(
          dataset_location=usgs_level1,
          allow_absolute_paths=True,
        ):
            ...

.. _COG: https://www.cogeo.org/


Including provenance
--------------------
Most datasets are processed from an existing input dataset and have the same spatial information as the input.
We can record them as source datasets, and the assembler can optionally copy any common metadata automatically::

   collection = Path('/some/output/collection/path')
   with DatasetAssembler(collection) as p:
      # We add a source dataset, asking to inherit the common properties
      # (eg. platform, instrument, datetime)
      p.add_source_path(level1_ls8_dataset_path, auto_inherit_properties=True)

      # Set our product information.
      # It's a GA product of "numerus-unus" ("the number one").
      p.producer = "ga.gov.au"
      p.product_family = "numerus-unus"
      p.dataset_version = "3.0.0"

      ...


In these situations, we often write our new pixels as a numpy array, inheriting the existing
:class:`grid spatial information <eodatasets3.GridSpec>` of our input dataset::

      # Write a measurement from a numpy array, using the source dataset's grid spec.
      p.write_measurement_numpy(
         "water",
         my_computed_numpy_array,
         GridSpec.from_dataset_doc(source_dataset),
         nodata=-999,
      )

Other ways to reference your source datasets:

* As an in-memory :class:`DatasetDoc <eodatasets3.DatasetDoc>` using :func:`p.add_source_dataset() <eodatasets3.DatasetPrepare.add_source_dataset>`
* Or as raw uuids, using :func:`p.note_source_datasets() <eodatasets3.DatasetPrepare.note_source_datasets>` (without property inheritance)


Creating documents in-memory
----------------------------

You may want to assemble metadata entirely in memory without touching the filesystem.

To do this, prepare a dataset as normal. You still need to give a dataset location, as paths
in the document will be relative to this location:

.. testsetup:: inmem

   from eodatasets3 import GridSpec

   import tempfile

   tmp_path = Path(tempfile.mkdtemp())

   grid_spec = GridSpec(shape=(7721, 7621),
      transform=Affine(30.0, 0.0, 241485.0, 0.0, -30.0, -2281485.0),
      crs=CRS.from_epsg(32656)
   )

   dataset_location = (tmp_path / 'test_dataset')
   measurement_path = dataset_location / "our_image_dont_read_it.tif"
{'fraction_non_alphanumeric': 0.10284391534391535, 'fraction_numerical': 0.03267195767195767, 'mean_word_length': 3.2249231628946635, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1594177', 'n_tokens_mistral': 4884, 'n_tokens_neox': 4459, 'n_words': 1478}