<jupyter_start><jupyter_text>Data IO (input/output) IntroductionESRF data (used to) come in (too many) different formats:* Specfile, EDF, HDF5* And specific detector formats: MarCCD, Pilatus CBF, Dectris Eiger, …HDF5 is now the standard ESRF data format so we will only focus on it today.Methods for accessing other file formats are described in the [io_spec_edf.ipynb](io_spec_edf.ipynb) notebook.  HDF5 What is HDF5 ?[HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) (for Hierarchical Data Format) is a file format to structure and store complex and large volumes of data. Why HDF5 ?* Hierarchical collection of data (directory and file, UNIX-like path)* High-performance (binary)* Portable file format (Standard exchange format for heterogeneous data)* Self-describing extensible types, rich metadata* Support data compression* Free ( & open source)* Adopted by a large number of institutes (NASA, LIGO, ...)* Adopted by most of the synchrotrons (ESRF, SOLEIL, Desy...)**Data can be mostly anything: image, table, graphs, documents**  HDF5 descriptionThe container is mostly structured with:* **File**: the root of the container* **Group**: a grouping structure containing groups or datasets* **Dataset**: a multidimensional array of data elements* And other features (links, attributes, datatypes, virtual datasets)![hdf5_class_diag](images/hdf5_model.png "hdf5 class diagram")  HDF5 exampleHere is an example of the file generated by [pyFAI](https://github.com/silx-kit/pyFAI)![hdf5_example](images/hdf5_example.png "hdf5 example")  Useful tools for HDF5 [HDFGroup tools](https://portal.hdfgroup.org/display/HDF5/HDF5+Tools+by+Category)Command line and desktop application: `h5ls`, `h5dump`, `hdfview````bash>>> h5ls -r my_first_one.h5    /                        Group    /data1                   Dataset {100, 100}    /group1                  Group    /group1/data2            Dataset {100, 100}```  [`h5glance`](https://github.com/European-XFEL/h5glance)Jupyter notebook and command line tool for browsing HDF5 files<jupyter_code># From jupyter notebook
from h5glance import H5Glance
H5Glance("data/water.h5")
%%bash
# From the command line
h5glance data/water.h5<jupyter_output><empty_output><jupyter_text>[`jupyterlab-h5web`](https://github.com/silx-kit/jupyterlab-h5web/)JupyterLab HDF5 file browser and viewer[Go to JupyterLab](/lab)<jupyter_code>from jupyterlab_h5web import H5Web

H5Web("data/water.h5")<jupyter_output><empty_output><jupyter_text>[`silx view`](http://www.silx.org/doc/silx/latest/applications/view.html)Desktop application file browser/viewer```bash>>> pip install silx>>> silx view my_file.h5```<jupyter_code>%%bash
# With silx view GUI
silx view data/water.h5<jupyter_output><empty_output><jupyter_text>h5py![h5py book](images/h5py.png "h5py book")[h5py](https://www.h5py.org/) is the python binding for accessing HDF5 files. Originally from [<NAME>](http://shop.oreilly.com/product/0636920030249.do) **Warning**: A major version of h5py has been released at the end of 2020 (3.0).Some code might not be compatible with it yet (end of Python2 support, [interface for storing & reading strings has changed](https://docs.h5py.org/en/stable/strings.html)...).<jupyter_code>import h5py

print('h5py:', h5py.version.version)
print('hdf5:', h5py.version.hdf5_version)<jupyter_output><empty_output><jupyter_text>How to read a HDF5 file from Python `h5py.File`* First open the file with [h5py.File](http://docs.h5py.org/en/stable/high/file.html):  ```  h5py.File('myfile.hdf5', mode)  ```  [opening modes](http://docs.h5py.org/en/stable/high/file.htmlopening-creating-files):| Mode    | Meaning                                                            ||---------|--------------------------------------------------------------------|| r       | Readonly, file must exist; *Default with h5py* **v3**              || r+      | Read/write, file must exist                                        || w       | Create file, truncate if exists                                    || w- or x | Create file, fail if exists                                        || a       | Read/write if exists, create otherwise; *Default with h5py* **v2** |<jupyter_code>import h5py

h5file = h5py.File('data/water.h5', 'r')<jupyter_output><empty_output><jupyter_text>`h5py.Group`* Then access the file content with a dictionary-like API, [h5py.Group](http://docs.h5py.org/en/stable/high/group.html):  - [`Group.keys()`](https://docs.h5py.org/en/stable/high/group.htmlh5py.Group.keys)  - [`Group.items()`](https://docs.h5py.org/en/stable/high/group.htmlh5py.Group.items)  - [`Group.values()`](https://docs.h5py.org/en/stable/high/group.htmlh5py.Group.values)<jupyter_code># Available names at the first level
list(h5file.keys())
# List 'entry_0000' group children
group = h5file['entry_0000']
dict(group.items())
# List 'entry_0000/4_azimuthal_integration' group children
list(group["4_azimuthal_integration"].values())
# List 'entry_0000/4_azimuthal_integration/results' group children
list(h5file["/entry_0000/4_azimuthal_integration/results"].values())
# Get a dataset from a sub group
h5dataset = h5file['/entry_0000/4_azimuthal_integration/results/I']
h5dataset<jupyter_output><empty_output><jupyter_text>Not very convenient for interactive browsing...  `h5py.Dataset`It mimics `numpy.ndarray`.The data is read from the file only when it is needed.<jupyter_code>h5dataset = h5file['/entry_0000/4_azimuthal_integration/results/I']
h5dataset
# Here we only read metadata from the dataset
print("Dataset:", h5dataset.shape, h5dataset.dtype, h5dataset.size)
# Read data from the file to a numpy.ndarray
subset = h5dataset[:5]  # Copy the selection to a numpy.ndarray
print('subset:', subset, '=> sum:', subset.sum())

data = h5dataset[()]  # Copy the whole dataset to a numpy.ndarray
print('data type:', type(data), '; shape', data.shape, '; min.:', data.min())
data[:5] = 0
print(data[:5])
print(h5dataset[:5])
# Remember to close the file
h5file.close()
print(data[:5])
print(subset)
# Once the file is closed, the Dataset no longer gives access to data
print(h5dataset)
print(h5dataset[:5])<jupyter_output><empty_output><jupyter_text>Context manager* Context managers guarantee that resources are released. In our case, it ensures that the HDF5 file is closed.* Usually used from the `with` statement.To safely access a HDF5 file, do:<jupyter_code>with h5py.File('data/water.h5', "r") as h5file:
    dataset = h5file['/entry_0000/4_azimuthal_integration/results/I']
    data = dataset[()]
print(dataset)
print(data[:5])<jupyter_output><empty_output><jupyter_text>How to write to a HDF5 file*There are several ways for writing groups and datasets. Here we will mostly focus on the 'dictionary-like' API.*Documentation: [Group](http://docs.h5py.org/en/stable/high/group.html) and [Dataset](http://docs.h5py.org/en/stable/high/dataset.html)<jupyter_code>import numpy
import h5py

data = numpy.random.random(10000)
data.shape = 100, 100

# create file, notice the mode='w', as 'write'
h5file = h5py.File('my_first_one.h5', mode='w')

# write data into a dataset from the root
h5file['/data1'] = data

# write data into a dataset from group1
h5file['/group1/data2'] = data

h5file.close()<jupyter_output><empty_output><jupyter_text>The same operation with a context manager:<jupyter_code>import numpy
import h5py

# Create 2D data
data = numpy.arange(100 * 100)
data.shape = 100, 100

with h5py.File('my_first_one.h5', mode='w') as h5file:

    # write data into a dataset from the root
    h5file['/data1'] = data

    # write data into a dataset from group1
    h5file['/group1/data2'] = data<jupyter_output><empty_output><jupyter_text>With [Group.create_dataset](https://docs.h5py.org/en/stable/high/group.htmlh5py.Group.create_dataset) and [Group.create_group](https://docs.h5py.org/en/stable/high/group.htmlh5py.Group.create_group):<jupyter_code>import numpy
import h5py

# Create 2D data
data = numpy.arange(100 * 100)
data.shape = 100, 100

# Notice the mode='w', as 'write'
with h5py.File('my_first_one.h5', mode='w') as h5file:

    # write data into a dataset from the root
    h5file.create_dataset('data1', data=data)

    # Or with a functional API
    grp1 = h5file.create_group("group1")
    grp1.create_dataset("data2", data=data)<jupyter_output><empty_output><jupyter_text>Exercice: Flat field correctionFlat-field correction is a technique used to improve quality in digital imaging.The goal is to normalize images and remove artifacts caused by variations in the pixel-to-pixel sensitivity of the detector and/or by distortions in the optical path. (see https://en.wikipedia.org/wiki/Flat-field_correction)$$ normalized = \frac{raw - dark}{flat - dark} $$* `normalized`: Image after flat field correction* `raw`: Raw image. It is acquired with the sample.* `flat`: Flat field image. It is the response given out by the detector for a uniform input signal. This image is acquired without the sample.* `dark`: Also named `background` or `dark current`. It is the response given out by the detector when there is no signal. This image is acquired without the beam. Here is a function implementing the flat field correction:*Note: make sure you execute the cell for defining this function*<jupyter_code>import numpy

def flatfield_correction(raw, flat, dark):
    """
    Apply a flat-field correction to a raw data using a flat and a dark.
    """
    # Make sure that the computation is done using float
    # to avoid type overflow or loss of precision
    raw = raw.astype(numpy.float32)
    flat = flat.astype(numpy.float32)
    dark = dark.astype(numpy.float32)
    # Do the computation
    return (raw - dark) / (flat - dark)<jupyter_output><empty_output><jupyter_text>**Note**: If you like to plot an image you can use `matplotlib`'s `imshow` function.The `%matplotlib` "magic" command should be called once first.<jupyter_code>%matplotlib inline

from matplotlib import pyplot as plt
import numpy
plt.imshow(numpy.random.random((20, 60)))<jupyter_output><empty_output><jupyter_text>Exercise 11. Browse the file ``data/ID16B_diatomee.h5``2. Get a single raw dataset, a flat field dataset and a dark image dataset from this file3. Apply the flat field correction4. Save the result into a new HDF5 fileIf you are stuck, the solution is provided in the file [solutions/exercise1.py](./solutions/exercise1.py)<jupyter_code>from jupyterlab_h5web import H5Web
H5Web("data/ID16B_diatomee.h5")
# or
from h5glance import H5Glance
H5Glance("data/ID16B_diatomee.h5")
import h5py

with h5py.File("data/ID16B_diatomee.h5", mode="r") as h5s:
    pass
    # this is a comment

    # step1: Read the data

    # raw_data_path = ...
    # raw_data = ...

    # flat_path = ...
    # flat = ...

    # dark_path = ...
    # dark = ...

# step2: Compute the result

# normalized = flatfield_correction(raw_data, flat, dark)

# step3: Save the result

# ...<jupyter_output><empty_output><jupyter_text>Exercise 21. Apply the flat field correction to all raw data available (use the same flat and dark for all the images)2. Save each result into different datasets of the same HDF5 fileIf you are stuck, the solution is provided in the file [solutions/exercise2.py](./solutions/exercise2.py)  Exercise 3From the previous exercise, we can see that the flat field correction was not very good for the last images.Another flat field was acquired at the end of the acquisition.We could use this information to compute a flat field closer to the image we want to normalize. It can be done with a linear interpolation of the flat images by using the name of the image as the interpolation factor (which varies between 0 and 500 in this case).1. For each raw data, compute the corresponding flat field using lineal interpolation (between `flatfield/0000` and `flatfield/0500`)2. Save each result into different datasets in a single HDF5 fileIf you are stuck, the solution is provided in the file [solutions/exercise3.py](./solutions/exercise3.py)  Advanced usage  Dataset compressionTL;DR: Install [hdf5plugin](https://github.com/silx-kit/hdf5plugin) and `import hdf5plugin`.HDF5 provides dataset compression support.With `h5py` GZIP and LZF compression are available (see [compression-filters](https://docs.h5py.org/en/stable/high/dataset.htmllossless-compression-filters)).Yet, there are many [third-party compression filters for HDF5](https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins) available.[hdf5plugin](https://github.com/silx-kit/hdf5plugin) allows usage of some of those compression filters with `h5py` (blosc, bitshuffle, lz4, FCIDECOMP, ZFP).<jupyter_code>import h5py
import hdf5plugin  # Allows to read dataset stored with supported compressions<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11872935996225821, 'fraction_numerical': 0.026812391885516592, 'mean_word_length': 4.447109207708779, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 72, 'https://': 17, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '635948', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4216, 'n_tokens_neox': 4022, 'n_words': 1239, 'path': 'python/io/io_hdf5.ipynb', 'repo_name': 'silx-kit/silx-training', 'size': '12727'}