<jupyter_start><jupyter_text>Brought to you by the dynamic trio  Reading the Root Group's Object HeaderThe primitives defined in the [HDF5 File Format Specification](https://www.hdfgroup.org/HDF5/doc/H5.format.html) are organized into three levels.> Level 0 contains basic information for identifying and defining information about the file. Level 1 information contains the information about the pieces of a file shared by many objects in the file (such as a B-trees and heaps). Level 2 is the rest of the file and contains all of the data objects, with each object partitioned into header information, also known as *metadata*, and data.In this installment, we skip level 1, and get a preview of a level 2 primitve. The superblock (level 0) contains the address of the root group's [*object header*](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlDataObject) (OHDR). What appears from the perspective of the HDF5 library as a collection of links, is represented in the file as a conglomerate of [*object header messages*](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlObjectHeaderMessages) and supporting level 1 primitives. There are about two dozen types of messages, which make up the "vocabulary" of an "object description language." Not all message combinations are grammatically correct, for example, an object header in which a [Group Info Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlGroupInfoMessage) were followed by a [Dataspace Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlDataspaceMessage) would be as nonsensical as *Quadruplicity drinks procrastination.* (<NAME>)We go through the usual motions to create a simple test file:<jupyter_code>(defparameter *file-name* "root-ohdr.h5")
(let ((fcpl (h5pcreate +H5P-FILE-CREATE+)))
     (h5pset-userblock fcpl (cffi:convert-to-foreign 524288 'hsize-t))
     (h5fclose (h5fcreate *file-name* +H5F-ACC-TRUNC+ fcpl +H5P-DEFAULT+))
     (h5pclose fcpl))
(defparameter *input-stream*
    (open *file-name* :direction :input :element-type '(unsigned-byte 8)))
(defparameter *file* (make-file *file-name*))
(add-superblock *input-stream* *file*)
(values)<jupyter_output><empty_output><jupyter_text>The address of the root group's object header is stored in the file superblock.<jupyter_code>(get-root-ohdr-address *file*)<jupyter_output><empty_output><jupyter_text>Except for the role of being the *origin* or base node of an HDF5 multi-graph, there is nothing special about the root group. It's an HDF5 group. In the `alcove` package, we have defined a simple function, `READ-OHDR`, for reading HDF5 object headers.<jupyter_code>(documentation 'read-ohdr 'function)
(loop for x in (read-ohdr *input-stream* (get-root-ohdr-address *file*) *file*)
   do (format t "~a~%" x))<jupyter_output>(OHDR-VERSION . 1)
(MESSAGE-COUNT . 1)
(REFERENCE-COUNT . 1)
(HEADER-SIZE . 24)
(MESSAGES
  ((MSG-TYPE . 17) (MSG-DATA-SIZE . 16) (MSG-FLAGS . 0)
   (MSG-DATA (NAME . Symbol Table) (V1-B-TREE-ADDRESS . 136)
    (LOCAL-HEAP-ADDRESS . 680))))<jupyter_text>As most other primitives in the HDF5 file format, the object header is versioned. There's only one message stored in this particular object header, and the HDF5 object has a reference count of 1. The message is a [Symbol Table Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlSymbolTableMessage), which tells us that the corresponding HDF5 object is indeed an (old-style) HDF5 group. Information about the links in the group is stored in two places: A B-tree manages references to link names and descriptions of link destinations, and the actual names and descriptions are stored on a local heap. We'll dig into reading those items in the next installment.<jupyter_code>(close *input-stream*)<jupyter_output><empty_output><jupyter_text>The Root Group's Object Header Version 2Let's take a look at the object header of a "newer style" HDF5 group.<jupyter_code>(setq *file-name* "root-ohdr.v2.h5")
(let ((fapl (h5pcreate +H5P-FILE-ACCESS+))
      (fcpl (h5pcreate +H5P-FILE-CREATE+)))
  (h5pset-libver-bounds fapl :H5F-LIBVER-LATEST :H5F-LIBVER-LATEST)
  (h5pset-userblock fcpl (cffi:convert-to-foreign 1024 'hsize-t))
  (h5fclose (h5fcreate *file-name* +H5F-ACC-TRUNC+ fcpl fapl))
  (h5pclose fcpl)
  (h5pclose fapl))
(setq *input-stream*
      (open *file-name* :direction :input :element-type '(unsigned-byte 8)))
(setq *file* (make-file *file-name*))
(add-superblock *input-stream* *file*)
(values)
(loop for x in (read-ohdr *input-stream* (get-root-ohdr-address *file*) *file*)
   do (format t "~a~%" x))<jupyter_output>(SIGNATURE . OHDR)
(OHDR-VERSION . 2)
(FLAGS . 32)
(ACCESS-TIME . 1457828958)
(MODIFICATION-TIME . 1457828958)
(CHANGE-TIME . 1457828958)
(BIRTH-TIME . 1457828958)
(SIZE-OF-CHUNK0 . 120)
(MESSAGES
  ((MSG-TYPE . 0) (MSG-DATA-SIZE . 88) (MSG-FLAGS . 0)
   (MSG-DATA (NAME . NIL)
    (DATA
     . #(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
         0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
         0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0))))
  ((MSG-TYPE . 10) (MSG-DATA-SIZE . 2) (MSG-FLAGS . 1)
   (MSG-DATA (NAME . Group Info) (VERSION . 0) (FLAGS . 0)))
  ((MSG-TYPE . 2) (MSG-DATA-SIZE . 18) (MSG-FLAGS . 0)
   (MSG-DATA (NAME . Link Info) (VERSION . 0) (FLAGS . 0)
    (FRACTAL-HEAP-ADDRESS . 18446744073709551615)
    (NAME-INDEX-V2-B-TREE-ADDRESS . 18446744073709551615))))
(CHECKSUM . #(215 128 248 180))<jupyter_text>There's a lot more going on in a [version 2 object header](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlV2ObjectHeaderPrefix)! This version saw the introduction of the following:- a signature- time stamps- the support for arbitrarily large attributes- facilities for the tracking and indexing of attributes by creation time- an object header checksumIn this particular case, the time stamps are stored, as is confirmed by a flag value of 32 (bit 5 is set). There are three messages stored in this object header.1. The [NIL Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlNILMessage) can be ignored (and has nothing to do with the Common Lisp constant `NIL`). 2. The [Group Info Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlGroupInfoMessage) confirms that we are dealing with a "new style" HDF5 group.3. The [Link Info Message](https://www.hdfgroup.org/HDF5/doc/H5.format.htmlLinkInfoMessage) stores references to variable size information about the group. In this particular example, there are no links in the group. We'll read that variable information in a future installment.<jupyter_code>(close *input-stream*)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11225571883048023, 'fraction_numerical': 0.04620512043629753, 'mean_word_length': 4.341423948220065, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 10, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '386313', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2436, 'n_tokens_neox': 2171, 'n_words': 815, 'path': 'notebooks/RootGroupOHDR.ipynb', 'repo_name': 'HDFGroup/alcove', 'size': '6605'}