<commit_before>Low-level file layout
=====================

The overall structure of a file is as follows (in order):

- :ref:`header`

- :ref:`comments`, optional

- :ref:`tree`, optional

- Zero or more :ref:`block`

- :ref:`block-index`, optional

ASDF is a hybrid text and binary format.  The header, tree and block
index are text, (specifically, in UTF-8 with DOS or UNIX-style
newlines), while the blocks are raw binary.

The low-level file layout is designed in such a way that the tree
section can be edited by hand, possibly changing its size, without
requiring changes in other parts of the file.  While such an operation
may invalidate the :ref:`block-index`, the format is designed so that
if the block index is removed or invalid, it may be regenerated by
"skipping along" the blocks in the file.

The same is not true for resizing a block, which has an explicit size
stored in the block header (except for, optionally, the last block).

Note also that, by design, an ASDF file containing no binary blocks is
also a completely standard and valid YAML file.

Additionally, the spec allows for extra unallocated space after the
tree and between blocks.  This allows libraries to more easily update
the files in place, since it allows expansion of certain areas without
rewriting of the entire file.

.. _header:

Header
------

All ASDF files must start with a short one-line header.  For example::

  #ASDF 0.1.0

It is made up of two parts, separated by white space characters:

  - **ASDF token**: The constant string ``#ASDF``. This can be used to
    quickly identify the file as an ASDF file by reading the first 5
    bytes.  It begins with a ``#`` so it will be treated as a YAML
    comment such that the :ref:`header` and the :ref:`tree` together
    form a valid YAML file.

  - **File format version**: The version of the low-level file format
    that this file was written with.  This version may differ from the
    version of the ASDF specification, and is only updated when a
    change is made that affects the layout of file.  It follows the
    `Semantic Versioning 2.0.0 <http://semver.org/spec/v2.0.0.html>`__
    specification. See :ref:`versioning-conventions` for more
    information about these versions.

.. _comments:

Comments
--------

Additional comment lines may appear between the Header and the Tree.

The use of comments here is intended for information for the ASDF
parser, and not information of general interest to the end user.  All
data of interest to the end user should be in the Tree.
<commit_msg>Restore EBNF about ASDF header<commit_after>Low-level file layout
=====================

The overall structure of a file is as follows (in order):

- :ref:`header`

- :ref:`comments`, optional

- :ref:`tree`, optional

- Zero or more :ref:`block`

- :ref:`block-index`, optional

ASDF is a hybrid text and binary format.  The header, tree and block
index are text, (specifically, in UTF-8 with DOS or UNIX-style
newlines), while the blocks are raw binary.

The low-level file layout is designed in such a way that the tree
section can be edited by hand, possibly changing its size, without
requiring changes in other parts of the file.  While such an operation
may invalidate the :ref:`block-index`, the format is designed so that
if the block index is removed or invalid, it may be regenerated by
"skipping along" the blocks in the file.

The same is not true for resizing a block, which has an explicit size
stored in the block header (except for, optionally, the last block).

Note also that, by design, an ASDF file containing no binary blocks is
also a completely standard and valid YAML file.

Additionally, the spec allows for extra unallocated space after the
tree and between blocks.  This allows libraries to more easily update
the files in place, since it allows expansion of certain areas without
rewriting of the entire file.

.. _header:

Header
------

All ASDF files must start with a short one-line header.  For example::

  #ASDF 0.1.0

It is made up of two parts, separated by white space characters:

  - **ASDF token**: The constant string ``#ASDF``. This can be used to
    quickly identify the file as an ASDF file by reading the first 5
    bytes.  It begins with a ``#`` so it will be treated as a YAML
    comment such that the :ref:`header` and the :ref:`tree` together
    form a valid YAML file.

  - **File format version**: The version of the low-level file format
    that this file was written with.  This version may differ from the
    version of the ASDF specification, and is only updated when a
    change is made that affects the layout of file.  It follows the
    `Semantic Versioning 2.0.0 <http://semver.org/spec/v2.0.0.html>`__
    specification. See :ref:`versioning-conventions` for more
    information about these versions.

The header in EBNF form::

    asdf_token = "#ASDF"
    header     = asdf_token " " format_version ["\r"] "\n"

.. _comments:

Comments
--------

Additional comment lines may appear between the Header and the Tree.

The use of comments here is intended for information for the ASDF
parser, and not information of general interest to the end user.  All
data of interest to the end user should be in the Tree.

{'fraction_non_alphanumeric': 0.07433594496464743, 'fraction_numerical': 0.004204089432447927, 'mean_word_length': 3.644188110026619, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '884081', 'n_tokens_mistral': 1445, 'n_tokens_neox': 1392, 'n_words': 798}