<commit_before>The zs2 file format
===================

File structure
--------------

Compression
^^^^^^^^^^^
``zs2`` files are `gzip-compressed`_ binary files. The first step to
decoding a file is to unpack the ``zs2`` file with a utility program 
such as ``gunzip`` or `7-zip`_, or opening the file using the 
`Python module`_ ``gzip``. This results in a binary data stream.

.. cssclass:: table-bordered

 +----+--------------------+----+
 |       gzip compression       |
 +----+--------------------+----+
 |    |                    |    |
 |    |     data stream    |    |
 |    |                    |    |
 +----+--------------------+----+
 

.. _gzip-compressed: https://en.wikipedia.org/wiki/Gzip
.. _7-zip: http://www.7-zip.org/
.. _Python module: https://docs.python.org/2/library/gzip.html

Data stream structure
^^^^^^^^^^^^^^^^^^^^^
The data stream starts with a header signature and
is followed by chunks.
A typical ``zs2`` file contains over 100,000 chunks.

 +--------------------+--------+
 | Header (4 bytes)            |
 +--------------------+--------+
 | Chunk 1                     |
 |                             |
 +--------------------+--------+
 | Chunk 2                     |
 |                             |
 +--------------------+--------+
 | ...                         |
 |                             |
 +--------------------+--------+
 | Chunk n                     |
 |                             |
 +--------------------+--------+
 
 
Byte order
~~~~~~~~~~
The `byte order`_ of the *binary file* is **little-endian**. For
example, a 32-bit representation of the integer value ``1`` 
in the data stream would be ``0x01 0x00 0x00 0x00``.

.. _byte order: https://en.wikipedia.org/wiki/Endianness

Header
^^^^^^
The *binary file* starts with the 4-byte signature ``0xAF 0xBE 0xAD 0xDE``, 
i.e. ``0xDEADBEAF`` in hexadecimal or ``3735928495`` in decimal. 

 +----------+----------+----------+----------+
 | Offset 0 | Offset 1 | Offset 2 | Offset 3 |
 +==========+==========+==========+==========+
 | ``0xAF`` | ``0xBE`` | ``0xAD`` | ``0xDE`` |
 +----------+----------+----------+----------+

This signature is followed immediately by the first chunk.

Chunks
^^^^^^
Chunks contain information on data stream structure, metadata or data.

There are two different chunk formats. One format is used by the
"End-of-Section" chunk, the other format is used by all other chunks.
The latter has the following structure:
an ASCII-encoded name of the chunk type, starting 
with one byte length giving the length of the name
in bytes (Section :ref:`section-ascii-string-definition`), 
followed by the data type of the particular
chunk (Section :ref:`section-data-types`) and actual chunk 
data. In contrast, the End-of-Section chunk is simply
a single byte of value ``0xFF``. Both chunk structures
can be discriminated between because chunk type names
do not start with ``0xFF``.

The following chunk structures are possible:
 
 +------------+------------+------------+------------+
 | Chunk type name         | Data type  | Chunk      |
 |                         | code       | data       |
 +============+============+============+============+
 | ASCII-encoded, at least | 1 byte     | 1 or more  |
 | 2 bytes                 |            | bytes      |
 +------------+------------+------------+------------+
 
or, for the "End-of-Section" chunk,
 
 +------------+------------+------------+------------+
 | End of     |            |            |            |
 | Section    |            |            |            |
 +============+============+============+============+
 | ``0xFF``   |            |            |            |
 |            |            |            |            |
 +------------+------------+------------+------------+
 
The total length of the chunk can be anywhere from 1 byte upward.
In particular, the total chunk length is generally not a 
multiple of 2 or 4 bytes.

.. note:: Less than 5% of the existing chunk types have no data type.

An example of a chunk with chunk type name ``ID`` would be: 

 +--------+--------+--------+--------+--------+--------+
 | Chunk type name          | Data   | Chunk data      |
 +--------+--------+--------+ type   +                 +
 | Length | Name            |        |                 |
 +--------+--------+--------+--------+--------+--------+
 | ``2``  | ``I``  | ``D``  |``0x66``| ``48154``       | 
 +========+========+========+========+========+========+
 | 1      | 2      | 3      | 4      | 5      | 6      | 
 +--------+--------+--------+--------+--------+--------+
 |``0x02``|``0x49``|``0x44``|``0x66``|``0x1A``|``0xBC``|
 +--------+--------+--------+--------+--------+--------+

.. _section-chunk-naming:

Chunk type naming
^^^^^^^^^^^^^^^^^
Chunk type names are ASCII strings defined in 
Section :ref:`section-ascii-string-definition`.
Chunk types names are chosen to be readable ASCII text, 
comprising mostly of underscore (``_``),
digits ``0`` to ``9``, and English letters ``A`` to ``Z`` and ``a`` to ``z``.
Very few chunk type names include other characters, in particular chunk type
names starting with ``nt&)m_``.
The chunk type name length is limited to 254 characters since an indicated length of
``255`` (``0xFF``) represents an "End-of-Section" chunk.
Also, chunk type names of length 0 (``0x00``) do not exist.

The three chunk types ``Key``, ``Elem``, and ``Val`` represent list items. 
Digits are used at the end of chunk type names to enumerate the list items.
Within each list, numbers are consecutive in decimal format, 
starting with zero. 
For example, the list element ``Elem0`` will be followed by ``Elem1``. 
``Elem9`` will be followed by ``Elem10`` etc. 
If a list has only one entry, the number will be zero (e.g. ``Key0``).


.. note:: By convention, most chunk type names start with a capital letter 
          ``A`` to ``Z`` and use *CamelCase* spelling for compound words 
          (i.e., approximately 95% of all chunk type names).
          Names are derived from either English or German language.
          The shortest chunk type names are ``x``, ``y``, ``X``, and ``Y``. 
          The longest chunk type name is
          ``AssignmentBetweenOrganizationDataAndTestProgramParamIds``
          at 55 characters.

Order of chunks
^^^^^^^^^^^^^^^
The order of some chunks is significant as they can establish
a partitioning into sections (chunks of data type ``0xDD`` start<commit_msg>editorial change regarding uncommon characters<commit_after>The zs2 file format
===================

File structure
--------------

Compression
^^^^^^^^^^^
``zs2`` files are `gzip-compressed`_ binary files. The first step to
decoding a file is to unpack the ``zs2`` file with a utility program 
such as ``gunzip`` or `7-zip`_, or opening the file using the 
`Python module`_ ``gzip``. This results in a binary data stream.

.. cssclass:: table-bordered

 +----+--------------------+----+
 |       gzip compression       |
 +----+--------------------+----+
 |    |                    |    |
 |    |     data stream    |    |
 |    |                    |    |
 +----+--------------------+----+
 

.. _gzip-compressed: https://en.wikipedia.org/wiki/Gzip
.. _7-zip: http://www.7-zip.org/
.. _Python module: https://docs.python.org/2/library/gzip.html

Data stream structure
^^^^^^^^^^^^^^^^^^^^^
The data stream starts with a header signature and
is followed by chunks.
A typical ``zs2`` file contains over 100,000 chunks.

 +--------------------+--------+
 | Header (4 bytes)            |
 +--------------------+--------+
 | Chunk 1                     |
 |                             |
 +--------------------+--------+
 | Chunk 2                     |
 |                             |
 +--------------------+--------+
 | ...                         |
 |                             |
 +--------------------+--------+
 | Chunk n                     |
 |                             |
 +--------------------+--------+
 
 
Byte order
~~~~~~~~~~
The `byte order`_ of the *binary file* is **little-endian**. For
example, a 32-bit representation of the integer value ``1`` 
in the data stream would be ``0x01 0x00 0x00 0x00``.

.. _byte order: https://en.wikipedia.org/wiki/Endianness

Header
^^^^^^
The *binary file* starts with the 4-byte signature ``0xAF 0xBE 0xAD 0xDE``, 
i.e. ``0xDEADBEAF`` in hexadecimal or ``3735928495`` in decimal. 

 +----------+----------+----------+----------+
 | Offset 0 | Offset 1 | Offset 2 | Offset 3 |
 +==========+==========+==========+==========+
 | ``0xAF`` | ``0xBE`` | ``0xAD`` | ``0xDE`` |
 +----------+----------+----------+----------+

This signature is followed immediately by the first chunk.

Chunks
^^^^^^
Chunks contain information on data stream structure, metadata or data.

There are two different chunk formats. One format is used by the
"End-of-Section" chunk, the other format is used by all other chunks.
The latter has the following structure:
an ASCII-encoded name of the chunk type, starting 
with one byte length giving the length of the name
in bytes (Section :ref:`section-ascii-string-definition`), 
followed by the data type of the particular
chunk (Section :ref:`section-data-types`) and actual chunk 
data. In contrast, the End-of-Section chunk is simply
a single byte of value ``0xFF``. Both chunk structures
can be discriminated between because chunk type names
do not start with ``0xFF``.

The following chunk structures are possible:
 
 +------------+------------+------------+------------+
 | Chunk type name         | Data type  | Chunk      |
 |                         | code       | data       |
 +============+============+============+============+
 | ASCII-encoded, at least | 1 byte     | 1 or more  |
 | 2 bytes                 |            | bytes      |
 +------------+------------+------------+------------+
 
or, for the "End-of-Section" chunk,
 
 +------------+------------+------------+------------+
 | End of     |            |            |            |
 | Section    |            |            |            |
 +============+============+============+============+
 | ``0xFF``   |            |            |            |
 |            |            |            |            |
 +------------+------------+------------+------------+
 
The total length of the chunk can be anywhere from 1 byte upward.
In particular, the total chunk length is generally not a 
multiple of 2 or 4 bytes.

.. note:: Less than 5% of the existing chunk types have no data type.

An example of a chunk with chunk type name ``ID`` would be: 

 +--------+--------+--------+--------+--------+--------+
 | Chunk type name          | Data   | Chunk data      |
 +--------+--------+--------+ type   +                 +
 | Length | Name            |        |                 |
 +--------+--------+--------+--------+--------+--------+
 | ``2``  | ``I``  | ``D``  |``0x66``| ``48154``       | 
 +========+========+========+========+========+========+
 | 1      | 2      | 3      | 4      | 5      | 6      | 
 +--------+--------+--------+--------+--------+--------+
 |``0x02``|``0x49``|``0x44``|``0x66``|``0x1A``|``0xBC``|
 +--------+--------+--------+--------+--------+--------+

.. _section-chunk-naming:

Chunk type naming
^^^^^^^^^^^^^^^^^
Chunk type names are ASCII strings defined in 
Section :ref:`section-ascii-string-definition`.
Chunk types names are chosen to be readable ASCII text, 
comprising mostly of underscore (``_``),
digits ``0`` to ``9``, and English letters ``A`` to ``Z`` and ``a`` to ``z``.
Very few chunk type names include other characters.
The chunk type name length is limited to 254 characters since an indicated length of
``255`` (``0xFF``) represents an "End-of-Section" chunk.
Also, chunk type names of length 0 (``0x00``) do not exist.

The three chunk types ``Key``, ``Elem``, and ``Val`` represent list items. 
Digits are used at the end of chunk type names to enumerate the list items.
Within each list, numbers are consecutive in decimal format, 
starting with zero. 
For example, the list element ``Elem0`` will be followed by ``Elem1``. 
``Elem9`` will be followed by ``Elem10`` etc. 
If a list has only one entry, the number will be zero (e.g. ``Key0``).


.. note:: By convention, most chunk type names start with a capital letter 
          ``A`` to ``Z`` and use *CamelCase* spelling for compound words 
          (i.e., approximately 95% of all chunk type names).
          Names are derived from either English or German language.
          The shortest chunk type names are ``x``, ``y``, ``X``, and ``Y``. 
          The longest chunk type name is
          ``AssignmentBetweenOrganizationDataAndTestProgramParamIds``
          at 55 characters.
		  Chunk type names with special characters are rare. Those names
		  may start with ``nt&)m_`` prepended to a common *CamelCase* name,
		  e.g. ``nt&)m_CompressionType``.

Order of chunks
^^^^^^^^^^^^^^^
The order of some chunks is significant as they can establish
a partitioning into sections (chunks of data type ``0xDD`` start
{'fraction_non_alphanumeric': 0.2641102641102641, 'fraction_numerical': 0.01848001848001848, 'mean_word_length': 1.6328806000405434, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '57655', 'n_tokens_mistral': 3681, 'n_tokens_neox': 3411, 'n_words': 1281}