<commit_before>+---------------------------------------------------------------------------+
| Copyright (c) 2008, <NAME>                                         |
| Distributed under the Terms of the 3clause BSD License - see LICENSE File |
| if you received this without a LICENSE file, please contact the Author    |
| (lars<at>chaotika<dot>org) to obtain one                                  |
+---------------------------------------------------------------------------+

                             The XPAK Data Format
                             ====================

Notes
=====
 Data Types
 ==========

  Integer: every offset or length(len) value in this documentation will be an
           unsigned 32bit integer in big endian byte order(32bit unsigned big
           int or uint32(big) ).

  String:  All Strings, Mentioned in this Documentation are ASCII encoded.

 Vertical Bars
 =============

  The vertical bars '|' are not part of the file format, they are merely used
  to illustrate how the offset values apply to the data.

Synopsis
========

  <tarball>:      |<-------xpak_offset------->|
             <tar>|<xpak><xpak_offset><"STOP">|

  <xpak>   : <"XPAKPACK"><index_len><data_len><index><data><"XPAKSTOP">

  <index>  : |<-------------index_len------------->|
             |<index1><index2><index3><index4><...>|

  <indexN> :           |<-name_len->|
             <name_len>|<   name   >|<data_offset><data_len>

  <data>   : |<-------------data_len------------>|
             |<-dataN_offset->|<-dataN_len->|
             |<     data     >|<  data_N   >|<data>|

Detailed
========

 Every gentoo binary package has a xpak attached to it which contains built
 time information like the use flags it was built with, the ebuild it was
 built from, the environmental variables, CFLAGs, CXXFLAGs, ....

 lets have a look at this XPAK:

 If you look at a gentoo binary package (binpkg) with a hex-editor you'll
 notice the that after the data, which belongs to the tarball you find a
 binary blob - the <xpak>, an offset which holds the bytes from the start of
 the <xpak> to the end of the file - <xpak_offset> and finally the String
 "STOP":

                            |<------xpak_offset------>|
                      <tbz2>|<xpak><xpak_offset>"STOP"|

 Here you see the tbz2 archive, and the attached xpak-blob, the xpak-offset<commit_msg>typos--<commit_after>+---------------------------------------------------------------------------+
| Copyright (c) 2008, <NAME>                                         |
| Distributed under the Terms of the 3clause BSD License - see LICENSE File |
| if you received this without a LICENSE file, please contact the Author    |
| (lars<at>chaotika<dot>org) to obtain one                                  |
+---------------------------------------------------------------------------+

                             The XPAK Data Format
                             ====================

Notes
=====
 Data Types
 ==========

  Integer: every offset or length(len) value in this documentation will be an
           unsigned 32bit integer in big endian byte order(32bit unsigned big
           int or uint32(big) ).

  String:  All Strings, Mentioned in this Documentation are ASCII encoded.

 Vertical Bars
 =============

  The vertical bars '|' are not part of the file format, they are merely used
  to illustrate how the offset values apply to the data.

Synopsis
========

  <tarball>:      |<-------xpak_offset------->|
             <tar>|<xpak><xpak_offset><"STOP">|

  <xpak>   : <"XPAKPACK"><index_len><data_len><index><data><"XPAKSTOP">

  <index>  : |<-------------index_len------------->|
             |<index1><index2><index3><index4><...>|

  <indexN> :           |<-name_len->|
             <name_len>|<   name   >|<data_offset><data_len>

  <data>   : |<-------------data_len------------>|
             |<-dataN_offset->|<-dataN_len->|
             |<     data     >|<  data_N   >|<data>|

Detailed
========

 Every gentoo binary package has a xpak attached to it which contains build
 time information like the use flags it was built with, the ebuild it was
 built from, the environmental variables, CFLAGs, CXXFLAGs, ....

 lets have a look at this XPAK:

 If you look at a gentoo binary package (binpkg) with a hex-editor you'll
 notice the that after the data, which belongs to the tarball you find a
 binary blob - the <xpak>, an offset which holds the bytes from the start of
 the <xpak> to the end of the file - <xpak_offset> and finally the String
 "STOP":

                            |<------xpak_offset------>|
                      <tbz2>|<xpak><xpak_offset>"STOP"|

 Here you see the tbz2 archive, and the attached xpak-blob, the xpak-offset
{'fraction_non_alphanumeric': 0.20608108108108109, 'fraction_numerical': 0.0071790540540540545, 'mean_word_length': 1.7365684575389948, 'pattern_counts': {'":': 2, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '705615', 'n_tokens_mistral': 1331, 'n_tokens_neox': 1218, 'n_words': 463}