<commit_before>SHORTSTRING 1 byte length header; that many bytes of UTF-8
VARINT      1-5 bytes, protobuf unsigned varint encoding
INT32       4 bytes, big-endian
BYTE        A single byte
BYTES       Raw bytes; no length header implied

Signature file
--------------

SHORTSTRING   checksum algorithm
VARINT        block size
SHORTSTRING   strong hash algorithm
VARINT        number of signatures per signature block (N)
VARINT        number of signatures in this block (n)                         \  repeated
INT32         weak hash   \ repeated n times                                  > until n != N
BYTES         stronghash  / (length of BYTES determined by strong hash alg)  /
BYTES         checksum: hash of everything after the checksum algorithm (length determined by that algorithm)

Patch file
----------

SHORTSTRING   checksum algorithm
VARINT        block size
BYTE          op (0 = block; 1 = data; 255 = end) \ repeated until op = 0
BLOCKDATA(op)                                     /
BYTES         checksum: hash of everything after the checksum algorithm (length determined by that algorithm)

BLOCK(block):
  VARINT      number of block from signature file

BLOCK(data):
  VARINT      number of bytes (must be <= block size)
  BYTES       that many bytes<commit_msg>Patches end with op = 255, not op = 0<commit_after>SHORTSTRING 1 byte length header; that many bytes of UTF-8
VARINT      1-5 bytes, protobuf unsigned varint encoding
INT32       4 bytes, big-endian
BYTE        A single byte
BYTES       Raw bytes; no length header implied

Signature file
--------------

SHORTSTRING   checksum algorithm
VARINT        block size
SHORTSTRING   strong hash algorithm
VARINT        number of signatures per signature block (N)
VARINT        number of signatures in this block (n)                         \  repeated
INT32         weak hash   \ repeated n times                                  > until n != N
BYTES         stronghash  / (length of BYTES determined by strong hash alg)  /
BYTES         checksum: hash of everything after the checksum algorithm (length determined by that algorithm)

Patch file
----------

SHORTSTRING   checksum algorithm
VARINT        block size
BYTE          op (0 = block; 1 = data; 255 = end) \ repeated until op = 255
BLOCKDATA(op)                                     /
BYTES         checksum: hash of everything after the checksum algorithm (length determined by that algorithm)

BLOCK(block):
  VARINT      number of block from signature file

BLOCK(data):
  VARINT      number of bytes (must be <= block size)
  BYTES       that many bytes
{'fraction_non_alphanumeric': 0.05893682588597843, 'fraction_numerical': 0.01386748844375963, 'mean_word_length': 2.040983606557377, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4754854', 'n_tokens_mistral': 660, 'n_tokens_neox': 619, 'n_words': 306}