<commit_before>Key spec.

The following grammar specifies the keys used for storing each DAG in a B-tree.

Basic Types

The following basic types are used as terminals in the rest of the grammar. Each
type must be serialized in little-endian format.

""      0 byte
byte    1 byte (8-bits)
lnum    6 bytes (48-bits unsigned integer LE)


Non-terminals

The following specifies the rest of the key grammar. Note that quoted strings
represent terminals, and should be interpreted with C semantics (e.g. "\x01"
represents the byte 0000 0001). Also note that we use the * operator as
shorthand for repetition (e.g. ("\x01"*2) is "\x01\x01"). When used as a unary
operator, * means that the repetition can occur 0 or more times.

key     ::=  name subkey ""    key is comprised of a name followed by a subkey
name    ::=  string            name of the keyspace
subkey  ::=
  |   "\x01" dskey             data store key type
  |   "\x02" ikey              i index key type
  |   "\x03" headkey           head index key type
ikey    ::=  lnum              i key
dskey   ::=  string lnum       data store key is comprised of the document id
                               plus an lnum
headkey ::=  string lnum       head key is comprised of the document id plus an
                               lnum which corresponds to the version of the
                               document
string  ::=  byte (byte*) "\x00"    String - The byte is the number bytes in the
                                    (byte*) + 1 (for the trailing '\x00'). The
                                    (byte*) is zero or more UTF-8 encoded
                                    characters. Can be at most 255 bytes.
<commit_msg>keyspec.txt: use new lbeint type in ikey<commit_after>Key spec.

The following grammar specifies the keys used for storing each DAG in a B-tree.

Basic Types

The following basic types are used as terminals in the rest of the grammar. Each
type must be serialized in little-endian format.

""      0 byte
byte    1 byte (8-bits)


Non-terminals

The following specifies the rest of the key grammar. Note that quoted strings
represent terminals, and should be interpreted with C semantics (e.g. "\x01"
represents the byte 0000 0001). Also note that we use the * operator as
shorthand for repetition (e.g. ("\x01"*2) is "\x01\x01"). When used as a unary
operator, * means that the repetition can occur 0 or more times.

key     ::=  name subkey ""    key is comprised of a name followed by a subkey
name    ::=  string            name of the keyspace
subkey  ::=
  |   "\x01" dskey             data store key type
  |   "\x02" ikey              i index key type
  |   "\x03" headkey           head index key type
ikey    ::=  lbeint            i key
dskey   ::=  string lbeint     data store key is comprised of the value id plus
                               an lbeint
headkey ::=  string lbeint     head key is comprised of the id of the value plus
                               an lbeint which corresponds to the version of the
                               value
lbeint  ::=  byte (byte*) ""    Large int - The byte is the number of bytes in
                                (byte*). The (byte*) is one to 255 bytes written
                                that should be interpreted as an unsigned
                                integer in big endian format.
string  ::=  byte (byte*) "\x00"    String - The byte is the number bytes in the
                                    (byte*) + 1 (for the trailing '\x00'). The
                                    (byte*) is zero or more UTF-8 encoded
                                    characters. Can be at most 255 bytes.

{'fraction_non_alphanumeric': 0.061664392905866304, 'fraction_numerical': 0.021282401091405185, 'mean_word_length': 1.7358208955223882, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2424650', 'n_tokens_mistral': 1025, 'n_tokens_neox': 937, 'n_words': 489}