<commit_before>
Notes:

- Storage ID's are unsigned 32-bit values encoded by io.permazen.util.UnsignedIntEncoder,
  which means they can have variable length from 1 to 5 bytes (but typically 3).

- Every key/value pair in a Permazen database should match one of the following.
  You can use the "jsck" CLI command to consistency check this.

****** OBJECT IDS (64 bits) *******

    +------------+----------------------------------------------------------+
    | Storage ID |                   Random Bytes                           |
    +------------+----------------------------------------------------------+

****** DATABASE META-DATA *******

- The meta-data area spans all keys with first byte 0x00

- Keys with first byte 0x00 and second byte 0x00 through 0xfe (inclusive)
  are reserved by Permazen and must not be modified.

- Keys with first byte 0x00 and second byte 0xff are ignored by Permazen
  and may be used for custom user applications.

Format Version

    +------+------+-----------------+    +-------------------+
    | 0x00 | 0x00 |   'JSimpleDB'   |    |   Format Version  |
    +------+------+-----------------+    +-------------------+

Recorded Schemas

    +------+------+-------------------+    +---------------------------+
    | 0x00 | 0x01 |   Version Number  |    |   Compressed Schema XML   |
    +------+------+-------------------+    +---------------------------+

Object Version Index

    +------+------+-------------------+---------------+    +---------+
    | 0x00 | 0x80 |   Version Number  |     ObjId     |    | (Empty) |
    +------+------+-------------------+---------------+    +---------+

User meta-data area

    +------+------+-----
    | 0x00 | 0xff | ...
    +------+------+-----

****** OBJECTS *******

    +------------+   +-------------------+----------------+-----------------+
    |  Object ID |   | Meta-Data Version | Object Version | Delete Notified |
    +------------+   +-------------------+----------------+-----------------+

****** SIMPLE AND COUNTER FIELDS *******

    +------------+---------------+    +---------------+
    |  Object ID |   Storage ID  |    |     Value     |  (non-default only)
    +------------+---------------+    +---------------+

****** COMPLEX FIELDS *******

Set Field Element

    +------------+---------------+---------------+    +---------+
    |  Object ID |   Storage ID  |    Element    |    | (Empty) |
    +------------+---------------+---------------+    +---------+

List Field Element

    +------------+---------------+---------------+    +---------------+
    |  Object ID |   Storage ID  |  List Index   |    |    Element    |
    +------------+---------------+---------------+    +---------------+

Map Field Entry

    +------------+---------------+---------------+    +---------------+
    |  Object ID |   Storage ID  |    Map Key    |    |   Map Value   |
    +------------+---------------+---------------+    +---------------+

****** SIMPLE INDEXES *******

Simple Field Index

    +------------+---------------+---------------+    +---------+
    | Storage ID |     Value     |     ObjId     |    | (Empty) |
    +------------+---------------+---------------+    +---------+

Set Element Field Index

    +------------+---------------+---------------+    +---------+
    | Storage ID |    Element    |     ObjId     |    | (Empty) |
    +------------+---------------+---------------+    +---------+

List Element Field Index

    +------------+---------------+---------------+--------------+    +---------+
    | Storage ID |    Element    |     ObjId     |  List Index  |    | (Empty) |
    +------------+---------------+---------------+--------------+    +---------+

Map Key Field Index

    +------------+---------------+---------------+    +---------+
    | Storage ID |    Map Key    |     ObjId     |    | (Empty) |
    +------------+---------------+---------------+    +---------+

Map Value Field Index

    +------------+---------------+---------------+--------------+    +---------+
    | Storage ID |   Map Value   |     ObjId     |    Map Key   |    | (Empty) |
    +------------+---------------+---------------+--------------+    +---------+

****** COMPOSITE INDEXES *******

    +------------+-------------+...+-------------+--------------+    +---------+
    | Storage ID |   Value 1   |...|   Value N   |     ObjId    |    | (Empty) |
    +------------+-------------+...+-------------+--------------+    +---------+
<commit_msg>Use box edge characters in layout file.<commit_after>This file describes how objects, indexes, and meta-data are mapped into the key/value store.

Notes:

- Storage ID's are unsigned 32-bit values encoded by io.permazen.util.UnsignedIntEncoder,
  which means when encoded they can have variable length from 1 to 5 bytes (but typically 3).

- Every key/value pair in a Permazen database should match one of the following.
  You can use the "jsck" CLI command to consistency check this.

****** OBJECT IDS (64 bits) *******

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Storage ID ┃                   Random Bytes                           ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

****** DATABASE META-DATA *******

- The meta-data area spans all keys with first byte 0x00

- Keys with first byte 0x00 and second byte 0x00 through 0xfe (inclusive)
  are reserved by Permazen and must not be modified.

- Keys with first byte 0x00 and second byte 0xff are ignored by Permazen
  and may be used for custom user applications.

Magic Cookie and Database Format Version

    ┏━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━━━━━┓
    ┃ 0x00 ┃ 0x00 ┃   'JSimpleDB'   ┃ -> ┃   Format Version  ┃
    ┗━━━━━━┻━━━━━━┻━━━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━━━━━┛

Recorded Schemas

    ┏━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ 0x00 ┃ 0x01 ┃  Version Number  ┃ -> ┃   Compressed Schema XML   ┃
    ┗━━━━━━┻━━━━━━┻━━━━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Object Version Index

    ┏━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃ 0x00 ┃ 0x80 ┃  Version Number  ┃   Object ID   ┃ -> ┃ (Empty) ┃
    ┗━━━━━━┻━━━━━━┻━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

User meta-data area

    ┏━━━━━━┳━━━━━━┳━━━━-
    ┃ 0x00 ┃ 0xff ┃ ...
    ┗━━━━━━┻━━━━━━┻━━━━-

****** OBJECTS *******

    ┏━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
    ┃  Object ID ┃ -> ┃ Meta-Data Version ┃ Object Version ┃ Delete Notified ┃
    ┗━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━┛

****** SIMPLE AND COUNTER FIELDS *******

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━┓
    ┃  Object ID ┃    Field ID   ┃ -> ┃     Value     ┃  (non-default only)
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━┛

****** COMPLEX FIELDS *******

Set Field Element

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Object ID ┃    Field ID   ┃    Element    ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

List Field Element

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━┓
    ┃  Object ID ┃    Field ID   ┃  List Index   ┃ -> ┃    Element    ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━┛

Map Field Entry

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━━━━━━━┓
    ┃  Object ID ┃    Field ID   ┃    Map Key    ┃ -> ┃   Map Value   ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━━━━━━━┛

****** SIMPLE INDEXES *******

Simple Field Index

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Field ID  ┃     Value     ┃   Object ID   ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

Set Element Field Index

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Field ID  ┃    Element    ┃   Object ID   ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

List Element Field Index

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Field ID  ┃    Element    ┃   Object ID   ┃  List Index  ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

Map Key Field Index

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Field ID  ┃    Map Key    ┃   Object ID   ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

Map Value Field Index

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Field ID  ┃   Map Value   ┃   Object ID   ┃    Map Key   ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

****** COMPOSITE INDEXES *******

    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━-┳...┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓    ┏━━━━━━━━━┓
    ┃  Index ID  ┃   Value 1   ┃...┃   Value N   ┃   Object ID  ┃ -> ┃ (Empty) ┃
    ┗━━━━━━━━━━━━┻━━━━━━━━━━━━━┻...┻━━━━━━━━━━━━━┻━━━━━━━━━━━━━━┛    ┗━━━━━━━━━┛

{'fraction_non_alphanumeric': 0.48012333443453364, 'fraction_numerical': 0.00969056271335756, 'mean_word_length': 1.3787323205866946, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4568046', 'n_tokens_mistral': 4340, 'n_tokens_neox': 5717, 'n_words': 697}