<commit_before><commit_msg>doc/problems: add documentation of significant issues in development<commit_after>jgfs2
problems


variable length btree items:
- breaks the normal btree guarantees about node space usage
  - see <NAME>'s comments wrt Btrfs for details
- current split algorithm is hopelessly broken
  - tree_insert doesn't try to split more than once
  - tree_insert can't communicate the amount of free space wanted (and the key)
    to node_split, which chould make use of such information
  - node_split tries to cut the *currently used* space in half; see prev point
- partial alleviation: don't use really big items
  - we won't store data inline like btrfs does
  - we must use DIR_ITEM items for name lookups (>255 bytes at max)
- bad performance: store (key, offset to data file)
  - pro: fixed size items
  - con: bad locality (where is the 'data file' gonna get put?)
  - con: ugly ugly ugly
- total fix: use s(b)trees (modification of B+ tree)
  - need to take a closer look at p and q constraints

free space tracking:
- ext tree has issues
  - when the ext tree is modified, it may need to alloc/dealloc an extent
  - therefore we have recursive situations that get very nasty, very quickly
- fs bitmap would work
  - makes the fs 'messier'
  - finding free extent of required size is slow

fs corruption:
- writing trees atomically requires COW, which is too complicated to justify
- tree 'commits' will significantly decrease the likelihood of tree corruption
- core fs functionality needs to choose order of operations carefully to
  minimize the impact of partial operations
{'fraction_non_alphanumeric': 0.04868913857677903, 'fraction_numerical': 0.0024968789013732834, 'mean_word_length': 3.9323076923076923, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6490814', 'n_tokens_mistral': 425, 'n_tokens_neox': 408, 'n_words': 238}