<commit_before>
Tango prototype TODO list
=========================

__ The current checkpoint implementation is fundamentally broken and
   needs a rewrite, or else.
   option 1: fix checkpoint to be 100% correct
   option 2: checkpointing is for the weak and the memory-constrained, so
             don't bother.  Instead, rip out the current checkpoint code,
             period. 

__ Checkpoint fix option #1: history splicing within the same OID?

__ Checkpoint fix option #2: checkpoint to a new OID, history writes to both
                             OIDs during the CP, then a marker in the old OID
                             to switch over to the new OID?

<commit_msg>Update README.tango.md with latest checkpoint implementation fix notes<commit_after>
Tango prototype TODO list
=========================

__ The current checkpoint implementation is fundamentally broken and
   needs a rewrite, or else.
   This issue is not mentioned at all in the Tango paper.

   option 1: fix checkpoint to be 100% correct
   option 2: checkpointing is for the weak and the memory-constrained, so
             don't bother.  Instead, rip out the current checkpoint code,
             period. 
   option 3: other

xx Checkpoint fix option #1: history splicing within the same OID?

xx Checkpoint fix option #2: checkpoint to a new OID, history writes to both
                             OIDs during the CP, then a marker in the old OID
                             to switch over to the new OID?

History splicing has a flaw that I belive just won't work.  The switch to a
new OID has problems with updates written to the old OID before and before the
new checkpoint has finished.

I believe that a checkpoint where:
    * all Tango writes, checkpoint and non-checkpoint alike, are noted with
      a checkpoint number.
    * that checkpoint number is strictly increasing
    * a new checkpoint has a new checkpoint number
    * scans ignore blocks with checkpoint numbers larger than the current
      active checkpoint #, until the checkpoint is complete.

... ought to work correctly.  
{'fraction_non_alphanumeric': 0.05696505505026329, 'fraction_numerical': 0.007180469123982767, 'mean_word_length': 2.7522441651705565, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2346445', 'n_tokens_mistral': 535, 'n_tokens_neox': 477, 'n_words': 287}