<commit_before>

The pieces are simdb, tbl, visualizer, node graph interface, execution loop
 
simdb.hpp     -  Lock free shared memory key value store. Currently working.

Future Work   -  Lock free robin hood hashing.

tbl.hpp           -   A vector+hash table data structure that is always a single span of contiguous memory and thus has no separate serialization. Useful as a general purpose high level data structure so that custom structures that and their serialization are not needed as often.   Next step is to nest tbl structures so that trees and compound data structures can be created easily as well.  Working but missing nested tables.

Future Work  - Nesting tables as pointers or offsets into contiguous memory. 

Visualizer:  -Separate process that draws textured 3d geometry from other processes. Uses simdb to get data which enables it to update at greater than 144 frames per second.  Currently working and in fact has been already been used to debug my commercial product.

Future Work:  Showing the vector, key-value, and nested tbl contents of a tbl structure.  Estimate: 8 days 

node graph interface - Initial Prototype - Still in it's infancy though shows that openGL + nanovg can be used to provide a fluid and great looking interface.

Execution loop:  The actual execution of the program being created. Takes data packets from a lock free queue, passes them to the node they go to, gets the outputs and puts them into the queue.     Not Yet Started  -  Previous Prototypes.








The individual components are simdb, tbl, visualizer, node graph interface, execution loop.  Most of the work would go into the node graph and execution loop.
 
node graph interface - Initial Prototype - Still in it's infancy, though shows that openGL + nanovg can be used to provide a fluid and great looking interface.

Execution loop:  The actual execution of the program being created. Takes data packets from a lock free queue, passes them to the node they go to, gets the outputs and puts them into the queue.     Not Yet Started  -  Previous Prototypes.

simdb.hpp  - 5 days: Lock free robin hood hashing.
tbl.hpp    - 5 days: Nesting tables as offsets into contiguous memory. 
Visualizer - 8 days: Showing the vector, key-value and nested tbl contents of a tbl structure.  





<commit_msg>better timeline<commit_after>

The pieces are simdb, tbl, visualizer, node graph interface, execution loop
 
simdb.hpp     -  Lock free shared memory key value store. Currently working.

Future Work   -  Lock free robin hood hashing.

tbl.hpp           -   A vector+hash table data structure that is always a single span of contiguous memory and thus has no separate serialization. Useful as a general purpose high level data structure so that custom structures that and their serialization are not needed as often.   Next step is to nest tbl structures so that trees and compound data structures can be created easily as well.  Working but missing nested tables.

Future Work  - Nesting tables as pointers or offsets into contiguous memory. 

Visualizer:  -Separate process that draws textured 3d geometry from other processes. Uses simdb to get data which enables it to update at greater than 144 frames per second.  Currently working and in fact has been already been used to debug my commercial product.

Future Work:  Showing the vector, key-value, and nested tbl contents of a tbl structure.  Estimate: 8 days 

node graph interface - Initial Prototype - Still in it's infancy though shows that openGL + nanovg can be used to provide a fluid and great looking interface.

Execution loop:  The actual execution of the program being created. Takes data packets from a lock free queue, passes them to the node they go to, gets the outputs and puts them into the queue.     Not Yet Started  -  Previous Prototypes.








The individual components are simdb, tbl, the visualizer, the node graph interface and the execution loop.  Most of the time would go into the node graph and execution loop. Some additional time could be used to polish tbl.hpp and simdb.hpp so that they could be released individually and used to promote the architecture as a whole.  All estimates are just as rough as any software estimate. 
 
Node graph interface - 3-4 Weeks: Still in it's infancy, though shows that openGL + nanovg can be used to provide a fluid and great looking interface. The goal would be:
 | A workable interface for building data flow graphs 
 | Dynamic reloading of nodes on change
 | Ability to visualize any types in the shared memory that can be converted to images or geometry
 | Ability to freeze execution on a node so that dynamic reloading runs the node and visualizes the result
 | The ability to load and save workspace files

Execution loop - 10 days:  
 | Define plug-in API for shared libaries
 | Take data packets from a lock free queue, pass to their destination node
 | Get the outputs and put them into the queue
 | Run message passing nodes with their messages when available


simdb.hpp  - 5 days: Lock free robin hood hashing.
tbl.hpp    - 5 days: Nesting tables as offsets into contiguous memory. 
Visualizer - 8 days: Showing the vector, key-value and nested tbl contents of a tbl structure.  






{'fraction_non_alphanumeric': 0.03151862464183381, 'fraction_numerical': 0.0038204393505253103, 'mean_word_length': 3.7044025157232703, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3555274', 'n_tokens_mistral': 1320, 'n_tokens_neox': 1209, 'n_words': 816}