<commit_before>Updates / Issues for Scala conversion
===========================================

Internal weight representation
------------------------------

The Python version (deliberately) has two modes of operation : Training and 'Real'.

During training, the 'weight' associated with every hash entry corresponds to the current count of plusses and minuses to that item, and 
the total/timestamp is there to record how long the weight has lingered in each state.

The reason for capturing the average weight (based on the overall training period) is that *honnibal* observes that 
the current count is too jittery : It's much more reasonable to use the overall average values during training
when it comes to 'real life' usage of the library.  

Hence the need for the ```average_weights()``` function, which effectively replaces the weights calculated during training with
an average weight figure.

However : This introduces a before/after split.  Once the ```average_weights()``` function is called, the perceptron(s) are in 
'real world' mode, and can't be trained further.

For the Scala version, the three integers (current, total, ts) are always preserved (and stored to disk as integers).  This has the following benefits :

 * The average weights, when required, are calculated on-the-fly (not a significant computation)

 * The perceptrons are always ready to do further training, since sufficient state is stored

 * Despite more data being stored, the disk space used is actually less
 
One argument for doing it this way is that it means that ```score()``` doesn't have to be written twice.  However, Scala is very comfortable with
juggling functions around, so the Scala ```score()``` method can simply be passed either the ```current``` or ```average``` function.


Internal dependency list representation
------------------------------------------------------------

Scala lists are quicker to prepend to, so instead of maintaining ```stack``` using ```.append()``` and ```stack[-1]```, 
the Scala version simply does ``` x :: stack``` and checks ```stack.head```.


Gold Move Logic
------------------------------------------------------------

The ```gold_moves``` logic is essentially ```Set(gold_moves) = Set( all_moves - (all_moves - valid_moves - bad_moves) )```.  
This makes sense in Python, which doesn't have a nice ```Set``` natively.  But in Scala, it can easily be flipped around.<commit_msg>Better space explanation<commit_after>Updates / Issues for Scala conversion
===========================================

Internal weight representation
------------------------------

The Python version (deliberately) has two modes of operation : Training and 'Real'.

During training, the 'weight' associated with every hash entry corresponds to the current count of plusses and minuses to that item, and 
the total/timestamp is there to record how long the weight has lingered in each state.

The reason for capturing the average weight (based on the overall training period) is that *honnibal* observes that 
the current count is too jittery : It's much more reasonable to use the overall average values during training
when it comes to 'real life' usage of the library.  

Hence the need for the ```average_weights()``` function, which effectively replaces the weights calculated during training with
an average weight figure.

However : This introduces a before/after split.  Once the ```average_weights()``` function is called, the perceptron(s) are in 
'real world' mode, and can't be trained further.

For the Scala version, the three integers (current, total, ts) are always preserved (and stored to disk as integers).  This has the following benefits :

 * The average weights, when required, are calculated on-the-fly (not a significant computation)

 * The perceptrons are always ready to do further training, since sufficient state is stored

 * Despite more data being stored (i.e. more of the important state), less disk space is actually used (since many entries are low-number-of-digit integers)
 
One argument for doing it this way is that it means that ```score()``` doesn't have to be written twice.  However, Scala is very comfortable with
juggling functions around, so the Scala ```score()``` method can simply be passed either the ```current``` or ```average``` function.


Internal dependency list representation
------------------------------------------------------------

Scala lists are quicker to prepend to, so instead of maintaining ```stack``` using ```.append()``` and ```stack[-1]```, 
the Scala version simply does ``` x :: stack``` and checks ```stack.head```.


Gold Move Logic
------------------------------------------------------------

The ```gold_moves``` logic is essentially ```Set(gold_moves) = Set( all_moves - (all_moves - valid_moves - bad_moves) )```.  
This makes sense in Python, which doesn't have a nice ```Set``` natively.  But in Scala, it can easily be flipped around.
{'fraction_non_alphanumeric': 0.15367528271405492, 'fraction_numerical': 0.0004038772213247173, 'mean_word_length': 3.4904805077062555, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2453985', 'n_tokens_mistral': 1210, 'n_tokens_neox': 1150, 'n_words': 646}