<commit_before><commit_msg>Create code.md<commit_after>Follow the following steps to write any new code:

1. write pseudocode as carefully as you can, typically using latex "algorithm" package.  there are several examples here: 
http://arxiv.org/abs/1609.05148.
the structure is always the same:

  1. input: type & size of each input
  2. output: type and size of each output
  3. function name
  4. step-by-step instructions, comments where appropriate
  
2. show pseudocode to a team-mate, ask them if they understand the input, the output, and the goal
3. describe simulations (possibly in pseudocode if necessary) that you will write to test the code
4. provide details of parameter values for each simulation setting
5. choose visualizations for qualitatively evaluating performance
6. specify metrics for quantitatively evaluating performance
8. write simulation code
9. plot simulations
9. write algorithm code
10. write qualitative evaluation code
11. qualitatively evaluate
12. write quantitative evaluation code
13. quantitatively evaluate simulations
14. get real data in exact same format as simulation expects
15. apply exact same code to real data, generating estimates, qualitative and quantitative results. 
{'fraction_non_alphanumeric': 0.0457516339869281, 'fraction_numerical': 0.027777777777777776, 'mean_word_length': 4.593607305936073, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3530553', 'n_tokens_mistral': 313, 'n_tokens_neox': 282, 'n_words': 170}