<commit_before>Generic looping through files which calls the filters

Store generated data a bit more obviously away from the source code
First run should make all the necessary directories automatically
Use some tmpfile function for temporary filenames

cvs remove removecomments.py
findallhocdaydebate.py should die if date string empty

My content grabber does this (horrifyingly) early on:
    # remove strange hyphens
    $all_content =~ s/&#150;/-/g;
    $all_content =~ s/&#151;/-/g;

Time stamps aren't all being caught e.g. search for pm in c2daydebatefixcolumnnumbers/daydebThursday13November2003.html
<commit_msg>*** empty log message ***<commit_after>Generic looping through files which calls the filters

Store generated data a bit more obviously away from the source code
First run should make all the necessary directories automatically
Use some tmpfile function for temporary filenames

cvs remove removecomments.py
findallhocdaydebate.py should die if date string empty

My content grabber does this (horrifyingly) early on:
    # remove strange hyphens
    $all_content =~ s/&#150;/-/g;
    $all_content =~ s/&#151;/-/g;


Time stamp exception:
Time stamps aren't all being caught e.g. search for 1.27 pm in c2daydebatefixcolumnnumbers/daydebThursday13November2003.html
  this is a rare case outside of a heading.  explicit repair will be required.  

{'fraction_non_alphanumeric': 0.05989773557341125, 'fraction_numerical': 0.021183345507669833, 'mean_word_length': 4.614754098360656, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6915612', 'n_tokens_mistral': 410, 'n_tokens_neox': 367, 'n_words': 168}