<commit_before># Todo

1. Allow user to supply aspec ratio and check for it with pip
2. ~~Check for content type with response.headers (and json)~~
3. ~~Restructure database, adding int time columns to both tables~~
5. ~~Generally make the database more sensible (maybe an ID column but keeping links unique)~~
4. Improve database querying, no reason to pull links that are months old (allow user to specify this in settings)
<commit_msg>Update todo<commit_after># Todo

1. Allow user to supply aspec ratio and check for it with pip
2. ~~Check for content type with response.headers (and json)~~
3. ~~Restructure database, adding int time columns to both tables~~
5. ~~Generally make the database more sensible (maybe an ID column but keeping links unique)~~
4. ~~Improve database querying, no reason to pull links that are months old (allow user to specify this in settings)~~

{'fraction_non_alphanumeric': 0.07289293849658314, 'fraction_numerical': 0.011389521640091117, 'mean_word_length': 4.707792207792208, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3758778', 'n_tokens_mistral': 229, 'n_tokens_neox': 229, 'n_words': 139}