<commit_before>IDEAS

- make end2end test use this ReportJob and make assertions about it
- some way to stop ensure the report job doesnt get stuck between 1000 url jobs
- create a relationship redirects_to for 302s and enqueue the redirect url
- change log {"msg"} to log "msg"
- can we change Page.retrieve_links_for(page) to be page.retrieve_links ?
- is there a better way to count all pages rather than iterating through all nodes?

CURRENT

- create a ReportJob which outputs a report as json<commit_msg>next: fix other end2end tests to use report, not TestJob<commit_after>IDEAS

- make end2end test use this ReportJob and make assertions about it
- some way to stop ensure the report job doesnt get stuck between 1000 url jobs
- create a relationship redirects_to for 302s and enqueue the redirect url
- change log {"msg"} to log "msg"
- can we change Page.retrieve_links_for(page) to be page.retrieve_links ?
- is there a better way to count all pages rather than iterating through all nodes?

CURRENT

- fix other end2end tests to use report, not TestJob
{'fraction_non_alphanumeric': 0.044172932330827065, 'fraction_numerical': 0.016917293233082706, 'mean_word_length': 4.325, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1587829', 'n_tokens_mistral': 304, 'n_tokens_neox': 285, 'n_words': 160}