<commit_before>Sitemap.xml Automated Testing Script using CasperJS

Usage instructions: casperjs sitemap_xml_testing.js --sitemap=<URL TO SITEMAP> --logfile=<LOG FILE NAME>

This script will attempt to crawl through a specified sitemap to check children pages for broken urls, 
images, css, and Javascript. Errors will be recorded to the logfile specified. This script requires 
casperjs to be installed.

Known issues:
-PhantomJS seems to crash after writing about 3KB of data to a single log file. This might be able to be fixed
by using multiple log files, or splitting data up differently
-PhantomJS crashes on certain pages, albeit consistently. Google suggests that it may be a PhantomJS issue.<commit_msg>Added known issues and cleaned up code a bit.<commit_after>Sitemap.xml Automated Testing Script using CasperJS

Usage instructions: casperjs sitemap_xml_testing.js --sitemap=<URL TO SITEMAP> --logfile=<LOG FILE NAME>

This script will attempt to crawl through a specified sitemap to check children pages for broken urls, 
images, css, and Javascript. Errors will be recorded to the logfile specified. This script requires 
casperjs to be installed.

Known issues:

-PhantomJS seems to crash after writing about 3KB of data to a single log file. This might be able to be fixed
by using multiple log files, or splitting data up differently.

-PhantomJS crashes on certain pages, albeit consistently. Google suggests that it may be a PhantomJS issue.
{'fraction_non_alphanumeric': 0.04249485949280329, 'fraction_numerical': 0.0013708019191226869, 'mean_word_length': 4.770750988142293, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4342462', 'n_tokens_mistral': 405, 'n_tokens_neox': 378, 'n_words': 211}