<commit_before>= CRAWLER!

This is a simple webcrawler written in Ruby. It's not really meant for anything
hardcore and was written originally to see how connected the pages of a site
are; i.e. can a user reach all of the pages we want them to be able to reach
through links.

=== Usage

./crawler.rb [options] URL

=== Options:

-t, --tags: Comma-separated list of html tags to pick out. Used this to find
		any extant "<b>" or "<i>" tags.

-x, --exclude: Comma-separated list of strings which, if present in a gathered
		URL, will cause that URL to be ignored. ".pdf" is a good one.


That's pretty much it. It will not extend itself beyond the initial host,
ignores query strings and fragments. It does not have any current function to
throttle itself and does not handle abnormal HTTP response codes at the moment.

As I say, basic.
<commit_msg>Fixed formatting<commit_after>= CRAWLER!

This is a simple webcrawler written in Ruby. It's not really meant for anything
hardcore and was written originally to see how connected the pages of a site
are; i.e. can a user reach all of the pages we want them to be able to reach
through links.

=== Usage

./crawler.rb [options] URL

=== Options:

-t, --tags: Comma-separated list of html tags to pick out. Used this to find any extant "<b>" or "<i>" tags.

-x, --exclude: Comma-separated list of strings which, if present in a gathered URL, will cause that URL to be ignored. ".pdf" is a good one.


That's pretty much it. It will not extend itself beyond the initial host,
ignores query strings and fragments. It does not have any current function to
throttle itself and does not handle abnormal HTTP response codes at the moment.

As I say, basic.

{'fraction_non_alphanumeric': 0.0695344725987036, 'fraction_numerical': 0.0, 'mean_word_length': 3.6906077348066297, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1911018', 'n_tokens_mistral': 493, 'n_tokens_neox': 471, 'n_words': 287}