<commit_before>============
TinyBot v0.1
============

Summary
=======

TinyBot is a tiny web crawler that checks the cohesiveness of your website.

It implements a breadth-first-search to find all links on every page it can find 
on your site. If it finds an external url, it will report it, but not crawl it.

It outputs a csv containing the pages and the amount of times a link was found
to that page. If you are trying to SEO, you'll want a high score for the pages
you want to show off the most.

Currently, the website and output have to be specified in the main java class.
This will be changed in the next update.

What's next
===========

- Inputs and outputs specified from command line
- Different output data structure
- More 'stuff'<commit_msg>Formatting corrected<commit_after>TinyBot v0.1
============

Summary
-------

TinyBot is a tiny web crawler that checks the cohesiveness of your website.

It implements a breadth-first-search to find all links on every page it can find 
on your site. If it finds an external url, it will report it, but not crawl it.

It outputs a csv containing the pages and the amount of times a link was found
to that page. If you are trying to SEO, you'll want a high score for the pages
you want to show off the most.

Currently, the website and output have to be specified in the main java class.
This will be changed in the next update.

What's next
-----------

- [ ] Inputs and outputs specified from command line
- [ ] Different output data structure
- [ ] More 'stuff'
{'fraction_non_alphanumeric': 0.08289473684210526, 'fraction_numerical': 0.002631578947368421, 'mean_word_length': 3.6513761467889907, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2418647', 'n_tokens_mistral': 411, 'n_tokens_neox': 392, 'n_words': 249}