<commit_before># SnapCrawl - crawl a website and take screenshots

`snapcrawl` is a command line utility for crawling a website and saving
screenshots. It is using [Runfile](https://github.com/DannyBen/runfile).

## Features

- Crawl a website to any given depth
- Save full screenshots of all crawled pages
- Skips capturing if the screenshot was already saved recently
- Uses local caching to avoid expensive crawl operations if not needed

## Install Gem Dependencies

Option 1, with bundler:

	$ bundle

Option 2, manually:

	$ gem install runfile
	$ gem install nokogiri
	$ gem install screencap

## Usage

	$ run --help
<commit_msg>add todo<commit_after># SnapCrawl - crawl a website and take screenshots

`snapcrawl` is a command line utility for crawling a website and saving
screenshots. It is using [Runfile](https://github.com/DannyBen/runfile).

## Features

- Crawl a website to any given depth
- Save full screenshots of all crawled pages
- Skips capturing if the screenshot was already saved recently
- Uses local caching to avoid expensive crawl operations if not needed

## Install Gem Dependencies

Option 1, with bundler:

	$ bundle

Option 2, manually:

	$ gem install runfile
	$ gem install nokogiri
	$ gem install screencap

## Usage

	$ run --help


## TODO

[ ] Add caching options to command line
[ ] Add ignore regexes to command line or config
[ ] Add exception handling for when snapping fails




{'fraction_non_alphanumeric': 0.061754385964912284, 'fraction_numerical': 0.002807017543859649, 'mean_word_length': 3.9686411149825784, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1233682', 'n_tokens_mistral': 438, 'n_tokens_neox': 388, 'n_words': 189}