<commit_before># Introduction to crawler

Introduction:

The crawler cannot entirely be built using lein deps. I pushed a few
commits to wtetzner's uri library that we use so we have to run
<code>lein install</code> on it to make things work:

<pre>
 git clone https://github.com/wtetzner/exploding-fish.git
 cd exploding-fish
 lein install
</pre>
<commit_msg>added natty install instructions<commit_after># Introduction to crawler

Introduction:

The crawler cannot entirely be built using lein deps. I pushed a few
commits to wtetzner's uri library that we use so we have to run
<code>lein install</code> on it to make things work:

<pre>
 git clone https://github.com/wtetzner/exploding-fish.git
 cd exploding-fish
 lein install
</pre>

Next, install natty:

<pre>
git clone https://github.com/joestelmach/natty.git
cd natty
export MAVEN_OPTS=-Xmx1024m
mvn install
</pre>

After this, this project can be run from the repl (currently the only
way to use it).
{'fraction_non_alphanumeric': 0.07700312174817898, 'fraction_numerical': 0.004162330905306972, 'mean_word_length': 4.465909090909091, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2426426', 'n_tokens_mistral': 311, 'n_tokens_neox': 300, 'n_words': 132}