<commit_before>./setup.sh
ruby download_gems.rb # Takes ~20 hours running 10 threads to download
115k gems

# run this to monitor how gems were download so far
while :; do clear; ls | grep "\.gem" | wc -l; sleep 60; done

# move the gems into gems/all
for f in `ls | grep "\.gem"` ; do mv $f gems/all/ ; done

ruby search_for_ip_addresses.rb 100 > results_for_ip_address_search.txt<commit_msg>Format the readme<commit_after>Getting started

    ./setup.sh

Download 115,000 gems. Takes ~20 hours running 10 threads to download

    ruby download_gems.rb

Run this to monitor how gems were download so far

    while :; do clear; ls | grep "\.gem" | wc -l; sleep 60; done

Move the gems into gems/all

    for f in `ls | grep "\.gem"` ; do mv $f gems/all/ ; done

Make sub directories

    ruby directory_generator.rb





ruby search_for_ip_addresses.rb 100 > results_for_ip_address_search.txt
{'fraction_non_alphanumeric': 0.08734602463605823, 'fraction_numerical': 0.030235162374020158, 'mean_word_length': 3.4477611940298507, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6973422', 'n_tokens_mistral': 344, 'n_tokens_neox': 286, 'n_words': 117}