<commit_before>= Robotex
== Obey Robots.txt

With one line of code, Robotex (pronounced like "robotics") will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.

Usage:

    robotex = Robotex.new "My User Agent"
    robotex.allowed?("http://www.example.com/foo")
    robotex.delay! # wait until any specified Crawl-Delay has passed

== Acknowledgements

Robotex is a modified version of <NAME>'s excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.
<commit_msg>[doc] add url as an argument to delay! method<commit_after>= Robotex
== Obey Robots.txt

With one line of code, Robotex (pronounced like "robotics") will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.

Usage:

    robotex = Robotex.new "My User Agent"
    robotex.allowed?("http://www.example.com/foo")
    robotex.delay!("http://www.example.com/foo") # wait until any specified Crawl-Delay has passed

== Acknowledgements

Robotex is a modified version of <NAME>'s excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.

{'fraction_non_alphanumeric': 0.07958199356913183, 'fraction_numerical': 0.0, 'mean_word_length': 4.102459016393443, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5949402', 'n_tokens_mistral': 368, 'n_tokens_neox': 365, 'n_words': 168}