<commit_before>========
Commands
========

ArchiveBot listens to commands prefixed with ``!``.

archive
=======

``!archive URL``, ``!a URL``
    begin recursive retrieval from a URL

::

    > !archive http://artscene.textfiles.com/litpacks/
    < Archiving http://artscene.textfiles.com/litpacks/.
    < Use !status 43z7a11vo6of3a7i173441dtc for updates, !abort
      43z7a11vo6of3a7i173441dtc to abort.

ArchiveBot does not ascend to parent links. This means that everything
under the ``litpacks`` directory will be downloaded. For example,
``/litpacks/hello.html`` will be downloaded but not ``/hello.html``.

If you leave out the trailing slash, eg ``/litpacks``, it will consider
that to be a file and download everything under ``/``.

Accepted parameters
+++++++++++++++++++

``--ignore-sets SET1,...,SETN``
    specify sets of URL patterns to ignore::

       > !archive http://example.blogspot.com/ncr --ignore-sets=blogs,forums
       < Archiving http://example.blogspot.com/ncr.
       < 14 ignore patterns loaded.
       < Use !status 5sid4pgxkiu6zynhbt3q1gi2s for updates, !abort
         5sid4pgxkiu6zynhbt3q1gi2s to abort.

``--no-offsite-links``
    do not download links to offsite pages::

        > !archive http://example.blogspot.com/ncr
        >    --no-offsite-links
        < Archiving http://example.blogspot.com/ncr.
        < Offsite links will not be grabbed.
        < Use !status 5sid4pgxkiu6zynhbt3q1gi2s for updates, !abort
          5sid4pgxkiu6zynhbt3q1gi2s to abort.

    ArchiveBot's default behavior with !archive is to recursively<commit_msg>doc: Note that URLs are case-sensitive<commit_after>========
Commands
========

ArchiveBot listens to commands prefixed with ``!``.

archive
=======

``!archive URL``, ``!a URL``
    begin recursive retrieval from a URL

::

    > !archive http://artscene.textfiles.com/litpacks/
    < Archiving http://artscene.textfiles.com/litpacks/.
    < Use !status 43z7a11vo6of3a7i173441dtc for updates, !abort
      43z7a11vo6of3a7i173441dtc to abort.

ArchiveBot does not ascend to parent links. This means that everything
under the ``litpacks`` directory will be downloaded. For example,
``/litpacks/hello.html`` will be downloaded but not ``/hello.html``.

If you leave out the trailing slash, eg ``/litpacks``, it will consider
that to be a file and download everything under ``/``.

URLs are treated as case-sensitive. ``/litpacks`` is different from
``/LitPacks``.

Accepted parameters
+++++++++++++++++++

``--ignore-sets SET1,...,SETN``
    specify sets of URL patterns to ignore::

       > !archive http://example.blogspot.com/ncr --ignore-sets=blogs,forums
       < Archiving http://example.blogspot.com/ncr.
       < 14 ignore patterns loaded.
       < Use !status 5sid4pgxkiu6zynhbt3q1gi2s for updates, !abort
         5sid4pgxkiu6zynhbt3q1gi2s to abort.

``--no-offsite-links``
    do not download links to offsite pages::

        > !archive http://example.blogspot.com/ncr
        >    --no-offsite-links
        < Archiving http://example.blogspot.com/ncr.
        < Offsite links will not be grabbed.
        < Use !status 5sid4pgxkiu6zynhbt3q1gi2s for updates, !abort
          5sid4pgxkiu6zynhbt3q1gi2s to abort.

    ArchiveBot's default behavior with !archive is to recursively
{'fraction_non_alphanumeric': 0.12870681748700702, 'fraction_numerical': 0.03362885967594008, 'mean_word_length': 3.310935441370224, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3168546', 'n_tokens_mistral': 1133, 'n_tokens_neox': 1064, 'n_words': 304}