<commit_before># WebArchiveBOT
PHP script to save external links to Internet Archive using the MediaWiki and Internet Archive API.

This experimental tool looks for New Files uploaded to Wikimedia Commons (or any Wiki), extract the external links, and queries to Internet Archive by Wayback Machine to save them. This tools runs continuously as a daemon, and is monitored by Bigbrother.

Page with latest 50 files archived at https://tools.wmflabs.org/webarchivebot

What is included:

* /bin The executable and libraries. For security reasons, keep it outside the public_html!
 * WebArchiveBOT.sh The (bootstarp) script. You should use it and fill it with the appopiate data
 * cli.php The The main script
 * class.php The classes library
 
* /public_html The files intended to be exposed to Internet (optional)
 * index.php The PHP/HTML script to display the latest pages parsed and links archived
 * doc/ The documentation generated with phpDocumentator (optional but useful)<commit_msg>Merging changes from Debug branch<commit_after># WebArchiveBOT
PHP script to save external links to Internet Archive using the MediaWiki and Internet Archive API.

This experimental tool looks for New Files uploaded to Wikimedia Commons (or any Wiki), extract the external links, and queries to Internet Archive by Wayback Machine to save them. This tools runs continuously as a daemon, and is monitored by Bigbrother.

Page with latest 50 files archived at https://tools.wmflabs.org/webarchivebot

## Instalaltion: 

* Run `git clone` to download the files:

    `git clone https://github.com/Amitie10g/WebArchiveBOT.git`

* Create a directory for the backend script (eg "bin") outside the WWW access, and,
  * Copy "WebArchiveBOT.sh" and edit it to add credentials and other settings
  * Symlink "cli.php" and "class.php"

* Create a directory inside the WWW access if you want, and,
  * Copy "index.php"
  * Symlink "template.php"

Ensure the "WebArchiveBOT.sh" has exec permissions
{'fraction_non_alphanumeric': 0.05569620253164557, 'fraction_numerical': 0.0030379746835443038, 'mean_word_length': 4.566197183098591, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4452994', 'n_tokens_mistral': 542, 'n_tokens_neox': 510, 'n_words': 270}