<commit_before>autowget
========

<h1> Overview: </h1>

Uses wget to get webpage and then saves to a file and then pushes changes to your github page or ftps file(s) to your server

<h1> Instructions: </h1>
<h3> What's needed: </h3>
<ul>
<li>Wget</li>
</ul>

`git clone https://www.github.com/EricLagerg/autowget.git`
`cd /path/to/autowget/`
`mv autowget /place/where/you/store/scripts/`
`chmod 755 autowget`

Also, in order for the logs to work, you need to 

`cd /var/log/ && touch autowget.log`
`chmod 666 autowget.log`

Permissions aren't set in stone. You'll just need autowget to be writable from whichever user is running the script, and you'll need autowget to be executable by whichever user you want running the script.



<h1> Documentation:</h1>

Fill out correct paths to wanted items. Make sure the git command "git push..." will automatically use either your SSH keys OR the correct username/password. Documentation can be found on GitHub's website, StackOverflow, or similar sites. If desired, move the autowget file to cron.daily/monthly/etc so you can let it run. Make sure permissions are correct. Additional sites can be fetched by adding the first line again, but changing the remote URL. Make sure to add more time to
the sleep counter if the sites are large (not 100% sure this is necrssary).

<h1> To-Do List: </h1>

<ul>
<li>Add email confirmation</li>
<li>Append fetched URL to master directory list</li>
</ul><commit_msg>removed incorrect line from documentation<commit_after>autowget
========

<h1> Overview: </h1>

Uses wget to get webpage and then saves to a file and then pushes changes to your github page or ftps file(s) to your server

<h1> Instructions: </h1>
<h3> What's needed: </h3>
<ul>
<li>Wget</li>
</ul>

`git clone https://www.github.com/EricLagerg/autowget.git`
`cd /path/to/autowget/`
`mv autowget /place/where/you/store/scripts/`
`chmod 755 autowget`

Also, in order for the logs to work, you need to 

`cd /var/log/ && touch autowget.log`
`chmod 666 autowget.log`

Permissions aren't set in stone. You'll just need autowget to be writable from whichever user is running the script, and you'll need autowget to be executable by whichever user you want running the script.



<h1> Documentation:</h1>

Fill out correct paths to wanted items. Make sure the git command "git push..." will automatically use either your SSH keys OR the correct username/password. Documentation can be found on GitHub's website, StackOverflow, or similar sites. If desired, move the autowget file to cron.daily/monthly/etc so you can let it run. Make sure permissions are correct. Additional sites can be fetched by adding the first line again, but changing the remote URL.

<h1> To-Do List: </h1>

<ul>
<li>Add email confirmation</li>
<li>Append fetched URL to master directory list</li>
</ul>
{'fraction_non_alphanumeric': 0.09723207948899928, 'fraction_numerical': 0.01242015613910575, 'mean_word_length': 4.269158878504673, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5336677', 'n_tokens_mistral': 925, 'n_tokens_neox': 866, 'n_words': 406}