<commit_before>pystemon
========
Monitoring tool for PasteBin-alike sites written in Python

Copyleft AGPLv3 - <NAME> - <EMAIL>  
Feel free to use the code, but please share the changes you've made

Features:
---------
* search for regular expressions in pasties
* flexible design, minimal effort to add another paste* site
* use custom download functions for complex pastie sites
* uses multiple threads per unique site to download the pastes
* waits a random time (within a range) before downloading the latest pastes, time customizable per site
* (optional) only trigger on X hits in the same pastie
* (optional) exclude matching pasties if exclusion regex matches
* (optional) allow additional email recipients per search pattern
* (optional) uses random User-Agents
* (optional) uses random proxies
* removes a proxy if it is unreliable (fails 5 times)
* (optional) compress saved files with Gzip. (no zip to limit external dependencies)

Python Dependencies
-------------------
* PyYAML
* BeautifulSoup
* PyMongo (For Optional Mongodb support)

Limitations:
------------
* Only HTTP proxies are allowed
* Only HTTP urls will use proxies

Usage
------
```
Usage: pystemon.py [options]
Options:
      -h, --help            show this help message and exit  
      -c FILE, --config=FILE  
                            load configuration from file  
      -d, --daemon          runs in background as a daemon (NOT IMPLEMENTED)  
      -s, --stats           display statistics about the running threads (NOT IMPLEMENTED)    
      -v                    outputs more information  

Default configuration file: /etc/pystemon.yaml or pystemon.yaml in current directory
``` <commit_msg>update readme with daemon support<commit_after>pystemon
========
Monitoring tool for PasteBin-alike sites written in Python

Copyleft AGPLv3 - <NAME> - <EMAIL>  
Feel free to use the code, but please share the changes you've made

Features:
---------
* search for regular expressions in pasties
* flexible design, minimal effort to add another paste* site
* use custom download functions for complex pastie sites
* uses multiple threads per unique site to download the pastes
* waits a random time (within a range) before downloading the latest pastes, time customizable per site
* (optional) only trigger on X hits in the same pastie
* (optional) exclude matching pasties if exclusion regex matches
* (optional) allow additional email recipients per search pattern
* (optional) uses random User-Agents
* (optional) uses random proxies
* removes a proxy if it is unreliable (fails 5 times)
* (optional) compress saved files with Gzip. (no zip to limit external dependencies)
* can run as daemon

Python Dependencies
-------------------
* PyYAML
* BeautifulSoup
* PyMongo (For Optional Mongodb support)

Limitations:
------------
* Only HTTP proxies are allowed
* Only HTTP urls will use proxies

Usage
------
```
Usage: pystemon.py [options]
Options:
      -h, --help            show this help message and exit  
      -c FILE, --config=FILE  
                            load configuration from file  
      -d, --daemon          runs in background as a daemon  
      -k, --kill            kill pystemon daemon
      -s, --stats           display statistics about the running threads (NOT IMPLEMENTED)    
      -v                    outputs more information  

Default configuration file: /etc/pystemon.yaml or pystemon.yaml in current directory
``` 
{'fraction_non_alphanumeric': 0.08704512372634643, 'fraction_numerical': 0.0011644832605531296, 'mean_word_length': 2.847704367301232, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5023723', 'n_tokens_mistral': 905, 'n_tokens_neox': 867, 'n_words': 423}