<commit_before>asf_releases_aggregator
=======================

Twitter [@asf_releases](http://twitter.com/asf_releases). An automated Twitted bot aggregator.
Retrieves the last releases and tweets them for you.

Uses a simple logger in the server, [Markmail hacky Python API](https://github.com/tupilabs/markmail),
and Tweepy.

## Configuration

The bot is configured through the *aggregator.cfg* file. It is a simple INI file, with field groups.
The following code listing contains an example configuration file.

```
[aggregator]
hour_difference=+12

[markmail]
max_pages=2

[twitter]
tweet_url_length=22
consumer_key=
consumer_secret=
access_key=
access_token=
```

Fields details.

* **hour_difference** UTC time diff in the server
* **max_pages** Maximum of pages to retrieve from Markmail
* **tweet_url_length** URL length to discount from tweet message length
* **consumer_key** Twitter consumer key
* **consumer_secret** Twitter consumer secret
* **access_key** Twitter access key
* **access_token** Twitter access token

## Execution details

1. reads configuration from an INI file with group fields. The file contains settings for
aggregator, Markmail and Twitter
2. reads the last execution time and subject used by the bot
3. retrieves Markmail messages, up to a maximum of pages (step #1)
4. finds messages which title matches to a REGEX, and that are more recent then the
last execution time (step #2)
5. posts a tweet for each message found
6. updates last execution time and subject (used by step #2)

Every step includes logging that can be found in the server.

## License

The code is licensed under the Apache License v2. The Markmail API used by this code, it licensed under
the GPL license. See LICENSE.txt for more.

## Infra

The bot is hosted at TupiLabs VPS servers, and the results can be see at 
[@asf_releases](http://twitter.com/asf_releases).<commit_msg>Add section about Dependencies<commit_after>asf_releases_aggregator
=======================

Twitter [@asf_releases](http://twitter.com/asf_releases). An automated Twitted bot aggregator.
Retrieves the last releases and tweets them for you.

Uses a simple logger in the server, [Markmail hacky Python API](https://github.com/tupilabs/markmail),
and Tweepy.

## Configuration

The bot is configured through the *aggregator.cfg* file. It is a simple INI file, with field groups.
The following code listing contains an example configuration file.

```
[aggregator]
hour_difference=+12

[markmail]
max_pages=2

[twitter]
tweet_url_length=22
consumer_key=
consumer_secret=
access_key=
access_token=
```

Fields details.

* **hour_difference** UTC time diff in the server
* **max_pages** Maximum of pages to retrieve from Markmail
* **tweet_url_length** URL length to discount from tweet message length
* **consumer_key** Twitter consumer key
* **consumer_secret** Twitter consumer secret
* **access_key** Twitter access key
* **access_token** Twitter access token

## Execution details

1. reads configuration from an INI file with group fields. The file contains settings for
aggregator, Markmail and Twitter
2. reads the last execution time and subject used by the bot
3. retrieves Markmail messages, up to a maximum of pages (step #1)
4. finds messages which title matches to a REGEX, and that are more recent then the
last execution time (step #2)
5. posts a tweet for each message found
6. updates last execution time and subject (used by step #2)

Every step includes logging that can be found in the server.

## Dependencies

For a complete list, see requirements.txt. You can install them all with pip by running
`pip install -r requirements.txt`, or use *conda* to manually install them with
[Anaconda](https://www.continuum.io/).

There is one dependency that is not a PIP package, but a Git submodule. You can initialise
it by running the following set of commands.

```
git submodule init
git submodule update
```

## License

The code is licensed under the Apache License v2. The Markmail API used by this code, it licensed under
the GPL license. See LICENSE.txt for more.

## Infra

The bot is hosted at TupiLabs VPS servers, and the results can be see at 
[@asf_releases](http://twitter.com/asf_releases).
{'fraction_non_alphanumeric': 0.08376963350785341, 'fraction_numerical': 0.007139457401237506, 'mean_word_length': 4.9113924050632916, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '437736', 'n_tokens_mistral': 1230, 'n_tokens_neox': 1157, 'n_words': 560}