<commit_before># Hydra
producer-consumer based python task engine


## When to use?

* when your backend task is too much, and want to done as soon as possible.

## Usage

    # define or import your retrieve_items and consume functions here

    from hydra.manager import Manager
    cfg_list = [{'retrieve_items':  retrieve_items, 'consume': consume_item, 'consumer_count':2, 'working_time': [('00:00', '23:59')]}]
    Manager.trigger(**cfg_list)

## How to define a retrieve_items fucntion?

* items are saved in db/mq or any broker.
* make sure do not get the duplicated items.(e.g. might use a global lock)

## how to define a consume function?

nothing special.
<commit_msg>modify readme.md<commit_after># Hydra
producer-consumer based python task engine


## When to use?

* when your backend task is too much, and want to be done as soon as possible.

## Usage

```python
# define or import your retrieve_items and consume functions here

from hydra.manager import Manager
cfg_list = [{'retrieve_items':  retrieve_items, 'consume': consume_item, 'consumer_count':2, 'working_time': [('00:00', '23:59')]}]
Manager.trigger(**cfg_list)
```

## How to define a retrieve_items fucntion?

* items are saved in db/mq or any broker.
* make sure do not get the duplicated items.(e.g. might use a global lock)

## how to define a consume function?

nothing special.

{'fraction_non_alphanumeric': 0.10337243401759531, 'fraction_numerical': 0.013196480938416423, 'mean_word_length': 4.46, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3450291', 'n_tokens_mistral': 440, 'n_tokens_neox': 411, 'n_words': 171}