<commit_before>==============
saferedisqueue
==============

A small wrapper around `Redis <http://www.redis.io>`_ (using the "standard"
`Python redis client lib <https://pypi.python.org/pypi/redis>`_) that provides
access to a FIFO queue and allows upstream code to mark pop'ed items as
processed successfully (ACK'ing) or unsucessfully (FAIL'ing).

Failed items are automatically requeued. Addionally a backup is kept for items
that were neither ack'ed nor fail'ed, i.e. in case the consumer crashes. The
backup items will be requeued as soon as one of the consumer(s) comes up
again.

All actions are atomic and it is safe to have multiple producers and consumers
accessing the same queue concurrently.


Version compatibility with redis.py
------------------------------------

===============      ===============
redis.py             saferedisqueue
===============      ===============
2.4.10 - 2.6.x       1.x
2.7.0 - 2.7.5        no compatible version
2.7.6 - 2.10.x       2.x
===============      ===============


Usage as a library
==================

>>> queue = SafeRedisQueue(name='test')
>>> queue.put("Hello World")
>>> queue.put("Foo bar")
>>> queue.get()
('595d43b2-2e49-4e96-a1d2-0848d1c7f0d3', 'Hello World')
>>> queue.get()
('1df060eb-b578-499d-bede-20db9da8184e', 'Foo bar')

Note: to be compatible with previous versions, 2 aliases `push/pop` exist. Start using the new `put/get` terminology as soon as possible since `push/pop` will be deleted in a future version.


ACK'ing and FAIL'ing
--------------------

>>> queue.put("Good stuff")
>>> queue.put("Bad stuff")
>>> uid_good, payload_good = queue.get()
>>> uid_bad, payload_bad = queue.get()
...
# process the payloads...
...
>>> queue.ack(uid_good)  # done with that one
>>> queue.fail(uid_bad)  # something didn't work out with that one, let's requeue
>>> uid, payload = queue.get()  # get again; we get the requeued payload again
>>> uid == uid_bad
True
...
# try again
...
>>> queue.ack(uid)  # now it worked; ACK the stuff now
<commit_msg>Update formatting in README.txt<commit_after>==============
saferedisqueue
==============

A small wrapper around `Redis <http://www.redis.io>`_ (using the "standard"
`Python redis client lib <https://pypi.python.org/pypi/redis>`_) that provides
access to a FIFO queue and allows upstream code to mark pop'ed items as
processed successfully (ACK'ing) or unsucessfully (FAIL'ing).

Failed items are automatically requeued. Addionally a backup is kept for items
that were neither ack'ed nor fail'ed, i.e. in case the consumer crashes. The
backup items will be requeued as soon as one of the consumer(s) comes up
again.

All actions are atomic and it is safe to have multiple producers and consumers
accessing the same queue concurrently.


Version compatibility with redis.py
------------------------------------

===============      ===============
redis.py             saferedisqueue
===============      ===============
2.4.10 - 2.6.x       1.x
2.7.0 - 2.7.5        no compatible version
2.7.6 - 2.10.x       2.x
===============      ===============


Usage as a library
==================

>>> queue = SafeRedisQueue(name='test')
>>> queue.put("Hello World")
>>> queue.put("Foo bar")
>>> queue.get()
('595d43b2-2e49-4e96-a1d2-0848d1c7f0d3', 'Hello World')
>>> queue.get()
('1df060eb-b578-499d-bede-20db9da8184e', 'Foo bar')

Note: to be compatible with previous versions, 2 aliases ``push/pop`` exist. Start using the new ``put/get`` terminology as soon as possible since ``push/pop`` will be deleted in a future version.


ACK'ing and FAIL'ing
--------------------

>>> queue.put("Good stuff")
>>> queue.put("Bad stuff")
>>> uid_good, payload_good = queue.get()
>>> uid_bad, payload_bad = queue.get()
...
# process the payloads...
...
>>> queue.ack(uid_good)  # done with that one
>>> queue.fail(uid_bad)  # something didn't work out with that one, let's requeue
>>> uid, payload = queue.get()  # get again; we get the requeued payload again
>>> uid == uid_bad
True
...
# try again
...
>>> queue.ack(uid)  # now it worked; ACK the stuff now

{'fraction_non_alphanumeric': 0.20831283817019183, 'fraction_numerical': 0.029513034923757994, 'mean_word_length': 3.4545454545454546, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 91, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2160972', 'n_tokens_mistral': 1340, 'n_tokens_neox': 1230, 'n_words': 437}