<commit_before>qless
=====

Qless is a powerful `Redis`-based job queueing system inspired by
[resque](https://github.com/defunkt/resque#readme),
but built on a collection of Lua scripts, maintained in the
[qless-core](https://github.com/seomoz/qless-core) repo.

Philosophy and Nomenclature
===========================
A `job` is a unit of work identified by a job id or `jid`. A `queue` can contain
several jobs that are scheduled to be run at a certain time, several jobs that are
waiting to run, and jobs that are currently running. A `worker` is a process on a
host, identified uniquely, that asks for jobs from the queue, performs some process
associated with that job, and then marks it as complete. When it's completed, it
can be put into another queue.

Jobs can only be in one queue at a time. That queue is whatever queue they were last
put in. So if a worker is working on a job, and you move it, the worker's request to
complete the job will be ignored.

A job can be `canceled`, which means it disappears into the ether, and we'll never
pay it any mind every again. A job can be `dropped`, which is when a worker fails
to heartbeat or complete the job in a timely fashion, or a job can be `failed`,
which is when a host recognizes some systematically problematic state about the
job. A worker should only fail a job if the error is likely not a transient one;
otherwise, that worker should just drop it and let the system reclaim it.

Features
========

1. __Jobs don't get dropped on the floor__ -- Sometimes workers drop jobs. Qless
  automatically picks them back up and gives them to another worker
1. __Tagging / Tracking__ -- Some jobs are more interesting than others. Track those
  jobs to get updates on their progress. Tag jobs with meaningful identifiers to
  find them quickly in the UI.
1. __Job Dependencies__ -- One job might need to wait for another job to complete
1. __Stats__ -- `qless` automatically keeps statistics about how long jobs wait
  to be processed and how long they take to be processed. Currently, we keep
  track of the count, mean, standard deviation, and a histogram of these times.
1. __Job data is stored temporarily__ -- Job info sticks around for a configurable
  amount of time so you can still look back on a job's history, data, etc.
1. __Priority__ -- Jobs with the same priority get popped in the order they were
  inserted; a higher priority means that it gets popped faster
1. __Retry logic__ -- Every job has a number of retries associated with it, which are
  renewed when it is put into a new queue or completed. If a job is repeatedly
  dropped, then it is presumed to be problematic, and is automatically failed.
1. __Web App__ -- With the advent of a Ruby client, there is a Sinatra-based web
  app that gives you control over certain operational issues
1. __Scheduled Work__ -- Until a job waits for a specified delay (defaults to 0),
  jobs cannot be popped by workers
1. __Recurring Jobs__ -- Scheduling's all well and good, but we also support
  jobs that need to recur periodically.
1. __Notifications__ -- Tracked jobs emit events on pubsub channels as they get
  completed, failed, put, popped, etc. Use these events to get notified of
  progress on jobs you're interested in.

Enqueing Jobs
=============
First things first, require `qless` and create a client. The client accepts all the
same arguments that you'd use when constructing a redis client.

``` ruby
require 'qless'

# Connect to localhost
client = Qless::Client.new
# Connect to somewhere else
client = Qless::Client.new(:host => 'foo.bar.com', :port => 1234)
```

Jobs should be classes or modules that define a `perform` method, which
must accept a single `job` argument:

``` ruby
class MyJobClass
  def self.perform(job)
    # job is an instance of `Qless::Job` and provides access to
    # job.data, a means to cancel the job (job.cancel), and more.
  end
end
```

Now you can access a queue, and add a job to that queue.

``` ruby
# This references a new or existing queue 'testing'
queue = client.queues['testing']
# Let's add a job, with some data. Returns Job ID
queue.put(MyJobClass, :hello => 'howdy')
# => "0c53b0404c56012f69fa482a1427ab7d"
# Now we can ask for a job
job = queue.pop
# => <Qless::Job 0c53b0404c56012f69fa482a1427ab7d (MyJobClass / testing)>
# And we can do the work associated with it!
job.perform
```

The job data must be serializable to JSON, and it is recommended
that you use a hash for it. See below for a list of the supported job options.

The argument returned by `queue.put` is the job ID, or jid. Every Qless
job has a unique jid, and it provides a means to interact with an
existing job:

``` ruby
# find an existing job by it's jid
job = client.jobs[jid]

# Query it to find out details about it:
job.klass # => the class of the job
job.queue # => the queue the job is in
job.data  # => the data for the job
job.history # => the history of what has happened to the job sofar
job.dependencies # => the jids of other jobs that must complete before this one
job.dependents # => the jids of other jobs that depend on this one
job.priority # => the priority of this job
job.tags # => array of tags for this job
job.original_retries # => the number of times the job is allowed to be retried
job.retries_left # => the number of retries left

# You can also change the job in various ways:
job.move("some_other_queue") # move it to a new queue
job.cancel # cancel the job
job.tag("foo") # add a tag
job.untag("foo") # remove a tag
```

Running A Worker
================

The Qless ruby worker was heavily inspired by Resque's worker,
but thanks to the power of the qless-core lua scripts, it is
*much* simpler and you are welcome to write your own (e.g. if
you'd rather save memory by not forking the worker for each job).

As with resque...

* The worker forks a child process for each job in order to provide
   resilience against memory leaks.
* The worker updates its procline with its status so you can see
  what workers are doing using `ps`.
* The worker registers signal handlers so that you can control it
  by sending it signals.
* The worker is given a list of queues to pop jobs off of.
* The worker logs out put based on `VERBOSE` or `VVERBOSE` (very
  verbose) environment variables.
* Qless ships with a rake task (`qless:work`) for running workers.
  It runs `qless:setup` before starting the main work loop so that
  users can load their environment in that task.<commit_msg>update the readme<commit_after>qless
=====

Qless is a powerful `Redis`-based job queueing system inspired by
[resque](https://github.com/defunkt/resque#readme),
but built on a collection of Lua scripts, maintained in the
[qless-core](https://github.com/seomoz/qless-core) repo.

Philosophy and Nomenclature
===========================
A `job` is a unit of work identified by a job id or `jid`. A `queue` can contain
several jobs that are scheduled to be run at a certain time, several jobs that are
waiting to run, and jobs that are currently running. A `worker` is a process on a
host, identified uniquely, that asks for jobs from the queue, performs some process
associated with that job, and then marks it as complete. When it's completed, it
can be put into another queue.

Jobs can only be in one queue at a time. That queue is whatever queue they were last
put in. So if a worker is working on a job, and you move it, the worker's request to
complete the job will be ignored.

A job can be `canceled`, which means it disappears into the ether, and we'll never
pay it any mind every again. A job can be `dropped`, which is when a worker fails
to heartbeat or complete the job in a timely fashion, or a job can be `failed`,
which is when a host recognizes some systematically problematic state about the
job. A worker should only fail a job if the error is likely not a transient one;
otherwise, that worker should just drop it and let the system reclaim it.

Features
========

1. __Jobs don't get dropped on the floor__ -- Sometimes workers drop jobs. Qless
  automatically picks them back up and gives them to another worker
1. __Tagging / Tracking__ -- Some jobs are more interesting than others. Track those
  jobs to get updates on their progress. Tag jobs with meaningful identifiers to
  find them quickly in the UI.
1. __Job Dependencies__ -- One job might need to wait for another job to complete
1. __Stats__ -- `qless` automatically keeps statistics about how long jobs wait
  to be processed and how long they take to be processed. Currently, we keep
  track of the count, mean, standard deviation, and a histogram of these times.
1. __Job data is stored temporarily__ -- Job info sticks around for a configurable
  amount of time so you can still look back on a job's history, data, etc.
1. __Priority__ -- Jobs with the same priority get popped in the order they were
  inserted; a higher priority means that it gets popped faster
1. __Retry logic__ -- Every job has a number of retries associated with it, which are
  renewed when it is put into a new queue or completed. If a job is repeatedly
  dropped, then it is presumed to be problematic, and is automatically failed.
1. __Web App__ -- With the advent of a Ruby client, there is a Sinatra-based web
  app that gives you control over certain operational issues
1. __Scheduled Work__ -- Until a job waits for a specified delay (defaults to 0),
  jobs cannot be popped by workers
1. __Recurring Jobs__ -- Scheduling's all well and good, but we also support
  jobs that need to recur periodically.
1. __Notifications__ -- Tracked jobs emit events on pubsub channels as they get
  completed, failed, put, popped, etc. Use these events to get notified of
  progress on jobs you're interested in.

Enqueing Jobs
=============
First things first, require `qless` and create a client. The client accepts all the
same arguments that you'd use when constructing a redis client.

``` ruby
require 'qless'

# Connect to localhost
client = Qless::Client.new
# Connect to somewhere else
client = Qless::Client.new(:host => 'foo.bar.com', :port => 1234)
```

Jobs should be classes or modules that define a `perform` method, which
must accept a single `job` argument:

``` ruby
class MyJobClass
  def self.perform(job)
    # job is an instance of `Qless::Job` and provides access to
    # job.data, a means to cancel the job (job.cancel), and more.
  end
end
```

Now you can access a queue, and add a job to that queue.

``` ruby
# This references a new or existing queue 'testing'
queue = client.queues['testing']
# Let's add a job, with some data. Returns Job ID
queue.put(MyJobClass, :hello => 'howdy')
# => "0c53b0404c56012f69fa482a1427ab7d"
# Now we can ask for a job
job = queue.pop
# => <Qless::Job 0c53b0404c56012f69fa482a1427ab7d (MyJobClass / testing)>
# And we can do the work associated with it!
job.perform
```

The job data must be serializable to JSON, and it is recommended
that you use a hash for it. See below for a list of the supported job options.

The argument returned by `queue.put` is the job ID, or jid. Every Qless
job has a unique jid, and it provides a means to interact with an
existing job:

``` ruby
# find an existing job by it's jid
job = client.jobs[jid]

# Query it to find out details about it:
job.klass # => the class of the job
job.queue # => the queue the job is in
job.data  # => the data for the job
job.history # => the history of what has happened to the job sofar
job.dependencies # => the jids of other jobs that must complete before this one
job.dependents # => the jids of other jobs that depend on this one
job.priority # => the priority of this job
job.tags # => array of tags for this job
job.original_retries # => the number of times the job is allowed to be retried
job.retries_left # => the number of retries left

# You can also change the job in various ways:
job.move("some_other_queue") # move it to a new queue
job.cancel # cancel the job
job.tag("foo") # add a tag
job.untag("foo") # remove a tag
```

Running A Worker
================

The Qless ruby worker was heavily inspired by Resque's worker,
but thanks to the power of the qless-core lua scripts, it is
*much* simpler and you are welcome to write your own (e.g. if
you'd rather save memory by not forking the worker for each job).

As with resque...

* The worker forks a child process for each job in order to provide
   resilience against memory leaks. Pass the `RUN_AS_SINGLE_PROCESS`
   environment variable to force Qless to not fork the child process.
   Single process mode should only be used in some test/dev
   environments.
* The worker updates its procline with its status so you can see
  what workers are doing using `ps`.
* The worker registers signal handlers so that you can control it
  by sending it signals.
* The worker is given a list of queues to pop jobs off of.
* The worker logs out put based on `VERBOSE` or `VVERBOSE` (very
  verbose) environment variables.
* Qless ships with a rake task (`qless:work`) for running workers.
  It runs `qless:setup` before starting the main work loop so that
  users can load their environment in that task.
{'fraction_non_alphanumeric': 0.07084032331859082, 'fraction_numerical': 0.009150526155253927, 'mean_word_length': 3.934161023325809, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1707418', 'n_tokens_mistral': 3740, 'n_tokens_neox': 3565, 'n_words': 2096}