<commit_before># Sidekiq::Grouping

<a href="https://evilmartians.com/?utm_source=sidekiq-grouping-gem">
<img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by <NAME>" width="236" height="54">
</a>

Allows to combine similar sidekiq jobs into groups to process them at once.

Useful for:
* Grouping asynchronous API index calls into bulks for bulk updating/indexing.
* Periodical batch updating of recently changing database counters.

## Usage

Create a worker:

```ruby
class ElasticBulkIndexWorker
  include Sidekiq::Worker

  sidekiq_options(
    queue: :group_by_size,
    batch_flush_size: 30,     # Jobs will be combined when queue size exceeds 30
    batch_flush_interval: 60, # Jobs will be combined every 60 seconds
    batch_unique: true,       # Prevents jobs with identical arguments to be enqueued
    retry: 5
  )

  def perform(group)
    client = Elasticsearch::Client.new
    client.bulk(body: group.flatten)
  end
end
```

Perform a jobs:

```ruby
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 5, _type: 'user' } })
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 6, _type: 'user' } })
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 7, _type: 'user' } })
...
```

This jobs will be grouped into a single job with the single argument:

```ruby
[
  [{ delete: { _index: 'test', _id: 5, _type: 'user' } }],
  [{ delete: { _index: 'test', _id: 6, _type: 'user' } }],
  [{ delete: { _index: 'test', _id: 7, _type: 'user' } }]
  ...
]
```

## Flush conditions

- If `batch_flush_size` option set - grouping will be performed when batched queue size exceeds this value.
- If `batch_flush_interval` option set - grouping will be performed every given interval.
- If both are set - grouping will be performed when any condition become true. For example, if `batch_flush_interval` is set to 60 and `batch_flush_size` is set to 5 - group task will be enqueued even just 3 tasks are in the queue at the end of minute.

## Web UI

![Web UI](web.png)

Add this line to your `config/routes.rb` to activate web UI:

```ruby
require "sidekiq/grouping/web"
```

## Configuration

```ruby
Sidekiq::Grouping::Config.poll_interval = 5     # Amount of time between polling batches
Sidekiq::Grouping::Config.max_batch_size = 5000 # Maximum batch size allowed
Sidekiq::Grouping::Config.lock_ttl = 1          # Timeout of lock set when batched job enqueues
```

## TODO

1. Add support redis_pool option.
2. Make able to work together with sidekiq-unique-jobs.

## Installation

Add this line to your application's Gemfile:
<commit_msg>README fix<commit_after># Sidekiq::Grouping

<a href="https://evilmartians.com/?utm_source=sidekiq-grouping-gem">
<img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54">
</a>

Allows to combine similar sidekiq jobs into groups to process them at once.

Useful for:
* Grouping asynchronous API index calls into bulks for bulk updating/indexing.
* Periodical batch updating of recently changing database counters.

## Usage

Create a worker:

```ruby
class ElasticBulkIndexWorker
  include Sidekiq::Worker

  sidekiq_options(
    queue: :group_by_size,
    batch_flush_size: 30,     # Jobs will be combined when queue size exceeds 30
    batch_flush_interval: 60, # Jobs will be combined every 60 seconds
    batch_unique: true,       # Prevents jobs with identical arguments to be enqueued
    retry: 5
  )

  def perform(group)
    client = Elasticsearch::Client.new
    client.bulk(body: group.flatten)
  end
end
```

Perform a jobs:

```ruby
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 5, _type: 'user' } })
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 6, _type: 'user' } })
ElasticBulkIndexWorker.perform_async({ delete: { _index: 'test', _id: 7, _type: 'user' } })
...
```

This jobs will be grouped into the single job with the single argument:

```ruby
[
  [{ delete: { _index: 'test', _id: 5, _type: 'user' } }],
  [{ delete: { _index: 'test', _id: 6, _type: 'user' } }],
  [{ delete: { _index: 'test', _id: 7, _type: 'user' } }]
  ...
]
```

## Grouping control

- If `batch_flush_size` option set - grouping will be performed when batched queue size exceeds this value or `Sidekiq::Grouping::Config.max_batch_size` (1000 by default).
- If `batch_flush_interval` option set - grouping will be performed every given interval.
- If both are set - grouping will be performed when any condition become true. For example, if `batch_flush_interval` is set to 60 and `batch_flush_size` is set to 5 - group task will be enqueued even just 3 tasks are in the queue at the end of the minute. In the other hand, if 3 jobs will be enqueued during 10 seconds - they will be grouped and enqueued immediately.

## Web UI

![Web UI](web.png)

Add this line to your `config/routes.rb` to activate web UI:

```ruby
require "sidekiq/grouping/web"
```

## Configuration

```ruby
Sidekiq::Grouping::Config.poll_interval = 5     # Amount of time between polling batches
Sidekiq::Grouping::Config.max_batch_size = 5000 # Maximum batch size allowed
Sidekiq::Grouping::Config.lock_ttl = 1          # Timeout of lock set when batched job enqueues
```

## TODO

1. Add support redis_pool option.
2. Make able to work together with sidekiq-unique-jobs.

## Installation

Add this line to your application's Gemfile:

{'fraction_non_alphanumeric': 0.11247481223667338, 'fraction_numerical': 0.013006045063198388, 'mean_word_length': 4.244956772334294, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2287441', 'n_tokens_mistral': 1867, 'n_tokens_neox': 1726, 'n_words': 624}