<story><title>Delayed Job vs. Sidekiq</title><url>https://blog.appsignal.com/2022/02/15/delayed-job-vs-sidekiq-which-is-better.html</url></story><parent_chain><item><author>berkes</author><text>I agree with most of the findings in that article, but I miss one very important one: consistency.&lt;p&gt;Jobs in your queue in the database can be committed in the same transaction that makes the mutation. With redis you have no such thing. Leading to (been there, it hurt) potentially missing jobs or queuing jobs that have no associated mutation in the db.&lt;p&gt;Furthermore, a big downside of redis is that it&amp;#x27;s promise of consistency is low. Mostly by design: redis is not meant to be the primary, one and only store of truth. Commonly, people store stuff in redis that can be re-built from database or some eventlog or so.
But not sidekiq. If your redis starts failing (been there, it hurts), e.g. though OOM errors, you are loosing data. Jobs, with their parameters, are dropped, often irrecoverably. And with Rails&amp;#x27; common set-up, this means you are irrecoverably loosing data: there simply is no way to find what emails you did not send, nor any way to re-queue them. There is no way to re-enqueue those &amp;quot;generateReport&amp;quot; jobs using the data at that point in time and so on.&lt;p&gt;Basically: Redis is quite certainly not the best tool to store your job-queues with associated data, in. Sidekiq is the less sturdy approach of both.</text></item></parent_chain><comment><author>scosman</author><text>Transactional consistency is nice. Maybe worth it for some low-volume jobs.&lt;p&gt;The downside that comes with it is using your primary DB as your queue is that the queues in an app tend to absorb all the exceptional conditions (queue grows to millions during worker outage, backfill which needs to process millions of items). Using your primary DB means all that disk usage, memory usage, and IO hit your primary DB in exceptional ways and impacts your app in more ways than just queue delays. It also scales much less linearly than other queue options.&lt;p&gt;A nice middle ground is to store all your data and state in your DB transactionally, and only queue tiny lightweight jobs with database primary keys (database: sentWelcomeMailDate=nil, queue: sentWelcomeMailForUser:1234).&lt;p&gt;This is another case for neither DJ or Sidekiq. A fully hosted option like SQS addresses the durability concerns of Sidekiq, and the scaling concerns of both. I’ve queued 1B+ jobs in SQS and watched the workers slowly burn through them over the course of a month.&lt;p&gt;DJ and Sidekiq can go quite far. I’ve used both and patched some perf issues on DJ. The lesson learned was queues at scale are hard distributed systems problems - you probably don’t want to have to know how they work, or host the distributed system yourself.</text></comment>
<story><title>Delayed Job vs. Sidekiq</title><url>https://blog.appsignal.com/2022/02/15/delayed-job-vs-sidekiq-which-is-better.html</url></story><parent_chain><item><author>berkes</author><text>I agree with most of the findings in that article, but I miss one very important one: consistency.&lt;p&gt;Jobs in your queue in the database can be committed in the same transaction that makes the mutation. With redis you have no such thing. Leading to (been there, it hurt) potentially missing jobs or queuing jobs that have no associated mutation in the db.&lt;p&gt;Furthermore, a big downside of redis is that it&amp;#x27;s promise of consistency is low. Mostly by design: redis is not meant to be the primary, one and only store of truth. Commonly, people store stuff in redis that can be re-built from database or some eventlog or so.
But not sidekiq. If your redis starts failing (been there, it hurts), e.g. though OOM errors, you are loosing data. Jobs, with their parameters, are dropped, often irrecoverably. And with Rails&amp;#x27; common set-up, this means you are irrecoverably loosing data: there simply is no way to find what emails you did not send, nor any way to re-queue them. There is no way to re-enqueue those &amp;quot;generateReport&amp;quot; jobs using the data at that point in time and so on.&lt;p&gt;Basically: Redis is quite certainly not the best tool to store your job-queues with associated data, in. Sidekiq is the less sturdy approach of both.</text></item></parent_chain><comment><author>cardy31</author><text>You can run Redis in a persistent setup. So each Redis instance has a replica and both master and replica write out to disk. You still don’t get transactions, but depending on your scale a transaction for every job you enqueue may be impossible.&lt;p&gt;If you’re small enough that you don’t knock over your DB by putting jobs in it then the transactions are nice!</text></comment>