<story><title>SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue</title><url>https://www.prequel.co/blog/sql-maxis-why-we-ditched-rabbitmq-and-replaced-it-with-a-postgres-queue</url></story><parent_chain><item><author>ryanjshaw</author><text>&amp;gt; I&amp;#x27;ve never seen this in years of dealing with RabbitMQ.&lt;p&gt;Did you do long running jobs like they did? It&amp;#x27;s a stereotype, but I don&amp;#x27;t think they used the technology correctly here -- you&amp;#x27;re not supposed to hold onto messages for &lt;i&gt;hours&lt;/i&gt; before acknowledging. They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking&amp;#x2F;completion handled inside... a database.</text></item><item><author>mark242</author><text>In summary -- their RabbitMQ consumer library and config is broken in that their consumers are fetching additional messages when they shouldn&amp;#x27;t. I&amp;#x27;ve never seen this in years of dealing with RabbitMQ. This caused a cascading failure in that consumers were unable to grab messages, rightfully, when only one of the messages was manually ack&amp;#x27;ed. Fixing this one fetch issue with their consumer would have fixed the entire problem. Switching to pg probably caused them to rewrite their message fetching code, which probably fixed the underlying issue.&lt;p&gt;It ultimately doesn&amp;#x27;t matter because of the low volume they&amp;#x27;re dealing with, but gang, &amp;quot;just slap a queue on it&amp;quot; gets you the same results as &amp;quot;just slap a cache on it&amp;quot; if you don&amp;#x27;t understand the tool you&amp;#x27;re working with. If they knew that some jobs would take hours and some jobs would take seconds, why would you not immediately spin up four queues. Two for the short jobs (one acting as a DLQ), and two for the long jobs (again, one acting as a DLQ). Your DLQ queues have a low TTL, and on expiration those messages get placed back onto the tail of the original queues. Any failure by your consumer, and that message gets dropped onto the DLQ and your overall throughput is determined by the number * velocity of your consumers, and not on your queue architecture.&lt;p&gt;This pg queue will last a very long time for them. Great! They&amp;#x27;re willing to give up the easy fanout architecture for simplicity, which again at their volume, sure, that&amp;#x27;s a valid trade. At higher volumes, they should go back to the drawing board.</text></item></parent_chain><comment><author>mark242</author><text>The short answer is &amp;quot;yes&amp;quot; but the questions that you should be asking are: A) How long am I willing to block the queue for additional consumers, B) How committed am I to getting close to exactly-once processing, and C) how tolerant of consumer failure should I be? Depending on the answer to those three questions is what drives your queue architecture. Note that this has nothing to do with time spent processing messages or &amp;quot;long running jobs&amp;quot;.&lt;p&gt;Assume that your producers will be able to spike and generate messages faster than your consumers can process them. This is normal! This is why you have a queue in the first place! If your jobs take 5 seconds or 5 hours, your strategy is influenced by the answers to those three questions. For example -- if you&amp;#x27;re willing to drop a message if a consumer gets power-cycled, then yeah, you&amp;#x27;d immediately ack the request and put it back onto a dead letter queue if your consumer runs into an exception. Alternatively, if you&amp;#x27;re unwilling to block and you want to be very tolerant of consumer failure, you&amp;#x27;d fan out your queues and have your consumers checking multiple queues in parallel. Etc etc etc, you get the drift.&lt;p&gt;Keep in mind also that this isn&amp;#x27;t specific to RabbitMQ! You&amp;#x27;d want to answer the same questions if you were using SQS, or if you were using Kafka, or if you were using 0mq, or if you were using Redis queues, or if you were using pg queues.</text></comment>
<story><title>SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue</title><url>https://www.prequel.co/blog/sql-maxis-why-we-ditched-rabbitmq-and-replaced-it-with-a-postgres-queue</url></story><parent_chain><item><author>ryanjshaw</author><text>&amp;gt; I&amp;#x27;ve never seen this in years of dealing with RabbitMQ.&lt;p&gt;Did you do long running jobs like they did? It&amp;#x27;s a stereotype, but I don&amp;#x27;t think they used the technology correctly here -- you&amp;#x27;re not supposed to hold onto messages for &lt;i&gt;hours&lt;/i&gt; before acknowledging. They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking&amp;#x2F;completion handled inside... a database.</text></item><item><author>mark242</author><text>In summary -- their RabbitMQ consumer library and config is broken in that their consumers are fetching additional messages when they shouldn&amp;#x27;t. I&amp;#x27;ve never seen this in years of dealing with RabbitMQ. This caused a cascading failure in that consumers were unable to grab messages, rightfully, when only one of the messages was manually ack&amp;#x27;ed. Fixing this one fetch issue with their consumer would have fixed the entire problem. Switching to pg probably caused them to rewrite their message fetching code, which probably fixed the underlying issue.&lt;p&gt;It ultimately doesn&amp;#x27;t matter because of the low volume they&amp;#x27;re dealing with, but gang, &amp;quot;just slap a queue on it&amp;quot; gets you the same results as &amp;quot;just slap a cache on it&amp;quot; if you don&amp;#x27;t understand the tool you&amp;#x27;re working with. If they knew that some jobs would take hours and some jobs would take seconds, why would you not immediately spin up four queues. Two for the short jobs (one acting as a DLQ), and two for the long jobs (again, one acting as a DLQ). Your DLQ queues have a low TTL, and on expiration those messages get placed back onto the tail of the original queues. Any failure by your consumer, and that message gets dropped onto the DLQ and your overall throughput is determined by the number * velocity of your consumers, and not on your queue architecture.&lt;p&gt;This pg queue will last a very long time for them. Great! They&amp;#x27;re willing to give up the easy fanout architecture for simplicity, which again at their volume, sure, that&amp;#x27;s a valid trade. At higher volumes, they should go back to the drawing board.</text></item></parent_chain><comment><author>xahrepap</author><text>I’ve used RabbitMq to do long running jobs. Jobs that take hours and hours to complete. Occasionally even 1-2 days.&lt;p&gt;It did take some configuring to get it working. Between acking appropriately and the prefetch (qos perhaps? Can’t remember, don’t have it in front of me). We were able to make it work. It was pretty straightforward it never even crossed my mind that this isn’t a correct use case for RMQ.&lt;p&gt;(Used the Java client.)</text></comment>