<story><title>SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue</title><url>https://www.prequel.co/blog/sql-maxis-why-we-ditched-rabbitmq-and-replaced-it-with-a-postgres-queue</url></story><parent_chain><item><author>nemothekid</author><text>&amp;gt;&lt;i&gt;To make all of this run smoothly, we enqueue and dequeue thousands of jobs every day.&lt;/i&gt;&lt;p&gt;If you your needs aren&amp;#x27;t that expensive, and you don&amp;#x27;t anticipate growing a ton, then it&amp;#x27;s probably a smart technical decision to minimize your operational stack. Assuming 10k&amp;#x2F;jobs a day, thats roughly 7 jobs per minute. Even the most unoptimized database should be able to handle this.</text></item></parent_chain><comment><author>hinkley</author><text>Years of being bullshitted have taught me to instantly distrust anyone who is telling me about how many things they do per day. Jobs or customers per day is something to tell you banker, or investors. For tech people it’s per second, per minute, maybe per hour, or self aggrandizement.&lt;p&gt;A million requests a day sounds really impressive, but it’s 12req&amp;#x2F;s which is not a lot. I had a project that needed 100 req&amp;#x2F;s ages ago.  That was considered a reasonably complex problem but not world class, and only because C10k was an open problem. Now you could do that with a single 8xlarge. You don’t even need a cluster.&lt;p&gt;10k tasks a day is 7 per minute. You could do that with Jenkins.</text></comment>
<story><title>SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue</title><url>https://www.prequel.co/blog/sql-maxis-why-we-ditched-rabbitmq-and-replaced-it-with-a-postgres-queue</url></story><parent_chain><item><author>nemothekid</author><text>&amp;gt;&lt;i&gt;To make all of this run smoothly, we enqueue and dequeue thousands of jobs every day.&lt;/i&gt;&lt;p&gt;If you your needs aren&amp;#x27;t that expensive, and you don&amp;#x27;t anticipate growing a ton, then it&amp;#x27;s probably a smart technical decision to minimize your operational stack. Assuming 10k&amp;#x2F;jobs a day, thats roughly 7 jobs per minute. Even the most unoptimized database should be able to handle this.</text></item></parent_chain><comment><author>MuffinFlavored</author><text>&amp;gt; Even the most unoptimized database should be able to handle this.&lt;p&gt;Anybody had any success running a queue on top of... sqlite?&lt;p&gt;With the way the sqlite file locking mechanisms work, are you basically guaranteed really low concurrency? You can have lots of readers but not really a lot of writers, and in order to pop a job off of the queue you need to have a process spinning waiting for work, move its status from &amp;quot;to do&amp;quot; to &amp;quot;in progress&amp;quot; and then &amp;quot;done&amp;quot; or &amp;quot;error&amp;quot;, which is sort of &amp;quot;write&amp;quot; heavy?&lt;p&gt;&amp;gt; An EXCLUSIVE lock is needed in order to write to the database file. Only one EXCLUSIVE lock is allowed on the file and no other locks of any kind are allowed to coexist with an EXCLUSIVE lock. In order to maximize concurrency, SQLite works to minimize the amount of time that EXCLUSIVE locks are held.&lt;p&gt;&amp;gt; You can avoid locks when reading, if you set database journal mode to Write-Ahead Logging (see: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.sqlite.org&amp;#x2F;wal.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.sqlite.org&amp;#x2F;wal.html&lt;/a&gt;).</text></comment>