<story><title>Queues should be empty</title><url>http://joshvoigts.com/articles/queues-should-be-empty/</url></story><parent_chain><item><author>mabbo</author><text>If a queue isn&amp;#x27;t approaching empty, then its input is exceeding its output. Therefore it&amp;#x27;s size is trending to infinity.&lt;p&gt;Since we don&amp;#x27;t have infinite space, we can expect eventually to lose some messages in this scenario.</text></item><item><author>orf</author><text>No they shouldn’t.&lt;p&gt;The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency.&lt;p&gt;If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers &lt;i&gt;waiting&lt;/i&gt; for new messages.&lt;p&gt;If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together.&lt;p&gt;In this case you &lt;i&gt;want&lt;/i&gt; it to &lt;i&gt;always&lt;/i&gt; have something to read from the queue and shut it down the moment this isn’t the case.</text></item></parent_chain><comment><author>klyrs</author><text>&amp;gt; If a queue isn&amp;#x27;t approaching empty, then its input is exceeding its output. Therefore it&amp;#x27;s size is trending to infinity.&lt;p&gt;This is literally the slippery slope fallacy.  You aren&amp;#x27;t accounting for minor fluctuations or finite timescales.  If you were driving a car, you might say &amp;quot;if you aren&amp;#x27;t steering into oncoming traffic then you&amp;#x27;re steering towards the ditch&amp;quot; and conclude that no cars will ever safely reach their destination.&lt;p&gt;If the only valid state for a queue is empty, then why waste time implementing a queue?</text></comment>
<story><title>Queues should be empty</title><url>http://joshvoigts.com/articles/queues-should-be-empty/</url></story><parent_chain><item><author>mabbo</author><text>If a queue isn&amp;#x27;t approaching empty, then its input is exceeding its output. Therefore it&amp;#x27;s size is trending to infinity.&lt;p&gt;Since we don&amp;#x27;t have infinite space, we can expect eventually to lose some messages in this scenario.</text></item><item><author>orf</author><text>No they shouldn’t.&lt;p&gt;The way I see it is that there are two opposing things you can optimise for that depend on queue depth: utilisation or latency.&lt;p&gt;If you care about processing each message as quickly as possible then queues should be empty. This often requires a higher cost and lower utilisation as you inevitably have idle workers &lt;i&gt;waiting&lt;/i&gt; for new messages.&lt;p&gt;If you care about utilisation, then you never want your queue to be empty while something is polling it. This might be some background task that runs on a GPU - every second that sits idle is wasted cash, and those tasks usually benefit heavily from batching inputs together.&lt;p&gt;In this case you &lt;i&gt;want&lt;/i&gt; it to &lt;i&gt;always&lt;/i&gt; have something to read from the queue and shut it down the moment this isn’t the case.</text></item></parent_chain><comment><author>User23</author><text>The Little’s law[1] offers considerable guidance here. In the shortish term it’s safe to model the system as being stationary and then you need to provision for seasonality[2]. Upward or downward trending demand after correcting for seasonality indicates that the system isn’t stationary and your choices are either to scale up or reject requests. Nevertheless the optimal queue size is not necessarily zero except in cases where extremely low latency is necessary. The optimal queue capacity for most web service type applications is one that trends toward almost but not quite full when seasonal requests are higher.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Little%27s_law&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Little%27s_law&lt;/a&gt;&lt;p&gt;[2] broadly construed. For example a service that gets a mass of requests at the top of every hour displays seasonality.</text></comment>