<story><title>Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale</title><url>https://code.facebook.com/posts/1499322996995183/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/</url></story><parent_chain></parent_chain><comment><author>cperciva</author><text>&lt;i&gt;The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links.&lt;/i&gt;&lt;p&gt;Not just connections which traverse aggregated links.  If you&amp;#x27;re load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets.&lt;p&gt;What I don&amp;#x27;t understand is why Facebook didn&amp;#x27;t simply fix their MRU pool: Switching from &amp;quot;most recent &lt;i&gt;response received&lt;/i&gt;&amp;quot; to &amp;quot;most recent &lt;i&gt;request sent&lt;/i&gt;&amp;quot; (out of the links which don&amp;#x27;t have a request already in progress, of course) would have flipped the effect from preferring overloaded links to avoiding them.</text></comment>
<story><title>Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale</title><url>https://code.facebook.com/posts/1499322996995183/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/</url></story><parent_chain></parent_chain><comment><author>jxf</author><text>This was a great investigation of what must surely have been a complicated, frustrating, and expensive problem. Awesome writeup.&lt;p&gt;I love reading about post-mortems like this, even if they&amp;#x27;re unlikely to happen at my startup, because the problem-solving techniques that get displayed tend to generalize to things of almost any size.</text></comment>