<story><title>An update on Sunday’s service disruption</title><url>https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption</url></story><parent_chain><item><author>jrockway</author><text>This answered all the questions I had.  I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the &amp;quot;default&amp;quot; or &amp;quot;bulk&amp;quot; traffic class being dropped would do it.&lt;p&gt;The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class.  It makes sense that internal blob storage is in the lowest &amp;quot;bulk&amp;quot; class.  Engineers building apps that depend on that know the limitations.  It makes less sense to put customer traffic in that class, though, when you have an SLA to meet for cloud storage.  People outside of Google have no idea that different tasks get different network priorities, and don&amp;#x27;t design their apps with that in mind.  (I run all my stuff on AWS and I have no idea what the failure mode when bandwidth is limited for things like EBS or S3.  It&amp;#x27;s probably the same as Google, but I can&amp;#x27;t design around it because I don&amp;#x27;t actually know what it looks like.)  But, of course, if everything is high priority, nothing is high priority.  I imagine that things in the highest traffic class kept working on Sunday, which is a good outcome.  If everything were in the highest class, then nothing would work.&lt;p&gt;(When I worked at Google, I spent a fair amount of time advocating for a higher traffic class for my application&amp;#x27;s traffic.  If my application still exists, I wonder if it was affected, or if the time I spent on that actually paid off.)</text></item></parent_chain><comment><author>dsfyu404ed</author><text>&amp;gt;Engineers building apps that depend on that know the limitations.&lt;p&gt;As someone who works at a slightly smaller tech company with of similar age with similar infrastructure I assure you this is not the case.  Engineers are building things that rely on other things that rely on other things.  There&amp;#x27;s a point where people don&amp;#x27;t know what their dependencies are.&lt;p&gt;I wouldn&amp;#x27;t be surprised if nobody actually knew there was customer traffic in this class until this happened.</text></comment>
<story><title>An update on Sunday’s service disruption</title><url>https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption</url></story><parent_chain><item><author>jrockway</author><text>This answered all the questions I had.  I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the &amp;quot;default&amp;quot; or &amp;quot;bulk&amp;quot; traffic class being dropped would do it.&lt;p&gt;The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class.  It makes sense that internal blob storage is in the lowest &amp;quot;bulk&amp;quot; class.  Engineers building apps that depend on that know the limitations.  It makes less sense to put customer traffic in that class, though, when you have an SLA to meet for cloud storage.  People outside of Google have no idea that different tasks get different network priorities, and don&amp;#x27;t design their apps with that in mind.  (I run all my stuff on AWS and I have no idea what the failure mode when bandwidth is limited for things like EBS or S3.  It&amp;#x27;s probably the same as Google, but I can&amp;#x27;t design around it because I don&amp;#x27;t actually know what it looks like.)  But, of course, if everything is high priority, nothing is high priority.  I imagine that things in the highest traffic class kept working on Sunday, which is a good outcome.  If everything were in the highest class, then nothing would work.&lt;p&gt;(When I worked at Google, I spent a fair amount of time advocating for a higher traffic class for my application&amp;#x27;s traffic.  If my application still exists, I wonder if it was affected, or if the time I spent on that actually paid off.)</text></item></parent_chain><comment><author>the-rc</author><text>I speculated it was something to do with the SDN, among others:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=20078433&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=20078433&lt;/a&gt;&lt;p&gt;It was unlikely to be fiber or a router failing, because there&amp;#x27;s enough redundancy at all sorts of levels (usually N+2 or better). Unless, that is, some nation state had been cutting multiple fibers at once.&lt;p&gt;This had the hallmark of some system blowing up, as you said. When it comes to QoS, it gets tricky. Gmail&amp;#x27;s frontend traffic should be at the highest priority, of course. But what about the replication traffic between your mailbox homes? What if a top level layer stalls or chokes when replication lags too much behind?&lt;p&gt;It&amp;#x27;s easier for stateless or less stateful systems like web search.</text></comment>