<story><title>Google Cloud Networking Incident Postmortem</title><url>https://status.cloud.google.com/incident/cloud-networking/19009</url></story><parent_chain><item><author>dnautics</author><text>&amp;gt; Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network.&lt;p&gt;Man that&amp;#x27;s got to suck.</text></item><item><author>carlsborg</author><text>I was curious to know how cascading failures in one region effected other regions. Impact was &amp;quot; ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1.&amp;quot;&lt;p&gt;Answer, and the root cause summarized:&lt;p&gt;Maintenance started in a physical location, and then &amp;quot;... the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. 
Those logical clusters also included network control jobs in other physical locations.&amp;quot;&lt;p&gt;So the automation equivalent of a human driven command that says &amp;quot;deschedule these core jobs in another region&amp;quot;.&lt;p&gt;Maybe someone needs to write a paper on Fault tolerance in the presence of Byzantine Automations (Joke. There was a satirical note on this subject posted here yesterday.)</text></item></parent_chain><comment><author>abbeyj</author><text>This reminded me of &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.usenix.org&amp;#x2F;system&amp;#x2F;files&amp;#x2F;1311_05-08_mickens.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.usenix.org&amp;#x2F;system&amp;#x2F;files&amp;#x2F;1311_05-08_mickens.pdf&lt;/a&gt;&lt;p&gt;&lt;pre&gt;&lt;code&gt;   I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.&lt;/code&gt;&lt;/pre&gt;</text></comment>
<story><title>Google Cloud Networking Incident Postmortem</title><url>https://status.cloud.google.com/incident/cloud-networking/19009</url></story><parent_chain><item><author>dnautics</author><text>&amp;gt; Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network.&lt;p&gt;Man that&amp;#x27;s got to suck.</text></item><item><author>carlsborg</author><text>I was curious to know how cascading failures in one region effected other regions. Impact was &amp;quot; ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1.&amp;quot;&lt;p&gt;Answer, and the root cause summarized:&lt;p&gt;Maintenance started in a physical location, and then &amp;quot;... the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. 
Those logical clusters also included network control jobs in other physical locations.&amp;quot;&lt;p&gt;So the automation equivalent of a human driven command that says &amp;quot;deschedule these core jobs in another region&amp;quot;.&lt;p&gt;Maybe someone needs to write a paper on Fault tolerance in the presence of Byzantine Automations (Joke. There was a satirical note on this subject posted here yesterday.)</text></item></parent_chain><comment><author>londons_explore</author><text>It isn&amp;#x27;t a worst-case though.   They should have had the capability to resolve this issue with &lt;i&gt;no&lt;/i&gt; network connectivity, which would be the worst case failure of the network control plane.</text></comment>