<story><title>Level 3 Global Outage</title><url>https://puck.nether.net/pipermail/outages/2020-August/013187.html</url></story><parent_chain><item><author>kitteh</author><text>Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn.&lt;p&gt;Edit: if you are a Level3 customer shut your sessions down to them.</text></item></parent_chain><comment><author>beagle3</author><text>History doesn&amp;#x27;t repeat, but it rhymes ....&lt;p&gt;There was a huge AT&amp;amp;T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly &amp;quot;everything not within the same area code&amp;quot;).&lt;p&gt;It was a bug. It wasn&amp;#x27;t a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for the original exchange to come back up, receive the crashy event back, and crash again.&lt;p&gt;The whole network was full of nodes crashing, causing their peers to crash, ad infinitum. In order to bring the network back up, they needed to either take everything down at the same time (and make sure all the queues are emptied), but even that wouldn&amp;#x27;t have made it stable, because a similar &amp;quot;patient 0&amp;quot; event would have brought the whole network down.&lt;p&gt;Once the problem was understood, they reverted to an earlier version which didn&amp;#x27;t have the bug, and the network re-stabilized.&lt;p&gt;The lore I grew up on is that this specific event was very significant in pushing and funding research into robust distributed systems, of which the best known result is Erlang and its ecosystem - originally built, and still mostly used, to make sure that phone exchanges don&amp;#x27;t break.&lt;p&gt;[0] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;users.csc.calpoly.edu&amp;#x2F;~jdalbey&amp;#x2F;SWE&amp;#x2F;Papers&amp;#x2F;att_collapse&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;users.csc.calpoly.edu&amp;#x2F;~jdalbey&amp;#x2F;SWE&amp;#x2F;Papers&amp;#x2F;att_collap...&lt;/a&gt;</text></comment>
<story><title>Level 3 Global Outage</title><url>https://puck.nether.net/pipermail/outages/2020-August/013187.html</url></story><parent_chain><item><author>kitteh</author><text>Massive reconvergence event in their network, causing edge router bgp sessions to bounce (due to cpu). Right now all their big peers are shutting down sessions with them to give level3s network the ability to reconverge. Prefixes announced to 3356 are frozen on their route reflectors and not getting withdrawn.&lt;p&gt;Edit: if you are a Level3 customer shut your sessions down to them.</text></item></parent_chain><comment><author>guerrilla</author><text>What is a reconvergence event? Is that what&amp;#x27;s described in your last sentence?</text></comment>