<story><title>CenturyLink 911 outage was caused by a single network card sending bad packets</title><url>https://twitter.com/GossiTheDog/status/1079144491238469638</url></story><parent_chain></parent_chain><comment><author>CydeWeys</author><text>I was visiting my SO&amp;#x27;s parents in Portland for the holidays when this happened.  Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn&amp;#x27;t going to work.  (At least their Internet through CenturyLink remained working.)&lt;p&gt;This also happened to coincide with them receiving a notice of a $50&amp;#x2F;month increase on their bill from CenturyLink, as well as my SO giving them a Roku for the holidays and setting them up with streaming services, which they got lots of practice using in the two whole days the cable TV didn&amp;#x27;t work.&lt;p&gt;Guess who&amp;#x27;s cutting the cord and switching to Internet-only service from Verizon FIOS at a much cheaper rate now?&lt;p&gt;This is what happens when you treat IT like a cost center and don&amp;#x27;t provide the necessary funds to tackle technical debt and keep your services up and running: You get huge costly outages revealing your basic incompetence and customers fleeing to superior competitors forever.</text></comment>
<story><title>CenturyLink 911 outage was caused by a single network card sending bad packets</title><url>https://twitter.com/GossiTheDog/status/1079144491238469638</url></story><parent_chain></parent_chain><comment><author>carlsborg</author><text>How does a single network card emitting bad packets effect other sites?&lt;p&gt;&amp;gt; investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card.&lt;p&gt;&amp;gt; Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes.&lt;p&gt;And then&lt;p&gt;&amp;gt; By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service.&lt;p&gt;And finally&lt;p&gt;&amp;gt; The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.&lt;p&gt;Looks like the root cause analysis has a way to go. Addendum says:&lt;p&gt;&amp;gt; The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. &lt;i&gt;&lt;/i&gt;&lt;i&gt;&lt;/i&gt;The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner.&lt;i&gt;&lt;/i&gt;&lt;i&gt;&lt;/i&gt; The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. &lt;i&gt;&lt;/i&gt;&lt;i&gt;At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment.&lt;/i&gt;&lt;i&gt;&lt;/i&gt; The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.</text></comment>