<story><title>Facebook-owned sites were down</title><url>https://www.facebook.com/</url></story><parent_chain><item><author>treesknees</author><text>They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don&amp;#x27;t lose their job.&lt;p&gt;Shareholders and other business leaders I&amp;#x27;m sure are much happier reporting this as a series of unfortunate technical failures (which I&amp;#x27;m sure is part of it) rather than a company-wide organizational failure. The fact they can&amp;#x27;t physically badge in the people who know the router configuration speaks to an organization that hasn&amp;#x27;t actually thought through all its failure modes. People aren&amp;#x27;t going to like that.  It&amp;#x27;s not uncommon to have the datacenter techs with access and the actual software folks restricted, but that being the reason one of the most popular services in the world has been down for nearly 3 hours now will raise a lot of questions.&lt;p&gt;Edit: I also hope this doesn&amp;#x27;t damage prospects for more Work From Home. If they couldn&amp;#x27;t get anyone who knew the configuration in because they all live a plane ride away from the datacenters, I could see managers being reluctant to have a completely remote team for situations where clearly physical access was needed.</text></item><item><author>Ueland</author><text>And there his account went poof, thanks for archiving.</text></item><item><author>guidopallemans</author><text>He just deleted all his updates.&lt;p&gt;user:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&lt;/a&gt;&lt;p&gt;some messages:&lt;p&gt;* This is a global outage for all FB-related services&amp;#x2F;infra (source: I&amp;#x27;m currently on the recovery&amp;#x2F;investigation team).&lt;p&gt;* Will try to provide any important&amp;#x2F;interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls.&lt;p&gt;* Update 1440 UTC: \&lt;p&gt;&lt;pre&gt;&lt;code&gt;    As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

    There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

    Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/code&gt;&lt;/pre&gt;</text></item><item><author>kossTKR</author><text>Reddit r&amp;#x2F;Sysadmin user that claims to be on the &amp;quot;Recovery Team&amp;quot; for this ongoing issue:&lt;p&gt;&amp;gt;&lt;i&gt;As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/i&gt;&lt;p&gt;User is providing live updates of the incident here:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like_facebook_is_down&amp;#x2F;hfd4dyv&amp;#x2F;?context=3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like...&lt;/a&gt;</text></item></parent_chain><comment><author>jart</author><text>Facebook should have had a panic room.&lt;p&gt;Operations teams normally have a special room with a secure connection for situations like this, so that production can be controlled in the event of bgp failure, nuclear war, etc. I could see physical presence being an issue if their bgp router depends on something like a crypto module in a locked cage, in which case there&amp;#x27;s always helicopters.&lt;p&gt;So if anything, Facebook&amp;#x27;s labor policies are about to become cooler.</text></comment>
<story><title>Facebook-owned sites were down</title><url>https://www.facebook.com/</url></story><parent_chain><item><author>treesknees</author><text>They were quoted on multiple news sites including Ars Technica. I would imagine they were not authorized to post that information. I hope they don&amp;#x27;t lose their job.&lt;p&gt;Shareholders and other business leaders I&amp;#x27;m sure are much happier reporting this as a series of unfortunate technical failures (which I&amp;#x27;m sure is part of it) rather than a company-wide organizational failure. The fact they can&amp;#x27;t physically badge in the people who know the router configuration speaks to an organization that hasn&amp;#x27;t actually thought through all its failure modes. People aren&amp;#x27;t going to like that.  It&amp;#x27;s not uncommon to have the datacenter techs with access and the actual software folks restricted, but that being the reason one of the most popular services in the world has been down for nearly 3 hours now will raise a lot of questions.&lt;p&gt;Edit: I also hope this doesn&amp;#x27;t damage prospects for more Work From Home. If they couldn&amp;#x27;t get anyone who knew the configuration in because they all live a plane ride away from the datacenters, I could see managers being reluctant to have a completely remote team for situations where clearly physical access was needed.</text></item><item><author>Ueland</author><text>And there his account went poof, thanks for archiving.</text></item><item><author>guidopallemans</author><text>He just deleted all his updates.&lt;p&gt;user:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&lt;/a&gt;&lt;p&gt;some messages:&lt;p&gt;* This is a global outage for all FB-related services&amp;#x2F;infra (source: I&amp;#x27;m currently on the recovery&amp;#x2F;investigation team).&lt;p&gt;* Will try to provide any important&amp;#x2F;interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls.&lt;p&gt;* Update 1440 UTC: \&lt;p&gt;&lt;pre&gt;&lt;code&gt;    As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

    There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

    Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/code&gt;&lt;/pre&gt;</text></item><item><author>kossTKR</author><text>Reddit r&amp;#x2F;Sysadmin user that claims to be on the &amp;quot;Recovery Team&amp;quot; for this ongoing issue:&lt;p&gt;&amp;gt;&lt;i&gt;As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/i&gt;&lt;p&gt;User is providing live updates of the incident here:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like_facebook_is_down&amp;#x2F;hfd4dyv&amp;#x2F;?context=3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like...&lt;/a&gt;</text></item></parent_chain><comment><author>polote</author><text>&amp;gt; an organization that hasn&amp;#x27;t actually thought through all its failure modes&lt;p&gt;Thinking about any potential things that can happen is impossible</text></comment>