<story><title>Facebook-owned sites were down</title><url>https://www.facebook.com/</url></story><parent_chain><item><author>kossTKR</author><text>Reddit r&amp;#x2F;Sysadmin user that claims to be on the &amp;quot;Recovery Team&amp;quot; for this ongoing issue:&lt;p&gt;&amp;gt;&lt;i&gt;As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/i&gt;&lt;p&gt;User is providing live updates of the incident here:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like_facebook_is_down&amp;#x2F;hfd4dyv&amp;#x2F;?context=3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like...&lt;/a&gt;</text></item></parent_chain><comment><author>guidopallemans</author><text>He just deleted all his updates.&lt;p&gt;user:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;old.reddit.com&amp;#x2F;user&amp;#x2F;ramenporn&lt;/a&gt;&lt;p&gt;some messages:&lt;p&gt;* This is a global outage for all FB-related services&amp;#x2F;infra (source: I&amp;#x27;m currently on the recovery&amp;#x2F;investigation team).&lt;p&gt;* Will try to provide any important&amp;#x2F;interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls.&lt;p&gt;* Update 1440 UTC: \&lt;p&gt;&lt;pre&gt;&lt;code&gt;    As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

    There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

    Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/code&gt;&lt;/pre&gt;</text></comment>
<story><title>Facebook-owned sites were down</title><url>https://www.facebook.com/</url></story><parent_chain><item><author>kossTKR</author><text>Reddit r&amp;#x2F;Sysadmin user that claims to be on the &amp;quot;Recovery Team&amp;quot; for this ongoing issue:&lt;p&gt;&amp;gt;&lt;i&gt;As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that&amp;#x27;s that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).
There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
Part of this is also due to lower staffing in data centers due to pandemic measures.&lt;/i&gt;&lt;p&gt;User is providing live updates of the incident here:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like_facebook_is_down&amp;#x2F;hfd4dyv&amp;#x2F;?context=3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;sysadmin&amp;#x2F;comments&amp;#x2F;q181fv&amp;#x2F;looks_like...&lt;/a&gt;</text></item></parent_chain><comment><author>pmlnr</author><text>&amp;gt; the people with physical access is separate from the people with knowledge of [...]&lt;p&gt;Welcome to the brave new world of troubleshooting. This will seriously bite us one day.</text></comment>