<story><title>Why Twitter didn’t go down: From a real Twitter SRE</title><url>https://matthewtejo.substack.com/p/why-twitter-didnt-go-down-from-a</url></story><parent_chain><item><author>itsoktocry</author><text>&amp;gt;&lt;i&gt;removing engineers won&amp;#x27;t instantly crash the product. It&amp;#x27;ll happen slowly&lt;/i&gt;&lt;p&gt;It&amp;#x27;s amazing to me how many people following the Twitter saga, some familiar with or &lt;i&gt;actually working&lt;/i&gt; in technology, thought that Twitter would crash within days of the engineers being fired.  And because it didn&amp;#x27;t, the job cuts are justified.</text></item><item><author>donkeyd</author><text>&amp;gt; the article might actually persuade me that it was all bloat&lt;p&gt;First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject.&lt;p&gt;Secondly, in any large company, the majority is &amp;#x27;bloat&amp;#x27;. It&amp;#x27;s security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth &amp;gt; stability. In a large organization, part of the bloat helps insure a certain amount of stability that&amp;#x27;s necessary to keep an organization alive.&lt;p&gt;If a product is mature enough, like Twitter seems to be, removing engineers won&amp;#x27;t instantly crash the product. It&amp;#x27;ll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That&amp;#x27;s when fixing the issues suddenly takes a lot more time.&lt;p&gt;If the current state of affairs at Twitter keeps up, it&amp;#x27;ll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.</text></item><item><author>resonious</author><text>&amp;gt; This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat.&lt;p&gt;I was partly expecting the rest of the article to explain to me why exactly it &lt;i&gt;wasn&amp;#x27;t&lt;/i&gt; just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that&amp;#x27;s really resilient to both hardware and software failures. If anything, the article might actually persuade me that it &lt;i&gt;was&lt;/i&gt; all bloat.</text></item></parent_chain><comment><author>matwood</author><text>I agree those were odd takes. I&amp;#x27;ve likened firing most of the engineers to taking your hands off the wheel in the car. It won&amp;#x27;t crash immediately, but it doesn&amp;#x27;t mean the car can go driverless.&lt;p&gt;With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerability comes out?</text></comment>
<story><title>Why Twitter didn’t go down: From a real Twitter SRE</title><url>https://matthewtejo.substack.com/p/why-twitter-didnt-go-down-from-a</url></story><parent_chain><item><author>itsoktocry</author><text>&amp;gt;&lt;i&gt;removing engineers won&amp;#x27;t instantly crash the product. It&amp;#x27;ll happen slowly&lt;/i&gt;&lt;p&gt;It&amp;#x27;s amazing to me how many people following the Twitter saga, some familiar with or &lt;i&gt;actually working&lt;/i&gt; in technology, thought that Twitter would crash within days of the engineers being fired.  And because it didn&amp;#x27;t, the job cuts are justified.</text></item><item><author>donkeyd</author><text>&amp;gt; the article might actually persuade me that it was all bloat&lt;p&gt;First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject.&lt;p&gt;Secondly, in any large company, the majority is &amp;#x27;bloat&amp;#x27;. It&amp;#x27;s security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth &amp;gt; stability. In a large organization, part of the bloat helps insure a certain amount of stability that&amp;#x27;s necessary to keep an organization alive.&lt;p&gt;If a product is mature enough, like Twitter seems to be, removing engineers won&amp;#x27;t instantly crash the product. It&amp;#x27;ll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That&amp;#x27;s when fixing the issues suddenly takes a lot more time.&lt;p&gt;If the current state of affairs at Twitter keeps up, it&amp;#x27;ll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.</text></item><item><author>resonious</author><text>&amp;gt; This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat.&lt;p&gt;I was partly expecting the rest of the article to explain to me why exactly it &lt;i&gt;wasn&amp;#x27;t&lt;/i&gt; just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that&amp;#x27;s really resilient to both hardware and software failures. If anything, the article might actually persuade me that it &lt;i&gt;was&lt;/i&gt; all bloat.</text></item></parent_chain><comment><author>Aeolun</author><text>&amp;gt; thought that Twitter would crash within days of the engineers being fired&lt;p&gt;Because they work for companies where the product would fail within days of them being fired themselves.</text></comment>