<story><title>Drgn: How the Linux kernel team at Facebook debugs the kernel at scale</title><url>https://developers.facebook.com/blog/post/2021/12/09/drgn-how-linux-kernel-team-meta-debugs-kernel-scale/</url></story><parent_chain><item><author>marcan_42</author><text>They don&amp;#x27;t care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications.&lt;p&gt;Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.</text></item><item><author>silisili</author><text>As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers.  I thought I&amp;#x27;d have a huge upper hand, understanding dark magic.  Come to find out most developers don&amp;#x27;t know and don&amp;#x27;t care about any of it, and have infected the whole community with the idea that none of it matters and servers and processes are just cattle.  &amp;#x2F;rant</text></item><item><author>rvnx</author><text>Makes me remember how Facebook operates memory leaks at scale; &amp;quot;won&amp;#x27;t fix, we just reboot servers when they crash&amp;quot;.</text></item></parent_chain><comment><author>nijave</author><text>Honestly I think this just rolls down from management.&lt;p&gt;&amp;quot;Oh you can spend $x * 80 hours debugging or $x * 2 hours putting a patch to constantly restart&amp;quot;&lt;p&gt;It&amp;#x27;s hard to quantify the impact of having a house of cards system with components constantly restarting to side steps bugs until the whole thing lights on fire in a cascading failure later (oh the service scheduler was down for 30 mins and things are restarting so much a massive amount of capacity went offline during that time)</text></comment>
<story><title>Drgn: How the Linux kernel team at Facebook debugs the kernel at scale</title><url>https://developers.facebook.com/blog/post/2021/12/09/drgn-how-linux-kernel-team-meta-debugs-kernel-scale/</url></story><parent_chain><item><author>marcan_42</author><text>They don&amp;#x27;t care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications.&lt;p&gt;Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.</text></item><item><author>silisili</author><text>As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers.  I thought I&amp;#x27;d have a huge upper hand, understanding dark magic.  Come to find out most developers don&amp;#x27;t know and don&amp;#x27;t care about any of it, and have infected the whole community with the idea that none of it matters and servers and processes are just cattle.  &amp;#x2F;rant</text></item><item><author>rvnx</author><text>Makes me remember how Facebook operates memory leaks at scale; &amp;quot;won&amp;#x27;t fix, we just reboot servers when they crash&amp;quot;.</text></item></parent_chain><comment><author>thomasjudge</author><text>I know you probably can&amp;#x27;t share details, but would be curious to hear a little more color around these two situations.</text></comment>