<story><title>Initial details about why CrowdStrike&apos;s CSAgent.sys crashed</title><url>https://twitter.com/patrickwardle/status/1814343502886477857</url></story><parent_chain><item><author>rco8786</author><text>Number 4 continues to be the most surprising bit to me. I could not fathom having a process that involves deploying to 8.5 million remote machines simultaneously.&lt;p&gt;Bugs in code I can almost always understand and forgive, even the ones that seem like they’d be obvious with hindsight. But this is just an egregious lack of the most basic rollout standards.</text></item><item><author>mtlmtlmtlmtl</author><text>There&amp;#x27;s at least five different things that went wrong simultaneously.&lt;p&gt;1. Poorly written code in the kernel module crashed the whole OS, and kept trying to parse the corrupted files, causing a boot loop. Instead of handling the error gracefully and deleting&amp;#x2F;marking the files as corrupt.&lt;p&gt;2. Either the corrupted files slipped through internal testing, or there is no internal testing.&lt;p&gt;3. Individual settings for when to apply such updates were apparently ignored. It&amp;#x27;s unclear whether this was a glitch or standard practice. Either way I consider it a bug(it&amp;#x27;s just a matter of whether it&amp;#x27;s a software bug or a bug in their procedures).&lt;p&gt;4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage.&lt;p&gt;5. Whatever caused the corruption in the first place, which is anyone&amp;#x27;s guess.</text></item><item><author>delta_p_delta_x</author><text>The moment I read &amp;#x27;it is a &lt;i&gt;content update&lt;/i&gt; that causes the BSOD, deleting it solves the problem&amp;#x27;, I was immediately willing to bet a hundred quid (for the non-British, that&amp;#x27;s £100) that it was a combination of said bad binary data and a poorly-written parser that didn&amp;#x27;t error out correctly upon reading invalid data (in this case, read an array of pointers, didn&amp;#x27;t verify that all of them were both non-null and pointed to valid data&amp;#x2F;code).&lt;p&gt;In the past ten years or so of having done somewhat serious computing and zero cybersecurity whatsoever, I have my mind concluded, feel free to disagree.&lt;p&gt;Approximately &lt;i&gt;100%&lt;/i&gt; of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so.&lt;p&gt;This includes everything from: decompression algorithms; font outline readers; image, video, and audio parsers; video game data parsers; XML and HTML parsers; the various certificate&amp;#x2F;signature&amp;#x2F;key parsers in OpenSSL (and derivatives); and now, this CrowdStrike content parser in its EDR program.&lt;p&gt;That wager stands, by the way, and I&amp;#x27;m happy to up the ante by £50 to account for my second theory.</text></item></parent_chain><comment><author>mbreese</author><text>For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t robust enough to recognize a corrupted file and not crash is inexcusable. Especially in kernel code that is so widely deployed.&lt;p&gt;If any one of the five points above hadn’t happened, this event would have been avoided. However, if number 1 had been addressed - any of the others could have happened (or all at the same time) and it would have been fine.&lt;p&gt;I understand that we should assume that bugs will be present anywhere, which is why staggered deployments are also important. If there had been staggered deployments, the. The damage would have happened, but it would have been localized. I think security people would argue against a staged deployment though, as if it were discovered what the new definitions protected against, an exploit could be developed quickly to put those servers that aren’t in the “canary” group at risk. (At least in theory — I can’t see how staggering deployment over a 6-12 hour window would have been that risky).</text></comment>
<story><title>Initial details about why CrowdStrike&apos;s CSAgent.sys crashed</title><url>https://twitter.com/patrickwardle/status/1814343502886477857</url></story><parent_chain><item><author>rco8786</author><text>Number 4 continues to be the most surprising bit to me. I could not fathom having a process that involves deploying to 8.5 million remote machines simultaneously.&lt;p&gt;Bugs in code I can almost always understand and forgive, even the ones that seem like they’d be obvious with hindsight. But this is just an egregious lack of the most basic rollout standards.</text></item><item><author>mtlmtlmtlmtl</author><text>There&amp;#x27;s at least five different things that went wrong simultaneously.&lt;p&gt;1. Poorly written code in the kernel module crashed the whole OS, and kept trying to parse the corrupted files, causing a boot loop. Instead of handling the error gracefully and deleting&amp;#x2F;marking the files as corrupt.&lt;p&gt;2. Either the corrupted files slipped through internal testing, or there is no internal testing.&lt;p&gt;3. Individual settings for when to apply such updates were apparently ignored. It&amp;#x27;s unclear whether this was a glitch or standard practice. Either way I consider it a bug(it&amp;#x27;s just a matter of whether it&amp;#x27;s a software bug or a bug in their procedures).&lt;p&gt;4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage.&lt;p&gt;5. Whatever caused the corruption in the first place, which is anyone&amp;#x27;s guess.</text></item><item><author>delta_p_delta_x</author><text>The moment I read &amp;#x27;it is a &lt;i&gt;content update&lt;/i&gt; that causes the BSOD, deleting it solves the problem&amp;#x27;, I was immediately willing to bet a hundred quid (for the non-British, that&amp;#x27;s £100) that it was a combination of said bad binary data and a poorly-written parser that didn&amp;#x27;t error out correctly upon reading invalid data (in this case, read an array of pointers, didn&amp;#x27;t verify that all of them were both non-null and pointed to valid data&amp;#x2F;code).&lt;p&gt;In the past ten years or so of having done somewhat serious computing and zero cybersecurity whatsoever, I have my mind concluded, feel free to disagree.&lt;p&gt;Approximately &lt;i&gt;100%&lt;/i&gt; of CVEs, crashes, bugs, slowdowns, and pain points of computing have to do with various forms of deserialising binary data back into machine-readable data structures. All because a) human programmers forget to account for edge cases, and b) imperative programming languages allow us to do so.&lt;p&gt;This includes everything from: decompression algorithms; font outline readers; image, video, and audio parsers; video game data parsers; XML and HTML parsers; the various certificate&amp;#x2F;signature&amp;#x2F;key parsers in OpenSSL (and derivatives); and now, this CrowdStrike content parser in its EDR program.&lt;p&gt;That wager stands, by the way, and I&amp;#x27;m happy to up the ante by £50 to account for my second theory.</text></item></parent_chain><comment><author>thundershart</author><text>Surely, CrowdStrike&amp;#x27;s safety posture for update rollouts is in serious need of improvement. No argument there.&lt;p&gt;But is there any responsibility for the clients consuming the data to have verified these updates prior to taking them in production? I haven&amp;#x27;t worn the sysadmin hat in a while now, but back when I was responsible for the upkeep of many thousands of machines, we&amp;#x27;d never have blindly consumed updates without at least a basic smoke test in a production-adjacent UAT type environment. Core OS updates, firmware updates, third party software, whatever -- all of it would get at least some cursory smoke testing before allowing it to hit production.&lt;p&gt;On the other hand, given EDR&amp;#x27;s real-world purpose and the speed at which novel attacks propagate, there&amp;#x27;s probably a compelling argument for always taking the latest definition&amp;#x2F;signature updates as soon as they&amp;#x27;re available, even in your production environments.&lt;p&gt;I&amp;#x27;m certainly not saying that CrowdStrike did nothing wrong here, that&amp;#x27;s clearly not the case. But if conventional wisdom says that you should kick the tires on the latest batch of OS updates from Microsoft in a test environment, maybe that same rationale should apply to EDR agents?</text></comment>