<story><title>ECC matters</title><url>https://www.realworldtech.com/forum/?threadid=198497&amp;curpostid=198647</url></story><parent_chain><item><author>nostrademons</author><text>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering &amp;quot;Not pushing for ECC memory.&amp;quot;&lt;p&gt;Google&amp;#x27;s initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software.  It turns out this is a terrible idea, because if you can&amp;#x27;t count on memory being robust against cosmic rays, you also can&amp;#x27;t count on the software being stored in that memory being robust against cosmic rays.  And when you have thousands of machines with petabytes of RAM, those bitflips do happen.  Google wasted many man-years tracking down corrupted GFS files and index shards before they finally bit the bullet and just paid for ECC.</text></item></parent_chain><comment><author>ksec</author><text>&amp;gt;I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering &amp;quot;Not pushing for ECC memory.&amp;quot;&lt;p&gt;Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they made a mistake without officially saying it?&lt;p&gt;I mean the whole world of Server or computer might not need ECC insanity was started entirely because of Google [1] [2] with news and articles published even in the early 00s [3]. And after that it has spread like wildfire and became a common accepted fact that even Google doesn&amp;#x27;t need ECC. Just like Apple were using custom ARM instruction to achieve their fast JS VM performance became a &amp;quot;fact&amp;quot;. ( For the last time, no they didn&amp;#x27;t ). And proponents of ECC memory has been fighting this misinformation like mad for decades. To the point giving up and only rant about every now and then. [3]&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;building-a-computer-the-google-way&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;building-a-computer-the-google...&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;to-ecc-or-not-to-ecc&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;to-ecc-or-not-to-ecc&amp;#x2F;&lt;/a&gt;&lt;p&gt;[3] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;danluu.com&amp;#x2F;why-ecc&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;danluu.com&amp;#x2F;why-ecc&amp;#x2F;&lt;/a&gt;</text></comment>
<story><title>ECC matters</title><url>https://www.realworldtech.com/forum/?threadid=198497&amp;curpostid=198647</url></story><parent_chain><item><author>nostrademons</author><text>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering &amp;quot;Not pushing for ECC memory.&amp;quot;&lt;p&gt;Google&amp;#x27;s initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software.  It turns out this is a terrible idea, because if you can&amp;#x27;t count on memory being robust against cosmic rays, you also can&amp;#x27;t count on the software being stored in that memory being robust against cosmic rays.  And when you have thousands of machines with petabytes of RAM, those bitflips do happen.  Google wasted many man-years tracking down corrupted GFS files and index shards before they finally bit the bullet and just paid for ECC.</text></item></parent_chain><comment><author>starfallg</author><text>Recent advances have blurred the lines a bit. The ECC memory that we all know and love is mainly side-band EEC, with the memory bus widened to accommodate the ECC bits driven by the memory controller. However as process size shrink, bit flips become more likely to the point that now many types of memory have on-die EEC, where the error correction is handled internally on the DRAM modules themselves. This is present on some DDR4 and DDR5 modules, but information on this is kept internal by the DRAM makers and not usually public.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;semiengineering.com&amp;#x2F;what-designers-need-to-know-about-error-correction-code-ecc-in-ddr-memories&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;semiengineering.com&amp;#x2F;what-designers-need-to-know-abou...&lt;/a&gt;&lt;p&gt;There has been a lot of debate regarding this that was summarised in this post -&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;to-ecc-or-not-to-ecc&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.codinghorror.com&amp;#x2F;to-ecc-or-not-to-ecc&amp;#x2F;&lt;/a&gt;</text></comment>