<story><title>SSD will fail at 40k power-on hours (2021)</title><url>https://www.cisco.com/c/en/us/support/docs/field-notices/705/fn70545.html</url></story><parent_chain><item><author>taneq</author><text>Yowch. The old &amp;quot;stagger your drive replacements, stagger your batches&amp;quot; thing might not be quite as outdated as we&amp;#x27;d like to think...</text></item><item><author>mkl</author><text>Not two SSDs, &lt;i&gt;four&lt;/i&gt;: two in the main server, and two in the backup server.</text></item><item><author>solardev</author><text>Wow, thanks for sharing. I didn&amp;#x27;t realize how closely related they were.&lt;p&gt;(TLDR For anyone wondering, &amp;quot;recent HN issues&amp;quot; means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.)&lt;p&gt;Edit: If you want to discuss that theory, it&amp;#x27;s probably better to do it in that other thread directly instead... dang and a person from M5 Hosting (HN&amp;#x27;s previous host) are both participating there.</text></item><item><author>dredmorbius</author><text>Possibly related to recent HN issues, see:  &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32031243&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32031243&lt;/a&gt;</text></item></parent_chain><comment><author>ekidd</author><text>I have definitely seen RAID arrays where the drives were all part of a single manufacturing batch, and multiple drives all failed in rapid succession. I think this can be caused by several things:&lt;p&gt;- Unless you periodically do full-drive reads, you may silently accumulate bad blocks across multiple drives in an array. When you finally detect a failed drive, you discover that other drives have also been failing for months.&lt;p&gt;- A full RAID rebuild is a high-stress event that tries to read every disk block on every drive, as rapidly as possible.&lt;p&gt;- And finally, some drive batches are just dodgy, and it may not take much to push them over. And if identically dodgy drives are all exposed to exactly the same thermal stress and the same I&amp;#x2F;O operations, then I guess they might fail close together?&lt;p&gt;Honestly, RAID arrays only buy you so much reliability. Hardware RAID controllers are another single point of failure. I once lost two drives &lt;i&gt;and&lt;/i&gt; a RAID controller all together during Christmas, which was not a fun time.&lt;p&gt;I do like the modern idea of S3-like storage, where data is replicated over several independent machines, and the controlling software can recover from losing entire servers (or even data centers). It&amp;#x27;s not a perfect match for everything, but it works great for lots of things.</text></comment>
<story><title>SSD will fail at 40k power-on hours (2021)</title><url>https://www.cisco.com/c/en/us/support/docs/field-notices/705/fn70545.html</url></story><parent_chain><item><author>taneq</author><text>Yowch. The old &amp;quot;stagger your drive replacements, stagger your batches&amp;quot; thing might not be quite as outdated as we&amp;#x27;d like to think...</text></item><item><author>mkl</author><text>Not two SSDs, &lt;i&gt;four&lt;/i&gt;: two in the main server, and two in the backup server.</text></item><item><author>solardev</author><text>Wow, thanks for sharing. I didn&amp;#x27;t realize how closely related they were.&lt;p&gt;(TLDR For anyone wondering, &amp;quot;recent HN issues&amp;quot; means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.)&lt;p&gt;Edit: If you want to discuss that theory, it&amp;#x27;s probably better to do it in that other thread directly instead... dang and a person from M5 Hosting (HN&amp;#x27;s previous host) are both participating there.</text></item><item><author>dredmorbius</author><text>Possibly related to recent HN issues, see:  &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32031243&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=32031243&lt;/a&gt;</text></item></parent_chain><comment><author>hackmiester</author><text>I had no idea anyone thought this was outdated. We certainly never stopped doing it. I think it is a timeless failsafe.</text></comment>