<story><title>Hard Drive Reliability Update – Sep 2014</title><url>https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014/</url></story><parent_chain><item><author>zaroth</author><text>Since annual failure rate is a function mostly of age, it would be interesting to see a line chart of cumulative failure rate vs age. But since new drives are continually being added to the population, there would be fewer drives in the data set as you moved up each curve.&lt;p&gt;I guess you could calculate confidence intervals at quarterly intervals, and so the error bars would get larger as age increases and &amp;#x27;n&amp;#x27; decreases.&lt;p&gt;How would you calculate the CI for failure rate? It&amp;#x27;s not binomial or poisson, since failure rate goes to 1 over time...&lt;p&gt;A little searching turns up &lt;a href=&quot;http://rmod.ee.duke.edu/statistics.htm&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;rmod.ee.duke.edu&amp;#x2F;statistics.htm&lt;/a&gt; which I&amp;#x27;m sure completely explains how to do this... (rolls eyes). I hate that this is how statistics is commonly taught. Knowing which distribution to use and applying it correctly can actually be intuitive if taught properly. It doesn&amp;#x27;t always need to be an exercise in alphabet soup &amp;#x2F; deriving from base principles.</text></item></parent_chain><comment><author>hyperbovine</author><text>Undergrad version: the lifespan of a single drive should be exponentially distributed with some rate lambda. The time to first failure from among k drives would then have an exponential distribution with rate k * lambda. The confidence interval for lambda is: &lt;a href=&quot;http://en.wikipedia.org/wiki/Exponential_distribution#Confidence_intervals&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Exponential_distribution#Confid...&lt;/a&gt;&lt;p&gt;Grad-level version: drive lifetime is exponential with an inhomogeneous (increasing, presumably) rate function lambda(t). Inferring lambda(t) is difficult without additional assumptions on the functional form. But potentially do-able.&lt;p&gt;Real-life version: None of the classical distribution fit that well. (&lt;a href=&quot;https://www.usenix.org/legacy/event/fast07/tech/schroeder/schroeder_html/&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.usenix.org&amp;#x2F;legacy&amp;#x2F;event&amp;#x2F;fast07&amp;#x2F;tech&amp;#x2F;schroeder&amp;#x2F;sc...&lt;/a&gt;)&lt;p&gt;(This is survival analysis: &lt;a href=&quot;http://en.wikipedia.org/wiki/Survival_analysis&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Survival_analysis&lt;/a&gt;)</text></comment>
<story><title>Hard Drive Reliability Update – Sep 2014</title><url>https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014/</url></story><parent_chain><item><author>zaroth</author><text>Since annual failure rate is a function mostly of age, it would be interesting to see a line chart of cumulative failure rate vs age. But since new drives are continually being added to the population, there would be fewer drives in the data set as you moved up each curve.&lt;p&gt;I guess you could calculate confidence intervals at quarterly intervals, and so the error bars would get larger as age increases and &amp;#x27;n&amp;#x27; decreases.&lt;p&gt;How would you calculate the CI for failure rate? It&amp;#x27;s not binomial or poisson, since failure rate goes to 1 over time...&lt;p&gt;A little searching turns up &lt;a href=&quot;http://rmod.ee.duke.edu/statistics.htm&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;rmod.ee.duke.edu&amp;#x2F;statistics.htm&lt;/a&gt; which I&amp;#x27;m sure completely explains how to do this... (rolls eyes). I hate that this is how statistics is commonly taught. Knowing which distribution to use and applying it correctly can actually be intuitive if taught properly. It doesn&amp;#x27;t always need to be an exercise in alphabet soup &amp;#x2F; deriving from base principles.</text></item></parent_chain><comment><author>jrapdx3</author><text>I&amp;#x27;m not a statistician, but what you are describing is a situation in which the survival of the &amp;quot;subjects&amp;quot; (SSDs) is the crucial variable.  How quickly the population declines is the question.&lt;p&gt;This is similar to studies in medical research, e.g., how long do subjects live after an experimental cancer treatment.  What I&amp;#x27;ve seen used is the Kaplan-Meier non-parametric method (Kaplan-Meier plot) for the purpose.&lt;p&gt;Maybe I&amp;#x27;m just used to seeing these plots, but I find the method conveys information very effectively.  More info can be found here: 
&lt;a href=&quot;https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Kaplan%E2%80%93Meier_estimator&lt;/a&gt;&lt;p&gt;I agree that presenting the SSD life data in that way would be a good idea.</text></comment>