<story><title>Disks lie. And the controllers that run them are partners in crime.</title><url>http://queue.acm.org/detail.cfm?id=2367378</url></story><parent_chain></parent_chain><comment><author>rarrrrrr</author><text>At SpiderOak, every storage cluster writes new customer data to three RAID6 volumes on 3 different machines, which wait for fdatasync, before we consider it written.&lt;p&gt;... and before that begins, the data is added to a giant ring buffer that records the last 30 TB of new writes, on machines with UPS.  If a rack power failure happens, the ring buffer is kept until storage audits complete.&lt;p&gt;I think every company that deals with large data they can&apos;t lose develops appropriate paranoia.&lt;p&gt;The behavior of hard drives is like decaying atoms.  You can&apos;t make accurate predictions about what any one of them will do.  Only in aggregate can you say something like &quot;the half life of this pile of hardware is 12 years&quot; or &quot;if we write this data N times we can reasonably expect to read it it again.&quot;</text></comment>
<story><title>Disks lie. And the controllers that run them are partners in crime.</title><url>http://queue.acm.org/detail.cfm?id=2367378</url></story><parent_chain></parent_chain><comment><author>ChuckMcM</author><text>During my tenure at NetApp I got to see all sorts of really really interesting disk problems and the lengths software had to go to reliably store data on them. Two scars burned into me from that were 1) disks suck, and 2) disks are not &apos;storage&apos;.&lt;p&gt;The first part is pretty easy to understand, storage manufacturers have competed for years in a commodity market where consumers often choose price per gigabyte over URER (the unrecoverable read error rate), further at the scale of the market small savings of cents adds up to better margins. And while the &apos;enterprise&apos; fibre channel and SCSI drives could (and did) command a hefty premium, the shift to SATA drives really put a crimp in the over all margin picture. So the disk manufacturers are stuck between the reliability of the drive and the cost of the drive. They surf the curve where it is just reliable enough to keep people buying drives.&lt;p&gt;This trend bites back, making the likely hood of an error while reading more and more probable. Not picking on Seagate here, they are all similar, but lets look at their Barracuda drive&apos;s spec sheet [1]. You will notice a parameter &apos;Nonrecoverable Read Errors per Bits read&apos;, and you&apos;ll see that its 1x10e14 across the sheet, from 3TB down to 250GB. It is a statistical thing, the whole magnetic field domain to digital bit pipeline is one giant analog loop of a error extraction. 1x10E14 &lt;i&gt;bits&lt;/i&gt;. So what does that mean? Lets say each byte is encoded in 10 bits on the disk. Three trillion bytes is 30 trillion bits in that case, or 3x10E13 bits. Best case, if you read that disk from one end to the other (like you were re-silvering a mirror from a RAID 10 setup) you have a 1 in 3 chance that one of those sectors won&apos;t be readable &lt;i&gt;for a perfectly working disk.&lt;/i&gt; Amazing isn&apos;t it? Trying to reconstruct a RAID5 group with 4 disks remaining out of 5, look out.&lt;p&gt;So physics is not on your side, but we&apos;ve got RAID 6 Chuck! Of course you do, and that is a good thing. But what about when you write to the disk, the disk replies &quot;Yeah sure boss! Got that on the platters now&quot; but it didn&apos;t actually right it? Now you&apos;ve got a parity failure waiting to bite you sitting on the drive, or worse, it &lt;i&gt;does&lt;/i&gt; write the sector but writes it somewhere else (saw this a number of times at NetApp as well). There are &lt;i&gt;three million lines of code&lt;/i&gt; in the disk firmware. One of the manufacturers (my memory says Maxtor) showed a demo where they booted Linux on the controller of the drive.&lt;p&gt;Bottom line is that the system works mostly, which is a huge win, and a lot of people blame the OS when the drive is at fault, another bonus for manufacturers, but unless your data is in a RAID 6 group or on at least 3 drives, its not really &apos;safe&apos; in the &quot;I am absolutely positive I could read this back&quot; sense of the word.&lt;p&gt;[1] &lt;a href=&quot;http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/disc/barracuda-ds1737-1-1111us.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/...&lt;/a&gt;</text></comment>