Hello, Upon reviewing the logs, the mystery has revealed itself as a puncture; also known as a write-hole or a type of double fault.  The problem is that the bad logical blocks have been spread and propagated throughout the array. In some cases, the puncture may have already occurred before replacing a bad drive. Unfortunately, when this happens, the only way to move forward is to backup all available data possible and then identify what data was unable to be backed up by reviewing your backup logs. You may be fortunate that the missing data is either obsolete or has been static for quite some time.  Review the technical information on the punctures here: www.dell.com/.../438291;s=biz&cs=555&l= Keep in mind that continued array operation or attempted hard drive rebuilds, may lead to additional Predictive Fail states on more drives or worse, complete failure of the array.  If you had already replaced the original failing drive, then the ongoing issues may purely be logical.  Any drives marked as Predictive Fail, should be replaced even though there may be nothing physically wrong them. Running Offline Diagnostics is the best way to determine drives' state before moving forward. You must delete the entire affected VD and replace the faulty drives. If this is your OS VD, then you will require a full OS re-installation.  Steps are as follows: · Verify a file-level backup. Do not use an image based backup. · From the RAID BIOS utility or from within OMSA (if your OS is not affected), Delete the VD. · Replace any failed or Predictive Failed drives. · Create a temporary Dissimilar RAID array from your current array configuration (ex: if you currently have a  RAID 5, then create two RAID 1’s). · Fast Initialize the temporary array(s). · Delete the temporary array(s). · Create your actual array. · Slowly or Fully initialize the array. ( Failure to do so will result in the puncture coming back to haunt you) · Re-install your OS and restore your data. This sounds tedious but it is the best recommendation to prevent similar issues from re-occurring. To further deter future issues,  it is recommended that you run consistency checks on all VD’s at least once of month during off-hours. Here is more information on Consistency Check and Background Patrol Reads. www.dell.com/.../SLN156918;l=en&s=bsd&cs=04 www.dell.com/.../ps1q06-20050212-Habas.pdf If you have multiple VDs... first, time them and spread out an automated schedule to run consistency checks. Here’s an example from an older Windows task scheduler. Refer to omconfig section for the pertinent information and use it accordingly based on your own OS. Syntax:  omconfig storage vdisk action=checkconsistency controller=id vdisk=id  EX: To run a consistency check on virtual disk 3 on controller 1, type: omconfig storage vdisk action=checkconsistency controller=1 vdisk=3  Please let us know if you have any further questions.
Gen 3 PE 1950 with 4 drives in RAID 5. This server also has a 6GBS HBA that attaches to a storage enclouse with 14 drives. We started having issues not being able to connect to the SQL DB's that reside on the storage enclosure. So i looked in the event logs and found: So i started looking at the storage enclosure immediately. After days of testing each physical drive in the enclosure, i login to OMSA on this server and see the internal drives connected to the PERC6/i labeled: So, it looks like we may have many drive issues including the enclosure (during the physical testing of 14 drives, 3 failed). So i am trying to decide the best way to resolve this issue. My thought, is start with the 4 internal drives, then move to the enclosure. Starting with the PERC6/i drives, it looks like there may be issues with 3 drives? If so, remove disk 0 and install another disk, use OMSA to add it to the array, let the array rebuild. Do this all the way to disk 3, yes?
Defunct Disks Everywhere!!