I have a synology DS411 device which has a raid5 array of 4 disks. I had what seems to be a one disk failure... which I have raised tickets with Synology I have no progress in last 3 weeks so this is my last resort.Here's what I have been able to dig up so far.############################################################ check mdstat###########################################################mgupta@DiskStation:/etc$ cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md2 : active raid5 sdb5[0] sdc5[2] sda5[1]      5846338944 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [UU_U]md1 : active raid1 sda2[0] sdb2[1] sdc2[2]      2097088 blocks [4/3] [UUU_]md0 : active raid1 sda1[1] sdb1[0]      2490176 blocks [4/2] [UU__]unused devices: <none>Hmm let's check the array status.############################################################ check array status###########################################################sudo mdadm --examine /dev/sda5/dev/sda5:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x0     Array UUID : 4eaec43d:77af7b98:5f64d654:4b1721a2           Name : DiskStation:2  (local to host DiskStation)  Creation Time : Sun Nov 18 12:57:42 2012     Raid Level : raid5   Raid Devices : 4 Avail Dev Size : 3897559680 (1858.50 GiB 1995.55 GB)     Array Size : 11692677888 (5575.50 GiB 5986.65 GB)  Used Dev Size : 3897559296 (1858.50 GiB 1995.55 GB)    Data Offset : 2048 sectors   Super Offset : 8 sectors          State : clean    Device UUID : 1acf0046:de3f9281:ce23b9c8:4173f4f5    Update Time : Wed Dec 14 12:26:33 2016       Checksum : c9902770 - correct         Events : 1898945         Layout : left-symmetric     Chunk Size : 64K   Device Role : Active device 1   Array State : AAAA ('A' == active, '.' == missing)mgupta@DiskStation:/etc$ sudo mdadm --examine /dev/sdb5/dev/sdb5:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x0     Array UUID : 4eaec43d:77af7b98:5f64d654:4b1721a2           Name : DiskStation:2  (local to host DiskStation)  Creation Time : Sun Nov 18 12:57:42 2012     Raid Level : raid5   Raid Devices : 4 Avail Dev Size : 3897559680 (1858.50 GiB 1995.55 GB)     Array Size : 11692677888 (5575.50 GiB 5986.65 GB)  Used Dev Size : 3897559296 (1858.50 GiB 1995.55 GB)    Data Offset : 2048 sectors   Super Offset : 8 sectors          State : clean    Device UUID : bd37470d:75dabe88:4d46f70e:cb9e4206    Update Time : Wed Dec 14 12:26:33 2016       Checksum : ee8f78b0 - correct         Events : 1898945         Layout : left-symmetric     Chunk Size : 64K   Device Role : Active device 0   Array State : AAAA ('A' == active, '.' == missing)mgupta@DiskStation:/etc$ sudo mdadm --examine /dev/sdc5/dev/sdc5:          Magic : a92b4efc        Version : 1.2    Feature Map : 0x0     Array UUID : 4eaec43d:77af7b98:5f64d654:4b1721a2           Name : DiskStation:2  (local to host DiskStation)  Creation Time : Sun Nov 18 12:57:42 2012     Raid Level : raid5   Raid Devices : 4 Avail Dev Size : 3897559680 (1858.50 GiB 1995.55 GB)     Array Size : 11692677888 (5575.50 GiB 5986.65 GB)  Used Dev Size : 3897559296 (1858.50 GiB 1995.55 GB)    Data Offset : 2048 sectors   Super Offset : 8 sectors          State : clean    Device UUID : 52820ab3:78740646:95548e58:29c187d9    Update Time : Wed Dec 14 12:26:33 2016       Checksum : ee728df2 - correct         Events : 1898945         Layout : left-symmetric     Chunk Size : 64K   Device Role : Active device 3   Array State : AA.A ('A' == active, '.' == missing)Hmm so /dev/sdc5 is missing a disk?    ############################################################ check events###########################################################mgupta@DiskStation:/etc$ sudo mdadm --examine /dev/sd[a-c]5 | egrep 'Event|/dev/sd'/dev/sda5:         Events : 1898945/dev/sdb5:         Events : 1898945/dev/sdc5:         Events : 1898945The event count on all disks looks good.. so maybe I can still get all my data back?############################################################ gentle array rebuild...###########################################################mgupta@DiskStation:/etc$ sudo mdadm -v --assemble --force /dev/md2 /dev/sda5 /dev/sdb5 /dev/sdc5 mdadm: looking for devices for /dev/md2mdadm: cannot open device /dev/sda5: Device or resource busymdadm: /dev/sda5 has no superblock - assembly aborted############################################################ Lets get info for all 3 disks###########################################################mgupta@DiskStation:/etc$ sudo smartctl -a /dev/sda5smartctl 6.5 (build date Apr 26 2016) [armv5tel-linux-2.6.32.12] (local build)Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org=== START OF INFORMATION SECTION ===Vendor:               WDCProduct:              WD20EARX-00PASB0Revision:             51.0User Capacity:        2,000,398,934,016 bytes [2.00 TB]Logical block size:   512 bytesSerial number:        WD-WMAZA7614058Device type:          diskLocal Time is:        Wed Dec 14 17:36:16 2016 ESTSMART support is:     Unavailable - device lacks SMART capability.=== START OF READ SMART DATA SECTION ===Current Drive Temperature:     0 CDrive Trip Temperature:        0 CError Counter logging not supported[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']Device does not support Self Test loggingmgupta@DiskStation:/etc$ sudo smartctl -a /dev/sdb5smartctl 6.5 (build date Apr 26 2016) [armv5tel-linux-2.6.32.12] (local build)Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org=== START OF INFORMATION SECTION ===Model Family:     Seagate Barracuda Green (AF)Device Model:     ST2000DL003-9VT166Serial Number:    6YD0RVPVLU WWN Device Id: 5 000c50 03def0174Firmware Version: CC32User Capacity:    2,000,398,934,016 bytes [2.00 TB]Sector Size:      512 bytes logical/physicalRotation Rate:    5900 rpmDevice is:        In smartctl database [for details use: -P show]ATA Version is:   ATA8-ACS T13/1699-D revision 4SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)Local Time is:    Wed Dec 14 17:36:22 2016 ESTSMART support is: Available - device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status:  (0x82) Offline data collection activity                    was completed without error.                    Auto Offline Data Collection: Enabled.Self-test execution status:      (   0) The previous self-test routine completed                    without error or no self-test has ever                     been run.Total time to complete Offline data collection:        (  612) seconds.Offline data collectioncapabilities:            (0x7b) SMART execute Offline immediate.                    Auto Offline data collection on/off support.                    Suspend Offline collection upon new                    command.                    Offline surface scan supported.                    Self-test supported.                    Conveyance Self-test supported.                    Selective Self-test supported.SMART capabilities:            (0x0003) Saves SMART data before entering                    power-saving mode.                    Supports SMART auto save timer.Error logging capability:        (0x01) Error logging supported.                    General Purpose Logging supported.Short self-test routine recommended polling time:    (   1) minutes.Extended self-test routinerecommended polling time:    ( 327) minutes.Conveyance self-test routinerecommended polling time:    (   2) minutes.SCT capabilities:          (0x30b7) SCT Status supported.                    SCT Feature Control supported.                    SCT Data Table supported.SMART Attributes Data Structure revision number: 10Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME                                                   FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE  1 Raw_Read_Error_Rate                                              0x000f   117   099   006    Pre-fail  Always       -       152709864  3 Spin_Up_Time                                                     0x0003   084   075   000    Pre-fail  Always       -       0  4 Start_Stop_Count                                                 0x0032   092   092   020    Old_age   Always       -       8491  5 Reallocated_Sector_Ct                                            0x0033   100   100   036    Pre-fail  Always       -       0  7 Seek_Error_Rate                                                  0x000f   052   036   030    Pre-fail  Always       -       1383036844200  9 Power_On_Hours                                                   0x0032   070   011   000    Old_age   Always       -       26495 10 Spin_Retry_Count                                                 0x0013   100   100   097    Pre-fail  Always       -       0 12 Power_Cycle_Count                                                0x0032   100   100   020    Old_age   Always       -       121183 Runtime_Bad_Block                                                0x0032   100   100   000    Old_age   Always       -       0184 End-to-End_Error                                                 0x0032   100   100   099    Old_age   Always       -       0187 Reported_Uncorrect                                               0x0032   100   100   000    Old_age   Always       -       0188 Command_Timeout                                                  0x0032   100   100   000    Old_age   Always       -       0189 High_Fly_Writes                                                  0x003a   100   100   000    Old_age   Always       -       0190 Airflow_Temperature_Cel                                          0x0022   056   049   045    Old_age   Always       -       44 (Min/Max 21/44)191 G-Sense_Error_Rate                                               0x0032   100   100   000    Old_age   Always       -       0192 Power-Off_Retract_Count                                          0x0032   100   100   000    Old_age   Always       -       64193 Load_Cycle_Count                                                 0x0032   096   096   000    Old_age   Always       -       8496194 Temperature_Celsius                                              0x0022   044   051   000    Old_age   Always       -       44 (0 19 0 0 0)195 Hardware_ECC_Recovered                                           0x001a   021   006   000    Old_age   Always       -       152709864197 Current_Pending_Sector                                           0x0012   100   100   000    Old_age   Always       -       0198 Offline_Uncorrectable                                            0x0010   100   100   000    Old_age   Offline      -       0199 UDMA_CRC_Error_Count                                             0x003e   200   200   000    Old_age   Always       -       0240 Head_Flying_Hours                                                0x0000   100   253   000    Old_age   Offline      -       17309 (200 184 0)241 Total_LBAs_Written                                               0x0000   100   253   000    Old_age   Offline      -       3099252822242 Total_LBAs_Read                                                  0x0000   100   253   000    Old_age   Offline      -       2293937892SMART Error Log Version: 1No Errors LoggedSMART Self-test log structure revision number 1No self-tests have been logged.  [To run self-tests, use: smartctl -t]SMART Selective self-test log data structure revision number 1 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS    1        0        0  Not_testing    2        0        0  Not_testing    3        0        0  Not_testing    4        0        0  Not_testing    5        0        0  Not_testingSelective self-test flags (0x0):  After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.mgupta@DiskStation:/etc$ sudo smartctl -a /dev/sdc5smartctl 6.5 (build date Apr 26 2016) [armv5tel-linux-2.6.32.12] (local build)Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org=== START OF INFORMATION SECTION ===Model Family:     Seagate Barracuda 7200.14 (AF)Device Model:     ST2000DM001-9YN164Serial Number:    Z1E0S6X0LU WWN Device Id: 5 000c50 05139e6f0Firmware Version: CC4BUser Capacity:    2,000,398,934,016 bytes [2.00 TB]Sector Sizes:     512 bytes logical, 4096 bytes physicalRotation Rate:    7200 rpmDevice is:        In smartctl database [for details use: -P show]ATA Version is:   ATA8-ACS T13/1699-D revision 4SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)Local Time is:    Wed Dec 14 17:36:48 2016 EST==> WARNING: A firmware update for this drive may be available,see the following Seagate web pages:http://knowledge.seagate.com/articles/en_US/FAQ/207931enhttp://knowledge.seagate.com/articles/en_US/FAQ/223651enSMART support is: Available - device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status:  (0x82) Offline data collection activity                    was completed without error.                    Auto Offline Data Collection: Enabled.Self-test execution status:      (   0) The previous self-test routine completed                    without error or no self-test has ever                     been run.Total time to complete Offline data collection:        (  575) seconds.Offline data collectioncapabilities:            (0x7b) SMART execute Offline immediate.                    Auto Offline data collection on/off support.                    Suspend Offline collection upon new                    command.                    Offline surface scan supported.                    Self-test supported.                    Conveyance Self-test supported.                    Selective Self-test supported.SMART capabilities:            (0x0003) Saves SMART data before entering                    power-saving mode.                    Supports SMART auto save timer.Error logging capability:        (0x01) Error logging supported.                    General Purpose Logging supported.Short self-test routine recommended polling time:    (   1) minutes.Extended self-test routinerecommended polling time:    ( 211) minutes.Conveyance self-test routinerecommended polling time:    (   2) minutes.SCT capabilities:          (0x3085) SCT Status supported.SMART Attributes Data Structure revision number: 10Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME                                                   FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE  1 Raw_Read_Error_Rate                                              0x000f   103   099   006    Pre-fail  Always       -       6147256  3 Spin_Up_Time                                                     0x0003   095   095   000    Pre-fail  Always       -       0  4 Start_Stop_Count                                                 0x0032   093   093   020    Old_age   Always       -       7563  5 Reallocated_Sector_Ct                                            0x0033   100   100   036    Pre-fail  Always       -       800  7 Seek_Error_Rate                                                  0x000f   074   060   030    Pre-fail  Always       -       25741443  9 Power_On_Hours                                                   0x0032   066   066   000    Old_age   Always       -       30275 10 Spin_Retry_Count                                                 0x0013   100   100   097    Pre-fail  Always       -       0 12 Power_Cycle_Count                                                0x0032   100   100   020    Old_age   Always       -       84183 Runtime_Bad_Block                                                0x0032   100   100   000    Old_age   Always       -       0184 End-to-End_Error                                                 0x0032   100   100   099    Old_age   Always       -       0187 Reported_Uncorrect                                               0x0032   099   099   000    Old_age   Always       -       1188 Command_Timeout                                                  0x0032   100   100   000    Old_age   Always       -       0 0 0189 High_Fly_Writes                                                  0x003a   096   096   000    Old_age   Always       -       4190 Airflow_Temperature_Cel                                          0x0022   053   049   045    Old_age   Always       -       47 (Min/Max 24/47)191 G-Sense_Error_Rate                                               0x0032   100   100   000    Old_age   Always       -       0192 Power-Off_Retract_Count                                          0x0032   100   100   000    Old_age   Always       -       50193 Load_Cycle_Count                                                 0x0032   097   097   000    Old_age   Always       -       7572194 Temperature_Celsius                                              0x0022   047   051   000    Old_age   Always       -       47 (0 22 0 0 0)197 Current_Pending_Sector                                           0x0012   100   100   000    Old_age   Always       -       0198 Offline_Uncorrectable                                            0x0010   100   100   000    Old_age   Offline      -       0199 UDMA_CRC_Error_Count                                             0x003e   200   200   000    Old_age   Always       -       0240 Head_Flying_Hours                                                0x0000   100   253   000    Old_age   Offline      -       15399h+51m+07.327s241 Total_LBAs_Written                                               0x0000   100   253   000    Old_age   Offline      -       25716789180543242 Total_LBAs_Read                                                  0x0000   100   253   000    Old_age   Offline      -       15598169133360SMART Error Log Version: 1No Errors LoggedSMART Self-test log structure revision number 1No self-tests have been logged.  [To run self-tests, use: smartctl -t]SMART Selective self-test log data structure revision number 1 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS    1        0        0  Not_testing    2        0        0  Not_testing    3        0        0  Not_testing    4        0        0  Not_testing    5        0        0  Not_testingSelective self-test flags (0x0):  After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.mgupta@DiskStation:/etc$ ############################################################ try stopped md2###########################################################mgupta@DiskStation:/etc$ sudo mdadm --stop /dev/md2mdadm: failed to stop array /dev/md2: Device or resource busyPerhaps a running process, mounted filesystem or active volume group?Can I just re-assemble it?############################################################ try reassemble###########################################################mgupta@DiskStation:/etc$ sudo mdadm --assemble --run --force /dev/md2 /dev/sda5 /dev/sdb5 /dev/sdc5mdadm: cannot open device /dev/sda5: Device or resource busymdadm: /dev/sda5 has no superblock - assembly abortedHmm... no superblock? Not sure what I can do now...############################################################ space history to see raid config###########################################################mgupta@DiskStation:/etc$ sudo more space/space_history_20161203_222319.xml<?xml version=1.0 encoding=UTF-8?><spaces>    <space path=/dev/vg1000/lv reference=/volume1 uuid=XVJ3ii-EoQW-Y4B3-DeVC-o0Yq-TqtR-7o2MCI device_type=1 drive_type=0 container_type=2 limited_raidgroup_num=12 >        <device>            <lvm path=/dev/vg1000 uuid=Jv3wdF-3ARY-qlN7-Rl5B-ctD2-DiOg-pli8Pz designed_pv_counts=1 status=normal total_size=5986647539712 free_size=0 pe_size=4194304 expansible=0 max_size=5846338944>                <raids>                    <raid path=/dev/md2 uuid=4eaec43d:77af7b98:5f64d654:4b1721a2 level=raid5 version=1.2>                        <disks>                            <disk status=normal dev_path=/dev/sda5 model=WD20EARX-00PASB0         serial=WD-WMAZA7614058 partition_version=7 partition_start=9453280 partition_size=3897561728 slot=1>                            </disk>                            <disk status=normal dev_path=/dev/sdb5 model=ST2000DL003-9VT166       serial=6YD0RVPV partition_version=7 partition_start=9453280 partition_size=3897561728 slot=0>                            </disk>                            <disk status=normal dev_path=/dev/sdc5 model=ST2000DM001-9YN164       serial=Z1E0S6X0 partition_version=7 partition_start=9453280 partition_size=3897561728 slot=3>                            </disk>                        </disks>                    </raid>                </raids>            </lvm>        </device>        <reference>            <volumes>                <volume path=/volume1 dev_path=/dev/vg1000/lv uuid=XVJ3ii-EoQW-Y4B3-DeVC-o0Yq-TqtR-7o2MCI type=ext4>                </volume>            </volumes>        </reference>    </space></spaces>############################################################ mdadm detail ###########################################################mgupta@DiskStation:/dev/vg1000$ sudo mdadm --detail --scanmdadm: cannot open /dev/md/0: No such file or directorymdadm: cannot open /dev/md/1: No such file or directoryARRAY /dev/md2 metadata=1.2 name=DiskStation:2 UUID=4eaec43d:77af7b98:5f64d654:4b1721a2ARRAY /dev/md/0_0 metadata=0.90 UUID=a4880af1:1fc10752:3017a5a8:c86610be############################################################ dmsetup###########################################################mgupta@DiskStation:/dev/vg1000$ sudo dmsetup lsvg1000-lv   (253:0)mgupta@DiskStation:/dev/vg1000$ sudo dmsetup tablevg1000-lv: 0 11692670976 linear 9:2 1152mgupta@DiskStation:/dev/vg1000$ ############################################################ cat /proc/partitions###########################################################mgupta@DiskStation:/etc$ cat /proc/partitionsmajor minor  #blocks  name   8        0 1953514584 sda   8        1    2490240 sda1   8        2    2097152 sda2   8        3          1 sda3   8        5 1948780864 sda5   8       16 1953514584 sdb   8       17    2490240 sdb1   8       18    2097152 sdb2   8       19          1 sdb3   8       21 1948780864 sdb5   8       32 1953514584 sdc   8       33    2490240 sdc1   8       34    2097152 sdc2   8       35          1 sdc3   8       37 1948780864 sdc5  31        0        512 mtdblock0  31        1       2048 mtdblock1  31        2       1280 mtdblock2  31        3         64 mtdblock3  31        4        128 mtdblock4  31        5         64 mtdblock5  31        6       4096 mtdblock6   9        0    2490176 md0   9        1    2097088 md1   9        2 5846338944 md2 253        0 5846335488 dm-0   9      127    2490176 md127What should I do next?
mdadm cannot start failed raid5 array synology
mdadm;raid5;synology