Most recent kernel where this bug did not occur: N/A Distribution: Gentoo Hardware Environment: 4 IDE disk, 0000:00:09.0 Mass storage controller: Promise Technology, Inc. 20269 (rev 02) Software Environment: Problem Description: RAID10 array, 2 of 4 drives failed (dmesg output will be attached), and now kernel remotes a BUG in the raid10 driver when attempting to rebuild the array. I was originally running 2.6.11.10 and had the same problem, so upgraded to 2.6.13 but it made no difference. Steps to reproduce:
Created attachment 5881 [details] errors logged These are all revelevant errors which lead up to this kernel bug.
------------[ cut here ]------------ kernel BUG at drivers/md/raid10.c:1448! invalid operand: 0000 [#1] Modules linked in: nfsd exportfs lockd sunrpc raid10 raid6 raid5 xor CPU: 0 EIP: 0060:[<f881a270>] Not tainted VLI EFLAGS: 00010246 (2.6.13) EIP is at sync_request+0x450/0x7d0 [raid10] eax: 00000000 ebx: f69d31c0 ecx: 00000002 edx: f0e27978 esi: 00000000 edi: 00000001 ebp: 00000002 esp: f24a1e68 ds: 007b es: 007b ss: 0068 Process md1_resync (pid: 9580, threadinfo=f24a0000 task=f7b8f040) Stack: f7bb4260 f0e27960 00000000 00000000 00000002 00000000 00000000 00000080 00000000 f0e27960 f7bb4260 c1bda600 00000000 00000000 c1bda6b0 c029154d c1bda600 00000000 f24a1ed0 00000001 f24a0000 f24a0000 00000000 00000000 Call Trace: [<c029154d>] md_do_sync+0x3ad/0x750 [<c0113245>] scheduler_tick+0x265/0x2f0 [<c02f5324>] schedule+0x324/0x5b0 [<c0290497>] md_thread+0x127/0x170 [<c0128ea0>] autoremove_wake_function+0x0/0x60 [<c0128ea0>] autoremove_wake_function+0x0/0x60 [<c0290370>] md_thread+0x0/0x170 [<c0100f65>] kernel_thread_helper+0x5/0x10 Code: fb 85 c0 74 07 8b 40 30 85 c0 75 33 ff 44 24 10 83 c2 0c 39 6c 24 10 7c e2 39 4c 24 10 74 0c 8b 5c 24 28 8b 53 08 e9 b9 fc ff ff <0f> 0b a8 05 35 ae 81 f8 8b 4c 24 28 8b 51 08 e9 a5 fc ff ff 8b
It looks like you lost to adjacent disks, so you have lots one disk worth of data (like losing both drives of a regular mirrored pair). However, the code obviously isn't dealing with this situation very well. I'll look into fixing it. It looks from the logs like hdf really has a hard error, but hde might just have been caught in the cross-fire (Being on the same channel). If you can still read hdf successfully, you could try mdadm /dev/md1 -Af /dev/hde1 /dev/hdg1 /dev/hdh1 (I'm guessing at 'md1' - use whatever it should be). That should re-assemble the array, and it should work again.
Figured as much, luckily nothing terribly important was lost. mdadm will quite happily re-assemble the array with the 2 remaining disks, but any attempt to add either of the 2 "failed" disks results in the kernel bug. I've got replacement disks, but will leave the array in this state to aid in any debugging.
Created attachment 5892 [details] Patch for mdadm-1.12.0 to properly assemble raid10 arrays with errors This patch to mdadm-1.12.0 will cause it not to try to assemble a raid10 with too many missing devices, and so --force will cause to try to included recently failed devices properly.
Created attachment 5893 [details] Patch for mdadm-2.0 to improve assembly of raid10s with missing devices Use this version of the patch if you are using mdadm 2.0
Created attachment 5894 [details] Patch for 2.6.13 to improve handling of errors in raid10 This patch 1/ requires there to be sufficient drives before starting a raid10 2/ will abort a recovery properly if there is no drive to recover from.
Hi, Could you please try the above patches, where relevant, and let me know if it makes things work again properly. Thanks, NeilBrown
Wow, thanks for the quick code response! I've patched (mdadm-1.12.0, 2.6.13) and rebooted successfully. Now when attempting to re-assemble the array I get the following from mdadm: mdadm: /dev/md1 assembled from 2 drives and 1 spare - not enough to start the array. That's irrespective of whether I specify /dev/hd{e,f,g,h}1, each of all 4 fully, or g & h and the 2 combinations of e & f on the command line. If I specify all 4 the inactive array is made up of f, g, and h. If I don't specify f, e gets used. If I attempt to add the missing drive, or remove a drive, I'm confronted with: mdadm: cannot get array info for /dev/md1 Stopping the array does work though. Nothing of note is logged, just stopped, bind, bind, bind.
Looks like you must have hot-added both 'failed' drives, so they forgot about their original place in the array, so 'mdadm -Af' cannot fix it. You will have to recreate the array. This won't hurt your data (provide you get it right...). As hdf is the one that appears to have really failed I'd leave that off, and mdadm --create /dev/md1 -l10 -n4 --chunk=whatever /dev/hde1 missing /dev/hdg1 /dev/hdh1 That should get it all back for you (providing hde1 isn't really bad). Then if you are confident that hdf1 is good, hot add it back into the array.
OK, so I re-created the array with 3 disks (f, g, and h (yes, f, the original "broken" f)), and it built perfectly. I then hot-added hde (the other "broken" one), and left it to rebuild. It succeeded! All my data is still there too! :) I now have a fully functional array that I'm leaving un-assembled until I get round to testing or replacing any actually faulty disks. I've seen drives being spuriously kicked out of arrays before, which is why I attempted to rebuild using the supposedly broken drives. Thanks very much for your time.
Glad it all worked out. I'll get the patches into the mainline kernel shortly and hopefully no-one else will have to suffer this particular bug. thanks, NeilBrown