Bug 5181

Created attachment 5881 [details]
errors logged

These are all revelevant errors which lead up to this kernel bug.

------------[ cut here ]------------ 
kernel BUG at drivers/md/raid10.c:1448! 
invalid operand: 0000 [#1] 
Modules linked in: nfsd exportfs lockd sunrpc raid10 raid6 raid5 xor 
CPU:    0 
EIP:    0060:[<f881a270>]    Not tainted VLI 
EFLAGS: 00010246   (2.6.13) 
EIP is at sync_request+0x450/0x7d0 [raid10] 
eax: 00000000   ebx: f69d31c0   ecx: 00000002   edx: f0e27978 
esi: 00000000   edi: 00000001   ebp: 00000002   esp: f24a1e68 
ds: 007b   es: 007b   ss: 0068 
Process md1_resync (pid: 9580, threadinfo=f24a0000 task=f7b8f040) 
Stack: f7bb4260 f0e27960 00000000 00000000 00000002 00000000 00000000 00000080 
       00000000 f0e27960 f7bb4260 c1bda600 00000000 00000000 c1bda6b0 c029154d 
       c1bda600 00000000 f24a1ed0 00000001 f24a0000 f24a0000 00000000 00000000 
Call Trace: 
 [<c029154d>] md_do_sync+0x3ad/0x750 
 [<c0113245>] scheduler_tick+0x265/0x2f0 
 [<c02f5324>] schedule+0x324/0x5b0 
 [<c0290497>] md_thread+0x127/0x170 
 [<c0128ea0>] autoremove_wake_function+0x0/0x60 
 [<c0128ea0>] autoremove_wake_function+0x0/0x60 
 [<c0290370>] md_thread+0x0/0x170 
 [<c0100f65>] kernel_thread_helper+0x5/0x10 
Code: fb 85 c0 74 07 8b 40 30 85 c0 75 33 ff 44 24 10 83 c2 0c 39 6c 24 10 7c 
e2 39 4c 24 10 74 0c 8b 5c 24 28 8b 53 08 e9 b9 fc ff ff <0f> 0b a8 05 35 ae 
 81 f8 8b 4c 24 28 8b 51 08 e9 a5 fc ff ff 8b 
 

It looks like you lost to adjacent disks, so you have lots one disk worth of 
data (like losing both drives of a regular mirrored pair).

However, the code obviously isn't dealing with this situation very well.
I'll look into fixing it.

It looks from the logs like hdf really has a hard error, but hde might just have
been caught in the cross-fire (Being on the same channel).

If you can still read hdf successfully, you could try

mdadm /dev/md1 -Af /dev/hde1 /dev/hdg1 /dev/hdh1
(I'm guessing at 'md1' - use whatever it should be).
That should re-assemble the array, and it should work again.

Figured as much, luckily nothing terribly important was lost. 
 
mdadm will quite happily re-assemble the array with the 2 remaining disks, but 
any attempt to add either of the 2 "failed" disks results in the kernel bug. 
 
I've got replacement disks, but will leave the array in this state to aid in 
any debugging. 

Created attachment 5892 [details]
Patch for mdadm-1.12.0 to properly assemble raid10 arrays with errors

This patch to mdadm-1.12.0 will cause it not to try to assemble a raid10
with too many missing devices, and so --force will cause to try to included
recently failed devices properly.

Created attachment 5893 [details]
Patch for mdadm-2.0 to improve assembly of raid10s with missing devices

Use this version of the patch if you are using mdadm 2.0

Created attachment 5894 [details]
Patch for 2.6.13 to improve handling of errors in raid10

This patch
  1/ requires there to be sufficient drives before starting a raid10
  2/ will abort a recovery properly if there is no drive to recover from.

Hi,
 Could you please try the above patches, where relevant, and let me know
if it makes things work again properly.

Thanks,
NeilBrown

Wow, thanks for the quick code response!  
  
I've patched (mdadm-1.12.0, 2.6.13) and rebooted successfully.  
Now when attempting to re-assemble the array I get the following from mdadm: 
mdadm: /dev/md1 assembled from 2 drives and 1 spare - not enough to start the 
array. 
That's irrespective of whether I specify /dev/hd{e,f,g,h}1, each of all 4 
fully, or g & h and the 2 combinations of e & f on the command line. 
If I specify all 4 the inactive array is made up of f, g, and h. If I don't 
specify f, e gets used. 
If I attempt to add the missing drive, or remove a drive, I'm confronted with: 
mdadm: cannot get array info for /dev/md1 
Stopping the array does work though. 
 
Nothing of note is logged, just stopped, bind, bind, bind. 

Looks like you must have hot-added both 'failed' drives, so they forgot about
their original place in the array, so 'mdadm -Af' cannot fix it.

You will have to recreate the array.  This won't hurt your data (provide you get
it right...). As hdf is the one that appears to have really failed I'd leave that 
off, and
   mdadm --create /dev/md1 -l10 -n4 --chunk=whatever /dev/hde1 missing /dev/hdg1
/dev/hdh1

That should get it all back for you (providing hde1 isn't really bad).

Then if you are confident that hdf1 is good, hot add it back into the array.

OK, so I re-created the array with 3 disks (f, g, and h (yes, f, the original  
"broken" f)), and it built perfectly.  
I then hot-added hde (the other "broken" one), and left it to rebuild. It  
succeeded! All my data is still there too! :)  
  
I now have a fully functional array that I'm leaving un-assembled until I get  
round to testing or replacing any actually faulty disks. I've seen drives 
being spuriously kicked out of arrays before, which is why I attempted to 
rebuild using the supposedly broken drives. 
 
Thanks very much for your time. 

Glad it all worked out.  I'll get the patches into the mainline kernel
shortly and hopefully no-one else will have to suffer this particular bug.

thanks,

NeilBrown