Bug 5181 - kernel BUG at drivers/md/raid10.c:1448!
kernel BUG at drivers/md/raid10.c:1448!
Status: RESOLVED CODE_FIX
Product: IO/Storage
Classification: Unclassified
Component: MD
i386 Linux
: P2 normal
Assigned To: Neil Brown
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-09-03 12:14 UTC by Mike Williams
Modified: 2005-09-05 16:53 UTC (History)
0 users

See Also:
Kernel Version: 2.6.13
Tree: Mainline
Regression: ---


Attachments
errors logged (40.81 KB, text/plain)
2005-09-03 12:18 UTC, Mike Williams
Details
Patch for mdadm-1.12.0 to properly assemble raid10 arrays with errors (4.15 KB, patch)
2005-09-04 18:16 UTC, Neil Brown
Details | Diff
Patch for mdadm-2.0 to improve assembly of raid10s with missing devices (5.84 KB, patch)
2005-09-04 18:18 UTC, Neil Brown
Details | Diff
Patch for 2.6.13 to improve handling of errors in raid10 (5.14 KB, patch)
2005-09-04 18:22 UTC, Neil Brown
Details | Diff

Description Mike Williams 2005-09-03 12:14:12 UTC
Most recent kernel where this bug did not occur: N/A 
Distribution: Gentoo   
Hardware Environment: 4 IDE disk, 0000:00:09.0 Mass storage controller:  
Promise Technology, Inc. 20269 (rev 02)  
Software Environment:   
Problem Description:  
RAID10 array, 2 of 4 drives failed (dmesg output will be attached), and now  
kernel remotes a BUG in the raid10 driver when attempting to rebuild the  
array.  
I was originally running 2.6.11.10 and had the same problem, so upgraded to  
2.6.13 but it made no difference.  
   
Steps to reproduce:
Comment 1 Mike Williams 2005-09-03 12:18:09 UTC
Created attachment 5881 [details]
errors logged

These are all revelevant errors which lead up to this kernel bug.
Comment 2 Mike Williams 2005-09-03 12:18:32 UTC
------------[ cut here ]------------ 
kernel BUG at drivers/md/raid10.c:1448! 
invalid operand: 0000 [#1] 
Modules linked in: nfsd exportfs lockd sunrpc raid10 raid6 raid5 xor 
CPU:    0 
EIP:    0060:[<f881a270>]    Not tainted VLI 
EFLAGS: 00010246   (2.6.13) 
EIP is at sync_request+0x450/0x7d0 [raid10] 
eax: 00000000   ebx: f69d31c0   ecx: 00000002   edx: f0e27978 
esi: 00000000   edi: 00000001   ebp: 00000002   esp: f24a1e68 
ds: 007b   es: 007b   ss: 0068 
Process md1_resync (pid: 9580, threadinfo=f24a0000 task=f7b8f040) 
Stack: f7bb4260 f0e27960 00000000 00000000 00000002 00000000 00000000 00000080 
       00000000 f0e27960 f7bb4260 c1bda600 00000000 00000000 c1bda6b0 c029154d 
       c1bda600 00000000 f24a1ed0 00000001 f24a0000 f24a0000 00000000 00000000 
Call Trace: 
 [<c029154d>] md_do_sync+0x3ad/0x750 
 [<c0113245>] scheduler_tick+0x265/0x2f0 
 [<c02f5324>] schedule+0x324/0x5b0 
 [<c0290497>] md_thread+0x127/0x170 
 [<c0128ea0>] autoremove_wake_function+0x0/0x60 
 [<c0128ea0>] autoremove_wake_function+0x0/0x60 
 [<c0290370>] md_thread+0x0/0x170 
 [<c0100f65>] kernel_thread_helper+0x5/0x10 
Code: fb 85 c0 74 07 8b 40 30 85 c0 75 33 ff 44 24 10 83 c2 0c 39 6c 24 10 7c 
e2 39 4c 24 10 74 0c 8b 5c 24 28 8b 53 08 e9 b9 fc ff ff <0f> 0b a8 05 35 ae 
 81 f8 8b 4c 24 28 8b 51 08 e9 a5 fc ff ff 8b 
 
Comment 3 Neil Brown 2005-09-03 23:04:08 UTC
It looks like you lost to adjacent disks, so you have lots one disk worth of 
data (like losing both drives of a regular mirrored pair).

However, the code obviously isn't dealing with this situation very well.
I'll look into fixing it.

It looks from the logs like hdf really has a hard error, but hde might just have
been caught in the cross-fire (Being on the same channel).

If you can still read hdf successfully, you could try

mdadm /dev/md1 -Af /dev/hde1 /dev/hdg1 /dev/hdh1
(I'm guessing at 'md1' - use whatever it should be).
That should re-assemble the array, and it should work again.
Comment 4 Mike Williams 2005-09-04 09:26:42 UTC
Figured as much, luckily nothing terribly important was lost. 
 
mdadm will quite happily re-assemble the array with the 2 remaining disks, but 
any attempt to add either of the 2 "failed" disks results in the kernel bug. 
 
I've got replacement disks, but will leave the array in this state to aid in 
any debugging. 
Comment 5 Neil Brown 2005-09-04 18:16:39 UTC
Created attachment 5892 [details]
Patch for mdadm-1.12.0 to properly assemble raid10 arrays with errors

This patch to mdadm-1.12.0 will cause it not to try to assemble a raid10
with too many missing devices, and so --force will cause to try to included
recently failed devices properly.
Comment 6 Neil Brown 2005-09-04 18:18:18 UTC
Created attachment 5893 [details]
Patch for mdadm-2.0 to improve assembly of raid10s with missing devices

Use this version of the patch if you are using mdadm 2.0
Comment 7 Neil Brown 2005-09-04 18:22:03 UTC
Created attachment 5894 [details]
Patch for 2.6.13 to improve handling of errors in raid10

This patch
  1/ requires there to be sufficient drives before starting a raid10
  2/ will abort a recovery properly if there is no drive to recover from.
Comment 8 Neil Brown 2005-09-04 18:23:25 UTC
Hi,
 Could you please try the above patches, where relevant, and let me know
if it makes things work again properly.

Thanks,
NeilBrown
Comment 9 Mike Williams 2005-09-04 19:34:58 UTC
Wow, thanks for the quick code response!  
  
I've patched (mdadm-1.12.0, 2.6.13) and rebooted successfully.  
Now when attempting to re-assemble the array I get the following from mdadm: 
mdadm: /dev/md1 assembled from 2 drives and 1 spare - not enough to start the 
array. 
That's irrespective of whether I specify /dev/hd{e,f,g,h}1, each of all 4 
fully, or g & h and the 2 combinations of e & f on the command line. 
If I specify all 4 the inactive array is made up of f, g, and h. If I don't 
specify f, e gets used. 
If I attempt to add the missing drive, or remove a drive, I'm confronted with: 
mdadm: cannot get array info for /dev/md1 
Stopping the array does work though. 
 
Nothing of note is logged, just stopped, bind, bind, bind. 
Comment 10 Neil Brown 2005-09-04 21:54:39 UTC
Looks like you must have hot-added both 'failed' drives, so they forgot about
their original place in the array, so 'mdadm -Af' cannot fix it.

You will have to recreate the array.  This won't hurt your data (provide you get
it right...). As hdf is the one that appears to have really failed I'd leave that 
off, and
   mdadm --create /dev/md1 -l10 -n4 --chunk=whatever /dev/hde1 missing /dev/hdg1
/dev/hdh1

That should get it all back for you (providing hde1 isn't really bad).

Then if you are confident that hdf1 is good, hot add it back into the array.


Comment 11 Mike Williams 2005-09-05 13:44:50 UTC
OK, so I re-created the array with 3 disks (f, g, and h (yes, f, the original  
"broken" f)), and it built perfectly.  
I then hot-added hde (the other "broken" one), and left it to rebuild. It  
succeeded! All my data is still there too! :)  
  
I now have a fully functional array that I'm leaving un-assembled until I get  
round to testing or replacing any actually faulty disks. I've seen drives 
being spuriously kicked out of arrays before, which is why I attempted to 
rebuild using the supposedly broken drives. 
 
Thanks very much for your time. 
Comment 12 Neil Brown 2005-09-05 16:53:00 UTC
Glad it all worked out.  I'll get the patches into the mainline kernel
shortly and hopefully no-one else will have to suffer this particular bug.

thanks,

NeilBrown

Note You need to log in before you can comment on or make changes to this bug.