Bug 9308 - forcedeth causes 7 hour boot delay
Summary: forcedeth causes 7 hour boot delay
Status: RESOLVED CODE_FIX
Alias: None
Product: Drivers
Classification: Unclassified
Component: Network (show other bugs)
Hardware: All Linux
: P1 blocking
Assignee: Ayaz Abdulla
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-11-05 09:43 UTC by Alex Howells
Modified: 2007-11-19 21:32 UTC (History)
5 users (show)

See Also:
Kernel Version: 2.6.24-rc1
Subsystem:
Regression: ---
Bisected commit-id:


Attachments
This patch removes the outer loop in mgmt unit detection code. (1.33 KB, patch)
2007-11-09 15:31 UTC, Ayaz Abdulla
Details | Diff

Description Alex Howells 2007-11-05 09:43:10 UTC
Kernel booting "fails" on a certain type of chassis we have in production use. It's got a forcedeth NIC on the board (two actually!); system is not responsive to sysrq combinations during its big pause, it's deadlocked in a loop for about seven hours after which it will boot perfectly normally.

Our chassis is:  

    http://www.supermicro.com/Aplus/system/1U/1011/AS-1011M-T2.cfm

dmesg output from the driver:

[   35.309834] forcedeth.c: Reverse Engineered nForce ethernet driver. Version
0.60.
[   35.317568] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 23
[   35.323305] ACPI: PCI Interrupt 0000:00:08.0[A] -> Link [LMAC] -> GSI 23
(level, low) -> IRQ 23
[   35.332026] forcedeth: using HIGHDMA
[   35.335620] 0000:00:08.0: link timer on.
[   35.339537] 0000:00:08.0: mgmt unit is running. mac in use 40000000.

Thanks :)
Comment 1 Daniel Drake 2007-11-05 09:49:05 UTC
Original downstream report:
https://bugs.gentoo.org/show_bug.cgi?id=197561

There's a codepath in the forcedeth probe routines that can potentially delay boot up to 7 hours.

Inside nv_probe:
	if (id->driver_data & DEV_HAS_MGMT_UNIT) {
[...]
			for (i = 0; i < 5000; i++) {
[...]
				if (nv_mgmt_acquire_sema(dev)) {


So, nv_mgmt_acquire_sema() may be called 5000 times.

Inside nv_mgmt_acquire_sema():
	for (i = 0; i < 10; i++) {
[...]
		msleep(500);
}

So, nv_mgmt_acquire_sema() may sleep for up to 5 seconds.

5 seconds * 5000 = almost 7 hours

Alex's hardware hits this exact code path: nv_mgmt_acquire_sema() never manages to acquire the semaphore, so is called 5000 times, and boot is delayed.

We tried disabling the loop, i.e.
			for (i = 0; i < 5000; i++) {
becomes:
			for (i = 0; i < 0; i++) {

In this case, the system boots as normal (no delay) and networking works fine.
Comment 2 Andrew Morton 2007-11-05 13:24:16 UTC
rofl.

This looks pretty simple to fix.
Comment 3 Alex Howells 2007-11-05 13:38:33 UTC
I'm not sure I see the architectural purpose to the loop if after running 5000 times and delaying my boot by seven hours, it fails through anyway without even a warning *and* my network interface works fine ....  :)

Unfortunately a fix more complex than simply removing the loop is beyond me!  Thus far I've not noticed any anomalous behaviour when the code within the loop isn't being run, but I've only been using it that way for 2-3 hours on <=4 boxes!

I'm more than happy to tinker and test any patches you guys come up with.
Comment 4 Daniel Drake 2007-11-05 16:34:33 UTC
Andrew, well, the question still remains, why did forcedeth on Alex's hardware fail to acquire this hardware-based semaphore even after 7 hours? I hope someone with more knowledge of the hardware can comment.

But yes, I agree that even if Alex didn't have this system in this strange situation, it should be regarded as a bug anyway that we have a codepath that can take 7 hours. :)
Comment 5 Ayaz Abdulla 2007-11-09 15:29:59 UTC
Thanks for catching this.

The IPMI firmware is holding the semaphore. This is most likely a bug in the firmware. You can try to upgrade the firmware by contacting Supermicro.

But, from a driver point of view, yes, we need to reduce the timeout :)

The extra outer loop is not needed.
Comment 6 Ayaz Abdulla 2007-11-09 15:31:03 UTC
Created attachment 13490 [details]
This patch removes the outer loop in mgmt unit detection code.
Comment 7 Ingo Molnar 2007-11-18 08:26:11 UTC
Ayaz, will you push this to Linus? It's still not upstream AFAICS.
Comment 8 Ayaz Abdulla 2007-11-19 12:07:05 UTC
I just sent the patch to netdev and stable kernel list.

Note You need to log in before you can comment on or make changes to this bug.