Bug 6706 - modprobe -r ohci1394 hangs or panics
Summary: modprobe -r ohci1394 hangs or panics
Status: CLOSED CODE_FIX
Alias: None
Product: Drivers
Classification: Unclassified
Component: IEEE1394 (show other bugs)
Hardware: i386 Linux
: P2 normal
Assignee: Stefan Richter
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-06-18 03:40 UTC by Stefan Richter
Modified: 2006-12-10 08:32 UTC (History)
0 users

See Also:
Kernel Version: 2.6.15, 2.6.16
Tree: Mainline
Regression: ---


Attachments
effectively reverts 2.6.16 patch "Hold the device's parent's lock during probe and remove" (3.20 KB, patch)
2006-11-19 03:56 UTC, Stefan Richter
Details | Diff
an unsane fix (672 bytes, patch)
2006-11-19 05:47 UTC, Stefan Richter
Details | Diff
another ugly workaround (2.85 KB, patch)
2006-11-20 16:24 UTC, Stefan Richter
Details | Diff
previous patch revised (3.25 KB, patch)
2006-11-21 13:51 UTC, Stefan Richter
Details | Diff

Description Stefan Richter 2006-06-18 03:40:46 UTC
Most recent kernel where this bug did not occur: unknown
Distribution: Mandrake 10.1
Hardware Environment: x86 K7 uniprocessor
two FireWire host adapters (0: TI based, 1394b, 1: VIA based, 1394a)
Software Environment: gcc 3.4.1, kernel.org's kernel
Problem Description:

When "modprobe ohci1394" is followed shortly by "modprobe -r ohci1394" (e.g. 1
second after the previous modprobe finished), one of the following may happen:

 - kernel panic due to exception in interrupt
   (happened on 2.6.15.x preempt uniprocessor)

   or

 - modprobe -r hangs in D state, as does knodemgrd_0
   (happened on 2.6.16.x preempt SMP on a uniprocessor machine,
   with original drivers as well as with 1394 drivers equivalent
   to 2.6.17-rc6-mm2 --- i.e. 2.6.17 will also be affected).

   The knodemgrd_1 slept interruptibly (S state) while the other
   slept uninterruptibly (D state) right after modprobe -r was
   issued. This happens with or without other nodes attached to
   the FireWire ports, with or without eth1394 loaded.

There is no problem if a longer pause is put between modprobe and modprobe -r,
for example 4 seconds.
Comment 1 Stefan Richter 2006-10-22 04:21:13 UTC
Didn't happen anymore when I tested again with Linux 2.6.19-rc2. I am too lazy
to find out which patch fixed that. On a quick glance, this fix in 2.6.18 looks
like it:
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff_plain;h=445151932e869fd76b23bccff75ae2a600ccf3c9
Comment 2 Stefan Richter 2006-11-18 15:37:58 UTC
Alas it's not fixed. I never get a panic but the deadlock between the modprobe
process and the knodemgrd kthread does still happen if the timing is right.
Tested with 2.6.19-rc.

# modprobe ohci1394 && modprobe -r ohci1394
works.

# modprobe ohci1394 && sleep 1 && modprobe -r ohci1394
gets stuck in uninterruptible sleep on kthread_stop(). This is trying to stop
the knodemgrd which uninterruptibly sleeps on bus_rescan_devices_helper() meanwhile.


Call trace of the modprobe -r context:
    kthread_stop		in kernel/kthread.c
    nodemgr_remove_host		in drivers/ieee1394/nodemgr.c
    __unregister_host		in drivers/ieee1394/highlevel.c
    highlevel_remove_host	in drivers/ieee1394/highlevel.c
    hpsb_remove_host		in drivers/ieee1394/hosts.c
    ohci1394_pci_remove		in drivers/ieee1394/ohci1394.c
    pci_device_remove		in pci/pci-driver.c
    __device_release_driver	in drivers/base/dd.c
    driver_detach		in drivers/base/dd.c

Call trace of the knodemgrd context:
    bus_rescan_devices_helper	in drivers/base/bus.c
    bus_rescan_devices		in drivers/base/bus.c
    nodemgr_node_probe		in drivers/ieee1394/nodemgr.c
    nodemgr_host_thread		in drivers/ieee1394/nodemgr.c


It seems the following is the culprit:

Since Linux 2.6.16, bus_rescan_devices_helper takes down(&dev->parent->sem) if a
parent device exists. This is true for all devices that are managed by nodemgr.
(FireWire ud's have ud's or ne's as parent, and FireWire ne's have hosts as
parent.) And yes, the call in driver_detach to __device_release_driver is
enclosed in down(&dev->sem).
Comment 3 Stefan Richter 2006-11-19 03:56:10 UTC
Created attachment 9564 [details]
effectively reverts 2.6.16 patch "Hold the device's parent's lock during probe and remove"

As expected, reverting the dev->parent->sem changes made in Linux 2.6.16 avoids
the deadlock. Of course we cannot simply revert it without wreaking havoc in
the USB subsystem.
Comment 4 Stefan Richter 2006-11-19 05:47:36 UTC
Created attachment 9566 [details]
an unsane fix

This prevents the deadlock. The lines in // are not required.

Note, although the deadlock is now gone for good according to repeated tests,
there is now a new oops in eth1394, logged as bug 7550. This could have been
the lock-up which I observed with kernels before 2.6.16.
Comment 5 Stefan Richter 2006-11-19 14:40:25 UTC
Discussion on the mailinglists: http://lkml.org/lkml/2006/11/18/140
Comment 6 Stefan Richter 2006-11-20 16:24:52 UTC
Created attachment 9577 [details]
another ugly workaround

This patch creates a dummy driver and binds it to all fw-host devices. That
way, bus_rescan_devices_helper will skip fw-host devices and won't block on
their parent's device semaphore.
Comment 7 Stefan Richter 2006-11-21 13:51:42 UTC
Created attachment 9585 [details]
previous patch revised

binds a dummy driver named "ieee1394" to fw-host devices, thus prevents
fw-hosts from being scanned by the driver core, thus prevents the deadlock
Comment 8 Stefan Richter 2006-12-10 08:32:13 UTC
fix was merged, will appear in Linux 2.6.20-rc1

Note You need to log in before you can comment on or make changes to this bug.