Bug 4916

Summary: USB mouse stops working after inserting USB storage device
Product: Drivers Reporter: Sander Bouwhuis (sanderb)
Component: Input DevicesAssignee: Dmitry Torokhov (dmitry.torokhov)
Status: CLOSED CODE_FIX    
Severity: normal CC: akpm, bunk, greg, llsousa, nacc, stern, vojtech
Priority: P2    
Hardware: i386   
OS: Linux   
Kernel Version: 2.6.12 Subsystem:
Regression: --- Bisected commit-id:
Attachments: Output of lspci
dmesg output, including any output when adding the storage device, removing the mouse and plugging the mouse back in
lsusb output at 1) boot with the mouse only; 2) with the mouse still plugged in and having added the disk; 3) just the disk plugged in (unplug disk) and 4) with the mouse plugged back in
/sys/kernel/debug/uhci/0000:00:1d.0 before the USB stick is inserted
/sys/kernel/debug/uhci/0000:00:1d.0 after USB stick is inserted
Complete dmesg
usbmon output of plugging in USB stick with mouse already inserted, then replugging the mouse
Remove bogus tests from HID driver
usbmon output of the same process with the provided patch applied
Manual device reset patch
dmesg output after going through the whole procedure
usbmon log of the whole procedure (inserting stick, issuing command, replugging)
usbmon
dmesg
Add HID device reset routine
rejected patch
Add HID device reset routine
Add error recovery to the USB HID driver
Add error recovery to the USB HID driver (updated)

Description Sander Bouwhuis 2005-07-19 14:38:36 UTC
Distribution: Gentoo     
    
Hardware Environment: ASUS M2400N notebook   
    
Software Environment: ?    
    
Problem Description: When my USB mouse is plugged in and I insert my USB memory  
stick in another USB slot, the mouse stops working and needs to be taken out  
and plugged in again to resume.  
     
Steps to reproduce:   
Insert USB mouse (it works)  
Insert USB memory device (mouse stops working)  
Pull out and reinsert mouse (it works again) 
 
I never committed a kernel bug before so I don't know what info I should 
provide. If somethign is needed, just ask and I'll be happy to provide it.
Comment 1 baldrick 2005-07-20 01:30:55 UTC
What is the output of lspci? 
Comment 2 Sander Bouwhuis 2005-07-20 01:37:09 UTC
Created attachment 5349 [details]
Output of lspci

Output of lspci
Comment 3 Nishanth Aravamudan 2005-07-20 10:43:35 UTC
Could you please give dmesg output, including any output when adding the storage
device, removing the mouse and plugging the mouse back in. This data should also
be in /var/log/messages. May also be useful to have the lsusb output at 1) boot
with the mouse only; 2) with the mouse still plugged in and having added the
disk; 3) just the disk plugged in (unplug disk) and 4) with the mouse plugged
back in.

Thanks,
Nish
Comment 4 Sander Bouwhuis 2005-07-21 00:02:09 UTC
Created attachment 5353 [details]
dmesg output, including any output when adding the storage
device, removing the mouse and plugging the mouse back in
Comment 5 Sander Bouwhuis 2005-07-21 00:15:48 UTC
Created attachment 5354 [details]
lsusb output at 1) boot
with the mouse only; 2) with the mouse still plugged in and having added the
disk; 3) just the disk plugged in (unplug disk) and 4) with the mouse plugged
back in

Hope it helps!

Sander
Comment 6 Alan Stern 2005-07-26 08:53:02 UTC
Can you try borrowing a someone else's USB mouse, to see if the same thing still
happens?

Also, when the mouse has stopped working, what happens if you rmmod usbhid and
then modprobe usbhid?  Does the mouse start to work again?
Comment 7 Sander Bouwhuis 2005-07-26 09:46:45 UTC
Yes, It does happen with an USB stick I borrowed from a friend   
   
When I try to rmmod and modprobe usbhid I get the following:  
  
$ sudo rmmod usbhid  
ERROR: Module usbhid does not exist in /proc/modules  
$ sudo modprobe usbhid  
FATAL: Module usbhid not found.  
  
Is that something I should have compiled in? When I search in menuconfig for 
'usbhid' I don't get any matches though.. 
  
Comment 8 Alan Stern 2005-07-26 11:47:39 UTC
But what happens if you use a different mouse (not a different stick)?

Apparently you have usbhid compiled into your kernel.  Its setting in menuconfig
is under Device Drivers -> USB support -> USB Human Interface Device (full HID)
support, in the USB Input Devices subsection.  You will also want to make sure
that HID input layer support is turned on.
Comment 9 Sander Bouwhuis 2005-07-26 13:51:05 UTC
Using another USB mouse did in fact work as it should, but another mouse of the  
type that I'm normally using didn't. So the problem is probably related to the  
type of mouse I'm using, but it's not my specific mouse that's broke.  
    
I recompiled my kernel with usbhid as a module, and got no errors on the 
rmmod/modprobe thing anymore, however it didn't make the mouse work. I still 
had to take it out before it worked again. 
Comment 10 Andrew Morton 2005-07-29 00:50:44 UTC
Well if one mouse type works and another doesn't then
it's likely to be some kernel compatibility problem.

I'm sure we have debugging support in the kernel which will
helkp us work out what's going on here.  Can someone please
guide Sander through it?

Who maintains the USB mouse stuff anyway?  Johannes Erdfelt?
He seems to not have a bugzilla account...
Comment 11 Alan Stern 2005-07-29 13:23:57 UTC
Here's something to try (I don't know if it will reveal anything useful, though).

Turn on USB verbose debugging in the kernel configuration (CONFIG_USB_DEBUG) and
rebuild the USB drivers.  Then reboot and mount a debugfs filesystem:

    mount -t debugfs none /sys/kernel/debug

Before you plug in the USB drive (while the mouse is still working), store a
copy of the "/sys/kernel/debug/uhci/0000:00:1d.0" file.  Then after you plug in
the drive (while the mouse isn't working), make another copy of that file.  Post
both copies, as well as a copy of the dmesg output going all the way back to the
boot-up, in this bug report.
Comment 12 Sander Bouwhuis 2005-07-29 14:28:11 UTC
hmm, I need a little more info here because it seems something is till missing.  
This is what I get when trying to mount debugfs after recompiling my kernel  
with USB_DEBUG and rebooting: 
 
23:17 0 /usr/src/linux # mount -t debugfs none /sys/kernel/debug 
mount: mount point /sys/kernel/debug does not exist 
 
So I try creating it: 
  
23:17 0 /home/sander # mkdir /sys/kernel/debug  
mkdir: cannot create directory `/sys/kernel/debug': Operation not permitted  
  
Then I tried making a mountpoint in /mnt, just to be sure:  
  
23:18 1 /home/sander # mkdir /mnt/debug  
23:18 0 /home/sander # mount -t debugfs none /mnt/debug  
mount: unknown filesystem type 'debugfs'  
  
And if it's any help, this is the contents of my /sys/kernel directory:  
  
23:22 0 /usr/src/linux # ls /sys/kernel -l  
total 0  
-r--r--r--  1 root root 4096 Jul 29 23:20 hotplug_seqnum  
  
I tried searching in menuconfig for debugfs but that got me nowhere. Is there  
anything else I should have included to make this work? 
Comment 13 Nishanth Aravamudan 2005-07-29 14:45:00 UTC
DEBUG_FS is enabled under Kernel Hacking --> Debug Filesystem (at least in
2.6.13-rc4). You may have been searching in menuconfig for debufs (which I did
the first time as well :) ).

Thanks, Nish
Comment 14 Sander Bouwhuis 2005-07-29 15:09:33 UTC
Created attachment 5413 [details]
/sys/kernel/debug/uhci/0000:00:1d.0 before the USB stick is inserted

Thanks, I found it. Here are the files.
Comment 15 Sander Bouwhuis 2005-07-29 15:10:34 UTC
Created attachment 5414 [details]
/sys/kernel/debug/uhci/0000:00:1d.0 after USB stick is inserted
Comment 16 Sander Bouwhuis 2005-07-29 15:12:23 UTC
Created attachment 5415 [details]
Complete dmesg

..and the last one. Hope it's useful!
Comment 17 Alan Stern 2005-07-29 19:14:39 UTC
I didn't see anything useful in the dmesg log, unfortunately.  But the UHCI
debugging files were helpful.  The first one contains these lines:

- skel_int8_qh
    [cecfb270] link (0ecfb152) element (0f1e6040)
      0: [cf1e6040] link (00000001) e3 LS IOC Active NAK Length=7ff MaxLen=3 DT1
EndPt=1 Dev=2, PID=69(IN) (buf=0c502000)

which are the entry for the interrupt URB used by the HID driver to get event
reports from the mouse.  Those lines were missing from the second file,
indicating that the URB completed (or was cancelled) but then was never
resubmitted (or failed in resubmission).

You may be able to get more information by using the usbmon facility.  It's one
of the drivers listed in the USB kernel configuration, and
Documentation/usb/usbmon.txt explains how to use it.

Beyond that I don't know what to suggest.  Maybe Vojtech will have some ideas.
Comment 18 Sander Bouwhuis 2005-08-05 04:30:23 UTC
Created attachment 5515 [details]
usbmon output of plugging in USB stick with mouse already inserted, then replugging the mouse
Comment 19 Alan Stern 2005-08-05 08:12:10 UTC
Here are the important lines from usbmon:

    ca7ab100 2376566963 S Ii:002:01 -115 4 D

The second field is a timestamp in microseconds.  So at 2376.566963 the HID
mouse driver submitted its event report interrupt URB.

    ca7ab100 2401970709 C Ii:002:01 -84 0

At 2401.970709, some 25 seconds later, the URB completed with -84 = -EILSEQ,
indicating a low-level error in the USB data transmission.  Apparently this
happened when you plugged in the stick, although I can't be certain because you
didn't include the usbmon log for the EHCI controller.  Anyway, whatever the
cause, this was enough to cause the HID driver to forget about the mouse.

    cecfdf80 2409908367 C Ii:001:01 0 1 D

At 2409.908367, some 8 seconds later still, the root hub status URB completed
because of the status change caused by you unplugging the mouse.  Then a second
later you plugged the mouse back in again.

My impression is that a hardware flaw in your USB controllers causes the UHCI
companion controller to receive an error when you plug in a new device.  The HID
driver should be able to cope with such errors by resubmitting its interrupt
URB, but it doesn't.

In fact, the code for the hid_irq_in, hid_irq_out, and hid_ctrl routines in
hid-core.c shows that the driver thinks -EILSEQ indicates a disconnection.  This
is wrong.  Yes, a disconnect can cause -EILSEQ errors, but so can other
problems.  When such an error occurs the driver should resubmit the URB, perhaps
after waiting a few tens of milliseconds.  The same is true for -EPROTO and
-ETIMEDOUT.

Vojtech, can you write a patch for the HID driver to fix this?
Comment 20 Alan Stern 2005-08-12 07:03:19 UTC
Created attachment 5616 [details]
Remove bogus tests from HID driver

This patch for 2.6.12 isn't the best one possible, but it ought to help solve
your problem.  Try it out and see what happens.
Comment 21 Sander Bouwhuis 2005-08-12 11:14:46 UTC
Nope, this patch doesn't seem to fix the problem :( 
Comment 22 Sander Bouwhuis 2005-08-12 11:20:38 UTC
Created attachment 5621 [details]
usbmon output of the same process with the provided patch applied

Is this of any help?
Comment 23 Alan Stern 2005-08-13 13:34:26 UTC
The log shows that whatever the problem was, it wasn't just a transient
occurrence.  It persisted from the time the mouse stopped working until you
replugged it.

I wonder if this isn't some sort of mechanical problem.  Like, maybe when you
plug in the memory device, it somehow disturbs the mouse plug connector just
enough to cause problems with signal transmission.  That could explain why some
brands of mouse work and others don't -- slight differences in the shapes of the
plug connectors.  What happens if you swap USB ports, or use different ports?

Also, I wonder whether it matters that the USB stick is running at high speed. 
What happens if you rmmod ehci-hcd before plugging in the stick?  (You may have
to rebuild your kernel with ehci-hcd as a removable module.)  Without it loaded,
the stick will be forced to run at full speed instead.
Comment 24 Sander Bouwhuis 2005-08-14 00:35:17 UTC
I only have two available USB slots on this laptop, and I've tried both  
combinations with the same effect. Furthermore, I doubt it's really a  
mechanical thing (or at least not an unsolvable one) because when I bought this  
machine it came preloaded with Windows XP Home which I used for about the first  
month and I don't remember having the problem there. I'm not 100% sure, but 
I've always used this USB stick a lot so I think I should have noticed if it 
had occurred. 
 
Removing ehci-hcd before plugging in the stick had no effect. 
Comment 25 Alan Stern 2005-08-14 07:03:28 UTC
It could even be some kind of electrical problem -- plugging in a second device
causes a slight voltage drop to the mouse, which then dies.  That also could
explain why some brands are affected but not others: differences in sensitivity.  

Ultimately the fix will have to be for the USB HID driver to reset the mouse.  I
assume that's what Windows does.  This will require a larger change than the
first patch, and I won't have time to work on it for a while.
Comment 26 Alan Stern 2005-08-21 08:15:33 UTC
Created attachment 5706 [details]
Manual device reset patch

You know, before changing the HID driver to make it reset the mouse, we should
check that a reset really will resuscitate the device.	This patch will give
you an easy way to reset the mouse by hand.  Once you've plugged in the stick
and the mouse has stopped working, as root do:

 echo reset >/sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/bConfigurationValue

(you better check that path to make sure it's correct).  The reset should show
up in the dmesg log, and you'll be able to tell if the mouse starts working
again.	If this does the job then it should be possible to add code to make the
HID driver do the equivalent thing automatically.
Comment 27 Sander Bouwhuis 2005-08-22 15:10:43 UTC
Created attachment 5731 [details]
dmesg output after going through the whole procedure

No, still nothing :( The mouse doesn't respond when I issue that command, but I
did have to change the path to the following:

echo reset > /sys/devices/pci0000:00/0000:00:1d.0/usb1/1-1/bConfigurationValue

Also, when I issued that command with just the mouse plugged in (while it was
working correctly) it stopped working too, so maybe the problem lies in that (I
doubt that should happen when resetting a device).

Attaching dmesg and usbmon again. And just for extra clarity: This is with the
first patch reversed and thus only the second patch applied.
Comment 28 Sander Bouwhuis 2005-08-22 15:12:24 UTC
Created attachment 5732 [details]
usbmon log of the whole procedure (inserting stick, issuing command, replugging)

this one may span about two minutes because I had to lookup the command again
Comment 29 Alan Stern 2005-08-23 10:09:49 UTC
Yes, the path will change depending on which port the mouse is plugged into.  I
had meant for you to use both patches in your testing, but since you say that
doing the reset while the mouse is working causes it stop working, there's no point.

Come to think of it, I should have realized the reset wouldn't work all by
itself.  When the mouse starts off, it initializes into Boot Protocol mode.  The
HID driver switches it over to Report Protocol mode and expects it to stay that
way.  But of course the reset switches it back, so it doesn't work any more.

This means that it will be necessary to make a full-blown change to the HID
driver.  Oh well.
Comment 30 Vojtech Pavlik 2005-08-23 10:55:30 UTC
Regarding boot protocol vs report protocol: On 99% of mice and keyboards
the switch doesn't do anything, because the devices use a HIDBP-compatible
layout of their reports.
Comment 31 Alan Stern 2005-08-23 11:39:26 UTC
Oh, I didn't realize that.  Then Sander, can you try running both tests over
again (that is, reset after the mouse stops working and reset while the mouse is
working) with both patches applied?
Comment 32 Sander Bouwhuis 2005-08-25 23:41:55 UTC
Same results. The mouse stops working on both occasions with both patches 
applied.. 
Comment 33 Alan Stern 2005-08-26 07:18:47 UTC
Please attach the two logs (dmesg and usbmon) for the experiment where you reset
the mouse while it is running.  For the moment let's not worry about the USB
stick.  After all, we've got to be able to reset the mouse before we can recover
from other problems.

By the way, have you ever tried plugging the mouse and stick into a different
computer to see if the same problem still occurs?
Comment 34 Sander Bouwhuis 2005-08-27 02:45:33 UTC
Created attachment 5779 [details]
usbmon

Yes, I've tried the same devices on another machine and got the same results,
however that machine was of the exact same type as this one so that's probably
not really useful. I don't have access to other Linux machines at the moment,
but when I run into a different one I'll try it.
Comment 35 Sander Bouwhuis 2005-08-27 02:46:28 UTC
Created attachment 5780 [details]
dmesg
Comment 36 Alan Stern 2005-08-29 09:30:06 UTC
Created attachment 5806 [details]
Add HID device reset routine

Here's a rather experimental patch to try.  You should remove the other two
patches before applying this one.  If it works, it will automatically reset the
mouse about a second after you plug in the storage device.

Vojtech, while working on this I noticed that in hid_irq_out and hid_ctrl the
Success case falls through to the -ESHUTDOWN case and so sets unplug to 1.  Is
it supposed to do that?
Comment 37 Vojtech Pavlik 2005-08-29 09:40:17 UTC
Thats definitely a bug, good catch!
Comment 38 Sander Bouwhuis 2005-08-29 11:35:59 UTC
Created attachment 5808 [details]
rejected patch

Applying this patch fails with the following message:

20:28 1 /usr/src/linux # patch -p1 <
/home/sander/misc/references/kernel-bug-4916/patch-3.txt
patching file drivers/usb/input/hid-core.c
Hunk #1 succeeded at 1000 (offset 92 lines).
Hunk #2 FAILED at 1033.
Hunk #3 FAILED at 1214.
Hunk #4 FAILED at 1260.
Hunk #5 FAILED at 1867.
Hunk #6 FAILED at 1895.
5 out of 6 hunks FAILED -- saving rejects to file
drivers/usb/input/hid-core.c.rej
patching file drivers/usb/input/hid.h
Comment 39 Alan Stern 2005-08-29 12:12:38 UTC
Created attachment 5809 [details]
Add HID device reset routine

Here's a version of the patch made against vanilla 2.6.12.  Make sure you have
removed all previous patches before trying to apply this one.
Comment 40 Andrew Morton 2005-09-14 23:18:57 UTC
Sander, did you get a chance to test Alan's patch?
Comment 41 Sander Bouwhuis 2005-09-17 08:54:50 UTC
I'm sorry, I've been away for a while and just got back. I need to catch up 
with stuff at home the next couple of days, but I'll try to test the patch this 
week. 
 
Sorry for the delay! 
Comment 42 Sander Bouwhuis 2005-12-21 09:08:32 UTC
shit, I totally forgot about this bug because I recently switched distros (from  
Gentoo to Kubuntu) and acquired another USB stick which didn't have these  
issues. I just tried and the old stick still has the problem in Kubuntu.  
However, I don't know how to test the patch(es) now anymore since Kubuntu just  
delivers a compiled kernel, and I'm just not that much into this stuff :)  
  
If you guys could give me any pointers on how to test this I'd be happy to try  
(promise) :)  
  
I'm really sorry about not responding anymore!  
Comment 43 Luis Sousa 2006-01-13 01:49:42 UTC
It seems that this problem is similar to the bugs 2303, 3028, 4305 and 4916, all
as NEW
Comment 44 Luis Sousa 2006-01-13 01:51:48 UTC
I just had the same problem on a machine using the pre-build kernel 2.6.8 from
Debian Sarge.
I just installed 2.6.11, from Debian Testing, hopping that it solves the problem.
Comment 45 Alan Stern 2006-01-14 09:30:28 UTC
Created attachment 7025 [details]
Add error recovery to the USB HID driver

Sander, try applying this patch to the 2.6.15 kernel source.  Even if you don't
have your distribution's source, you can always download 2.6.15 from
www.kernel.org.
Comment 46 Luis Sousa 2006-01-19 01:19:36 UTC
I just installed 2.6.12-1-686 (pre-build from Debian) and I have the 
same problem.
I have two machines with the same mouse and this only happens in one of 
them.
The machine with problems have a PII 300MHz, on a board ASUS P2L97.

Comment 47 Alan Stern 2006-01-19 06:52:29 UTC
Luis, try running the 2.6.15 kernel together with the patch in attachment 7025 [details].
Comment 48 Sander Bouwhuis 2006-01-19 07:43:42 UTC
http://bugzilla.kernel.org/attachment.cgi?id=7025&action=view works for me,  
after plugging in my USB disk the mouse pointer hangs for 1-2 seconds (which I  
assume is some kind of timeout) and then continues as normal. Thanks! 
Comment 49 Alan Stern 2006-01-25 13:44:05 UTC
Created attachment 7149 [details]
Add error recovery to the USB HID driver (updated)

This is a slightly updated version of the earlier patch.  Please try it and
make sure it works okay.  If it does, I'll submit it for inclusion in the
kernel.
Comment 50 Sander Bouwhuis 2006-01-25 14:53:13 UTC
Yes. Works the same. Slight delay, after 1-2 seconds it works perfectly. 
Comment 51 Luis Sousa 2006-01-30 08:56:14 UTC
Is it better to use this version instead of the previous one?
I patched the kernel with the previous one and apparently it's all fine :)

>This is a slightly updated version of the earlier patch.  Please try it and
>make sure it works okay.  If it does, I'll submit it for inclusion in the
>kernel.
>

Comment 52 Alan Stern 2006-01-30 09:08:57 UTC
Yes, the patch in attachment #7149 [details] is slightly better than the earlier one.  The
original version contained a small mistake.
Comment 53 Greg Kroah-Hartman 2006-02-14 17:32:56 UTC
Is Alan's patch required to get this device working on 2.6.16-rc3?

If so, Dmitry, do you feel it is solid enough to send this late in the 2.6.16-rc
series, or should I wait until 2.6.16 is out?
Comment 54 Dmitry Torokhov 2006-02-14 19:56:21 UTC
The patch looks very reasonable but it is pretty big and has a potential of 
disrupting HID and thus affecting a lot of people. This is not a recent 
regression and number of affected boxes seems to be limited so I'd feel more 
safe if we could wait for 2.6.17 to open. 
Comment 55 Sander Bouwhuis 2006-04-24 00:25:22 UTC
So, is this one going in any time soon?
Comment 56 Alan Stern 2006-04-24 06:55:41 UTC
It's already in 2.6.17-rc1.
Comment 57 Adrian Bunk 2006-11-12 12:22:42 UTC
The patch from this bug was included in 2.6.17.