Bug 15580

Created attachment 25610 [details]
lspci -vvv

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Fri, 19 Mar 2010 11:40:10 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=15580
> 
>            Summary: ehci_hcd has to be reloaded after disconnect/connect
>                     cycle of USB audio card
>            Product: Drivers
>            Version: 2.5
>     Kernel Version: 2.6.31 to 2.6.33
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: USB
>         AssignedTo: greg@kroah.com
>         ReportedBy: eusou15@yahoo.com
>         Regression: No
> 
> 
> Created an attachment (id=25609)
>  --> (http://bugzilla.kernel.org/attachment.cgi?id=25609)
> lsusb -vvv
> 
> Hi,
> 
> I have a WinTV NOVA-T DVB-T USB2 adapter (dvb_usb_dib0700) and a Native
> Instruments Audio4DJ USB sound card (snd-usb-caiaq).
> 
> Both devices work wonderfully when I come from a fresh boot (I can
> watch tv with the audio routed through the sound card). However, once
> I disconnected the audio card and reconnect again, I get this regular
> but not periodic interference on the audio card.
> By "regular and not periodic" I mean that the time elapsed between
> each interference is random, from 0.5 to 30 seconds, but it keeps
> happening. The interference is a kind of crack and pop which affects
> all output channels of the audio card.
> 
> >From now on (until I fresh boot again), I can reconnect the audio card
> as many times as I want, rmmod and modprobe its module again and I
> always get the interference. Note that unplugging and plugging the DVB
> adapter makes no difference, and if I fresh boot, connect both
> devices, disconnect the DVB card and reconnect, I do not get
> interference - only when I do the same with the audio card.
> 
> After much research I found the culprit - apparently if I rmmod and
> modprobe the ehci_usb device the interference is gone without having
> to fresh boot (of course before this I unload the modules for the dvb
> and audio devices).
> 
> I was to able to reproduce this problem in kernels 2.6.31 to 2.6.33 (I
> haven't tried any others). I WAS NOT able to reproduce on a ICH4
> laptop I have.
> 
> In conclusion:
> 
> Procedure 1:
> - fresh boot
> - connect both devices
> - everything works OK
> 
> Procedure 2:
> - (same as Procedure 1)
> - disconnect audio card and reconnect
> - interference starts until I reload all relevant modules and ehci_hcd
> 
> Procedure 3:
> - (same as Procedure 1)
> - disconnect DVB-T card and reconnect
> - everything works OK
> 
> My laptop is a Lenovo T400.
> 
> Attached are the lsusb and lspci -vvv. Please let me know if you need
> any more information.
>

Created attachment 25655 [details]
usbmon logs - see comment

This shows one usbmon log for normal operation (normal), other with interference (interference) and possibly a more useful one which shows how the interference starts in the audio card after I connect the DVB adapter (interference2).
All of them are taken on the audio card bus.

Sorry for the size of the logs, but I had to capture more than one event and they are pretty sparse (10 to 20 seconds between them).

Reply-To: pedrib@gmail.com

This is messing with my mind. I've been desperately trying to find a
pattern here. Now it appears the interference only happens if I unplug
and replug the DVB-T adapter.

Anyway, I have created an attachment to the original bug report with
usbmon logs from the audio card.
One of them shows normal operation, the other shows the interference
I'm talking about.
A third one (called interference2) is probably more useful - it shows
how the interference starts when I disconnect and reconnect the DVB
adapter.

To reply to Alan's question, sometimes when I unload and reload the
module the interference starts, sometimes it doesn't. Like I said,
seems completely random!

http://bugzilla.kernel.org/show_bug.cgi?id=15580

On Thu, 18 Mar 2010, Pedro Ribeiro wrote:

> After much research I found the culprit - apparently if I rmmod and
> modprobe the ehci_usb device the interference is gone without having
> to fresh boot (of course before this I unload the modules for the dvb
> and audio devices).
>
> I was to able to reproduce this problem in kernels 2.6.31 to 2.6.33 (I
> haven't tried any others). I WAS NOT able to reproduce on a ICH4
> laptop I have.
>
> The DVB-T adapter uses the dvb_usb_dib0700 module and the audio card
> uses the snd-usb-caiaq module.

It's hard to say what could be causing the problem here.  The best
approach I can think of is to collect a pair of usbmon traces: one
while the problem occurs and one while everything works correctly.  
(See the kernel source file Documentation/usb/usbmon.txt for
instructions.)  Attach the two traces to the bug report.  They don't 
have to be very long; just enough to cover some occurrences of the 
sound interference.

Also, you should attach a dmesg log (with CONFIG_USB_DEBUG enabled)  
showing what happens when you unload and reload ehci-hcd, and what
happens when you unplug and replug the audio card.

Does the same problem occur if you unload and reload the audio driver 
instead of unplugging and replugging the audio card?

Alan Stern

On Tue, 23 Mar 2010, Pedro Ribeiro wrote:

> This is messing with my mind. I've been desperately trying to find a
> pattern here. Now it appears the interference only happens if I unplug
> and replug the DVB-T adapter.
> 
> Anyway, I have created an attachment to the original bug report with
> usbmon logs from the audio card.
> One of them shows normal operation, the other shows the interference
> I'm talking about.
> A third one (called interference2) is probably more useful - it shows
> how the interference starts when I disconnect and reconnect the DVB
> adapter.

I didn't see anything especially unusual in the usbmon traces, except
that a little under half of the packets have length 0.  But that was
true in the normal trace as well.

(One little bug in the sound card did show up, but it isn't related to 
your problem.  The string-0 descriptor is invalid -- it contains an 
extra byte.)

Unfortunately the data sets are too large to examine very closely.

> To reply to Alan's question, sometimes when I unload and reload the
> module the interference starts, sometimes it doesn't. Like I said,
> seems completely random!

That's different from what you said originally -- you said that the
interference would start only when you unplugged and replugged the
audio card (and now when you unplug and replug the DVB-T adapter).  
But unloading and reloading the driver is a software event, not a
hardware event, so the fact that it also can cause the interference to
start may be significant.  Although I don't know just what it means...

Is it still true that if you don't unplug either card or unload the 
drivers then the interference never starts?

Were you able to collect the dmesg log with CONFIG_USB_DEBUG enabled?

Alan Stern

Reply-To: pedrib@gmail.com

On 23 March 2010 19:32, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Tue, 23 Mar 2010, Pedro Ribeiro wrote:
>
>> This is messing with my mind. I've been desperately trying to find a
>> pattern here. Now it appears the interference only happens if I unplug
>> and replug the DVB-T adapter.
>>
>> Anyway, I have created an attachment to the original bug report with
>> usbmon logs from the audio card.
>> One of them shows normal operation, the other shows the interference
>> I'm talking about.
>> A third one (called interference2) is probably more useful - it shows
>> how the interference starts when I disconnect and reconnect the DVB
>> adapter.
>
> I didn't see anything especially unusual in the usbmon traces, except
> that a little under half of the packets have length 0.  But that was
> true in the normal trace as well.
>
> (One little bug in the sound card did show up, but it isn't related to
> your problem.  The string-0 descriptor is invalid -- it contains an
> extra byte.)
>
> Unfortunately the data sets are too large to examine very closely.
>
>> To reply to Alan's question, sometimes when I unload and reload the
>> module the interference starts, sometimes it doesn't. Like I said,
>> seems completely random!
>
> That's different from what you said originally -- you said that the
> interference would start only when you unplugged and replugged the
> audio card (and now when you unplug and replug the DVB-T adapter).
> But unloading and reloading the driver is a software event, not a
> hardware event, so the fact that it also can cause the interference to
> start may be significant.  Although I don't know just what it means...
>
> Is it still true that if you don't unplug either card or unload the
> drivers then the interference never starts?
>
> Were you able to collect the dmesg log with CONFIG_USB_DEBUG enabled?
>
> Alan Stern
>
>

I have some really interesting developments. I think I narrowed it down.

So I ALWAYS get the interference when I:
- load the snd-usb-caiaq module
- load the dvb_usb_dib0700 module

In this order. If its in the opposite order, nothing happens.

So now I'm 100% sure this happens when loading and unloading modules,
which like you said means its something in software, much to my relief
(means its solvable). But now I have no idea where to look. usbmon
logs show nothing and I'm attaching dmesg - but I don't think there is
any interesting information there.

One thing I forgot to say is probably very important is that the audio
card only produces the interference when it is being used. So this
means that when it is sitting idle (no usbmon output) there is no
interference. However, as soon as I start using it (open it with ALSA,
or starting a JACK audio server) the cracks and pops start. When I
stop the application using it, the cracks and pops stop - and the
usbmon output too.

I've tried capturing a usbmon log on the DVB adapter part, but no
output is produced when the interference starts. Note that I do not
need to be using the DVB adapter (streaming from it) in order for the
interference to appear. But as soon as I unload the dvb_usb_dib0700
module, it stops the cracks and pops in the audio.

Created attachment 25665 [details]
dmesg with CONFIG_USB_DEBUG

Reply-To: pedrib@gmail.com

Just to clear things up (hopefully) the behaviour described above is
only for the first time, when I boot the system. Once I load the
modules in the order specified above, I always get the interference
even if I unload them all and reload them NOT in that order - what I
mean is once the damage is done, its irreversible - I can't use both
at the same time.

I'm really sorry for the mess that this bug report is - not only it is
driving me nuts, but also really hard to pin down in which
circumstances things happen.

On Tue, 23 Mar 2010, Pedro Ribeiro wrote:

> Just to clear things up (hopefully) the behaviour described above is
> only for the first time, when I boot the system. Once I load the
> modules in the order specified above, I always get the interference
> even if I unload them all and reload them NOT in that order - what I
> mean is once the damage is done, its irreversible - I can't use both
> at the same time.

This is very puzzling, and I don't see any reason for the behavior
you're describing.  The only thing I can think of is that somehow the
audio card gets into a bad state which persists until it is reset.

You can test this by using the attached usbreset program.  Start by 
triggering the interference, and then stop using both cards (don't 
stream any TV and don't stream any audio).  In fact, to be safe, you 
should unload both drivers.

While the cards are idle, use the program to reset the audio card
(you'll have to run it as root).  You just tell it the name of the USB
device to reset.  For example, according to your dmesg log the audio
card was device 3 on bus 1, so you would do:

	sudo ./usbreset /dev/bus/usb/001/003

Then reload the audio driver, try streaming some normal ALSA audio,
and see if the interference has gone away.

Alan Stern

Reply-To: pedrib@gmail.com

Actually after some quick testing, I can see it does not work :(

The only solution till now is still to unload and reload ehci_hcd. I
don't even need to reload the drivers for this!

Pedro

Reply-To: pedrib@gmail.com

>
> This is very puzzling, and I don't see any reason for the behavior
> you're describing.  The only thing I can think of is that somehow the
> audio card gets into a bad state which persists until it is reset.
>
> You can test this by using the attached usbreset program.  Start by
> triggering the interference, and then stop using both cards (don't
> stream any TV and don't stream any audio).  In fact, to be safe, you
> should unload both drivers.
>
> While the cards are idle, use the program to reset the audio card
> (you'll have to run it as root).  You just tell it the name of the USB
> device to reset.  For example, according to your dmesg log the audio
> card was device 3 on bus 1, so you would do:
>
>        sudo ./usbreset /dev/bus/usb/001/003
>
> Then reload the audio driver, try streaming some normal ALSA audio,
> and see if the interference has gone away.
>
> Alan Stern
>

This usbreset seems to work, even without reloading the audio driver.
I have to do some further testing (read - day to day use coupled with
stress testing) to be 100% sure. Unfortunately I'm going away for a
couple of weeks now and I can't take my devices with me.

I'll get back to you ASAP after further testing.

Your help is much appreciated, and I thank you very much.

Pedro

Please take notice that the two comments above are swapped (something is wrong with my mail system).

The usbreset DOES NOT work and the only way to is to reload ehci_hcd and uhci_hcd.

Reply-To: pedrib@gmail.com

On 24 March 2010 15:37, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Tue, 23 Mar 2010, Pedro Ribeiro wrote:
>
>> Just to clear things up (hopefully) the behaviour described above is
>> only for the first time, when I boot the system. Once I load the
>> modules in the order specified above, I always get the interference
>> even if I unload them all and reload them NOT in that order - what I
>> mean is once the damage is done, its irreversible - I can't use both
>> at the same time.
>
> This is very puzzling, and I don't see any reason for the behavior
> you're describing.  The only thing I can think of is that somehow the
> audio card gets into a bad state which persists until it is reset.
>
> You can test this by using the attached usbreset program.  Start by
> triggering the interference, and then stop using both cards (don't
> stream any TV and don't stream any audio).  In fact, to be safe, you
> should unload both drivers.
>
> While the cards are idle, use the program to reset the audio card
> (you'll have to run it as root).  You just tell it the name of the USB
> device to reset.  For example, according to your dmesg log the audio
> card was device 3 on bus 1, so you would do:
>
>        sudo ./usbreset /dev/bus/usb/001/003
>
> Then reload the audio driver, try streaming some normal ALSA audio,
> and see if the interference has gone away.
>
> Alan Stern
>

Hi Alan,

It is with great pleasure to announce that I have made some progress.

One thing I'm 100% sure now is that this bug ONLY occurs with 64 bit
kernel and userland. With a i686 kernel/userland, there is no
interference. This is always reproducible.

I downloaded Ubuntu 10.04 Beta 1 live CD's for i386 and amd64 and
decided to try them out. With the amd64 one I get the interference,
with the i386 I do not!

I tested with another platform (a friend's laptop) and I can reproduce
the results: with amd64 I get the interference, with i386 nothing. His
chipset is also ICH9. Unfortunately I don't have any other platform
where I can test it. Please let me know if you need the lspci and
lsusb for his platform.

So does this ring a bell? I had no idea there was a difference in USB
handling in the kernel between 32 and 64 bit!

Regards,
Pedro

http://bugzilla.kernel.org/show_bug.cgi?id=15580

On Tue, 6 Apr 2010, Pedro Ribeiro wrote:

> So does this ring a bell? I had no idea there was a difference in USB
> handling in the kernel between 32 and 64 bit!

As far as I know, there is no difference.  Certainly there's not 
_supposed_ to be any difference!

What happens if you run a 64-bit kernel with a 32-bit userspace?

Alan Stern

Reply-To: pedrib@gmail.com

On 6 April 2010 15:45, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Tue, 6 Apr 2010, Pedro Ribeiro wrote:
>
>> So does this ring a bell? I had no idea there was a difference in USB
>> handling in the kernel between 32 and 64 bit!
>
> As far as I know, there is no difference.  Certainly there's not
> _supposed_ to be any difference!
>
> What happens if you run a 64-bit kernel with a 32-bit userspace?
>
> Alan Stern
>
>

Unfortunately I don't have a spare hard disk or disk space so I can't
install a 64 bit kernel with 32 bit apps - do you know of any liveCD
which has this configuration?

I made some progress today. I talked to the developer of the DVB-T
card driver and he made me discover that this also happens with a USB
pendrive when it gets mounted or unmounted. So this is probably a
snd-usb-caiaq bug.
But to make matters worse, now I'm not even sure if unloading ehci_hcd
solves the problem - I had to reboot yesterday because I could not
make the interference go away with this method.

As you can see from the messages above in this thread this is becoming
very confusing. So I'm going to summarize where we stand.

So where is what I know FOR SURE:

- this happens on two different ICH9 platforms, not only on my laptop
- only with 64 bit kernel / userland
- not reproducible on 32 bit kernel / userland
- the interference can be caused by both the DVB card when it is
operating or a pendrive when it gets mounted or unmounted (but not
when reading or writing to the partition in the pendrive)
- upon fresh boot, there is no interference. I unplug and replug the
audio card and the interference starts


And where is what I SUSPECT BUT I'M NOT SURE:
- unloading and reloading the snd-usb-caiaq module triggers the
interference without unplugging and replugging
- unloading and reloading ehci_hcd solves the issue till the next
unplug and replug of the audio card

Any of the assumptions in previous posts can and should be ignored.
Once again, sorry for the mess, but you can see how hard and long this
is taking to debug.

Regards,
Pedro

Reply-To: pedrib@gmail.com

On 6 April 2010 23:05, Pedro Ribeiro <pedrib@gmail.com> wrote:
> On 6 April 2010 15:45, Alan Stern <stern@rowland.harvard.edu> wrote:
>> On Tue, 6 Apr 2010, Pedro Ribeiro wrote:
>>
>>> So does this ring a bell? I had no idea there was a difference in USB
>>> handling in the kernel between 32 and 64 bit!
>>
>> As far as I know, there is no difference.  Certainly there's not
>> _supposed_ to be any difference!
>>
>> What happens if you run a 64-bit kernel with a 32-bit userspace?
>>
>> Alan Stern
>>
>>
>
> Unfortunately I don't have a spare hard disk or disk space so I can't
> install a 64 bit kernel with 32 bit apps - do you know of any liveCD
> which has this configuration?
>
> I made some progress today. I talked to the developer of the DVB-T
> card driver and he made me discover that this also happens with a USB
> pendrive when it gets mounted or unmounted. So this is probably a
> snd-usb-caiaq bug.
> But to make matters worse, now I'm not even sure if unloading ehci_hcd
> solves the problem - I had to reboot yesterday because I could not
> make the interference go away with this method.
>
> As you can see from the messages above in this thread this is becoming
> very confusing. So I'm going to summarize where we stand.
>
> So where is what I know FOR SURE:
>
> - this happens on two different ICH9 platforms, not only on my laptop
> - only with 64 bit kernel / userland
> - not reproducible on 32 bit kernel / userland
> - the interference can be caused by both the DVB card when it is
> operating or a pendrive when it gets mounted or unmounted (but not
> when reading or writing to the partition in the pendrive)
> - upon fresh boot, there is no interference. I unplug and replug the
> audio card and the interference starts
>
>
> And where is what I SUSPECT BUT I'M NOT SURE:
> - unloading and reloading the snd-usb-caiaq module triggers the
> interference without unplugging and replugging
> - unloading and reloading ehci_hcd solves the issue till the next
> unplug and replug of the audio card
>
> Any of the assumptions in previous posts can and should be ignored.
> Once again, sorry for the mess, but you can see how hard and long this
> is taking to debug.
>
> Regards,
> Pedro
>

Good news: I've talked to the snd-usb-caiaq developer and he gave me a
patch which appears to solve the problem. I'm going to test it for a
couple of days just to make sure before I OK him.

Pedro

I'm guessing that was the following commit which was merged for v2.6.34-rc7:

commit 073900a28d95c75a706bf40ebf092ea048c7b236
Author: Daniel Mack <daniel@caiaq.de>
Date:   Mon Apr 12 13:17:25 2010 +0200

    USB: rename usb_buffer_alloc() and usb_buffer_free()

Hi,

all that commit did was to rename the functions. 

I've just verified that with 2.6.38 the bug is still present. Therefore I am re-opening the bug.

Thanks for your patience.

Pedro

That indeed seems fishy :| ... I really don't know how I came up with that commit... sorry. Maybe I found a commit but fat-fingered the copy-n-paste... But then again, it wouldn't still be present in 2.6.38... 

Thanks for testing with 2.6.38!

Can you post the patch you mentioned in comment #17 ?

Hi Florian,

sure, I'm attaching the patch. It has to be applied with -l.

Regards,
Pedro

Created attachment 51202 [details]
The patch by Daniel Mack that solves the problem

Apply this patch with -l

Who takes care of merging that patch? Has Daniel submitted that patch somewhere?

Hi Florian,

I'm very sorry for the late reply, I will answer more promptly onwards

This problem is still present in 2.6.38.3. Nobody has taken care of the patch and Daniel hasn't submitted because there was resistance by the USB maintainers to submit the patch. You can follow the discussion here: http://lkml.org/lkml/2010/4/7/87

I understand their resistance to including the patch since I'm the only person that has reported it. However, you should take into consideration that this is a real problem and I've tested with another laptop which showed the same behaviour.

One thing to take into consideration is that probably not many people use Linux with this sound card so the problem doesn't get reported. Once again, I can say that the patch solves this problem 100%.

Regards,
Pedro

Well, it just stuck at the point to implement kmalloc-variant of usb buffer allocation API.  The rename of usb_alloc_buffer() / usb_free_buffer() was already done.

Or maybe an easier option would be to implement an API giving the gfp flag for the corresponding controller to pass to kmalloc, like the patch below.  Pedro could you check whether it works?

Created attachment 59102 [details]
Patch to add usb_get_gfp() including the fix for caiaq

Hi Takashi,

thanks for your efforts. I built and installed 2.6.38.6 with your patch. 

However, when the module is loaded upon connecting the device I get a fatal error. I've captured the stack trace using KDB (sorry for the bad photo quality) and I'm attaching it to this thread.

Regards,
Pedro

Created attachment 59632 [details]
Picture with the stack trace when caiaq module is loaded with Takashi's patch

Takashi, do you plan to continue working on this or do you want me to have a look? It seems like kmalloc with usb_get_gfp() returns a different type of memory than usb_alloc_coherent() does, but taking a quick look, I can't tell why.

I'd like to get this issue solved, as there are more people reporting this issue, and they all say that manually applying the patch from http://lkml.org/lkml/2010/5/7/61 fixes their trouble.


Thanks,
Daniel

Patch: https://lkml.org/lkml/2011/8/10/52