Hi, for several months and still with the latest Ubuntu version I have severe trouble with my Toshiba Notebook (and found in forums that others have the same problem): When writing to USB-connected flash memory (USB-Stick/pendrive or camera flash cards like compact flash oder SD connected with a USB card reader), the process of writing gets quickly extremely slow and finally blocks completely. I see the problem on another notebook with intel chipset as well, but not with my computers having ATI controllers. Most often, there is no error message at all in the logs or dmesg. But sometimes there's a backtrace. See my bug report for Ubuntu under https://bugs.launchpad.net/ubuntu/+source/linux/+bug/493199 to find a detailed system and hardware description and one of theses backtraces. I guess that there is some timing or interrupt problem which finally results in a deadlock. regards
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Wed, 9 Dec 2009 23:04:21 GMT bugzilla-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=14778 > > Summary: USB (intel chipset) with external mass storage > extremely slow or blocking > Product: IO/Storage > Version: 2.5 > Kernel Version: 2.6.31-16-generic (Ubuntu 9.10) > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: high > Priority: P1 > Component: Other > AssignedTo: io_other@kernel-bugs.osdl.org > ReportedBy: hadmut@danisch.de > Regression: No > > > Hi, > > for several months and still with the latest Ubuntu version I have severe > trouble with my Toshiba Notebook (and found in forums that others have the > same > problem): > > When writing to USB-connected flash memory (USB-Stick/pendrive or camera > flash > cards like compact flash oder SD connected with a USB card reader), the > process > of writing gets quickly extremely slow and finally blocks completely. I see > the > problem on another notebook with intel chipset as well, but not with my > computers having ATI controllers. > > Most often, there is no error message at all in the logs or dmesg. But > sometimes there's a backtrace. > > See my bug report for Ubuntu under > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/493199 > > to find a detailed system and hardware description and one of theses > backtraces. > > > I guess that there is some timing or interrupt problem which finally results > in > a deadlock. > I'll reassign this to USB. The launchpad.net bug report is horridly wordwrapped. It looks like requests got lost - perhaps IO completion interrupts didn't happen, or didn't signal completion.
> > http://bugzilla.kernel.org/show_bug.cgi?id=14778 > > > > Summary: USB (intel chipset) with external mass storage > > extremely slow or blocking > > Hi, > > > > for several months and still with the latest Ubuntu version I have severe > > trouble with my Toshiba Notebook (and found in forums that others have the > same > > problem): > > > > When writing to USB-connected flash memory (USB-Stick/pendrive or camera > flash > > cards like compact flash oder SD connected with a USB card reader), the > process > > of writing gets quickly extremely slow and finally blocks completely. I see > the > > problem on another notebook with intel chipset as well, but not with my > > computers having ATI controllers. > > > > Most often, there is no error message at all in the logs or dmesg. But > > sometimes there's a backtrace. Get a usbmon trace showing what happens starting from before the device is plugged in (instructions are in the kernel source file Documentation/usb/usbmon.txt). Attach the output to this bug report and let us know when it's ready. It'll give us a place to start from. Alan Stern
Hi, Alan Stern wrote: > Get a usbmon trace showing what happens starting from before the device > is plugged in (instructions are in the kernel source file > Documentation/usb/usbmon.txt). Attach the output to this bug report > and let us know when it's ready. It'll give us a place to start from. > (Andrew asked me to reply be e-mail to all, not through the web interface. ) I just produced the usbmon trace, (250k compressed) and put it to http://www.danisch.de/tmp/mon_0u_out.bz2 (no other usb device had been in use while doing the test) I tried to copy a 623 MByte file onto a 2GB SD flash card with a dos file system. As always, the first 130 MByte were copied extremely fast (maybe that went just into the fs/block cache and had not yet been written to the device. Then the du or ls-lF show the file staying at 130 mb for a while, then it very slowly reaches 137 and 141, and then it is as good as frozen. When repeating this several times (with fresh reboots), the problem always occured around 130 MByte. I guess that this is what goes into the writeout buffers fast, while the problems seem to occur very early. With several tests I had the problem that the card could not be unmounted. After rebooting and pulling the card, the file, even the file name entry, were gone, or actually never written to the flash card. Some time ago with other tasks I could not even write small files onto USB pendrives. regards Hadmut
On Thu, 10 Dec 2009, Hadmut Danisch wrote: > I just produced the usbmon trace, (250k compressed) and put it to > > http://www.danisch.de/tmp/mon_0u_out.bz2 The advantage of attaching the trace to the bug report is that it will still be available for people to look at long after you have removed it from your server. > (no other usb device had been in use while doing the test) That's not correct. The trace shows that a device attached to port 2 (the card reader was on port 3) was quite active while your test was running. > I tried to copy a 623 MByte file onto a 2GB SD flash card with a dos > file system. > > As always, the first 130 MByte were copied extremely fast (maybe that > went just into the fs/block cache and had not yet been written > to the device. Then the du or ls-lF show the file staying at 130 mb for > a while, then it very slowly reaches 137 and 141, and then it is as good > as frozen. The trace shows two things. First, some program is probing your flash card much more frequently than it should, about once every 30 ms. Normally hal probes once every 2 seconds, so I don't know what's causing all these extra probes. But luckily they don't interfere with the data transfer very much. The real problem appears to be a bug in your hardware. Every five seconds or so the reader appears to crash and has to be reset. This takes around three seconds, during which time no data can be transferred. During the periods when it was working, I estimate the reader was handling about 3.5 MB/s. The trace does not show the transfer stopping completely. It was still going strong at the end of the trace. Maybe the bug is in the USB hardware on your computer instead of the card reader. I can't tell which. Alan Stern
Alan Stern wrote: > The advantage of attaching the trace to the bug report is that it will > still be available for people to look at long after you have removed it > from your server. > If 250k is not a waste of storage, I'll upload. > >> (no other usb device had been in use while doing the test) >> > > That's not correct. The trace shows that a device attached to port 2 > (the card reader was on port 3) was quite active while your test was > running. > There is a builtin WLAN adapter that is connected through USB (and unfortunately the kernel driver completely ignores the physical WLAN switch, so it is always turned on) but I assumed that it will not be active if I tell the Network Manager (lousy piece of software) to stop the WLAN. sorry for that. > > The trace does not show the transfer stopping completely. It was still > going strong at the end of the trace. > > Maybe the bug is in the USB hardware on your computer instead of the > card reader. I can't tell which. > > Thanks for the hint. Well, the reader worked perfectly on other machines, and the notebook has trouble with several devices like USB pendrivers, so if there's a damage, it must be the computer. But I have two notebooks with intel chipsets and both have similar problems, so I would be astonished if both of them were broken. And if I remember correctly, I could fix the problem by stepping back to an older kernel when the problem occured for the first time, but that's long time ago. :-( regards Hadmut
Created attachment 24148 [details] compressed usbmon output
On Thu, 10 Dec 2009, Hadmut Danisch wrote: > Well, the reader worked perfectly on other machines, and the notebook > has trouble with several devices like USB pendrivers, so if there's a > damage, it must be the computer. > > But I have two notebooks with intel chipsets and both have similar > problems, so I would be astonished if both of them were broken. > And if I remember correctly, I could fix the problem by stepping back to > an older kernel when the problem occured for the first time, but that's > long time ago. :-( Take a look at bug #14785; it seems to be a duplicate of this one. In each case the transfer fails during a CSW. If you can find an earlier kernel that didn't have this problem, please let us know. Better yet, find the first version where the problem manifests. Alan Stern
It's unlikely that this bug describes the same chipset as in bug #14785. The p55 express chipset is a fairly new one for Intels LGA1186 bus. And until now I haven't discovered any failures when using only one usb-storge device with the p55-chipset (and no slow operations).
Hi, Alan Stern wrote: > If you can find an earlier kernel that didn't have this problem, please > let us know. Better yet, find the first version where the problem > manifests. > Unfortunately I couldn't reproduce it, because it was working with an Ubuntu 8.04 release, where the regular kernel did not boot from the disk. I had replaced that with a self compiled kernel at that time, but didn't keep the kernel conf. But the more I test, the more I am confused. I have tested six (!) different multi-card readers (those cheap x-in-1 readers that have CF, SD, microSD, MMS, ... slots and usually occupy three or four /dev/sd* devices), and they all cause trouble as described (one even reports write-enabled SD-cards as write-protected). On the other hand, I have tested three card readers made for one particular card type only (SanDisk MicroMate for SD, a no-name SD-Expresscard-USB-Reader and a no-name CF-Expresscard-USB-Reader) that worked well, but only if none of the multi-card readers had been connected since last reboot (or at least that's what it appeared as). Maybe there's some problem with these multicard readers that brings something out of order. regards Hadmut
Created attachment 24164 [details] Problems of a distinct machine with an ATI chipset and a 16GB USB pendrive These are (similar) problems seen on a completely disctinct machine with different chipset (Ubuntu 9.10 as well) and an (encrypted) 16GB USB Stick.
Hi, the more experiments I do, the more problems I see. I found similar problems on two completely distinct machines (both with Ubuntu 9.10 and 2.6.31 kernel) with an (encrypted with cryptsetup) 16GB USB pendrive. Both problems seem to be similar so it would prove that it is not caused by hardware problems. I've just attached a dmesg output to the bug report page. See also ubuntu bug 183839 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/183839, which describes exactly that problem, but claims to be fixed. I just tried to compile a 2.6.32 plain kernel from kernel.org with ubuntu's config file, but it could not boot and complained that the ATA devices were not ready, and I am too busy right now to spend the whole weekend doing tests. But somewhere (maybe in a patch applied to the kernel by the ubuntu team) is something wrong with external usb mass storage. regards Hadmut
On Sat, 12 Dec 2009, Hadmut Danisch wrote: > I found similar problems on two completely distinct machines (both with > Ubuntu 9.10 and 2.6.31 kernel) with an (encrypted with cryptsetup) 16GB > USB pendrive. Both problems seem to be similar so it would prove that it > is not caused by hardware problems. I've just attached a dmesg output to > the bug report page. The dmesg output doesn't contain enough details to see what's going on. A usbmon trace is needed. Alan Stern
Created attachment 24185 [details] compressed usbmon output Now this is the usbmon output for the problem with the 16GB USB pendrive
Alan Stern wrote: > On Sat, 12 Dec 2009, Hadmut Danisch wrote: > > >> I found similar problems on two completely distinct machines (both with >> Ubuntu 9.10 and 2.6.31 kernel) with an (encrypted with cryptsetup) 16GB >> USB pendrive. Both problems seem to be similar so it would prove that it >> is not caused by hardware problems. I've just attached a dmesg output to >> the bug report page. >> > > The dmesg output doesn't contain enough details to see what's going on. > A usbmon trace is needed. > I just uploaded the usbmon trace to the bug report. (machine uses usb keyboard and mouse, so the bus is never quiet) I started mkfs.ext3 /dev/sdg and the mkfs programm froze while writing Inode-table 60/121. But it seems as if nothing had actually been written to the pendrive. This was completely distinct hardware (different computers, different flash memory). Should I really have four computers all with broken USB ports? regards Hadmut
On Mon, 14 Dec 2009, Hadmut Danisch wrote: > I just uploaded the usbmon trace to the bug report. (machine uses usb > keyboard and mouse, so the bus is never quiet) Actually there's something like a card reader which continually gets probed. It would be nice if you could stop that activity. > I started mkfs.ext3 /dev/sdg and the mkfs programm froze while writing > Inode-table 60/121. > But it seems as if nothing had actually been written to the pendrive. Almost nothing -- 4096 bytes did get written. But no more. > This was completely distinct hardware (different computers, different > flash memory). Should I really have four computers all with broken USB > ports? Or else different broken flash drives? I don't know. All I can say is that the usbmon trace shows that the drive you were using stopped working as soon as the computer tried to do a large write. Maybe the drive just can't handle transfers that are too large. You can reduce the transfer size by setting the drive's max_sectors value (the file will be /sys/block/sdg/device/max_sectors, if the drive is /dev/sdg). Trying writing 64 to that file before using the drive. Alan Stern
Created attachment 24190 [details] usb mon output of pen drive problem
Alan Stern wrote: > > > Actually there's something like a card reader which continually gets > probed. It would be nice if you could stop that activity. > That was a builtin multicard reader, that I did not use, but was still connected. I had to open the case and pull the usb cable, and replace the usbmon trace with a new one. Problem continues after reboot. > Or else different broken flash drives? I don't know. Meanwhile I see the problem with four different computers, four different card readers, a heap of flash cards, and usb pendrives. I don't believe they are all broken. > All I can say is > that the usbmon trace shows that the drive you were using stopped > working as soon as the computer tried to do a large write. > The pendrive works perfectly on my digital video recorder, and the flash cards work on my cameras. I can't believe that all my devices are broken. > Maybe the drive just can't handle transfers that are too large. You > can reduce the transfer size by setting the drive's max_sectors value > (the file will be /sys/block/sdg/device/max_sectors, if the drive is > /dev/sdg). Trying writing 64 to that file before using the drive. > Even 32 doesn't help. regards Hadmut
On Tue, 15 Dec 2009, Hadmut Danisch wrote: > > All I can say is > > that the usbmon trace shows that the drive you were using stopped > > working as soon as the computer tried to do a large write. > > > The pendrive works perfectly on my digital video recorder, and the flash > cards work on my cameras. I can't believe that all my devices are broken. > > > > Maybe the drive just can't handle transfers that are too large. You > > can reduce the transfer size by setting the drive's max_sectors value > > (the file will be /sys/block/sdg/device/max_sectors, if the drive is > > /dev/sdg). Trying writing 64 to that file before using the drive. > > > > Even 32 doesn't help. Maybe the problem is that the computer sends data before the flash drive is ready to accept it. You can try delaying the data transfer. Edit drivers/usb/storage/transport.c. In usb_stor_Bulk_transport(), near line 1068, you'll find this code: /* Some USB-IDE converter chips need a 100us delay between the * command phase and the data phase. Some devices need a little * more than that, probably because of clock rate inaccuracies. */ if (unlikely(us->fflags & US_FL_GO_SLOW)) udelay(125); Comment out the "if" line so that the udelay is always executed. You can also try increasing the duration above 125 us. Alan Stern
Hi Alan, Alan Stern wrote: > Maybe the problem is that the computer sends data before the flash > drive is ready to accept it. You can try delaying the data transfer. > > Edit drivers/usb/storage/transport.c. In usb_stor_Bulk_transport(), > > No, that didn't help. But I guess I got somewhat further. That particular usb pendrive was working for some time on my main desktop computer. I had created an encrypted device (without partition table, directly onto /dev/sdx) and put an ext3 filesystem inside. I could write some amount of data (some months ago) and then the file system hung, but I didn't care much about. Some weeks later I tried to create a new file system, and that's where the writing problems began. Now I had that particular writing problems with this USB pendrive on five different computers, alle with the latest Ubuntu. An mkfs.ext2 /dev/sdx failed on all machines the very same way. But then, when I used the pendrive in a Windows machine, I could easily format and use it with Windows. So the pendrive definitely is not broken - at least not completely. Surprisingly, after formatting it with Windows with a fat filesystem, it worked on all those Ubuntu machines, now I can write the files (onto fat). And now, the mkfs.ext2 /dev/sdx works as well (what did not terminate formerly). And even the large file can now be copied onto the stick. My guess is that the internal controller of the USB pendrive is somehow incompatible with Linux, encryption or ext2 filesystem. It is known that flash memory cards and drives have internal controllers that continuosly replace memory segments with each other to have them worn down equally. To do so, some of these controllers are able to "understand" dos/fat filesystems and use them as part of their strategy. I assume that I ran that controller with an encrypted filesystem into some error mode. Or hit some memory segments that are broken (it's a 16GB drive), where reformatting with vfat remixed the page order of those segments. I am not sure whether this is a problem of the USB drive (or its controller) or the Linux machine. Best regards (and merry Christmas) Hadmut
On Tue, 22 Dec 2009, Hadmut Danisch wrote: > That particular usb pendrive was working for some time on my main > desktop computer. I had created an encrypted device (without partition > table, directly onto /dev/sdx) and put an ext3 filesystem inside. I > could write some amount of data (some months ago) and then the file > system hung, but I didn't care much about. Some weeks later I tried to > create a new file system, and that's where the writing problems began. > > Now I had that particular writing problems with this USB pendrive on > five different computers, alle with the latest Ubuntu. An > mkfs.ext2 /dev/sdx failed on all machines the very same way. You didn't try mkdosfs, did you? > But then, when I used the pendrive in a Windows machine, I could easily > format and use it with Windows. So the pendrive definitely is not broken > - at least not completely. > > Surprisingly, after formatting it with Windows with a fat filesystem, it > worked on all those Ubuntu machines, now I can write the files (onto > fat). And now, the mkfs.ext2 /dev/sdx works as well (what did not > terminate formerly). And even the large file can now be copied onto the > stick. > > > > My guess is that the internal controller of the USB pendrive is somehow > incompatible with Linux, encryption or ext2 filesystem. It is known that > flash memory cards and drives have internal controllers that continuosly > replace memory segments with each other to have them worn down equally. > To do so, some of these controllers are able to "understand" dos/fat > filesystems and use them as part of their strategy. > > I assume that I ran that controller with an encrypted filesystem into > some error mode. Or hit some memory segments that are broken (it's a > 16GB drive), where reformatting with vfat remixed the page order of > those segments. That certainly is possible. > I am not sure whether this is a problem of the USB drive (or its > controller) or the Linux machine. I'd guess it's not a Linux problem. However the only way to be certain is to mess up the pendrive again in the same way and then see if a Linux computer can successfully reinstall a VFAT filesystem. If you don't want to bother with further testing, you can just close out the bug report. > Best regards (and merry Christmas) Merry Christmas to you. Alan Stern
Hi, I'm not sure, if it's relevant to this thread, but in case on Toshiba a210-15j (AMD x86_64) is circa from 2.6.31 (I'm not sure, sorry) problems with USB. If is writed on USB flash, data gets corrupted, uvcvideo webcam sometimes work, sometimes not. Mouse isn't recognized sometimes. Everything related to USB is weird. (when was linux first installed - circa Kubuntu 8.10 or 9.04, everything worked fine)
Created attachment 24400 [details] dmesg_usb_err Of course some slowdown when copying to usbflash and then freeze (not system, only copy, occured too). sdc is completly ok.
So it looks like this bug report can be closed out, right? The problem was due to the USB pendrive's internal firmware. Alan Stern
yes, closing it out, thanks.