Distribution: Debian Hardware Environment: USB Reader is a "Mediamover" SD Card is a Kingston MMC Mobile Dual-voltage 2GB, with 2034584 printed on it. Problem Description: Only 1GB of the 2GB SD card is available. Steps to reproduce: Plug in the card. I attached the debug dump. I noticed that if I delete the partition table, Windows only sees a 512 byte physical drive. I guess that confirms the suspicion that Windows refers to the partition table to calculate the physical size of the device (the 512 bytes would correspond to the MBR which is assumed to be present).
Created attachment 12717 [details] usb-storage Debug log Debug log
It may have been unclear above, but before deleting the partition table, Windows did see the card as 2GB.
Created attachment 12718 [details] Picture of SD card
USB was the right component all along. This is a USB storage device, not a direct MMC controller.
Reply-To: akpm@linux-foundation.org > On Wed, 5 Sep 2007 14:05:05 -0700 (PDT) bugme-daemon@bugzilla.kernel.org > wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=8983 > > Summary: 2GB SD card only shows 1GB in Linux > Product: Drivers > Version: 2.5 > KernelVersion: 2.6.22.6 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: USB > AssignedTo: greg@kroah.com > ReportedBy: nemesis@icequake.net > > > Distribution: Debian > > Hardware Environment: > USB Reader is a "Mediamover" > SD Card is a Kingston MMC Mobile Dual-voltage 2GB, with 2034584 printed on > it. > > Problem Description: > Only 1GB of the 2GB SD card is available. > > Steps to reproduce: > Plug in the card. I attached the debug dump. > > I noticed that if I delete the partition table, Windows only sees a 512 > byte physical drive. I guess that confirms the suspicion that Windows > refers to the partition table to calculate the physical size of the > device (the 512 bytes would correspond to the MBR which is assumed to be > present).
Interesting idea regarding Windows trusting the partition table size! We had a similar bug reported at Gentoo a while ago: https://bugs.gentoo.org/show_bug.cgi?id=149584 By description this bug is very similar, and actually I went through the usbmon logs from Windows trying to figure out where it might be discovering the size to be 2GB. I did not ever figure out why -- all of the standard USB mass storage commands reported 1GB. Once you have reformatted this card to be 1GB or less, is there any way to make it 2GB again without manually messing with the partition table? I guess that if you were to try to partition and format it under windows, it would create a partition sized 1gb.
I think this is a known problem in many SD card readers. The technique for handling 2 GB cards is different from the way 1 GB cards are handled, and if the reader isn't aware of this then it can't access anything beyond the first GB. By the way, how come your usb-storage debug log omits several of the messages from the SCSI subsystem? There should be messages about the capacity and partition entries for drive sdf.
Yes, the card gives 1024 bytes blocksize. A smart reader doubles the LBA count to compensate for the 512 byte block size of the USB mass storage class driver. Windows seems to work around dumb readers in a IMHO stupid way by using the partition table. Maybe a better workaround is to issue probe reads. This was strictly a debug log so it doesn't mention non-debug things, I'll attach the missing parts.
Sep 5 15:35:59 localhost kernel: usb 1-2: new full speed USB device using uhci_hcd and address 2 Sep 5 15:35:59 localhost kernel: usb 1-2: configuration #1 chosen from 1 choice Sep 5 15:35:59 localhost kernel: Initializing USB Mass Storage driver... Sep 5 15:35:59 localhost kernel: scsi2 : SCSI emulation for USB Mass Storage devices Sep 5 15:35:59 localhost kernel: usbcore: registered new interface driver usb-storage Sep 5 15:35:59 localhost kernel: USB Mass Storage support registered. Sep 5 15:36:04 localhost kernel: scsi 2:0:0:0: Direct-Access Generic USB READER 1 0100 PQ: 0 ANSI: 0 CCS Sep 5 15:36:04 localhost kernel: sd 2:0:0:0: [sde] Attached SCSI removable disk Sep 5 15:36:04 localhost kernel: scsi 2:0:0:1: Direct-Access Generic USB READER 3 0100 PQ: 0 ANSI: 0 CCS Sep 5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware sectors (1007 MB) Sep 5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] Write Protect is off Sep 5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware sectors (1007 MB) Sep 5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] Write Protect is off Sep 5 15:36:04 localhost kernel: sdf:<7>usb-storage: queuecommand called Sep 5 15:36:05 localhost kernel: unknown partition table Sep 5 15:36:05 localhost kernel: sd 2:0:0:1: [sdf] Attached SCSI removable disk Sep 5 15:36:05 localhost kernel: scsi 2:0:0:2: Direct-Access Generic USB READER 5 0100 PQ: 0 ANSI: 0 CCS Sep 5 15:36:05 localhost kernel: sd 2:0:0:2: [sdg] Attached SCSI removable disk Sep 5 15:36:05 localhost kernel: scsi 2:0:0:3: Direct-Access Generic USB READER 2 0100 PQ: 0 ANSI: 0 CCS Sep 5 15:36:05 localhost kernel: sd 2:0:0:3: [sdh] Attached SCSI removable disk Sep 5 15:51:58 localhost -- MARK --
sorry, I meant to attach a file.
Created attachment 12736 [details] dmesg
I suppose you're talking about this: Sep 5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware sectors (1007 MB) Note that the 512-byte block size isn't built into the usb-storage driver; it comes directly from the card reader. So apparently the reader reports a block size of 512 instead of the actual value of 1024, but at the same time it keeps the block count equal to 1967616 instead of doubling it to compensate. Are you really certain that Windows works around this bug? Have you checked that it is possible to store 2 GB of actual data on the card and read it back without corruption? Or does it turn out that the second GB of data ends up overwriting the first GB?
I meant the card reader returns 512 byte blocks probably as a workaround for Windows 98, not for usb-storage. Yes, Windows works around this bug and the card works fine, according to the owner.
I'm not sure what we can do to ameliorate this problem. Create a sysfs attribute allowing the user to override the device capacity with the correct value? That doesn't seem very robust, especially when you imagine switching different cards in and out of the reader. I do not like the idea of probing to determine the size. No doubt we would quickly find that with some hardware, trying to read beyond the end will crash the card reader.
Perhaps a reader blacklist can be accumulated, but it may grow large. The only sane thing to do seems to be to emulate the insane behavior of windows that enabled this broken hardware to exist in the first place. But then you run the risk of crashing the hardware, when someone puts an invalid/malicious partition table on a card that causes linux to access beyond the end of the device. Maybe we can consider it okay to crash the USB reader when the card is bogus, as long as the crash doesn't propagate beyond the hardware in question? That's the same approach we take with 3D acceleration hardware. But then you have other devices on the bus to consider. This is pathetic.
Crashing the card reader when a card is bogus isn't a good idea -- how would you manage to repair the card? Relying on the data in the partition table isn't as easy as it sounds, anyhow. For instance, a card doesn't need to have a standard partition sector. I agree that the situation is ridiculous. You could try asking the manufacturer how to reformat a zeroed-out card to recover the lost GB. After all, that's a problem even with Windows.
Well, I didn't mean crash the reader *after* mounting rw. I meant possibly crash the reader by 1) Using the partition table instead of the device to determine its size, and 2) Verifying the pt size is correct with a probe read. This would happen *before* ever allowing a mount to occur. :-)
Yes, I understood. Still, if you have a bad partition table entry and you want to fix it up, you won't be able to if your reader crashes as soon as you insert the card. I don't know of anything we can safely do to do work around the bug. If the manufacturer doesn't have any suggestions, you should go ahead and close this bug report (mark it WILL NOT FIX or something like that).
What would be the drawback to a reader blacklist, besides that it would generally only be a reactive solution and not one that benefits non-bug-reporting users?
I don't understand your question. Who would a reader blacklist help? And how?
... the people who have such a broken reader, in the case where we know the reader doesn't crash when probing nonexistent card region. People who have a broken reader that crashes when probing would be SOL.
So you're talking about a whitelist then, not a blacklist -- the readers which can survive an apparently invalid read request. It might help, except for the fact that I don't know of any reasonable way to probe the sectors listed in the partition table. Perhaps it could be done, but it would be a tremendous hack and very unappealing to lots of people. You'd have to have somebody who is very familiar with the block layer, such as Jens Axboe (the maintainer for the block layer).
Can we close this bug with a "Won't fix" resolution?
Please don't won't fix, but do something about. Importance: As you said, the problem is with many card readers, and 2 GB is the most common size now, and will stay so, given that they are cheap and larger ones are SDHC and don't with many cams/devices. SD card readers are a very common operation for mom and dad users. In other words, this is a bug many people run into. It leads to *data loss*. When writing more than 1 GB, all my file entries disappered, the directories stayed, and the space was occupied (1900 MB used, 5 MB free, 0 files, 1 dir). I ran into this before, but blamed the SD card. It took me several hours of noticing the problem, redoing, trying to fix it, searching until I found this. In short: This is serious. Please do something about it. Either - do the ugly Windows workaround of looking at the partition table, or - at the very least ensure that the user gets a dialog box in their face (dmesg does not count) that their SD card reader is broken. they will destroy their data and telling them to buy a new reader. - or whatever smart solution you come up with But please fix this ASAP. This is highly visible on the user level and makes people think that Linux destorys their data.
Requesting to please change Severity to High based on the fact that this causes data loss.
Andrew & Greg: Brief summary of the problem: Broken USB card readers report that 2-GB cards have only a 1-GB capacity. Linux believes the reader and loses the ability to access data beyond the 1-GB point. Windows believes the partition table instead, and is able to use the entire card. This is not a USB problem; it needs to be handled in the block or gendisk layers. Can this bug be reassigned?
now reassigned.
Problem is, I don't know who can/will fix this - partition table management is an unowned black hole full of drive-by patchings. If we can possibly position this as a fatfs problem then perhaps I can persuade hirofumi to look at it ;) Can you please describe the problem more fully? What does "Linux believes the reader" mean? Something in the block layer does a device query and then sets the device size? If so, where does this happen? What would be a suitable fix? Perhaps a fatfs mount option which says "use the partition table and not the device query results"? Thanks.
The revalidate_disk method is called whenever a new disk is registered using add_disk(). The SCSI version reads the drive capacity directly from the device, and in this case the card reader returns 1 GB. At the same time, the partition code in fs/partitions/check.c reads the partition table. If an entry in the partition table indicates that a partition extends beyond the end of the device's capacity then the kernel ignores the excess. In this case it won't allow accesses beyond the 1-GB capacity limit (I don't know where the limit check is made). However the 1-GB limit is bogus, the result of a bug in the card reader. The SD card itself actually has 2 GB of storage and the partition table reflects this. So if there were a fatfs mount option that changed disk->capacity to match the partition table, it ought to solve the problem.
Oh, um... yes, IIRC, SD/MMC has 2gb boundary, and old MMC driver of Linux was also having this problem. It seems this USB reader also has same bug. Well, anyway, the following warning was outputed? if (from + size > get_capacity(disk)) { printk(" %s: p%d exceeds device capacity\n", disk->disk_name, p); } Also, if you mount and use it, what error happen? I guess the limit check is on block layer. BTW, how did you write the partition? Windows also can't write?
Created attachment 16693 [details] dmesg and fdisk output > the following warning was outputed? > printk(" %s: p%d exceeds device capacity\n", Yes, log and fdisk attached.
> if you mount and use it, what error happen? No user-visible error message IIRC (as said, dmesg does not count as user-visible), but when you write more than 1 GB, all files disappear (including those below 1 GB), at least in my case. FWIW, I have never used this disk with Windows. The FAT on my disk was from manufacturer, it was preformatted (as FAT16 it seems). But the problem is obivously below that, on disk level, even below partitions. Please note that whatever solution you use, it needs to work without explicit user action, as users will not be aware of the problem.
> The revalidate_disk method > The SCSI version reads the drive capacity directly from the > device, and in this case the card reader returns 1 GB. At the same time, the > partition code in fs/partitions/check.c reads the partition table. How about a special case here: If you detect a card reader, and it reports 1 GB, and the partition size differs, trust the partition size?
Unfortunately there's no way for the kernel to tell whether a particular device is a card reader.
I'm a bit confused. > > if you mount and use it, what error happen? > > No user-visible error message IIRC (as said, dmesg does not count as > user-visible), but when you write more than 1 GB, all files disappear > (including those below 1 GB), at least in my case. So, was dmesg outputed? And how can you write beyond 1gb? > FWIW, I have never used this disk with Windows. The FAT on my disk was from > manufacturer, it was preformatted (as FAT16 it seems). But the problem is > obivously below that, on disk level, even below partitions. Um. If so, why do you think you can use 2gb cards on this reader? The your reader is really supporting 2gb SD/MMC? (The disk means the SD/MMC card, not the reader?) > Please note that whatever solution you use, it needs to work without explicit > user action, as users will not be aware of the problem. This is a broken reader. We don't know what happen on a broken reader. I think, at least the _kernel_ will not allow it without user's request.
> So, was dmesg outputed? Yes, see attachment. > And how can you write beyond 1gb? You keep copying files using Konqueror. IIRC, the partition is mounted as 2 GB. > why do you think you can use 2gb cards on this reader? Because that's what SD specs - up to 2 GB. And because it presumably works on Windows.
(In reply to comment #36) > > So, was dmesg outputed? > > Yes, see attachment. The log is the following in "dmesg and fdisk output"? sdd: rw=0, want=3967354, limit=1985024 Buffer I/O error on device sdd1, logical block 3967104 attempt to access beyond end of device sdd: rw=0, want=3967355, limit=1985024 Buffer I/O error on device sdd1, logical block 3967105 If so, the partition code doesn't check a limit, and also FAT donen't. > > And how can you write beyond 1gb? > > You keep copying files using Konqueror. IIRC, the partition is mounted as 2 > GB. Ok, I guess it was I/O error finally like above logs. > > why do you think you can use 2gb cards on this reader? > > Because that's what SD specs - up to 2 GB. And because it presumably works on > Windows. Ah, it is not right. SD specs is the spec of SD, not the reader. I know and had the reader which can't handle 2gb SD/MMC, of course even on Windows. If you really want to send a request to more lower layer (maybe, more lower layer doesn't check), the following test patch will do. But NOTE, I think it may be dangerous on some devices. diff -puN block/blk-core.c~debug block/blk-core.c --- linux-2.6/block/blk-core.c~debug 2008-07-03 04:22:54.000000000 +0900 +++ linux-2.6-hirofumi/block/blk-core.c 2008-07-03 04:23:00.000000000 +0900 @@ -1278,6 +1278,8 @@ static inline int bio_check_eod(struct b { sector_t maxsector; + return 0; + if (!nr_sectors) return 0; _
2GB cards are interesting because they are the only size of SD cards which have a block size of 1024 instead of 512. (READ_BL_LEN in the CSD register of the SD card). Some older USB devices fail to read this register and thus fail to report the size correctly. (They do not conform to the SD card 1.1 spec, presumably because they were manufactured before that version came out.) It would be interesting to try to write larger blocks or more sectors than reported by the USB device, but I suspect that these USB card readers will not handle this. > Windows did see the card as 2GB. That does not mean that Windows was able to read and write 2GB worth of data (without corruption). I do not think the sectors above 1GB are addressable through the USB device (but I could be wrong). This will be hard to verify/fix without one of the offending USB devices. Does anyone have a link where we can buy one, or does this happen only on older hardware, which is no longer for sale?