Bug 8983 - 2GB SD card only shows 1GB in Linux
Summary: 2GB SD card only shows 1GB in Linux
Status: REJECTED INSUFFICIENT_DATA
Alias: None
Product: IO/Storage
Classification: Unclassified
Component: Block Layer (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Jens Axboe
URL:
Keywords:
Depends on:
Blocks: USB
  Show dependency tree
 
Reported: 2007-09-05 14:05 UTC by Ryan Underwood
Modified: 2009-03-23 11:36 UTC (History)
9 users (show)

See Also:
Kernel Version: 2.6.22.6
Tree: Mainline
Regression: ---


Attachments
usb-storage Debug log (40.66 KB, application/x-gzip)
2007-09-05 14:06 UTC, Ryan Underwood
Details
Picture of SD card (18.98 KB, image/jpeg)
2007-09-05 14:10 UTC, Ryan Underwood
Details
dmesg (1.77 KB, text/plain)
2007-09-06 08:13 UTC, Ryan Underwood
Details
dmesg and fdisk output (2.64 KB, text/plain)
2008-07-02 05:49 UTC, Ben Bucksch
Details

Description Ryan Underwood 2007-09-05 14:05:05 UTC
Distribution: Debian

Hardware Environment:
USB Reader is a "Mediamover"
SD Card is a Kingston MMC Mobile Dual-voltage 2GB, with 2034584 printed on it.

Problem Description:
Only 1GB of the 2GB SD card is available.

Steps to reproduce:
Plug in the card.  I attached the debug dump.

I noticed that if I delete the partition table, Windows only sees a 512
byte physical drive.  I guess that confirms the suspicion that Windows
refers to the partition table to calculate the physical size of the
device (the 512 bytes would correspond to the MBR which is assumed to be
present).
Comment 1 Ryan Underwood 2007-09-05 14:06:07 UTC
Created attachment 12717 [details]
usb-storage Debug log

Debug log
Comment 2 Ryan Underwood 2007-09-05 14:08:53 UTC
It may have been unclear above, but before deleting the partition table, Windows did see the card as 2GB.
Comment 3 Ryan Underwood 2007-09-05 14:10:46 UTC
Created attachment 12718 [details]
Picture of SD card
Comment 4 Pierre Ossman 2007-09-05 22:13:24 UTC
USB was the right component all along. This is a USB storage device, not a direct MMC controller.
Comment 5 Anonymous Emailer 2007-09-06 00:08:59 UTC
Reply-To: akpm@linux-foundation.org

> On Wed,  5 Sep 2007 14:05:05 -0700 (PDT) bugme-daemon@bugzilla.kernel.org
> wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=8983
> 
>            Summary: 2GB SD card only shows 1GB in Linux
>            Product: Drivers
>            Version: 2.5
>      KernelVersion: 2.6.22.6
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: USB
>         AssignedTo: greg@kroah.com
>         ReportedBy: nemesis@icequake.net
> 
> 
> Distribution: Debian
> 
> Hardware Environment:
> USB Reader is a "Mediamover"
> SD Card is a Kingston MMC Mobile Dual-voltage 2GB, with 2034584 printed on
> it.
> 
> Problem Description:
> Only 1GB of the 2GB SD card is available.
> 
> Steps to reproduce:
> Plug in the card.  I attached the debug dump.
> 
> I noticed that if I delete the partition table, Windows only sees a 512
> byte physical drive.  I guess that confirms the suspicion that Windows
> refers to the partition table to calculate the physical size of the
> device (the 512 bytes would correspond to the MBR which is assumed to be
> present).
Comment 6 Daniel Drake 2007-09-06 06:49:44 UTC
Interesting idea regarding Windows trusting the partition table size! We had a similar bug reported at Gentoo a while ago:

https://bugs.gentoo.org/show_bug.cgi?id=149584

By description this bug is very similar, and actually I went through the usbmon logs from Windows trying to figure out where it might be discovering the size to be 2GB. I did not ever figure out why -- all of the standard USB mass storage commands reported 1GB.

Once you have reformatted this card to be 1GB or less, is there any way to make it 2GB again without manually messing with the partition table? I guess that if you were to try to partition and format it under windows, it would create a partition sized 1gb.
Comment 7 Alan Stern 2007-09-06 07:18:18 UTC
I think this is a known problem in many SD card readers.  The technique for handling 2 GB cards is different from the way 1 GB cards are handled, and if the reader isn't aware of this then it can't access anything beyond the first GB.

By the way, how come your usb-storage debug log omits several of the messages from the SCSI subsystem?  There should be messages about the capacity and partition entries for drive sdf.
Comment 8 Ryan Underwood 2007-09-06 08:04:02 UTC
Yes, the card gives 1024 bytes blocksize.

A smart reader doubles the LBA count to compensate for the 512 byte block size of the USB mass storage class driver.

Windows seems to work around dumb readers in a IMHO stupid way by using the partition table.  Maybe a better workaround is to issue probe reads.

This was strictly a debug log so it doesn't mention non-debug things, I'll attach the missing parts.
Comment 9 Ryan Underwood 2007-09-06 08:11:44 UTC
Sep  5 15:35:59 localhost kernel: usb 1-2: new full speed USB device using uhci_hcd and address 2
Sep  5 15:35:59 localhost kernel: usb 1-2: configuration #1 chosen from 1 choice
Sep  5 15:35:59 localhost kernel: Initializing USB Mass Storage driver...
Sep  5 15:35:59 localhost kernel: scsi2 : SCSI emulation for USB Mass Storage devices
Sep  5 15:35:59 localhost kernel: usbcore: registered new interface driver usb-storage
Sep  5 15:35:59 localhost kernel: USB Mass Storage support registered.
Sep  5 15:36:04 localhost kernel: scsi 2:0:0:0: Direct-Access     Generic  USB READER     1 0100 PQ: 0 ANSI: 0 CCS
Sep  5 15:36:04 localhost kernel: sd 2:0:0:0: [sde] Attached SCSI removable disk
Sep  5 15:36:04 localhost kernel: scsi 2:0:0:1: Direct-Access     Generic  USB READER     3 0100 PQ: 0 ANSI: 0 CCS
Sep  5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware sectors (1007 MB)
Sep  5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] Write Protect is off
Sep  5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware sectors (1007 MB)
Sep  5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] Write Protect is off
Sep  5 15:36:04 localhost kernel:  sdf:<7>usb-storage: queuecommand called
Sep  5 15:36:05 localhost kernel:  unknown partition table
Sep  5 15:36:05 localhost kernel: sd 2:0:0:1: [sdf] Attached SCSI removable disk
Sep  5 15:36:05 localhost kernel: scsi 2:0:0:2: Direct-Access     Generic  USB READER     5 0100 PQ: 0 ANSI: 0 CCS
Sep  5 15:36:05 localhost kernel: sd 2:0:0:2: [sdg] Attached SCSI removable disk
Sep  5 15:36:05 localhost kernel: scsi 2:0:0:3: Direct-Access     Generic  USB READER     2 0100 PQ: 0 ANSI: 0 CCS
Sep  5 15:36:05 localhost kernel: sd 2:0:0:3: [sdh] Attached SCSI removable disk
Sep  5 15:51:58 localhost -- MARK --
Comment 10 Ryan Underwood 2007-09-06 08:12:30 UTC
sorry, I meant to attach a file.
Comment 11 Ryan Underwood 2007-09-06 08:13:09 UTC
Created attachment 12736 [details]
dmesg
Comment 12 Alan Stern 2007-09-06 09:05:16 UTC
I suppose you're talking about this:

Sep  5 15:36:04 localhost kernel: sd 2:0:0:1: [sdf] 1967616 512-byte hardware
sectors (1007 MB)

Note that the 512-byte block size isn't built into the usb-storage driver; it comes directly from the card reader.  So apparently the reader reports a block size of 512 instead of the actual value of 1024, but at the same time it keeps the block count equal to 1967616 instead of doubling it to compensate.

Are you really certain that Windows works around this bug?  Have you checked that it is possible to store 2 GB of actual data on the card and read it back without corruption?  Or does it turn out that the second GB of data ends up overwriting the first GB?
Comment 13 Ryan Underwood 2007-09-06 11:17:26 UTC
I meant the card reader returns 512 byte blocks probably as a workaround for Windows 98, not for usb-storage.

Yes, Windows works around this bug and the card works fine, according to the owner.
Comment 14 Alan Stern 2008-01-24 11:42:52 UTC
I'm not sure what we can do to ameliorate this problem.  Create a sysfs attribute allowing the user to override the device capacity with the correct value?  That doesn't seem very robust, especially when you imagine switching different cards in and out of the reader.

I do not like the idea of probing to determine the size.  No doubt we would quickly find that with some hardware, trying to read beyond the end will crash the card reader.
Comment 15 Ryan Underwood 2008-01-24 12:33:04 UTC
Perhaps a reader blacklist can be accumulated, but it may grow large.

The only sane thing to do seems to be to emulate the insane behavior of windows that enabled this broken hardware to exist in the first place.

But then you run the risk of crashing the hardware, when someone puts an invalid/malicious partition table on a card that causes linux to access beyond the end of the device.

Maybe we can consider it okay to crash the USB reader when the card is bogus, as long as the crash doesn't propagate beyond the hardware in question?  That's the same approach we take with 3D acceleration hardware.  But then you have other devices on the bus to consider.

This is pathetic.
Comment 16 Alan Stern 2008-01-24 14:40:58 UTC
Crashing the card reader when a card is bogus isn't a good idea -- how would you manage to repair the card?

Relying on the data in the partition table isn't as easy as it sounds, anyhow.  For instance, a card doesn't need to have a standard partition sector.

I agree that the situation is ridiculous.  You could try asking the manufacturer how to reformat a zeroed-out card to recover the lost GB.  After all, that's a problem even with Windows.
Comment 17 Ryan Underwood 2008-01-24 17:14:40 UTC
Well, I didn't mean crash the reader *after* mounting rw.

I meant possibly crash the reader by 1) Using the partition table instead of the device to determine its size, and 2) Verifying the pt size is correct with a probe read.  This would happen *before* ever allowing a mount to occur. :-)
Comment 18 Alan Stern 2008-01-25 07:05:31 UTC
Yes, I understood.  Still, if you have a bad partition table entry and you want to fix it up, you won't be able to if your reader crashes as soon as you insert the card.

I don't know of anything we can safely do to do work around the bug.  If the manufacturer doesn't have any suggestions, you should go ahead and close this bug report (mark it WILL NOT FIX or something like that).
Comment 19 Ryan Underwood 2008-02-14 14:52:15 UTC
What would be the drawback to a reader blacklist, besides that it would generally only be a reactive solution and not one that benefits non-bug-reporting users?
Comment 20 Alan Stern 2008-02-14 15:15:45 UTC
I don't understand your question.  Who would a reader blacklist help?  And how?
Comment 21 Ryan Underwood 2008-02-14 15:26:01 UTC
... the people who have such a broken reader, in the case where we know the reader doesn't crash when probing nonexistent card region.

People who have a broken reader that crashes when probing would be SOL.
Comment 22 Alan Stern 2008-02-14 19:36:46 UTC
So you're talking about a whitelist then, not a blacklist -- the readers which can survive an apparently invalid read request.  It might help, except for the fact that I don't know of any reasonable way to probe the sectors listed in the partition table.  Perhaps it could be done, but it would be a tremendous hack and very unappealing to lots of people.  You'd have to have somebody who is very familiar with the block layer, such as Jens Axboe (the maintainer for the block layer).
Comment 23 Alan Stern 2008-05-08 11:49:26 UTC
Can we close this bug with a "Won't fix" resolution?
Comment 24 Ben Bucksch 2008-06-30 14:37:56 UTC
Please don't won't fix, but do something about.

Importance: As you said, the problem is with many card readers, and 2 GB is the most common size now, and will stay so, given that they are cheap and larger ones are SDHC and don't with many cams/devices. SD card readers are a very common operation for mom and dad users. In other words, this is a bug many people run into.

It leads to *data loss*. When writing more than 1 GB, all my file entries disappered, the directories stayed, and the space was occupied (1900 MB used, 5 MB free, 0 files, 1 dir).

I ran into this before, but blamed the SD card.
It took me several hours of noticing the problem, redoing, trying to fix it, searching until I found this.

In short: This is serious. Please do something about it. Either
- do the ugly Windows workaround of looking at the partition table, or
- at the very least ensure that the user gets a dialog box in their face (dmesg does not count) that their SD card reader is broken. they will destroy their data and telling them to buy a new reader.
- or whatever smart solution you come up with
But please fix this ASAP. This is highly visible on the user level and makes people think that Linux destorys their data.
Comment 25 Ben Bucksch 2008-06-30 14:39:44 UTC
Requesting to please change Severity to High based on the fact that this causes data loss.
Comment 26 Alan Stern 2008-07-01 08:13:56 UTC
Andrew & Greg:

Brief summary of the problem: Broken USB card readers report that 2-GB cards have only a 1-GB capacity.  Linux believes the reader and loses the ability to access data beyond the 1-GB point.  Windows believes the partition table instead, and is able to use the entire card.

This is not a USB problem; it needs to be handled in the block or gendisk layers.
Can this bug be reassigned?
Comment 27 Greg Kroah-Hartman 2008-07-01 08:23:52 UTC
now reassigned.
Comment 28 Andrew Morton 2008-07-01 13:01:42 UTC
Problem is, I don't know who can/will fix this - partition table management
is an unowned black hole full of drive-by patchings.

If we can possibly position this as a fatfs problem then perhaps I can
persuade hirofumi to look at it ;)

Can you please describe the problem more fully?  What does "Linux believes the reader" mean?  Something in the block layer does a device query and then sets the device size? If so, where does this happen?

What would be a suitable fix?  Perhaps a fatfs mount option which says "use the partition table and not the device query results"?

Thanks.
Comment 29 Alan Stern 2008-07-01 14:00:10 UTC
The revalidate_disk method is called whenever a new disk is registered using add_disk().  The SCSI version reads the drive capacity directly from the device, and in this case the card reader returns 1 GB.  At the same time, the partition code in fs/partitions/check.c reads the partition table.

If an entry in the partition table indicates that a partition extends beyond the end of the device's capacity then the kernel ignores the excess.  In this case it won't allow accesses beyond the 1-GB capacity limit (I don't know where the limit check is made).

However the 1-GB limit is bogus, the result of a bug in the card reader.  The SD card itself actually has 2 GB of storage and the partition table reflects this.  So if there were a fatfs mount option that changed disk->capacity to match the partition table, it ought to solve the problem.
Comment 30 OGAWA Hirofumi 2008-07-01 16:51:44 UTC
Oh, um... yes, IIRC, SD/MMC has 2gb boundary, and old MMC driver of Linux
was also having this problem. It seems this USB reader also has same bug.

Well, anyway, the following warning was outputed?

	if (from + size > get_capacity(disk)) {
		printk(" %s: p%d exceeds device capacity\n",
			disk->disk_name, p);
	}

Also, if you mount and use it, what error happen? I guess the limit
check is on block layer.

BTW, how did you write the partition? Windows also can't write?
Comment 31 Ben Bucksch 2008-07-02 05:49:23 UTC
Created attachment 16693 [details]
dmesg and fdisk output

> the following warning was outputed?
>                printk(" %s: p%d exceeds device capacity\n",

Yes, log and fdisk attached.
Comment 32 Ben Bucksch 2008-07-02 05:55:15 UTC
> if you mount and use it, what error happen?

No user-visible error message IIRC (as said, dmesg does not count as user-visible), but when you write more than 1 GB, all files disappear (including those below 1 GB), at least in my case.

FWIW, I have never used this disk with Windows. The FAT on my disk was from manufacturer, it was preformatted (as FAT16 it seems). But the problem is obivously below that, on disk level, even below partitions.

Please note that whatever solution you use, it needs to work without explicit user action, as users will not be aware of the problem.
Comment 33 Ben Bucksch 2008-07-02 05:57:17 UTC
> The revalidate_disk method
> The SCSI version reads the drive capacity directly from the
> device, and in this case the card reader returns 1 GB.  At the same time, the
> partition code in fs/partitions/check.c reads the partition table.

How about a special case here: If you detect a card reader, and it reports 1 GB, and the partition size differs, trust the partition size?
Comment 34 Alan Stern 2008-07-02 07:36:55 UTC
Unfortunately there's no way for the kernel to tell whether a particular device is a card reader.
Comment 35 OGAWA Hirofumi 2008-07-02 12:46:21 UTC
I'm a bit confused.

> > if you mount and use it, what error happen?
> 
> No user-visible error message IIRC (as said, dmesg does not count as
> user-visible), but when you write more than 1 GB, all files disappear
> (including those below 1 GB), at least in my case.

So, was dmesg outputed? And how can you write beyond 1gb?

> FWIW, I have never used this disk with Windows. The FAT on my disk was from
> manufacturer, it was preformatted (as FAT16 it seems). But the problem is
> obivously below that, on disk level, even below partitions.

Um. If so, why do you think you can use 2gb cards on this reader?
The your reader is really supporting 2gb SD/MMC?
(The disk means the SD/MMC card, not the reader?)

> Please note that whatever solution you use, it needs to work without explicit
> user action, as users will not be aware of the problem.

This is a broken reader. We don't know what happen on a broken reader.
I think, at least the _kernel_ will not allow it without user's request.
Comment 36 Ben Bucksch 2008-07-02 20:21:59 UTC
> So, was dmesg outputed?

Yes, see attachment.

> And how can you write beyond 1gb?

You keep copying files using Konqueror. IIRC, the partition is mounted as 2 GB.

> why do you think you can use 2gb cards on this reader?

Because that's what SD specs - up to 2 GB. And because it presumably works on Windows.
Comment 37 OGAWA Hirofumi 2008-07-02 21:50:10 UTC
(In reply to comment #36)
> > So, was dmesg outputed?
> 
> Yes, see attachment.

The log is the following in "dmesg and fdisk output"?

sdd: rw=0, want=3967354, limit=1985024
Buffer I/O error on device sdd1, logical block 3967104
attempt to access beyond end of device
sdd: rw=0, want=3967355, limit=1985024
Buffer I/O error on device sdd1, logical block 3967105

If so, the partition code doesn't check a limit,
and also FAT donen't.

> > And how can you write beyond 1gb?
> 
> You keep copying files using Konqueror. IIRC, the partition is mounted as 2
> GB.

Ok, I guess it was I/O error finally like above logs.

> > why do you think you can use 2gb cards on this reader?
> 
> Because that's what SD specs - up to 2 GB. And because it presumably works on
> Windows.

Ah, it is not right. SD specs is the spec of SD, not the reader. I know and had
the reader which can't handle 2gb SD/MMC, of course even on Windows.



If you really want to send a request to more lower layer (maybe, more lower layer
doesn't check), the following test patch will do. But NOTE, I think it may be
dangerous on some devices.

diff -puN block/blk-core.c~debug block/blk-core.c
--- linux-2.6/block/blk-core.c~debug	2008-07-03 04:22:54.000000000 +0900
+++ linux-2.6-hirofumi/block/blk-core.c	2008-07-03 04:23:00.000000000 +0900
@@ -1278,6 +1278,8 @@ static inline int bio_check_eod(struct b
 {
 	sector_t maxsector;
 
+	return 0;
+
 	if (!nr_sectors)
 		return 0;
 
_
Comment 38 Gordon Hopper 2008-07-30 09:03:04 UTC
2GB cards are interesting because they are the only size of SD cards which have a block size of 1024 instead of 512.  (READ_BL_LEN in the CSD register of the SD card).  Some older USB devices fail to read this register and thus fail to report the size correctly.  (They do not conform to the SD card 1.1 spec, presumably because they were manufactured before that version came out.)

It would be interesting to try to write larger blocks or more sectors than reported by the USB device, but I suspect that these USB card readers will not handle this.

> Windows did see the card as 2GB.

That does not mean that Windows was able to read and write 2GB worth of data (without corruption).  I do not think the sectors above 1GB are addressable through the USB device (but I could be wrong).

This will be hard to verify/fix without one of the offending USB devices.  Does anyone have a link where we can buy one, or does this happen only on older hardware, which is no longer for sale?

Note You need to log in before you can comment on or make changes to this bug.