Bug 4882

This email may help explain what happened.

-------------------------------------------------------------------------

Date: Sun, 17 Jul 2005 02:54:36 +0900
From: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
To: Alan Stern <stern@rowland.harvard.edu>
Cc: Andrew Morton <akpm@osdl.org>
Subject: Re: Fw: Re: [linux-usb-devel] Fw: [Bugme-new] [Bug 4882] New:
    degraded usb storage (FAT32) write speed with -o sync on kernel > 2.6.11

Andrew Morton <akpm@osdl.org> writes:

> This isn't a USB problem; it's a question about the behavior of the 
> filesystem code.  It should be forwarded to the people who develop the 
> FAT32 driver.

Support of sync options was added to 2.6.12, so this is not degraded things.
Older fatfs was just ignoring the sync option.

Sync option is for robustness, it is used to control the writing order
of metadata. As side effect, it become the cause of slowing the
performance of writing very much.  Because, to controling the order by
synchronous update is needing the many medata-updates than
delayed-writes.

I don't know what "ignoring the sync option" is supposed to mean. However,
behaviour of mounting in 2.6.11 with sync is different than mounting in 2.6.12
without sync. 2.6.11 with sync still writes-as-it-goes, which is pretty much how
I would expect sync to behave.

I was just at OLS and it was suggested that the write buffer size in the new
sync code is "incompatible" with USB devices, requiring excessive I/O.

I asked Hirofumi for more details on changed sync behavior, his reply follows:

---

From: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Date: Mon, 25 Jul 2005 20:07:49 +0900

> Anyway, mounting FAT filesystems with -o sync worked for me, at least I
> experienced unbuffered writes and almost none filesystem corruptions, which
> were the primary reason I started using -o sync for FATs on USB Flash drives.

Old fatfs was not using sync option (MS_SYNCHRONOUS) and O_SYNC at
all. So, sync option or O_SYNC affects only the user data (VFS is
synchronizing user data).  In other word, old fatfs is synchronizing
the half.

New fatfs with MS_SYNCHRONOUS and O_SYNC, it is synchronizing the meta
data too (directory, etc.). New fatfs is synchronizing all data
safely, but this is very slow.

*** Bug 5009 has been marked as a duplicate of this bug. ***

Using old fatfs code solves this problem.

Then is any solution here ? Or only thing that i can is come back to 2.6.11 ?

Mounting with -o sync is always going to be slower than without it, there's no
getting around that.  Maybe it could run somewhat faster than it does now, but
it will never be as fast.  Nor will it be as fast as -o sync was under 2.6.11,
because the 2.6.11 VFAT code didn't bother to handle -o sync correctly.

The best way for you to speed up your data transfers is to mount the filesystems
without -o sync.  There's no real harm in doing so, provided you remember always
to unmount the filesystem before unplugging the USB device.

@ Alan:

sync might have not been correctly handled in previous kernels, but it worked
very well. Certainly better than right now.

Mounting without -osync causes bad userland stuff. For example, Nautilus has no
concept of file copy progress to a usb stick.

I don't know what the distros will do, but when they get to the current kernels,
they will have to patch it with the old code to make it in any way a sane user
experience.

But how can i patch it ? Do you know how ? If, then give please here the patch.

komodo, copy next directories and files from 2.6.11-xxx to 2.6.12 kernel
tree&and recompile it: 
     fs/fat/
     fs/vfat/
     include/linux/msdos_fs.h

Vitja thanx, but i have found right now two articles and now i am a little
confused if is good idea to use sync. Do you know something anout that ?

http://readlist.com/lists/vger.kernel.org/linux-kernel/22/111748.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0506.0/0468.html

I've been using -o sync all the time till 2.6.12, as it solved frequent
filesystem corruptions I had with USB storage devices. Now I've dropped -o sync,
on 2.6.13-rc3 no problems so far. I'm using various flash USB pen drives and
media cards (MS, SD/MMC, CF). [The only problem I see now is the progress bar
issue, as my grandmother will see all her data has been copied almost instantly,
but still has to wait several minutes on unmounting the device for apparently
"no reason"].

I'm new to bugzilla so please forgive my if I do something wrong :). And also
please forgive my not so good english too.

Anyway, I can confirm this problem in 2.6.12.5 kernel (i tried vanilla kernel
and gentoo's). Copying a file to USB 2.0 (512MB) memory stick (Creative MuVO
512MB FS: FAT32) without the "-o sync" option and unmounting the file system
right after the copy shows that I'm getting around 3MB/s write speeds which is
very good.

But using newer kernels and the "-o sync" option simply destroys write speeds.
Using sync I'm getting around 200kB/s write speeds. I do understand that using
sync option of course slows down the process somewhat, but going from 3MB/s to
ridiculous 200kB/s! That can't be right? Going back to older 2.6.10 kernel and
I'm getting around 1.5MB/s write speed withn sync and again 3MB/s without it.

I also noticed that my stick lightens up when there is a transfer going on (read
or write). Without the "sync" -option and unmounting the file system right after
the cp -command the stick lightens up indicating that there is a transfer going
on and the light stays ON constantly till the cache is emptied.

With "-o sync" the stick light turns ON for a moment, then shuts down almost
immediately and then there is a pause. Light turns ON up for a moment, turns
off, pause, lights up, pause and so on.. during these pauses there doesn't seem
to be any action going on, at least the stick doesn't indicate anything happening.

I can't explaing this behavior any better with my english and with my knowledge
of Linux. This happens on my AMD XP 3200+, Abit NF7-s 2.0 (nforce2 ultra 400
chipset) machine.

I made some detailed timing experiments with my high-speed USB flash device, a
Memorex Travelmate.  Starting with an empty VFAT-16 filesystem, the experiment
amounted to:

	dd if=/dev/zero bs=1K count=10 of=/mnt/usb/a
	umount /mnt/usb/a

where the device had previously been mounted either with or without -o sync. 
This writes a file containing 10 KB, or 20 sectors of data.  Also the root
directory and the two FATs get updated.

Without -o sync the test took about 280 ms.  With -o sync it took about 2040 ms
-- over 2 seconds!  To understand this difference, we must examine the
individual transfers.  (Note that the maximum transfer capacity of a high-speed
USB connection for bulk data is 52 KB/ms; everything else is overhead.)

Without -o sync, the sequence of write operations went like this:

	Write 1 sector to FAT 1
	Write 20 data sectors
	Write 1 sector to FAT 2
	Write 1 sector to the root directory

With -o sync, the sequence went like this:

{	Write 1 sector to root directory
	Write 1 sector to FAT 1
	Write 1 sector to FAT 2
	Write 2 data sectors at i, i+1
	Write 2 data sectors at i+2, i+3
	Write 1 sector to FAT 2
	Write 1 sector to root directory
	Write 1 sector to FAT 1
} repeated with i = 0, 4, 8, 12, 16.

Evidently the metadata (the root directory and the FATs) get updated either
before or after every pair of data sectors is written.  So the overall pattern
is: write 2 data sectors plus 2 FAT sectors and a directory sector, repeated 10
times.

Obviously that's an awful lot of writes.  Even more important is the time
required for each write.  These are the delays caused by the device itself;
overhead due to Linux processing was insignificant.  It turned out (and I don't
understand why) that writes to different sectors did not all take the same
amount of time.

	Writes to data sectors took between 1 and 4 ms per sector.
	Writes to a root directory sector took about 5 ms.
	Writes to a FAT sector took about 95 ms!

Putting that together with the information above, look what we get.  Without -o
sync:

	Write 1 sector to FAT 1:		95 ms
	Write 20 data sectors:			80 ms
	Write 1 sector to FAT 2:		95 ms
	Write 1 sector to the root directory:	 5 ms
	Total:				       275 ms

which agrees pretty well with the measured time.  With -o sync, there were 10
repeats of:

	Write 1 sector to root directory	 5 ms
	Write 1 sector to FAT 1			95 ms
	Write 1 sector to FAT 2			95 ms
	Write 2 data sectors:			 2 ms
	Total:				       197 ms multiplied by 10

This shows pretty clearly why writing is so much slower with -o sync than
without it, at least on my device.

Another interesting thing to note is that as I ran the test over and over, the
time required to write a data sector increased.  At the end it was taking 40 ms
for each data sector.  This strongly suggests that repeated access to a single
location on the flash device is not a good idea, and mounting with -o sync is a
very bad idea indeed.  Of course, other devices might not behave the same as mine.

Downstream bug report: http://bugs.gentoo.org/show_bug.cgi?id=99039

I think people would really like to see a mount option that causes data sectors
to be written synchronously but caches metadata (directories and FATs) writes
for a few seconds or until a file is created or closed, whichever comes first.

Is there any combination of options that already does this?  If not, would it be
possible to add a new option?  Maybe call it something like "datasync".

Closing, as this is a documented issue.

I think this bug should be reconsidered. Slowness is one thing, but how are you
going to explain to users that their flash drives will be destroyed after
writing to it once?

Flash memory blocks can only be written to about 10.000 times. If you copy a
file of some hunderds MB's, the block on which the FAT resides is busted
immediatly. Bye Bye flash disk. 

And of course, this behaviour can't be good for mechanical drive-heads, because
it has to swing back to the FAT all the time.

Windows also has synchronous writes for floppies etc, but there the speed is
normal. So, it can be done.

Thing is, this isn't really a kernel bug; it's a problem with userspace
utilities that mount flash devices with "-o sync".  At most, this is a feature
request for a new VFAT mount option that would duplicate the way "-o sync" used
to work.

But I can't think of a single reason why I would want this kind of sync option
on my vfat drives. I destroys flash discs, wears down harddrives and is very
very slow. I don't know how the old sync option worked, but if that's more like
how windows does it, I would prefer that.

Windows only updates the FAT after each file is complete. Writing to a floppy
disc makes this appearent. When writing one big file, you hear the head moving
to a new sector all the time, and when the file is complete, it moves back, to
write the FAT. When you write a lot of small files, you hear the head going back
all the time. This is how I would prefer the sync option to work.

@ Comment #19 (Alan Stern):

You cannot possibly call a significant regression bug a "feature request." Fact
is that something used to work better than it does not, not dependent of whether
it's the "right" solution or not.

Complain to the VFAT developers.  If they get enough flak, maybe they'll do
something about it.