Bug 6380

Created attachment 7845 [details]
output of xfs_check for occasion 3 from the bug report

Created attachment 7846 [details]
output of xfs_repair for occasion 3 from the bug report

I suspected defective RAM on my IBM ThinkPad, so I ran memtest86 for an hour. 
It reported to errors. I have 128 MB that were originally in that ThinkPad + 
256 Kingston RAM that is made with timings especially for IBM ThinkPad T23.

I will now be running 2.6.15 again and report whether I get corruption again. I 
think I won't.

The other bugreport I think this is related to with complete link is:
http://bugzilla.kernel.org/show_bug.cgi?id=6180

Does bugzilla generate a link for this? bug #6180

I know this is not a plain vanilla kernel, cause I use sws2, but I used 2.6.15 
with sws2 since months already without XFS corruption three times a week. I 
would love to test it without sws2, but as I mentioned I have no time to afford 
to restore my backup again and again.

Created attachment 7847 [details]
kernel config of 2.6.16.4 under which I had occasion 2 and 3 from the bug report

Created attachment 7848 [details]
kernel config of the lastest 2.6.15 kernel I had in use

I did not have a XFS corruption problem with that kernel, nor with earlier
2.6.15 kernels

Created attachment 7849 [details]
output of lscpi-nvv for my IBM ThinkPad T23

Classic symptoms of the write cache being enabled on your drive.
Switch it off, or try a recent kernel with the -o barrier option
(this will be on by defult in 2.6.17).

cheers.

Many thanks for your prompt answer.

Indeed, write cache should have been switched on, according to this

root@deepdance:~ -> hdparm -i /dev/hda | grep WriteCache
 AdvancedPM=yes: mode=0x80 (128) WriteCache=enabled

and this:

root@deepdance:~ -> cat /etc/hdparm.conf
[...]
/dev/hda {
       mult_sect_io = 16
       write_cache = on
       dma = on
       apm = 0
       acoustic_management = 128
       io32_support = 3
       keep_settings_over_reset = on
       interrupt_unmask = on
}

I know no way to query it directly.

I switched it to off immediately using hdparm -W 0 and set it to off in the 
hdparm.conf file as well.

I take your comment that this should be set to off with kernel 2.6.15 as well. 
Well I know SmartFilesystem for AmigaOS relies on data to be flushed to disc to 
be written immediately before the call returns in order to ensure a certain 
order for atomic writes. So this is the case with XFS as well?

Well ok, Documentation/block/barrier.txt sheds some light on this - just for 
other readers of this bug:

"There are four cases,

i. No write-back cache.  Keeping requests ordered is enough.

ii. Write-back cache but no flush operation.  There's no way to
gurantee physical-medium commit order.  This kind of devices can't to
I/O barriers.

iii. Write-back cache and flush operation but no FUA (forced unit
access).  We need two cache flushes - before and after the barrier
request."

Ok, so either barriers on or write cache off. Got this.

"-o barrier" is a mount option? I do not find it documentated anywhere.

Any hints on why it may worked quite well with 2.6.15 but I got three 
corruptions in one week with 2.6.16? Just coincidence or was there some write 
cache related changes in 2.6.16? During 2.6.15 time I had quite some 3D savage 
DRI driver lockups without any data loss.

I lowered severity to normal as it seems from your comment that it is a 
misconfiguration on my side. Feel free to raise it again it is seems approbiate 
to you. When according to hdparm -i /dev/hda the drive seems to default to 
WriteCache enabled it may have severe implications. At least the default 
setting should never burn any data --> kernel 2.6.17 with -o barrier on by 
default.

I will test kernel 2.6.16.4 with write cache off in hdparm and when that works, 
I may try with barrier on and write cache on - I am still a bit scared ATM. 
When both works that bug can be closed. A hint in the xfs.txt readme would 
still be in order IMHO until 2.6.17 is standard.

Hard drive in my laptop:

root@deepdance:~ -> smartctl -i /dev/hda
smartctl version 5.34 [i686-pc-linux-gnu] Copyright (C) 2002-5 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Travelstar 5K80 family
Device Model:     HTS548060M9AT00
Serial Number:    MRLB21L4G6G3DC
Firmware Version: MGBOA50A
User Capacity:    60.011.642.880 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 3a
Local Time is:    Thu Apr 13 14:18:21 2006 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Regards,
Martin

How do I query the drive whether write caching is on or off to make sure that 
hdparm -W 0 /dev/hda or my entry in /etc/hdparm.conf actually worked the way it 
should? hdparm -W does not seem to be able to query the drive status regarding 
write cache. Any hints?

One additional thought: Can XFS detect common cases where write operation is 
dangerous (ATM only write cache on with barrier off comes to my mind) and do a 
readonly mount in that case issuing an error message explaining why it does so?

IMHO a filesystem should follow that better safe than sorry strategy where 
possible when it comes to the risk of data corruption. And at least here it 
shouldn't have any serious performance impact.

I tested about one week in total with write cache off and kernel 2.6.16 (with 
2.6.16.4 and 2.6.16.11). I had no further XFS crashes.

I am now using 2.6.15 again, cause ALSA sound output doesnt work after suspend 
to disk with 2.6.16 (different bug I know).

Just for information for readers of this bugreport:

<ul>
  <li>linux-xfs mailing list: <br /><a 
href="http://oss.sgi.com/archives/linux-xfs/2005-09/msg00044.html">TAKE 
912426 - write barrier support</a><br /><a 
href="http://oss.sgi.com/archives/linux-xfs/2005-11/msg00164.html">TAKE 
912426 - disable barriers by default</a></li>
  <li>linux-kernel mailing list: <br /> <a 
href="http://marc.theaimsgroup.com/?l=linux-kernel&m=113802736929484">[PATCH] 
enable XFS write barrier</a></li>
</ul>

There is a FAQ entry about the write cache issue available now in the XFS faq 
from SGI:

http://oss.sgi.com/projects/xfs/faq.html#wcache

I like to try it with kernel 2.6.17 and then also with enabled write caches 
again, but I want to wait a little bit longer until I switch to 2.6.17, since 
its rather new at the moment. (At all its a productively used system, no test 
machine).

Humm, can anybody explain or give some pointers about this write cache
effect?

I mean - I don't seem to understand how the corruption can occur
while having system *continuously* online.

I am assuming that after each 'event' fs was repaired and machine was rebooted.

[or maybe that isn't the case here?]

Thanks!

There is some kernel documentation about the write barrier stuff. I have no 
depacked kernel 2.6.16 at hand here currently, but you should be able to find 
it by using find -name "*barrier*" or grep -ir "barrier" *. That explains the 
issue quite nicely as does the SGI FAQ I posted before.

But actually even after I reading it I do not understand this issue completely.

I do not understand why I got three crashes with 2.6.16 in one week while with 
2.6.15 it worked quite stable. It was not perfect with 2.6.15, but at least I 
only got XFS corruption rarely after a DRI savage driver crash or when suspend 
to disk did not work correctly - when the machine was not online as you say. 
Actually XFS survived most of those crashes nicely. With 2.6.16 at least once - 
when I used kdissert - the kernel just went down while I was using the machine 
regurlarily (no 3D stuff and no suspend to disk issues). Even when kdissert / 
KDE somehow managed to crash X.org the kernel should still be alive and X.org 
should have been restarted. So either kernel 2.6.16 was a lot more unstable 
than 2.6.15 in the beginning or XFS had an issue with enabled write cache that 
happened while it was running and not only on power outages and kernel crashes. 
I had no kernel crashes while regular use with 2.6.16 when I disabled write 
cache what may point at the second alternative.

I repaired the filesystem after each event either by using xfs_repair or when 
damage was to big by replaying a backup via rsync.

Anyway I think its best to test with 2.6.17 again with barrier functionality 
and write cache enabled. I will do so once 2.6.17 matured a bit more and I do 
not hear about new issues, cause this is a production machine and I loose quite 
some time on each filesystem crash that happens.

Hello, Ok, I had a three week test period with 2.6.17.1 + the xfs-fix for 
kernel bug #6757 (that one is really needed and IMHO should go into a stable 
kernel patch as soon as possible!) + sws2 2.2.6. One week with disabled write 
caches, one week with enabled write caches and barrier mount option mentioned 
in /etc/fstab, one week with enabled write caches and barrier mount option not 
mentioned in /etc/fstab thus specifically testing whether its really the 
default now. 

No problems. xfs_check on the root partition showed three agi unlinked bucket 
that xfs_repair fixed but from what I know these are no real defects. If that 
shouldn't happen tough, something till needs to be fixed. Please tell me if 
thats the case.

I added to that three tests with switching off the computer while writing data 
to a XFS partition:

1) rsync -a /usr/src /destination/partition
2) ddrescue /dev/hda1 /destination/partition
3) 1 + 2 + rm -rf /that/usr/src/directory-from-test-one. The rm job was 
completed a second before I switched off the laptop, but I am sure that the 
other jobs were still running

Result: No problems. No single line of output in xfs_check after each of the 
three tests.

So I am pretty much convinced that XFS is working really stable now with write 
caches given that the patch from kernel bug #6757 which is unrelated to the 
write cache issue is applied.

Thank you, guys! 

Regards, Martin