Bug 5004

Summary: NFS slowdown problem
Product: File System Reporter: Heikki Orsila (heikki.orsila)
Component: NFSAssignee: Trond Myklebust (trondmy)
Status: REJECTED INVALID    
Severity: normal    
Priority: P2    
Hardware: i386   
OS: Linux   
Kernel Version: 2.6.13-rc5 Subsystem:
Regression: --- Bisected commit-id:

Description Heikki Orsila 2005-08-05 12:26:45 UTC
Most recent kernel where this bug did not occur: 2.6.12
Distribution: Gentoo
Hardware Environment:
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8385 [K8T800 AGP] Host Bridge
(rev 01)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800 South]
0000:00:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000
Controller (PHY/Link)
0000:00:0d.0 RAID bus controller: Silicon Image, Inc. (formerly CMD Technology
Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
0000:00:0e.0 Ethernet controller: 3Com Corporation 3c940 10/100/1000Base-T
[Marvell] (rev 12)
0000:00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID
Controller (rev 80)
0000:00:0f.1 IDE interface: VIA Technologies, Inc.
VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800
South]
0000:00:11.5 Multimedia audio controller: VIA Technologies, Inc.
VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
0000:01:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G400 AGP (rev 85)
Software Environment:
Linux e275b 2.6.13-rc5 #3 Wed Aug 3 16:39:51 EEST 2005 x86_64 AMD Athlon(tm) 64
Processor 3000+ AuthenticAMD GNU/Linux
 
Gnu C                  3.3.4
Gnu make               3.80
binutils               2.15.92.0.2
util-linux             2.12i
mount                  2.12i
module-init-tools      3.0
e2fsprogs              1.38
jfsutils               1.1.4
reiserfsprogs          line
reiser4progs           line
xfsprogs               2.6.25
nfs-utils              1.0.6
Linux C Library        2.3.5
Dynamic linker (ldd)   2.3.5
Procps                 3.2.5
Net-tools              1.60
Kbd                    1.12
Sh-utils               5.2.1
udev                   058
Modules Loaded         

Problem Description:
After seeing this:

 http://readlist.com/lists/vger.kernel.org/linux-kernel/26/132893.html

I immediately tested in my system. I didn't get an OOPS though, but a
significant slowdown on NFS operation has happened between 2.6.12 and
2.6.13-rc5. My gigabit network cards are different E1000 on the sending machine,
and SK98 (with new driver!) on the receiving NFS server.

Steps to reproduce:

Here's what I did:

$ dd if=/dev/zero of=zeros bs=1000000 count=256
$ tar cvf - zeros |pmr -t 1 |tcppipe - e275b 2345
zeros
bandwidth: 56.87 MiB/s  bytes: 59686912
bandwidth: 50.57 MiB/s  bytes: 112766976
bandwidth: 27.65 MiB/s  bytes: 141791232
bandwidth: 27.16 MiB/s  bytes: 170299392
bandwidth: 27.16 MiB/s  bytes: 198807552
bandwidth: 16.48 MiB/s  bytes: 216100864
bandwidth: 27.06 MiB/s  bytes: 244502528
average bandwidth: 32.94 MiB/s                       
total bytes: 256010240

So all fine, just sending a tar stream in TCP is efficient and fast. Works as it
used to work. However, now doing it over NFS:

$ tar cvf - zeros |pmr -t 1 |tar xvf - -C /path/to/nfs/
zeros
zeros
bandwidth: 132.92 MiB/s bytes: 139651072
bandwidth: 1.11 MiB/s   bytes: 215980032
average bandwidth: 3.65 MiB/s                        
total bytes: 256010240

The pmr process (which measures command line bandwidth) stalled for a long
period of time, which implies that the writing tar process blocked for a very
long time.

Reading from the nfs server gives 17 MiB/s so it is fine also.
Comment 1 Heikki Orsila 2005-08-05 12:42:43 UTC
One addition: I don't have CPU frequency scaling enabled.
Comment 2 Heikki Orsila 2005-08-05 13:27:25 UTC
Sorry, it seems the same thing happened already in 2.6.12. The slowness of
writing still bothers me, but I can't show that there is anything else than huge
lack of performance with writes compared to reads. Marking the bug as INVALID.
Comment 3 Heikki Orsila 2005-08-05 13:31:47 UTC
My performance problems were solved by mounting as NFS v3 rather than v2 (both
versions over TCP). Sorry again.