Bug 5004 - NFS slowdown problem
Summary: NFS slowdown problem
Status: REJECTED INVALID
Alias: None
Product: File System
Classification: Unclassified
Component: NFS (show other bugs)
Hardware: i386 Linux
: P2 normal
Assignee: Trond Myklebust
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-08-05 12:26 UTC by Heikki Orsila
Modified: 2005-08-05 13:31 UTC (History)
0 users

See Also:
Kernel Version: 2.6.13-rc5
Subsystem:
Regression: ---
Bisected commit-id:


Attachments

Description Heikki Orsila 2005-08-05 12:26:45 UTC
Most recent kernel where this bug did not occur: 2.6.12
Distribution: Gentoo
Hardware Environment:
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8385 [K8T800 AGP] Host Bridge
(rev 01)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800 South]
0000:00:07.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000
Controller (PHY/Link)
0000:00:0d.0 RAID bus controller: Silicon Image, Inc. (formerly CMD Technology
Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
0000:00:0e.0 Ethernet controller: 3Com Corporation 3c940 10/100/1000Base-T
[Marvell] (rev 12)
0000:00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID
Controller (rev 80)
0000:00:0f.1 IDE interface: VIA Technologies, Inc.
VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1
Controller (rev 81)
0000:00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800
South]
0000:00:11.5 Multimedia audio controller: VIA Technologies, Inc.
VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
DRAM Controller
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
0000:01:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G400 AGP (rev 85)
Software Environment:
Linux e275b 2.6.13-rc5 #3 Wed Aug 3 16:39:51 EEST 2005 x86_64 AMD Athlon(tm) 64
Processor 3000+ AuthenticAMD GNU/Linux
 
Gnu C                  3.3.4
Gnu make               3.80
binutils               2.15.92.0.2
util-linux             2.12i
mount                  2.12i
module-init-tools      3.0
e2fsprogs              1.38
jfsutils               1.1.4
reiserfsprogs          line
reiser4progs           line
xfsprogs               2.6.25
nfs-utils              1.0.6
Linux C Library        2.3.5
Dynamic linker (ldd)   2.3.5
Procps                 3.2.5
Net-tools              1.60
Kbd                    1.12
Sh-utils               5.2.1
udev                   058
Modules Loaded         

Problem Description:
After seeing this:

 http://readlist.com/lists/vger.kernel.org/linux-kernel/26/132893.html

I immediately tested in my system. I didn't get an OOPS though, but a
significant slowdown on NFS operation has happened between 2.6.12 and
2.6.13-rc5. My gigabit network cards are different E1000 on the sending machine,
and SK98 (with new driver!) on the receiving NFS server.

Steps to reproduce:

Here's what I did:

$ dd if=/dev/zero of=zeros bs=1000000 count=256
$ tar cvf - zeros |pmr -t 1 |tcppipe - e275b 2345
zeros
bandwidth: 56.87 MiB/s  bytes: 59686912
bandwidth: 50.57 MiB/s  bytes: 112766976
bandwidth: 27.65 MiB/s  bytes: 141791232
bandwidth: 27.16 MiB/s  bytes: 170299392
bandwidth: 27.16 MiB/s  bytes: 198807552
bandwidth: 16.48 MiB/s  bytes: 216100864
bandwidth: 27.06 MiB/s  bytes: 244502528
average bandwidth: 32.94 MiB/s                       
total bytes: 256010240

So all fine, just sending a tar stream in TCP is efficient and fast. Works as it
used to work. However, now doing it over NFS:

$ tar cvf - zeros |pmr -t 1 |tar xvf - -C /path/to/nfs/
zeros
zeros
bandwidth: 132.92 MiB/s bytes: 139651072
bandwidth: 1.11 MiB/s   bytes: 215980032
average bandwidth: 3.65 MiB/s                        
total bytes: 256010240

The pmr process (which measures command line bandwidth) stalled for a long
period of time, which implies that the writing tar process blocked for a very
long time.

Reading from the nfs server gives 17 MiB/s so it is fine also.
Comment 1 Heikki Orsila 2005-08-05 12:42:43 UTC
One addition: I don't have CPU frequency scaling enabled.
Comment 2 Heikki Orsila 2005-08-05 13:27:25 UTC
Sorry, it seems the same thing happened already in 2.6.12. The slowness of
writing still bothers me, but I can't show that there is anything else than huge
lack of performance with writes compared to reads. Marking the bug as INVALID.
Comment 3 Heikki Orsila 2005-08-05 13:31:47 UTC
My performance problems were solved by mounting as NFS v3 rather than v2 (both
versions over TCP). Sorry again.

Note You need to log in before you can comment on or make changes to this bug.