Latest working kernel version: 2.6.26-rc3 Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch) Distribution: gentoo Hardware Environment: Various 32 and 64 bit Intel and AMD machines with various PATA and SATA disks and various network interfaces. Problem Description: Here's my situation. I've recently upgraded the kernels on ~30 computers at work (from 2.6.21 to 2.6.27). These computers are used to build and test software we develop. We speed up the building process using distcc. However, after the kernel upgrade, the builds are much much slower. The preprocessing stage seems to be at least 10 times slower. As evidence of this slowdown I am attaching two images created using distccmon-gnome. Both snapshots were taken shortly after starting builds in a clean sandbox. The only difference is the kernel. "fast.png" was generated while running kernel 2.6.25.20. "slow.png" was generated with 2.6.26. The light purple sections indicate the preprocessing times for each file. This slowdown is observed on both 32 and 64 bit computers and using either gcc or the intel compiler. (The intel compiler builds do not use distcc, but that are also slower.) Strangely enough, it's still faster to use an NFS mounted sandbox on a machine with an older kernel than the same sandbox on the local machine with a newer kernel. (This suggests to me that it is neither a disk or network IO problem.) I've used git bisect to narrow it down to a single commit: # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching This is the first commit after v2.6.26-rc3. I'm not an experienced git user, so I don't know how to narrow it down further. Distccmon-gnome snapshots from the bisection process are similar to the ones noted above. Naturally, I would like the newer kernels to have similar performance to the older kernels. I will be attaching various items. Let me know what other information might be helpful. Steps to reproduce: distccmon-gnome & # using a makefile setup to use distcc: make -j 5 all # note preprocessing times in distccmon
Created attachment 20033 [details] distccmon-gnome snapshot from earlier (faster) kernel
Created attachment 20034 [details] distccmon-gnome snapshot from later (slower) kernel
Created attachment 20035 [details] the git bisect log
Created attachment 20036 [details] the kernel config file This configuration was used for both the v2.6.26-rc3 and the last few iterations in the bisection process.
Created attachment 20037 [details] the output from /proc/cpuinfo
Created attachment 20038 [details] output from the ver_linux script Note, this output was generated while running kernel 2.6.25.20.
Reassigned to NFS, marked as a regression.
Reply-To: akpm@linux-foundation.org (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Wed, 28 Jan 2009 11:29:52 -0800 (PST) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=12564 > > Summary: poor performance while preprocessing source code > Product: IO/Storage Thanks for the report. > Version: 2.5 > KernelVersion: 2.6.28.2 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Other > AssignedTo: io_other@kernel-bugs.osdl.org > ReportedBy: steven@ngls.net > > > Latest working kernel version: 2.6.26-rc3 > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch) (huge performance regression in NFS) > Distribution: gentoo > > Hardware Environment: > Various 32 and 64 bit Intel and AMD machines with various > PATA and SATA disks and various network interfaces. > > Problem Description: > Here's my situation. I've recently upgraded the kernels > on ~30 computers at work (from 2.6.21 to 2.6.27). These > computers are used to build and test software we develop. > We speed up the building process using distcc. However, > after the kernel upgrade, the builds are much much slower. > The preprocessing stage seems to be at least 10 times > slower. > As evidence of this slowdown I am attaching two images created > using distccmon-gnome. Both snapshots were taken shortly > after starting builds in a clean sandbox. The only difference > is the kernel. "fast.png" was generated while running > kernel 2.6.25.20. "slow.png" was generated with 2.6.26. > The light purple sections indicate the preprocessing times > for each file. > This slowdown is observed on both 32 and 64 bit computers > and using either gcc or the intel compiler. (The intel compiler > builds do not use distcc, but that are also slower.) Strangely > enough, it's still faster to use an NFS mounted sandbox on a > machine with an older kernel than the same sandbox on the local > machine with a newer kernel. (This suggests to me that it > is neither a disk or network IO problem.) > I've used git bisect to narrow it down to a single commit: > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' > and/or 'actimeo=0' turn off attribute caching And thanks for bisecting it. > This is the first commit after v2.6.26-rc3. I'm not an > experienced git user, so I don't know how to narrow it down > further. Distccmon-gnome snapshots from the bisection > process are similar to the ones noted above. > Naturally, I would like the newer kernels to have similar > performance to the older kernels. > I will be attaching various items. Let me know what other > information might be helpful. > > Steps to reproduce: > distccmon-gnome & > # using a makefile setup to use distcc: > make -j 5 all > # note preprocessing times in distccmon > Something I don't understand from this: If your normal setup is using distcc then what role does NFS have to play? I mean, distcc kind-of replaces NFS in this workload. Perhaps you could briefly describe the topology/data-flow/etc so that we can see how NFS could be a bottleneck here? Thanks.
If you are worried about performance, why are you using noac/actimeo=0? The commit you point to fixed a bug in which the noac/actimeo=0 was using cached metadata in situations where it should have been retrieving the data from the server. We fully expect a significant performance drop when the client is forced to send more GETATTR requests to the server, and as I said, that was precisely the point of this fix.
On Wed, 28 Jan 2009 13:29:42 -0800 Andrew Morton <akpm@linux-foundation.org> wrote: > > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Wed, 28 Jan 2009 11:29:52 -0800 (PST) > bugme-daemon@bugzilla.kernel.org wrote: > > > http://bugzilla.kernel.org/show_bug.cgi?id=12564 > > > > Summary: poor performance while preprocessing source code > > Product: IO/Storage > > Thanks for the report. > > > Version: 2.5 > > KernelVersion: 2.6.28.2 > > Platform: All > > OS/Version: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: Other > > AssignedTo: io_other@kernel-bugs.osdl.org > > ReportedBy: steven@ngls.net > > > > > > Latest working kernel version: 2.6.26-rc3 > > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch) > > (huge performance regression in NFS) > > > Distribution: gentoo > > > > Hardware Environment: > > Various 32 and 64 bit Intel and AMD machines with various > > PATA and SATA disks and various network interfaces. > > > > Problem Description: > > Here's my situation. I've recently upgraded the kernels > > on ~30 computers at work (from 2.6.21 to 2.6.27). These > > computers are used to build and test software we develop. > > We speed up the building process using distcc. However, > > after the kernel upgrade, the builds are much much slower. > > The preprocessing stage seems to be at least 10 times > > slower. > > As evidence of this slowdown I am attaching two images created > > using distccmon-gnome. Both snapshots were taken shortly > > after starting builds in a clean sandbox. The only difference > > is the kernel. "fast.png" was generated while running > > kernel 2.6.25.20. "slow.png" was generated with 2.6.26. > > The light purple sections indicate the preprocessing times > > for each file. > > This slowdown is observed on both 32 and 64 bit computers > > and using either gcc or the intel compiler. (The intel compiler > > builds do not use distcc, but that are also slower.) Strangely > > enough, it's still faster to use an NFS mounted sandbox on a > > machine with an older kernel than the same sandbox on the local > > machine with a newer kernel. (This suggests to me that it > > is neither a disk or network IO problem.) > > I've used git bisect to narrow it down to a single commit: > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' > > and/or 'actimeo=0' turn off attribute caching > > And thanks for bisecting it. > It's a pretty small patch. The obvious question is: "What mount options are you using on your NFS mounts?" > > This is the first commit after v2.6.26-rc3. I'm not an > > experienced git user, so I don't know how to narrow it down > > further. Distccmon-gnome snapshots from the bisection > > process are similar to the ones noted above. > > Naturally, I would like the newer kernels to have similar > > performance to the older kernels. > > I will be attaching various items. Let me know what other > > information might be helpful. > > > > Steps to reproduce: > > distccmon-gnome & > > # using a makefile setup to use distcc: > > make -j 5 all > > # note preprocessing times in distccmon > > > > Something I don't understand from this: > > If your normal setup is using distcc then what role does NFS have to > play? I mean, distcc kind-of replaces NFS in this workload. > > Perhaps you could briefly describe the topology/data-flow/etc so that > we can see how NFS could be a bottleneck here? > > Thanks. > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
On Wed, 2009-01-28 at 13:29 -0800, Andrew Morton wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Wed, 28 Jan 2009 11:29:52 -0800 (PST) > bugme-daemon@bugzilla.kernel.org wrote: > > > http://bugzilla.kernel.org/show_bug.cgi?id=12564 > > > > Summary: poor performance while preprocessing source code > > Product: IO/Storage > > Thanks for the report. > > > Version: 2.5 > > KernelVersion: 2.6.28.2 > > Platform: All > > OS/Version: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: Other > > AssignedTo: io_other@kernel-bugs.osdl.org > > ReportedBy: steven@ngls.net > > > > > > Latest working kernel version: 2.6.26-rc3 > > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch) > > (huge performance regression in NFS) > > > Distribution: gentoo > > > > Hardware Environment: > > Various 32 and 64 bit Intel and AMD machines with various > > PATA and SATA disks and various network interfaces. > > > > Problem Description: > > Here's my situation. I've recently upgraded the kernels > > on ~30 computers at work (from 2.6.21 to 2.6.27). These > > computers are used to build and test software we develop. > > We speed up the building process using distcc. However, > > after the kernel upgrade, the builds are much much slower. > > The preprocessing stage seems to be at least 10 times > > slower. > > As evidence of this slowdown I am attaching two images created > > using distccmon-gnome. Both snapshots were taken shortly > > after starting builds in a clean sandbox. The only difference > > is the kernel. "fast.png" was generated while running > > kernel 2.6.25.20. "slow.png" was generated with 2.6.26. > > The light purple sections indicate the preprocessing times > > for each file. > > This slowdown is observed on both 32 and 64 bit computers > > and using either gcc or the intel compiler. (The intel compiler > > builds do not use distcc, but that are also slower.) Strangely > > enough, it's still faster to use an NFS mounted sandbox on a > > machine with an older kernel than the same sandbox on the local > > machine with a newer kernel. (This suggests to me that it > > is neither a disk or network IO problem.) > > I've used git bisect to narrow it down to a single commit: > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' > > and/or 'actimeo=0' turn off attribute caching > > And thanks for bisecting it. I think it might be possible to bisect further, but I don't know how get git to do it. > > This is the first commit after v2.6.26-rc3. I'm not an > > experienced git user, so I don't know how to narrow it down > > further. Distccmon-gnome snapshots from the bisection > > process are similar to the ones noted above. > > Naturally, I would like the newer kernels to have similar > > performance to the older kernels. > > I will be attaching various items. Let me know what other > > information might be helpful. > > > > Steps to reproduce: > > distccmon-gnome & > > # using a makefile setup to use distcc: > > make -j 5 all > > # note preprocessing times in distccmon > > > > Something I don't understand from this: > > If your normal setup is using distcc then what role does NFS have to > play? I mean, distcc kind-of replaces NFS in this workload. I don't believe that this actually has anything to do with NFS. The only way I could see NFS coming into this is the fact that I use amd to automount home directories. The commit mentioned above modifies just over 100 files, very few of which appear related to NFS. I've attached the diifs. It appears to be a number of fixes rolled up into one patch. This is why I believe it might be possible to continue bisecting with git. But, to do that I suspect I would need to use a different repository and/or tag or version names. BTW, the diffs were generated with "git diff v2.6.26-rc3" after having done the bisection. This is why it looks to me like this one commit includes multiple updates. Since all bisection iterations were bad, the diffs against the good version must be the diffs of the one commit. Of course, I reserve the right to have made a mistake or not understand things completely. There was one other detail that seemed strange. In the last four bisection iterations, the kernel identified itself as -rc2 instead of -rc3. I assumed an update accidentally changed it back. > > Perhaps you could briefly describe the topology/data-flow/etc so that > we can see how NFS could be a bottleneck here? > > Thanks.
Reply-To: akpm@linux-foundation.org On Wed, 28 Jan 2009 14:43:37 -0800 Steven Patrick <steven@ngls.net> wrote: > > > Problem Description: > > > Here's my situation. I've recently upgraded the kernels > > > on ~30 computers at work (from 2.6.21 to 2.6.27). These > > > computers are used to build and test software we develop. > > > We speed up the building process using distcc. However, > > > after the kernel upgrade, the builds are much much slower. > > > The preprocessing stage seems to be at least 10 times > > > slower. > > > As evidence of this slowdown I am attaching two images created > > > using distccmon-gnome. Both snapshots were taken shortly > > > after starting builds in a clean sandbox. The only difference > > > is the kernel. "fast.png" was generated while running > > > kernel 2.6.25.20. "slow.png" was generated with 2.6.26. > > > The light purple sections indicate the preprocessing times > > > for each file. > > > This slowdown is observed on both 32 and 64 bit computers > > > and using either gcc or the intel compiler. (The intel compiler > > > builds do not use distcc, but that are also slower.) Strangely > > > enough, it's still faster to use an NFS mounted sandbox on a > > > machine with an older kernel than the same sandbox on the local > > > machine with a newer kernel. (This suggests to me that it > > > is neither a disk or network IO problem.) > > > I've used git bisect to narrow it down to a single commit: > > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' > > > and/or 'actimeo=0' turn off attribute caching > > > > And thanks for bisecting it. > > I think it might be possible to bisect further, but I > don't know how get git to do it. OK.. > > > This is the first commit after v2.6.26-rc3. I'm not an > > > experienced git user, so I don't know how to narrow it down > > > further. Distccmon-gnome snapshots from the bisection > > > process are similar to the ones noted above. > > > Naturally, I would like the newer kernels to have similar > > > performance to the older kernels. > > > I will be attaching various items. Let me know what other > > > information might be helpful. > > > > > > Steps to reproduce: > > > distccmon-gnome & > > > # using a makefile setup to use distcc: > > > make -j 5 all > > > # note preprocessing times in distccmon > > > > > > > Something I don't understand from this: > > > > If your normal setup is using distcc then what role does NFS have to > > play? I mean, distcc kind-of replaces NFS in this workload. > > I don't believe that this actually has anything to do > with NFS. The only way I could see NFS coming into this is the > fact that I use amd to automount home directories. > The commit mentioned above modifies just over 100 files, very > few of which appear related to NFS. I've attached the diifs. > It appears to be a number of fixes rolled up into one patch. > This is why I believe it might be possible to continue bisecting > with git. But, to do that I suspect I would need to use a > different repository and/or tag or version names. > BTW, the diffs were generated with "git diff v2.6.26-rc3" > after having done the bisection. This is why it looks to me > like this one commit includes multiple updates. Since all > bisection iterations were bad, the diffs against the good > version must be the diffs of the one commit. Of course, I > reserve the right to have made a mistake or not understand > things completely. > There was one other detail that seemed strange. In the > last four bisection iterations, the kernel identified > itself as -rc2 instead of -rc3. I assumed an update > accidentally changed it back. Heaven knows. OK, "This is the first commit after v2.6.26-rc3", so NFS is innocent. Presumably your git bisect landed on a big merge commit. I'm afaid I'm not the person who can tell you how to fix this - it hasn't happened to me in a while, but I don't know what I did to prevent it happening :( Here's the diffstat from your diffs.txt.gz: arch/m68k/configs/multi_defconfig | 1269 ------------------------- b/MAINTAINERS | 4 b/Makefile | 2 b/arch/arm/common/locomo.c | 66 - b/arch/arm/kernel/armksyms.c | 2 b/arch/arm/kernel/arthur.c | 2 b/arch/arm/mach-omap1/board-palmte.c | 2 b/arch/arm/mach-omap1/board-palmz71.c | 2 b/arch/arm/mach-omap2/board-2430sdp.c | 1 b/arch/arm/mach-omap2/board-apollon.c | 1 b/arch/arm/mach-omap2/board-generic.c | 1 b/arch/arm/mach-omap2/board-h4.c | 1 b/arch/arm/mach-omap2/clock.c | 4 b/arch/arm/mach-omap2/clock34xx.h | 21 b/arch/arm/mach-omap2/cm-regbits-34xx.h | 1 b/arch/arm/mach-omap2/mailbox.c | 25 b/arch/arm/mach-omap2/prm.h | 2 b/arch/arm/mach-orion5x/dns323-setup.c | 2 b/arch/arm/mach-orion5x/kurobox_pro-setup.c | 2 b/arch/arm/mach-pxa/colibri.c | 3 b/arch/arm/mach-pxa/spitz.c | 1 b/arch/arm/mm/proc-arm925.S | 2 b/arch/arm/mm/proc-arm926.S | 2 b/arch/arm/mm/proc-arm940.S | 2 b/arch/arm/mm/proc-arm946.S | 2 b/arch/arm/plat-omap/clock.c | 10 b/arch/arm/plat-omap/dma.c | 2 b/arch/arm/plat-omap/mailbox.c | 1 b/arch/blackfin/mach-bf527/boards/ezkit.c | 2 b/arch/m68k/Kconfig | 9 b/arch/m68k/configs/amiga_defconfig | 159 +-- b/arch/m68k/configs/apollo_defconfig | 140 -- b/arch/m68k/configs/atari_defconfig | 143 +- b/arch/m68k/configs/bvme6000_defconfig | 138 -- b/arch/m68k/configs/hp300_defconfig | 142 +- b/arch/m68k/configs/mac_defconfig | 144 +- b/arch/m68k/configs/mvme147_defconfig | 138 -- b/arch/m68k/configs/mvme16x_defconfig | 138 -- b/arch/m68k/configs/q40_defconfig | 159 +-- b/arch/m68k/configs/sun3_defconfig | 140 -- b/arch/m68k/configs/sun3x_defconfig | 140 -- b/arch/m68k/kernel/head.S | 2 b/arch/m68k/kernel/setup.c | 15 b/arch/powerpc/platforms/pasemi/misc.c | 7 b/arch/sparc64/defconfig | 40 b/arch/sparc64/mm/init.c | 2 b/arch/x86/kernel/process.c | 36 b/arch/x86/mm/pat.c | 2 b/drivers/block/amiflop.c | 6 b/drivers/block/z2ram.c | 2 b/drivers/char/snsc_event.c | 2 b/drivers/char/vme_scc.c | 4 b/drivers/i2c/busses/i2c-amd756.c | 2 b/drivers/i2c/busses/i2c-nforce2.c | 28 b/drivers/i2c/chips/max6875.c | 3 b/drivers/i2c/i2c-core.c | 22 b/drivers/ide/legacy/macide.c | 3 b/drivers/input/keyboard/hilkbd.c | 4 b/drivers/input/misc/hp_sdc_rtc.c | 5 b/drivers/input/serio/hp_sdc_mlc.c | 5 b/drivers/input/serio/q40kbd.c | 2 b/drivers/media/video/cs5345.c | 7 b/drivers/media/video/cs53l32a.c | 10 b/drivers/media/video/cx18/cx18-i2c.c | 9 b/drivers/media/video/cx25840/cx25840-core.c | 7 b/drivers/media/video/et61x251/et61x251_core.c | 2 b/drivers/media/video/ivtv/ivtv-i2c.c | 13 b/drivers/media/video/m52790.c | 9 b/drivers/media/video/msp3400-driver.c | 17 b/drivers/media/video/saa7115.c | 40 b/drivers/media/video/saa7127.c | 9 b/drivers/media/video/saa717x.c | 9 b/drivers/media/video/sn9c102/sn9c102_core.c | 2 b/drivers/media/video/tuner-core.c | 17 b/drivers/media/video/upd64031a.c | 6 b/drivers/media/video/upd64083.c | 6 b/drivers/media/video/vp27smpx.c | 9 b/drivers/media/video/wm8739.c | 7 b/drivers/media/video/wm8775.c | 7 b/drivers/media/video/zc0301/zc0301_core.c | 2 b/drivers/media/video/zoran_device.c | 2 b/drivers/media/video/zoran_driver.c | 2 b/drivers/net/82596.c | 7 b/drivers/net/apne.c | 3 b/drivers/net/mac89x0.c | 3 b/drivers/net/macmace.c | 3 b/drivers/net/sun3lance.c | 3 b/drivers/net/wireless/atmel.c | 2 b/drivers/video/Kconfig | 4 b/drivers/video/amifb.c | 4 b/drivers/video/dnfb.c | 3 b/drivers/video/hpfb.c | 2 b/drivers/video/pxafb.c | 5 b/fs/befs/endian.h | 2 b/fs/nfs/inode.c | 7 b/include/asm-arm/arch-omap/common.h | 4 b/include/asm-arm/arch-omap/control.h | 2 b/include/asm-arm/arch-omap/mmc.h | 24 b/include/asm-arm/arch-sa1100/irqs.h | 2 b/include/asm-arm/hardware/locomo.h | 19 b/include/asm-m68k/bug.h | 4 b/include/asm-m68k/io.h | 44 b/include/asm-m68k/setup.h | 2 b/include/asm-m68k/uaccess.h | 6 b/include/linux/i2c.h | 7 b/include/linux/i2c/pcf857x.h | 3 106 files changed, 833 insertions(+), 2743 deletions(-) Now, distcc has a long track record of tripping up bugs and regressions in the networking code - it's very clever that way. But it doesn't look like there were any relevant networking changes in that window. So I think the next step is to ask you to continue with (or redo) that bisection search. Is there someone who is more git-competent than I who could help with this please?
On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote: > > It's a pretty small patch. The obvious question is: "What mount > options > are you using on your NFS mounts?" > Well, I didn't think that I was using any NFS mounts. But, it turns out I was in a way. Apparently, amd does its top level mounts with noac. So, where this (using an automounted path): cd /h/spo/work make -j 5 would be slow. This (using a non-automounted path): cd /home/spo/work make -j 5 showed the faster performance I was expecting. I will mark my bug as invalid and decide what, if anything, I want to do about amd. Thanks for pointing me in the right direction. Steven
Reply-To: akpm@linux-foundation.org On Wed, 28 Jan 2009 17:02:36 -0800 Steven Patrick <steven@ngls.net> wrote: > On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote: > > > > It's a pretty small patch. The obvious question is: "What mount > > options > > are you using on your NFS mounts?" > > > > Well, I didn't think that I was using any NFS mounts. > But, it turns out I was in a way. Apparently, amd does its top > level mounts with noac. So, where this (using an automounted path): > cd /h/spo/work > make -j 5 > would be slow. This (using a non-automounted path): > cd /home/spo/work > make -j 5 > showed the faster performance I was expecting. > I will mark my bug as invalid and decide what, if anything, > I want to do about amd. > Thanks for pointing me in the right direction. > This had me all mystified until I clicked on the bugzilla link: : ------- Comment #9 From Trond Myklebust 2009-01-28 13:58:50 [reply] ------- : : If you are worried about performance, why are you using noac/actimeo=0? : : The commit you point to fixed a bug in which the noac/actimeo=0 was : using cached metadata in situations where it should have been : retrieving the data from the server. We fully expect a significant : performance drop when the client is forced to send more GETATTR : requests to the server, and as I said, that was precisely the point of : this fix. Fair enough. However. We surprised one user (yourself), and we will presumably surprise others. Or, worse, we will slow down people's stuff and they won't even notice. Should we perhaps warn people about this? A mount-time printk telling them that moac/actime=0 is slow?
On Wed, 2009-01-28 at 17:37 -0800, Andrew Morton wrote: > On Wed, 28 Jan 2009 17:02:36 -0800 > Steven Patrick <steven@ngls.net> wrote: > > > On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote: > > > > > > It's a pretty small patch. The obvious question is: "What mount > > > options > > > are you using on your NFS mounts?" > > > > > > > Well, I didn't think that I was using any NFS mounts. > > But, it turns out I was in a way. Apparently, amd does its top > > level mounts with noac. So, where this (using an automounted path): > > cd /h/spo/work > > make -j 5 > > would be slow. This (using a non-automounted path): > > cd /home/spo/work > > make -j 5 > > showed the faster performance I was expecting. > > I will mark my bug as invalid and decide what, if anything, > > I want to do about amd. > > Thanks for pointing me in the right direction. > > > > This had me all mystified until I clicked on the bugzilla link: > > > : ------- Comment #9 From Trond Myklebust 2009-01-28 13:58:50 [reply] > ------- > : > : If you are worried about performance, why are you using noac/actimeo=0? > : > : The commit you point to fixed a bug in which the noac/actimeo=0 was > : using cached metadata in situations where it should have been > : retrieving the data from the server. We fully expect a significant > : performance drop when the client is forced to send more GETATTR > : requests to the server, and as I said, that was precisely the point of > : this fix. > > Fair enough. > > > > However. We surprised one user (yourself), and we will presumably > surprise others. Or, worse, we will slow down people's stuff and they > won't even notice. > > Should we perhaps warn people about this? A mount-time printk telling > them that moac/actime=0 is slow? No. The current nfs manpages (dated 2 November 2007) have a whole section on data and metadata coherence and about the trade-offs involved. That section is explicitly referenced in the description of the 'noac' mount option. It explains that 'noac' is basically about disabling caching in order to achieve approximate cache coherent behaviour when multiple NFS clients need to access files that are being modified frequently on the server. Note that 'noac' has _never_ been intended as a default NFS mount option. The kernel default is rather to cache attributes as aggressively as possible for scalability and performance reasons. All other NFS mount programs therefore require 'noac' to be explicitly set by the user. I see no reason to punish those users with extra printk spam.