Bug 12564

Summary: poor performance while preprocessing source code
Product: File System Reporter: Steven Patrick (steven)
Component: NFSAssignee: io_other
Status: REJECTED INVALID    
Severity: normal CC: kernel
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.28.2 Subsystem:
Regression: Yes Bisected commit-id:
Attachments: distccmon-gnome snapshot from earlier (faster) kernel
distccmon-gnome snapshot from later (slower) kernel
the git bisect log
the kernel config file
the output from /proc/cpuinfo
output from the ver_linux script

Description Steven Patrick 2009-01-28 11:29:51 UTC
Latest working kernel version: 2.6.26-rc3
Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

Distribution: gentoo

Hardware Environment:
  Various 32 and 64 bit Intel and AMD machines with various
PATA and SATA disks and various network interfaces.

Problem Description:
  Here's my situation.  I've recently upgraded the kernels
on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
computers are used to build and test software we develop.  
We speed up the building process using distcc.  However,
after the kernel upgrade, the builds are much much slower.
The preprocessing stage seems to be at least 10 times
slower.
  As evidence of this slowdown I am attaching two images created
using distccmon-gnome.  Both snapshots were taken shortly 
after starting builds in a clean sandbox.  The only difference
is the kernel.  "fast.png" was generated while running
kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
The light purple sections indicate the preprocessing times
for each file.  
  This slowdown is observed on both 32 and 64 bit computers
and using either gcc or the intel compiler. (The intel compiler
builds do not use distcc, but that are also slower.)  Strangely 
enough, it's still faster to use an NFS mounted sandbox on a
machine with an older kernel than the same sandbox on the local
machine with a newer kernel.  (This suggests to me that it
is neither a disk or network IO problem.)
  I've used git bisect to narrow it down to a single commit:
# bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching
This is the first commit after v2.6.26-rc3.  I'm not an
experienced git user, so I don't know how to narrow it down
further.  Distccmon-gnome snapshots from the bisection
process are similar to the ones noted above.
  Naturally, I would like the newer kernels to have similar
performance to the older kernels.
  I will be attaching various items.  Let me know what other 
information might be helpful.

Steps to reproduce:
distccmon-gnome &
# using a makefile setup to use distcc:
make -j 5 all
# note preprocessing times in distccmon
Comment 1 Steven Patrick 2009-01-28 11:31:37 UTC
Created attachment 20033 [details]
distccmon-gnome snapshot from earlier (faster) kernel
Comment 2 Steven Patrick 2009-01-28 11:32:36 UTC
Created attachment 20034 [details]
distccmon-gnome snapshot from later (slower) kernel
Comment 3 Steven Patrick 2009-01-28 11:33:20 UTC
Created attachment 20035 [details]
the git bisect log
Comment 4 Steven Patrick 2009-01-28 11:36:19 UTC
Created attachment 20036 [details]
the kernel config file

  This configuration was used for both the v2.6.26-rc3
and the last few iterations in the bisection process.
Comment 5 Steven Patrick 2009-01-28 11:37:49 UTC
Created attachment 20037 [details]
the output from /proc/cpuinfo
Comment 6 Steven Patrick 2009-01-28 11:39:14 UTC
Created attachment 20038 [details]
output from the ver_linux script

  Note, this output was generated while running
kernel 2.6.25.20.
Comment 7 Andrew Morton 2009-01-28 13:30:13 UTC
Reassigned to NFS, marked as a regression.
Comment 8 Anonymous Emailer 2009-01-28 13:30:14 UTC
Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12564
> 
>            Summary: poor performance while preprocessing source code
>            Product: IO/Storage

Thanks for the report.

>            Version: 2.5
>      KernelVersion: 2.6.28.2
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: io_other@kernel-bugs.osdl.org
>         ReportedBy: steven@ngls.net
> 
> 
> Latest working kernel version: 2.6.26-rc3
> Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)
 
> Distribution: gentoo
> 
> Hardware Environment:
>   Various 32 and 64 bit Intel and AMD machines with various
> PATA and SATA disks and various network interfaces.
> 
> Problem Description:
>   Here's my situation.  I've recently upgraded the kernels
> on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> computers are used to build and test software we develop.  
> We speed up the building process using distcc.  However,
> after the kernel upgrade, the builds are much much slower.
> The preprocessing stage seems to be at least 10 times
> slower.
>   As evidence of this slowdown I am attaching two images created
> using distccmon-gnome.  Both snapshots were taken shortly 
> after starting builds in a clean sandbox.  The only difference
> is the kernel.  "fast.png" was generated while running
> kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> The light purple sections indicate the preprocessing times
> for each file.  
>   This slowdown is observed on both 32 and 64 bit computers
> and using either gcc or the intel compiler. (The intel compiler
> builds do not use distcc, but that are also slower.)  Strangely 
> enough, it's still faster to use an NFS mounted sandbox on a
> machine with an older kernel than the same sandbox on the local
> machine with a newer kernel.  (This suggests to me that it
> is neither a disk or network IO problem.)
>   I've used git bisect to narrow it down to a single commit:
> # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> and/or 'actimeo=0' turn off attribute caching

And thanks for bisecting it.

> This is the first commit after v2.6.26-rc3.  I'm not an
> experienced git user, so I don't know how to narrow it down
> further.  Distccmon-gnome snapshots from the bisection
> process are similar to the ones noted above.
>   Naturally, I would like the newer kernels to have similar
> performance to the older kernels.
>   I will be attaching various items.  Let me know what other 
> information might be helpful.
> 
> Steps to reproduce:
> distccmon-gnome &
> # using a makefile setup to use distcc:
> make -j 5 all
> # note preprocessing times in distccmon
> 

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to
play?  I mean, distcc kind-of replaces NFS in this workload.

Perhaps you could briefly describe the topology/data-flow/etc so that
we can see how NFS could be a bottleneck here?

Thanks.
Comment 9 Trond Myklebust 2009-01-28 13:58:50 UTC
If you are worried about performance, why are you using noac/actimeo=0?

The commit you point to fixed a bug in which the noac/actimeo=0 was using
cached metadata in situations where it should have been retrieving the data
from the server.
We fully expect a significant performance drop when the client is forced to send
more GETATTR requests to the server, and as I said, that was precisely the point
of this fix.
Comment 10 Jeff Layton 2009-01-28 14:05:41 UTC
On Wed, 28 Jan 2009 13:29:42 -0800
Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=12564
> > 
> >            Summary: poor performance while preprocessing source code
> >            Product: IO/Storage
> 
> Thanks for the report.
> 
> >            Version: 2.5
> >      KernelVersion: 2.6.28.2
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Other
> >         AssignedTo: io_other@kernel-bugs.osdl.org
> >         ReportedBy: steven@ngls.net
> > 
> > 
> > Latest working kernel version: 2.6.26-rc3
> > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)
> 
> (huge performance regression in NFS)
>  
> > Distribution: gentoo
> > 
> > Hardware Environment:
> >   Various 32 and 64 bit Intel and AMD machines with various
> > PATA and SATA disks and various network interfaces.
> > 
> > Problem Description:
> >   Here's my situation.  I've recently upgraded the kernels
> > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > computers are used to build and test software we develop.  
> > We speed up the building process using distcc.  However,
> > after the kernel upgrade, the builds are much much slower.
> > The preprocessing stage seems to be at least 10 times
> > slower.
> >   As evidence of this slowdown I am attaching two images created
> > using distccmon-gnome.  Both snapshots were taken shortly 
> > after starting builds in a clean sandbox.  The only difference
> > is the kernel.  "fast.png" was generated while running
> > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > The light purple sections indicate the preprocessing times
> > for each file.  
> >   This slowdown is observed on both 32 and 64 bit computers
> > and using either gcc or the intel compiler. (The intel compiler
> > builds do not use distcc, but that are also slower.)  Strangely 
> > enough, it's still faster to use an NFS mounted sandbox on a
> > machine with an older kernel than the same sandbox on the local
> > machine with a newer kernel.  (This suggests to me that it
> > is neither a disk or network IO problem.)
> >   I've used git bisect to narrow it down to a single commit:
> > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > and/or 'actimeo=0' turn off attribute caching
> 
> And thanks for bisecting it.
> 

It's a pretty small patch. The obvious question is: "What mount options
are you using on your NFS mounts?"


> > This is the first commit after v2.6.26-rc3.  I'm not an
> > experienced git user, so I don't know how to narrow it down
> > further.  Distccmon-gnome snapshots from the bisection
> > process are similar to the ones noted above.
> >   Naturally, I would like the newer kernels to have similar
> > performance to the older kernels.
> >   I will be attaching various items.  Let me know what other 
> > information might be helpful.
> > 
> > Steps to reproduce:
> > distccmon-gnome &
> > # using a makefile setup to use distcc:
> > make -j 5 all
> > # note preprocessing times in distccmon
> > 
> 
> Something I don't understand from this:
> 
> If your normal setup is using distcc then what role does NFS have to
> play?  I mean, distcc kind-of replaces NFS in this workload.
> 
> Perhaps you could briefly describe the topology/data-flow/etc so that
> we can see how NFS could be a bottleneck here?
> 
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Comment 11 Steven Patrick 2009-01-28 14:44:12 UTC
On Wed, 2009-01-28 at 13:29 -0800, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 28 Jan 2009 11:29:52 -0800 (PST)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=12564
> > 
> >            Summary: poor performance while preprocessing source code
> >            Product: IO/Storage
> 
> Thanks for the report.
> 
> >            Version: 2.5
> >      KernelVersion: 2.6.28.2
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Other
> >         AssignedTo: io_other@kernel-bugs.osdl.org
> >         ReportedBy: steven@ngls.net
> > 
> > 
> > Latest working kernel version: 2.6.26-rc3
> > Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)
> 
> (huge performance regression in NFS)
>  
> > Distribution: gentoo
> > 
> > Hardware Environment:
> >   Various 32 and 64 bit Intel and AMD machines with various
> > PATA and SATA disks and various network interfaces.
> > 
> > Problem Description:
> >   Here's my situation.  I've recently upgraded the kernels
> > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > computers are used to build and test software we develop.  
> > We speed up the building process using distcc.  However,
> > after the kernel upgrade, the builds are much much slower.
> > The preprocessing stage seems to be at least 10 times
> > slower.
> >   As evidence of this slowdown I am attaching two images created
> > using distccmon-gnome.  Both snapshots were taken shortly 
> > after starting builds in a clean sandbox.  The only difference
> > is the kernel.  "fast.png" was generated while running
> > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > The light purple sections indicate the preprocessing times
> > for each file.  
> >   This slowdown is observed on both 32 and 64 bit computers
> > and using either gcc or the intel compiler. (The intel compiler
> > builds do not use distcc, but that are also slower.)  Strangely 
> > enough, it's still faster to use an NFS mounted sandbox on a
> > machine with an older kernel than the same sandbox on the local
> > machine with a newer kernel.  (This suggests to me that it
> > is neither a disk or network IO problem.)
> >   I've used git bisect to narrow it down to a single commit:
> > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > and/or 'actimeo=0' turn off attribute caching
> 
> And thanks for bisecting it.

  I think it might be possible to bisect further, but I
don't know how get git to do it.  

> > This is the first commit after v2.6.26-rc3.  I'm not an
> > experienced git user, so I don't know how to narrow it down
> > further.  Distccmon-gnome snapshots from the bisection
> > process are similar to the ones noted above.
> >   Naturally, I would like the newer kernels to have similar
> > performance to the older kernels.
> >   I will be attaching various items.  Let me know what other 
> > information might be helpful.
> > 
> > Steps to reproduce:
> > distccmon-gnome &
> > # using a makefile setup to use distcc:
> > make -j 5 all
> > # note preprocessing times in distccmon
> > 
> 
> Something I don't understand from this:
> 
> If your normal setup is using distcc then what role does NFS have to
> play?  I mean, distcc kind-of replaces NFS in this workload.

  I don't believe that this actually has anything to do
with NFS.  The only way I could see NFS coming into this is the
fact that I use amd to automount home directories.
  The commit mentioned above modifies just over 100 files, very
few of which appear related to NFS.  I've attached the diifs.
It appears to be a number of fixes rolled up into one patch.
This is why I believe it might be possible to continue bisecting
with git.  But, to do that I suspect I would need to use a
different repository and/or tag or version names.
  BTW, the diffs were generated with "git diff v2.6.26-rc3"
after having done the bisection.  This is why it looks to me
like this one commit includes multiple updates.  Since all
bisection iterations were bad, the diffs against the good
version must be the diffs of the one commit.  Of course, I 
reserve the right to have made a mistake or not understand 
things completely.
  There was one other detail that seemed strange.  In the
last four bisection iterations, the kernel identified
itself as -rc2 instead of -rc3.  I assumed an update
accidentally changed it back.

> 
> Perhaps you could briefly describe the topology/data-flow/etc so that
> we can see how NFS could be a bottleneck here?
> 
> Thanks.
Comment 12 Anonymous Emailer 2009-01-28 15:04:41 UTC
Reply-To: akpm@linux-foundation.org

On Wed, 28 Jan 2009 14:43:37 -0800
Steven Patrick <steven@ngls.net> wrote:

> > > Problem Description:
> > >   Here's my situation.  I've recently upgraded the kernels
> > > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > > computers are used to build and test software we develop.  
> > > We speed up the building process using distcc.  However,
> > > after the kernel upgrade, the builds are much much slower.
> > > The preprocessing stage seems to be at least 10 times
> > > slower.
> > >   As evidence of this slowdown I am attaching two images created
> > > using distccmon-gnome.  Both snapshots were taken shortly 
> > > after starting builds in a clean sandbox.  The only difference
> > > is the kernel.  "fast.png" was generated while running
> > > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > > The light purple sections indicate the preprocessing times
> > > for each file.  
> > >   This slowdown is observed on both 32 and 64 bit computers
> > > and using either gcc or the intel compiler. (The intel compiler
> > > builds do not use distcc, but that are also slower.)  Strangely 
> > > enough, it's still faster to use an NFS mounted sandbox on a
> > > machine with an older kernel than the same sandbox on the local
> > > machine with a newer kernel.  (This suggests to me that it
> > > is neither a disk or network IO problem.)
> > >   I've used git bisect to narrow it down to a single commit:
> > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > > and/or 'actimeo=0' turn off attribute caching
> > 
> > And thanks for bisecting it.
> 
>   I think it might be possible to bisect further, but I
> don't know how get git to do it.  

OK..

> > > This is the first commit after v2.6.26-rc3.  I'm not an
> > > experienced git user, so I don't know how to narrow it down
> > > further.  Distccmon-gnome snapshots from the bisection
> > > process are similar to the ones noted above.
> > >   Naturally, I would like the newer kernels to have similar
> > > performance to the older kernels.
> > >   I will be attaching various items.  Let me know what other 
> > > information might be helpful.
> > > 
> > > Steps to reproduce:
> > > distccmon-gnome &
> > > # using a makefile setup to use distcc:
> > > make -j 5 all
> > > # note preprocessing times in distccmon
> > > 
> > 
> > Something I don't understand from this:
> > 
> > If your normal setup is using distcc then what role does NFS have to
> > play?  I mean, distcc kind-of replaces NFS in this workload.
> 
>   I don't believe that this actually has anything to do
> with NFS.  The only way I could see NFS coming into this is the
> fact that I use amd to automount home directories.
>   The commit mentioned above modifies just over 100 files, very
> few of which appear related to NFS.  I've attached the diifs.
> It appears to be a number of fixes rolled up into one patch.
> This is why I believe it might be possible to continue bisecting
> with git.  But, to do that I suspect I would need to use a
> different repository and/or tag or version names.
>   BTW, the diffs were generated with "git diff v2.6.26-rc3"
> after having done the bisection.  This is why it looks to me
> like this one commit includes multiple updates.  Since all
> bisection iterations were bad, the diffs against the good
> version must be the diffs of the one commit.  Of course, I 
> reserve the right to have made a mistake or not understand 
> things completely.
>   There was one other detail that seemed strange.  In the
> last four bisection iterations, the kernel identified
> itself as -rc2 instead of -rc3.  I assumed an update
> accidentally changed it back.

Heaven knows.

OK, "This is the first commit after v2.6.26-rc3", so NFS is innocent. 
Presumably your git bisect landed on a big merge commit.

I'm afaid I'm not the person who can tell you how to fix this - it
hasn't happened to me in a while, but I don't know what I did to
prevent it happening :(

Here's the diffstat from your diffs.txt.gz:

 arch/m68k/configs/multi_defconfig              | 1269 -------------------------
 b/MAINTAINERS                                  |    4 
 b/Makefile                                     |    2 
 b/arch/arm/common/locomo.c                     |   66 -
 b/arch/arm/kernel/armksyms.c                   |    2 
 b/arch/arm/kernel/arthur.c                     |    2 
 b/arch/arm/mach-omap1/board-palmte.c           |    2 
 b/arch/arm/mach-omap1/board-palmz71.c          |    2 
 b/arch/arm/mach-omap2/board-2430sdp.c          |    1 
 b/arch/arm/mach-omap2/board-apollon.c          |    1 
 b/arch/arm/mach-omap2/board-generic.c          |    1 
 b/arch/arm/mach-omap2/board-h4.c               |    1 
 b/arch/arm/mach-omap2/clock.c                  |    4 
 b/arch/arm/mach-omap2/clock34xx.h              |   21 
 b/arch/arm/mach-omap2/cm-regbits-34xx.h        |    1 
 b/arch/arm/mach-omap2/mailbox.c                |   25 
 b/arch/arm/mach-omap2/prm.h                    |    2 
 b/arch/arm/mach-orion5x/dns323-setup.c         |    2 
 b/arch/arm/mach-orion5x/kurobox_pro-setup.c    |    2 
 b/arch/arm/mach-pxa/colibri.c                  |    3 
 b/arch/arm/mach-pxa/spitz.c                    |    1 
 b/arch/arm/mm/proc-arm925.S                    |    2 
 b/arch/arm/mm/proc-arm926.S                    |    2 
 b/arch/arm/mm/proc-arm940.S                    |    2 
 b/arch/arm/mm/proc-arm946.S                    |    2 
 b/arch/arm/plat-omap/clock.c                   |   10 
 b/arch/arm/plat-omap/dma.c                     |    2 
 b/arch/arm/plat-omap/mailbox.c                 |    1 
 b/arch/blackfin/mach-bf527/boards/ezkit.c      |    2 
 b/arch/m68k/Kconfig                            |    9 
 b/arch/m68k/configs/amiga_defconfig            |  159 +--
 b/arch/m68k/configs/apollo_defconfig           |  140 --
 b/arch/m68k/configs/atari_defconfig            |  143 +-
 b/arch/m68k/configs/bvme6000_defconfig         |  138 --
 b/arch/m68k/configs/hp300_defconfig            |  142 +-
 b/arch/m68k/configs/mac_defconfig              |  144 +-
 b/arch/m68k/configs/mvme147_defconfig          |  138 --
 b/arch/m68k/configs/mvme16x_defconfig          |  138 --
 b/arch/m68k/configs/q40_defconfig              |  159 +--
 b/arch/m68k/configs/sun3_defconfig             |  140 --
 b/arch/m68k/configs/sun3x_defconfig            |  140 --
 b/arch/m68k/kernel/head.S                      |    2 
 b/arch/m68k/kernel/setup.c                     |   15 
 b/arch/powerpc/platforms/pasemi/misc.c         |    7 
 b/arch/sparc64/defconfig                       |   40 
 b/arch/sparc64/mm/init.c                       |    2 
 b/arch/x86/kernel/process.c                    |   36 
 b/arch/x86/mm/pat.c                            |    2 
 b/drivers/block/amiflop.c                      |    6 
 b/drivers/block/z2ram.c                        |    2 
 b/drivers/char/snsc_event.c                    |    2 
 b/drivers/char/vme_scc.c                       |    4 
 b/drivers/i2c/busses/i2c-amd756.c              |    2 
 b/drivers/i2c/busses/i2c-nforce2.c             |   28 
 b/drivers/i2c/chips/max6875.c                  |    3 
 b/drivers/i2c/i2c-core.c                       |   22 
 b/drivers/ide/legacy/macide.c                  |    3 
 b/drivers/input/keyboard/hilkbd.c              |    4 
 b/drivers/input/misc/hp_sdc_rtc.c              |    5 
 b/drivers/input/serio/hp_sdc_mlc.c             |    5 
 b/drivers/input/serio/q40kbd.c                 |    2 
 b/drivers/media/video/cs5345.c                 |    7 
 b/drivers/media/video/cs53l32a.c               |   10 
 b/drivers/media/video/cx18/cx18-i2c.c          |    9 
 b/drivers/media/video/cx25840/cx25840-core.c   |    7 
 b/drivers/media/video/et61x251/et61x251_core.c |    2 
 b/drivers/media/video/ivtv/ivtv-i2c.c          |   13 
 b/drivers/media/video/m52790.c                 |    9 
 b/drivers/media/video/msp3400-driver.c         |   17 
 b/drivers/media/video/saa7115.c                |   40 
 b/drivers/media/video/saa7127.c                |    9 
 b/drivers/media/video/saa717x.c                |    9 
 b/drivers/media/video/sn9c102/sn9c102_core.c   |    2 
 b/drivers/media/video/tuner-core.c             |   17 
 b/drivers/media/video/upd64031a.c              |    6 
 b/drivers/media/video/upd64083.c               |    6 
 b/drivers/media/video/vp27smpx.c               |    9 
 b/drivers/media/video/wm8739.c                 |    7 
 b/drivers/media/video/wm8775.c                 |    7 
 b/drivers/media/video/zc0301/zc0301_core.c     |    2 
 b/drivers/media/video/zoran_device.c           |    2 
 b/drivers/media/video/zoran_driver.c           |    2 
 b/drivers/net/82596.c                          |    7 
 b/drivers/net/apne.c                           |    3 
 b/drivers/net/mac89x0.c                        |    3 
 b/drivers/net/macmace.c                        |    3 
 b/drivers/net/sun3lance.c                      |    3 
 b/drivers/net/wireless/atmel.c                 |    2 
 b/drivers/video/Kconfig                        |    4 
 b/drivers/video/amifb.c                        |    4 
 b/drivers/video/dnfb.c                         |    3 
 b/drivers/video/hpfb.c                         |    2 
 b/drivers/video/pxafb.c                        |    5 
 b/fs/befs/endian.h                             |    2 
 b/fs/nfs/inode.c                               |    7 
 b/include/asm-arm/arch-omap/common.h           |    4 
 b/include/asm-arm/arch-omap/control.h          |    2 
 b/include/asm-arm/arch-omap/mmc.h              |   24 
 b/include/asm-arm/arch-sa1100/irqs.h           |    2 
 b/include/asm-arm/hardware/locomo.h            |   19 
 b/include/asm-m68k/bug.h                       |    4 
 b/include/asm-m68k/io.h                        |   44 
 b/include/asm-m68k/setup.h                     |    2 
 b/include/asm-m68k/uaccess.h                   |    6 
 b/include/linux/i2c.h                          |    7 
 b/include/linux/i2c/pcf857x.h                  |    3 
 106 files changed, 833 insertions(+), 2743 deletions(-)

Now, distcc has a long track record of tripping up bugs and regressions
in the networking code - it's very clever that way.  But it doesn't
look like there were any relevant networking changes in that window.

So I think the next step is to ask you to continue with (or redo) that
bisection search.

Is there someone who is more git-competent than I who could help with
this please?
Comment 13 Steven Patrick 2009-01-28 17:03:24 UTC
On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> 
> It's a pretty small patch. The obvious question is: "What mount
> options
> are you using on your NFS mounts?"
> 

  Well, I didn't think that I was using any NFS mounts.
But, it turns out I was in a way.  Apparently, amd does its top
level mounts with noac.  So, where this (using an automounted path):
    cd /h/spo/work
    make -j 5
would be slow.  This (using a non-automounted path):
    cd /home/spo/work
    make -j 5
showed the faster performance I was expecting.
  I will mark my bug as invalid and decide what, if anything,
I want to do about amd.
  Thanks for pointing me in the right direction.

Steven
Comment 14 Anonymous Emailer 2009-01-28 17:37:38 UTC
Reply-To: akpm@linux-foundation.org

On Wed, 28 Jan 2009 17:02:36 -0800
Steven Patrick <steven@ngls.net> wrote:

> On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> > 
> > It's a pretty small patch. The obvious question is: "What mount
> > options
> > are you using on your NFS mounts?"
> > 
> 
>   Well, I didn't think that I was using any NFS mounts.
> But, it turns out I was in a way.  Apparently, amd does its top
> level mounts with noac.  So, where this (using an automounted path):
>     cd /h/spo/work
>     make -j 5
> would be slow.  This (using a non-automounted path):
>     cd /home/spo/work
>     make -j 5
> showed the faster performance I was expecting.
>   I will mark my bug as invalid and decide what, if anything,
> I want to do about amd.
>   Thanks for pointing me in the right direction.
> 

This had me all mystified until I clicked on the bugzilla link:


:  ------- Comment  #9 From Trond Myklebust  2009-01-28 13:58:50  [reply] -------
: 
: If you are worried about performance, why are you using noac/actimeo=0?
: 
: The commit you point to fixed a bug in which the noac/actimeo=0 was
: using cached metadata in situations where it should have been
: retrieving the data from the server.  We fully expect a significant
: performance drop when the client is forced to send more GETATTR
: requests to the server, and as I said, that was precisely the point of
: this fix.

Fair enough.



However.  We surprised one user (yourself), and we will presumably
surprise others.  Or, worse, we will slow down people's stuff and they
won't even notice.

Should we perhaps warn people about this?  A mount-time printk telling
them that moac/actime=0 is slow?
Comment 15 Trond Myklebust 2009-01-29 05:28:28 UTC
On Wed, 2009-01-28 at 17:37 -0800, Andrew Morton wrote:
> On Wed, 28 Jan 2009 17:02:36 -0800
> Steven Patrick <steven@ngls.net> wrote:
> 
> > On Wed, 2009-01-28 at 17:05 -0500, Jeff Layton wrote:
> > > 
> > > It's a pretty small patch. The obvious question is: "What mount
> > > options
> > > are you using on your NFS mounts?"
> > > 
> > 
> >   Well, I didn't think that I was using any NFS mounts.
> > But, it turns out I was in a way.  Apparently, amd does its top
> > level mounts with noac.  So, where this (using an automounted path):
> >     cd /h/spo/work
> >     make -j 5
> > would be slow.  This (using a non-automounted path):
> >     cd /home/spo/work
> >     make -j 5
> > showed the faster performance I was expecting.
> >   I will mark my bug as invalid and decide what, if anything,
> > I want to do about amd.
> >   Thanks for pointing me in the right direction.
> > 
> 
> This had me all mystified until I clicked on the bugzilla link:
> 
> 
> :  ------- Comment  #9 From Trond Myklebust  2009-01-28 13:58:50  [reply]
> -------
> : 
> : If you are worried about performance, why are you using noac/actimeo=0?
> : 
> : The commit you point to fixed a bug in which the noac/actimeo=0 was
> : using cached metadata in situations where it should have been
> : retrieving the data from the server.  We fully expect a significant
> : performance drop when the client is forced to send more GETATTR
> : requests to the server, and as I said, that was precisely the point of
> : this fix.
> 
> Fair enough.
> 
> 
> 
> However.  We surprised one user (yourself), and we will presumably
> surprise others.  Or, worse, we will slow down people's stuff and they
> won't even notice.
> 
> Should we perhaps warn people about this?  A mount-time printk telling
> them that moac/actime=0 is slow?

No. The current nfs manpages (dated 2 November 2007) have a whole
section on data and metadata coherence and about the trade-offs
involved. That section is explicitly referenced in the description of
the 'noac' mount option. It explains that 'noac' is basically about
disabling caching in order to achieve approximate cache coherent
behaviour when multiple NFS clients need to access files that are being
modified frequently on the server.

Note that 'noac' has _never_ been intended as a default NFS mount
option. The kernel default is rather to cache attributes as aggressively
as possible for scalability and performance reasons.
All other NFS mount programs therefore require 'noac' to be explicitly
set by the user. I see no reason to punish those users with extra printk
spam.