Bug 14024

Summary:	%sys usage increase to 100%
Product:	Memory Management	Reporter:	Yohan Tordjman (ytordjman)
Component:	Other	Assignee:	Andrew Morton (akpm)
Status:	RESOLVED OBSOLETE
Severity:	blocking	CC:	alan, ytordjman
Priority:	P1
Hardware:	x86-64
OS:	Linux
Kernel Version:	2.6.29.6 / 2.6.30.5 / 2.6.31-rc6-git4 / 31-rc7-git2	Subsystem:
Regression:	No	Bisected commit-id:
Attachments:	kernel config cpuinfo munin cpu graph munin memory graph readprofile1 readprofile2 readprofile3 readprofile4 readprofile - 2009-08-27 - before drop_caches - delay 10s after a reset readprofile - 2009-08-27 - before drop_caches - delay 60s after a reset readprofile - 2009-08-27 - after drop_caches - delay 10s after a reset readprofile - 2009-08-27 - after drop_caches - delay 60s after a reset 2009-09-03 - with the debug/profiling kernel 2009-09-03 - 60s readprofile

Description Yohan Tordjman 2009-08-20 09:51:27 UTC

The system use xinetd to bind ~6K process, that works with many nfs mounts, and one USB card for some caching...

At normal load the %sys is ~5%, 
in 3/4 days with the same workload, its grows to 100%...

With a
  echo 3 > /proc/sys/vm/drop_caches

the %sys load goes to normal state (~5%)

Comment 1 Yohan Tordjman 2009-08-20 09:51:58 UTC

Created attachment 22784 [details]
kernel config

Comment 2 Yohan Tordjman 2009-08-20 09:52:16 UTC

Created attachment 22785 [details]
cpuinfo

Comment 3 Yohan Tordjman 2009-08-20 09:54:26 UTC

Created attachment 22786 [details]
munin cpu graph

Comment 4 Yohan Tordjman 2009-08-20 09:54:46 UTC

Created attachment 22787 [details]
munin memory graph

Comment 5 Yohan Tordjman 2009-08-20 09:57:16 UTC

Comment on attachment 22786 [details]
munin cpu graph

the down at 10h is when i did the drop_caches

Comment 6 Yohan Tordjman 2009-08-20 09:57:35 UTC

Comment on attachment 22787 [details]
munin memory graph

the down at 10h is when i did the drop_caches

Comment 7 Yohan Tordjman 2009-08-21 10:11:36 UTC

This morning, sys load is yet about ~65%

freeing pagecache (echo 1) does nothing

freeing dentries and inodes (echo 2) does the sys load goes down at ~5%

Comment 8 Yohan Tordjman 2009-08-26 11:48:35 UTC

Created attachment 22859 [details]
readprofile1

Comment 9 Yohan Tordjman 2009-08-26 11:48:52 UTC

Created attachment 22860 [details]
readprofile2

Comment 10 Yohan Tordjman 2009-08-26 11:49:07 UTC

Created attachment 22861 [details]
readprofile3

Comment 11 Yohan Tordjman 2009-08-26 11:49:21 UTC

Created attachment 22862 [details]
readprofile4

Comment 12 Yohan Tordjman 2009-08-26 11:51:02 UTC

(delay between profiles = 10mn)

readprofile1 & 2 normal after reboot

readprofile3 & 4 => 4 cores are fully used by the system...

Comment 13 Yohan Tordjman 2009-08-27 08:07:18 UTC

I post 4 new readprofiles of today 2009-08-27...
I cleaned the counters before getting the reports...

The firsts 2 are before drop_caches, 2 others are after...

It seems that rpcauth_lookup_credcache takes all the time...

Comment 14 Yohan Tordjman 2009-08-27 08:08:26 UTC

Created attachment 22874 [details]
readprofile - 2009-08-27 - before drop_caches - delay 10s after a reset

Comment 15 Yohan Tordjman 2009-08-27 08:08:48 UTC

Created attachment 22875 [details]
readprofile - 2009-08-27 - before drop_caches - delay 60s after a reset

Comment 16 Yohan Tordjman 2009-08-27 08:09:06 UTC

Created attachment 22876 [details]
readprofile - 2009-08-27 - after drop_caches - delay 10s after a reset

Comment 17 Yohan Tordjman 2009-08-27 08:09:25 UTC

Created attachment 22877 [details]
readprofile - 2009-08-27 - after drop_caches - delay 60s after a reset

Comment 18 Yohan Tordjman 2009-09-03 13:31:03 UTC

Created attachment 22986 [details]
2009-09-03 - with the debug/profiling kernel

Comment 19 Yohan Tordjman 2009-09-03 13:38:21 UTC

Created attachment 22987 [details]
2009-09-03 - 60s readprofile