Bug 14024 - %sys usage increase to 100%
Summary: %sys usage increase to 100%
Status: RESOLVED OBSOLETE
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Other (show other bugs)
Hardware: x86-64 Linux
: P1 blocking
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-20 09:51 UTC by Yohan Tordjman
Modified: 2013-12-10 16:48 UTC (History)
2 users (show)

See Also:
Kernel Version: 2.6.29.6 / 2.6.30.5 / 2.6.31-rc6-git4 / 31-rc7-git2
Subsystem:
Regression: No
Bisected commit-id:


Attachments
kernel config (7.32 KB, application/x-gzip)
2009-08-20 09:51 UTC, Yohan Tordjman
Details
cpuinfo (3.96 KB, text/plain)
2009-08-20 09:52 UTC, Yohan Tordjman
Details
munin cpu graph (34.75 KB, image/x-png)
2009-08-20 09:54 UTC, Yohan Tordjman
Details
munin memory graph (48.78 KB, image/x-png)
2009-08-20 09:54 UTC, Yohan Tordjman
Details
readprofile1 (93.15 KB, application/octet-stream)
2009-08-26 11:48 UTC, Yohan Tordjman
Details
readprofile2 (94.32 KB, application/octet-stream)
2009-08-26 11:48 UTC, Yohan Tordjman
Details
readprofile3 (108.88 KB, application/octet-stream)
2009-08-26 11:49 UTC, Yohan Tordjman
Details
readprofile4 (109.00 KB, application/octet-stream)
2009-08-26 11:49 UTC, Yohan Tordjman
Details
readprofile - 2009-08-27 - before drop_caches - delay 10s after a reset (28.63 KB, application/octet-stream)
2009-08-27 08:08 UTC, Yohan Tordjman
Details
readprofile - 2009-08-27 - before drop_caches - delay 60s after a reset (42.99 KB, application/octet-stream)
2009-08-27 08:08 UTC, Yohan Tordjman
Details
readprofile - 2009-08-27 - after drop_caches - delay 10s after a reset (28.46 KB, application/octet-stream)
2009-08-27 08:09 UTC, Yohan Tordjman
Details
readprofile - 2009-08-27 - after drop_caches - delay 60s after a reset (41.60 KB, application/octet-stream)
2009-08-27 08:09 UTC, Yohan Tordjman
Details
2009-09-03 - with the debug/profiling kernel (34.19 KB, image/x-png)
2009-09-03 13:31 UTC, Yohan Tordjman
Details
2009-09-03 - 60s readprofile (41.60 KB, text/plain)
2009-09-03 13:38 UTC, Yohan Tordjman
Details

Description Yohan Tordjman 2009-08-20 09:51:27 UTC
The system use xinetd to bind ~6K process, that works with many nfs mounts, and one USB card for some caching...

At normal load the %sys is ~5%, 
in 3/4 days with the same workload, its grows to 100%...

With a
  echo 3 > /proc/sys/vm/drop_caches

the %sys load goes to normal state (~5%)
Comment 1 Yohan Tordjman 2009-08-20 09:51:58 UTC
Created attachment 22784 [details]
kernel config
Comment 2 Yohan Tordjman 2009-08-20 09:52:16 UTC
Created attachment 22785 [details]
cpuinfo
Comment 3 Yohan Tordjman 2009-08-20 09:54:26 UTC
Created attachment 22786 [details]
munin cpu graph
Comment 4 Yohan Tordjman 2009-08-20 09:54:46 UTC
Created attachment 22787 [details]
munin memory graph
Comment 5 Yohan Tordjman 2009-08-20 09:57:16 UTC
Comment on attachment 22786 [details]
munin cpu graph

the down at 10h is when i did the drop_caches
Comment 6 Yohan Tordjman 2009-08-20 09:57:35 UTC
Comment on attachment 22787 [details]
munin memory graph

the down at 10h is when i did the drop_caches
Comment 7 Yohan Tordjman 2009-08-21 10:11:36 UTC
This morning, sys load is yet about ~65%

freeing pagecache (echo 1) does nothing

freeing dentries and inodes (echo 2) does the sys load goes down at ~5%
Comment 8 Yohan Tordjman 2009-08-26 11:48:35 UTC
Created attachment 22859 [details]
readprofile1
Comment 9 Yohan Tordjman 2009-08-26 11:48:52 UTC
Created attachment 22860 [details]
readprofile2
Comment 10 Yohan Tordjman 2009-08-26 11:49:07 UTC
Created attachment 22861 [details]
readprofile3
Comment 11 Yohan Tordjman 2009-08-26 11:49:21 UTC
Created attachment 22862 [details]
readprofile4
Comment 12 Yohan Tordjman 2009-08-26 11:51:02 UTC
(delay between profiles = 10mn)

readprofile1 & 2 normal after reboot

readprofile3 & 4 => 4 cores are fully used by the system...
Comment 13 Yohan Tordjman 2009-08-27 08:07:18 UTC
I post 4 new readprofiles of today 2009-08-27...
I cleaned the counters before getting the reports...

The firsts 2 are before drop_caches, 2 others are after...

It seems that rpcauth_lookup_credcache takes all the time...
Comment 14 Yohan Tordjman 2009-08-27 08:08:26 UTC
Created attachment 22874 [details]
readprofile - 2009-08-27 - before drop_caches - delay 10s after a reset
Comment 15 Yohan Tordjman 2009-08-27 08:08:48 UTC
Created attachment 22875 [details]
readprofile - 2009-08-27 - before drop_caches - delay 60s after a reset
Comment 16 Yohan Tordjman 2009-08-27 08:09:06 UTC
Created attachment 22876 [details]
readprofile - 2009-08-27 - after drop_caches - delay 10s after a reset
Comment 17 Yohan Tordjman 2009-08-27 08:09:25 UTC
Created attachment 22877 [details]
readprofile - 2009-08-27 - after drop_caches - delay 60s after a reset
Comment 18 Yohan Tordjman 2009-09-03 13:31:03 UTC
Created attachment 22986 [details]
2009-09-03 - with the debug/profiling kernel
Comment 19 Yohan Tordjman 2009-09-03 13:38:21 UTC
Created attachment 22987 [details]
2009-09-03 - 60s readprofile

Note You need to log in before you can comment on or make changes to this bug.