Subject : Why is kslowd accumulating so much CPU time?
Submitter : "Theodore Ts'o" <email@example.com>
Date : 2010-06-09 18:36
Message-ID : E1OMQ88-0002a1-Gb@closure.thunk.org
References : http://marc.info/?l=linux-kernel&m=127610857819033&w=4
This entry is being used for tracking a regression from 2.6.. Please don't
close it until the problem is fixed in the mainline.
Author: Dave Airlie <firstname.lastname@example.org>
Date: Tue Jun 1 09:09:06 2010 +1000
drm/kms: disable/enable poll around switcheroo on/off
Because we aren't in a suspend state the poll will still run when we have switcherooed a card off.
Signed-off-by: Dave Airlie <email@example.com>
First-Bad-Commit : fbf81762e385d3d45acad057b654d56972acf58c
On Friday, July 23, 2010, Nick Bowler wrote:
> On 13:47 Fri 23 Jul , Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a summary report
> > of recent regressions.
> > The following bug entry is on the current list of known regressions
> > from 2.6.34. Please verify if it still should be listed and let the
> tracking team
> > know (either way).
> > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=16265
> > Subject : Why is kslowd accumulating so much CPU time?
> > Submitter : Theodore Ts'o <firstname.lastname@example.org>
> > Date : 2010-06-09 18:36 (45 days old)
> I actually haven't been able to reproduce with kernels since a couple
> weeks ago (still reproducible if I go back to 2.6.35-rc3, though).
> Seems to be fixed somehow, can anyone else confirm?
I'm still experiencing what I believe to be this bug on 2.6.35-rc6. The mouse stalls every few seconds and the kslowd threads use a lot of cpu when the mouse stalls. The problem started after upgrading from 2.6.34.
I'm using an intel 965GM connected with HDMI.
I'm having this problem with kernel 184.108.40.206 in archlinux. I'm willing to provide debugging information, I just don't know what should I post. I have a lenovo sl300 laptop with intel integrated graphics (HD4500 I believe).
Ok, we understand what's going on here now. Every 10s the non-hotplug capable outputs are polled for connection/disconnection events. For a certain class of analog device on some hardware this is very slow and CPU intensive.
See also https://bugs.freedesktop.org/show_bug?id=29536
The first workaround proposed is just to disable polling via a module parameter. The eventual solution will likely be a mix of finer grained locking and cheaper, non-destructive polling.
PS: The link is http://bugs.freedesktop.org/show_bug.cgi?id=29536
Handled-By: Chris Wilson <email@example.com>
The "automatic workaround" does not work for me. I needed to set the
module parameter to "disable polling". There is a similar bug
Bug 18802 kworker: high CPU usage -> system sluggish
*** Bug 18802 has been marked as a duplicate of this bug. ***
Is this issue still visible on 2.6.37?
I'm closing this for now, please reopen or shout if this issue is still unresolved for you.
This should have been fixed for intel since 2.6.36-rc5:
Author: Chris Wilson <firstname.lastname@example.org>
Date: Tue Sep 14 11:07:23 2010 +0100
drm: Use a nondestructive mode for output detect when polling (v2)
for nouveau it's probably fixed since v2.6.37-rc3:
Author: Francisco Jerez <email@example.com>
Date: Thu Oct 21 17:43:08 2010 +0200
drm/nouveau: Use "force" to decide if analog load detection is ok or not.
and for radeon since v2.6.37-rc1:
Author: Dave Airlie <firstname.lastname@example.org>
Date: Tue Oct 26 12:55:52 2010 +1000
drm/radeon/kms: don't poll dac load detect.
This issue is still present for me with Intel 4500 MHD (intel driver 2.14.0, xorg-server 220.127.116.111, mesa 7.9.1, libdrm 2.4.23) with gentoo-sources 2.6.37. I've read this announcement and disabled my drm_kms_helper.poll=N kernel boot param but the issue has come up after few minutes after booting. System was sluggish and kworker process appeared eating 60% CPU.
(In reply to comment #11)
> This issue is still present for me with Intel 4500 MHD (intel driver 2.14.0,
> xorg-server 18.104.22.1681, mesa 7.9.1, libdrm 2.4.23) with gentoo-sources 2.6.37.
> I've read this announcement and disabled my drm_kms_helper.poll=N kernel boot
> param but the issue has come up after few minutes after booting. System was
> sluggish and kworker process appeared eating 60% CPU.
Different culprit - you neither have the affected hardware nor the responsible hotplug polling enabled.
I have been redirected here from https://bugzilla.kernel.org/show_bug.cgi?id=18802 so it is probably not a duplicate of this bug because the workaround there did work for me.
You write in comment #11 about 'disabling the boot drm_kms_helper.poll=N'...
Did you mean with that, that the problem is in 2.6.37 (final) still present and only solved by putting drm_kms_helper.poll=N on the commandline...
Even with polling disabled via the commandline drm_kms_helper.poll=N you experience the symptoms?
I've had this drm_kms_helper.poll=N kernel workaround in my grub since I've discovered it in 18802 bug (around 2.6.36-r2 i think) because I've suffered by the udev drm polling storm in 2.6.34, kslowd in 2.6.35 and kworker in 2.6.36. Since I've applied it everything was working smooth. I read the announcement about bug 18802 being duplicate of this bug and it being resolved as of 2.6.36-rc5 so I have removed this kernel setting and rebooted with my 2.6.37 kernel hoping that everything should be fine. But in few minutes the same symptoms appeared so I reverted back to drm_kms_helper.poll=N kernel option and everything seems all right again (with the workaround). I can provide more info on my HW or SW by request.
You need to quantify your observations here. When polling is enabled, then every 10s the worker will be awoken and consume CPU time checking that all the displays are the same as before. Unless on your machine it takes a ridiculous amount of CPU time (say greater than 5%), it is just the cost of the ability for your computer to automatically detect display changes.
The real bug is the latency it induces... But I digress.
It doesn't seem to me as sane to consume this amount of CPU time to poll for anything (see attached screenshot). I figured a way how to reproduce this, I've enabled RandR polling in KDE, attached LCD TV via HDMI and played a video in VLC. After few mouse clicks in the menu of VLC I've witnessed jerky mouse movement and later my X has frozen completely. I've switched to console and found out that when I unpluged HDMI the kworker process went quiet.
I was able to take one screenshot but as my X froze I saw 3 kworker processes running, eating all CPU time rendering my system useless (no chance to take screenshot at that moment).
I repeat I'm running latest gentoo-sources-2.6.37.
Created attachment 44072 [details]
kslowd in 2.6.37
You should definetely reopen this as I still have issues on 2.6.38. Will provide anything to help you resolve it. See following attachement...
Created attachment 53912 [details]
kworker polling hogs CPU on 2.6.38
kworker polling hogs CPU on 2.6.38
HP ProBook 4510s
00:02.0 VGA compatible controller : Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07) (prog-if 00 [VGA controller])
Subsystem: Hewlett-Packard Company Device [103c:3072]
Flags: bus master, fast devsel, latency 0, IRQ 44
Memory at d0000000 (64-bit, non-prefetchable) [size=4M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
I/O ports at 50f0 [size=8]
Expansion ROM at <unassigned> [disabled]
Capabilities:  MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 3
Kernel driver in use: i915
Attached external LCD TV at 720p via HDMI cable
Please do a "perf top" to see what tasklet is actually being run by kslowd.
This is while kslowd consumed approx. 75% CPU:
PerfTop: 0 irqs/sec kernel:-nan% exact: -nan% [1000Hz cycles], (all, 2 CPUs)
3215.00 86.8% delay_tsc /lib/modules/2.6.38-ck-BKL/build/vmlinux
163.00 4.4% read_hpet /lib/modules/2.6.38-ck-BKL/build/vmlinux
110.00 3.0% set_clock /lib/modules/2.6.38-ck-BKL/build/vmlinux
55.00 1.5% get_clock /lib/modules/2.6.38-ck-BKL/build/vmlinux
45.00 1.2% get_data /lib/modules/2.6.38-ck-BKL/build/vmlinux
28.00 0.8% rb_next /lib/modules/2.6.38-ck-BKL/build/vmlinux
7.00 0.2% _raw_spin_lock_irqsave /lib/modules/2.6.38-ck-BKL/build/vmlinux
7.00 0.2% hpet_legacy_next_event /lib/modules/2.6.38-ck-BKL/build/vmlinux
7.00 0.2% set_data /lib/modules/2.6.38-ck-BKL/build/vmlinux
6.00 0.2% __pthread_mutex_lock_internal /lib64/libpthread-2.11.3.so
5.00 0.1% acpi_os_read_port /lib/modules/2.6.38-ck-BKL/build/vmlinux