Bug 10919

Summary: [regression] display dimming is slow and laggy - Acer Travelmate 661lci
Product: ACPI Reporter: Rafael J. Wysocki (rjw)
Component: ECAssignee: Alexey Starikovskiy (astarikovskiy)
Status: CLOSED CODE_FIX    
Severity: normal CC: acpi-bugzilla, alan-jenkins, maxi
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.26-rc6 Subsystem:
Regression: Yes Bisected commit-id:
Bug Depends on:    
Bug Blocks: 10492    
Attachments: dmesg
dmesg from 2.6.26-rc8 with patch
disable gpe during while pending query
output of dmesg
same patch, semicolon removed
silence EC mode switch, 3rd attempt
4th attempt
Incremental fix: re-enable interrupts in acpi_ec_wait
Dmesg with patch V4 applied showing "missing confirmations"
wake up regardless of mode
Incremental fix: storm workaround also requires polling mode
Patch 1/4 : Don't issue the burst disable command if EC exits the burst mode
Patch 2/4: Clear the query_pending bit only after processing EC notification event
Patch 3/4: Simplify EC working flowchart and always enable EC GPE
patch 4/4: Add some udelay in EC GPE handler to avoid EC GPE interrupt storm
patch vs 2.6.27-rc7

Description Rafael J. Wysocki 2008-06-15 04:16:11 UTC
Subject    : [regression] display dimming is slow and laggy
Submitter  : Maximilian Engelhardt <maxi@daemonizer.de>
Date       : 2008-06-14 22:31
References : http://marc.info/?l=linux-kernel&m=121348428828320&w=4

This entry is being used for tracking a regression from 2.6.25.  Please don't
close it until the problem is fixed in the mainline.
Comment 1 Maximilian Engelhardt 2008-06-17 06:39:56 UTC
Created attachment 16526 [details]
dmesg

seems like dmesg got lost in my mail somewhere, here it is again.
Comment 2 Len Brown 2008-06-17 10:04:03 UTC
> ACPI: EC: GPE storm detected, disabling EC GPE
Comment 3 Alexey Starikovskiy 2008-06-27 02:37:23 UTC
please check if last patch in 10724 bug report helps.
Comment 4 Maximilian Engelhardt 2008-06-27 19:28:39 UTC
Created attachment 16648 [details]
dmesg from 2.6.26-rc8 with patch

I assume you mean the patch from comment 36 in bug 10724.
I tried it on top of 2.6.26-rc8 and it seems to fix the issue. However I see some of these strange messages in dmesg now (full dmesg is attached):
ACPI: EC: non-query interrupt received, switching to interrupt mode
Comment 5 Maximilian Engelhardt 2008-06-27 19:40:06 UTC
And I just tried an unpatched 2.6.26-rc8 kernel and it still has the issue.
Comment 6 Maximilian Engelhardt 2008-07-06 06:09:06 UTC
This bug is still present in the 2.6.26-rc9 kernel.
Comment 7 Rafael J. Wysocki 2008-07-13 11:57:43 UTC
Handled-By : Alexey Starikovskiy <astarikovskiy@suse.de>
Comment 8 Alexey Starikovskiy 2008-07-13 14:49:12 UTC
Created attachment 16806 [details]
disable gpe during while pending query

This patch should make switch silent...
Comment 9 Maximilian Engelhardt 2008-07-15 10:41:09 UTC
Alexey,
your patch does work fine so far.
Thanks.
Comment 10 Alexey Starikovskiy 2008-07-15 12:36:50 UTC
Good, let's mark it resolved.
Thanks for report and testing!
Comment 11 Maximilian Engelhardt 2008-07-17 06:27:27 UTC
Created attachment 16865 [details]
output of dmesg

Hm, looks I was too fast reporting this as fixed. Using your patch dimming the display does indeed work without problems, but I'm getting this message again:
ACPI: EC: non-query interrupt received, switching to interrupt mode

I've checked my logs and it really didn't appear after a few boots when I was reporting this as fixed. Now it's there again. I'm still running the same kernel.

I'll attach my dmesg output in case it might be useful
Comment 12 Alexey Starikovskiy 2008-07-17 07:32:35 UTC
Maximilian,
This could only happen if you applied previous version of the patch.
With the last patch the offending printk is guarded with the check for GPE_STORM bit, which is set once and never cleared:

if (printk_ratelimit() &&
    !test_bit(EC_FLAGS_GPE_STORM, &ec->flags));
    pr_info(PREFIX "non-query interrupt received,"
	           " switching to interrupt mode\n");
please make sure you have this check.
Comment 13 Alan Jenkins 2008-07-17 08:37:22 UTC
(In reply to comment #12)
> Maximilian,
> This could only happen if you applied previous version of the patch.
> With the last patch the offending printk is guarded with the check for
> GPE_STORM bit, which is set once and never cleared:
> 
> if (printk_ratelimit() &&
>     !test_bit(EC_FLAGS_GPE_STORM, &ec->flags));
>     pr_info(PREFIX "non-query interrupt received,"
>                    " switching to interrupt mode\n");
> please make sure you have this check.

The check isn't guarding the printk though.  It's guarding an empty statement, thanks to a stray semicolon :-).
Comment 14 Alexey Starikovskiy 2008-07-17 09:26:27 UTC
Created attachment 16866 [details]
same patch, semicolon removed

Thanks,
Here is an updated patch
Comment 15 Alan Jenkins 2008-07-17 09:46:15 UTC
*** Bug 11089 has been marked as a duplicate of this bug. ***
Comment 16 Alan Jenkins 2008-07-17 12:11:36 UTC
Better, but it's still noisy - I get printk_ratelimit() messages with no accompanying message!  printk_ratelimit() needs to go last:

if (!test_bit(EC_FLAGS_GPE_STORM, &ec->flags) &&
     printk_ratelimit())
     pr_info(PREFIX "non-query interrupt received,"
                    " switching to interrupt mode\n");
Comment 17 Alexey Starikovskiy 2008-07-17 12:27:22 UTC
Created attachment 16869 [details]
silence EC mode switch, 3rd attempt

Hope it works now...
Comment 18 Maximilian Engelhardt 2008-07-18 03:52:19 UTC
hm, still not good. I'm using you patch (comment #17) with 2.6.26 final and I get a kernel Oops quite early in the boot process, i. e. before the framebuffer, so I had to disable the fb to actually see it.
It doesn't happen with an unpatched 2.6.26 kernel.
If it's useful I can try to get a digicam and take a picture of the Oops, but I don't have one here right now.
Comment 19 Alexey Starikovskiy 2008-07-18 08:36:55 UTC
Could you write down just the function names?
Comment 20 Maximilian Engelhardt 2008-07-18 13:26:42 UTC
here is some output I wrote down:

BUG: unable to handle kernel NULL pointer dereference at 00000001
[...]
EIP is at acpi_ec_gpe_handler
[...]
acpi_av_gpe_dispatch
acpi_ev_gpe_detect
acpi_ev_sci_xrupt_handler
acpi_irq
handle_IRQ_event
handle_level_irq
do_IRQ
default_idle
ktime_get
default_idle
default_idle
common_interrupt
default_idle
default_idle
default_idle
cpu_idle
start_kernel
unknown_bootoption
Comment 21 Alexey Starikovskiy 2008-07-18 13:41:14 UTC
Created attachment 16880 [details]
4th attempt

Thanks for listing... It's my fault again, please check new patch.
Comment 22 Maximilian Engelhardt 2008-07-20 12:45:17 UTC
Thanks, using that patch everything looks fine for now.
Comment 23 Alan Jenkins 2008-07-21 10:34:02 UTC
Created attachment 16926 [details]
Incremental fix: re-enable interrupts in acpi_ec_wait

Alexey: I retested your patch (v4) on my laptop and I noticed "missing confirmations, switch off interrupt mode".  This fix-up makes it go away for me.
Comment 24 Alexey Starikovskiy 2008-07-21 14:26:11 UTC
Alan,
no, this is wrong -- by this you enable GPE too early. Could you please make a log of ec.c with #define DEBUG uncommented and this string?
Comment 25 Alan Jenkins 2008-07-21 15:05:44 UTC
Created attachment 16929 [details]
Dmesg with patch V4 applied showing "missing confirmations"

Sorry, that was the wrong patch.  I only meant to send the first hunk, adding two lines to re-enable the GPE in acpi_ec_wait, as per the description.  I can't see anything wrong with that - but I've been wrong before :-).

Here's the dmesg as requested.  With #define DEBUG uncommented, I don't get a GPE storm on boot - so I trigger it manually by pressing some hotkeys (which auto-repeat).
Comment 26 Alexey Starikovskiy 2008-07-22 01:45:43 UTC
Created attachment 16931 [details]
wake up regardless of mode

hm, may be this would help?
Comment 27 Alan Jenkins 2008-07-23 06:02:03 UTC
Created attachment 16948 [details]
Incremental fix: storm workaround also requires polling mode

No, that doesn't help.

I think if you're going to disable interrupts temporarily during queries (or while queries are pending), then you've got to switch to polling mode as well.  Like this.  You can't wait for interrupts during a query if you've disabled them :-).
Comment 28 Maximilian Engelhardt 2008-08-07 03:39:44 UTC
Any news here?
Anything where I can help?

This bug is still present in 2.6.27-rc2
Comment 29 ykzhao 2008-09-07 22:51:58 UTC
Created attachment 17671 [details]
Patch 1/4 : Don't issue the burst disable command if EC exits the burst mode
Comment 30 ykzhao 2008-09-07 22:54:52 UTC
Created attachment 17672 [details]
Patch 2/4: Clear the query_pending bit only after processing EC notification event
Comment 31 ykzhao 2008-09-07 23:00:08 UTC
Created attachment 17673 [details]
Patch 3/4: Simplify EC working flowchart and always enable EC GPE
Comment 32 ykzhao 2008-09-07 23:02:09 UTC
Created attachment 17674 [details]
patch 4/4: Add some udelay in EC GPE handler to avoid EC GPE interrupt storm
Comment 33 ykzhao 2008-09-07 23:05:45 UTC
Hi, Maximillian
    Do you have opportunity to try the attached four patches on the latest kernel(2.6.27-rc5) and see whether the system can work well?
    
    It will also be great if you can try the patch in bug9998,#c76 on the 2.6.27-rc5 kernel.
    Thanks.
Comment 34 Maximilian Engelhardt 2008-09-09 16:13:31 UTC
> ------- Comment #33 from yakui.zhao@intel.com  2008-09-07 23:05 -------
> Hi, Maximillian
>     Do you have opportunity to try the attached four patches on the latest
> kernel(2.6.27-rc5) and see whether the system can work well?
>
>     It will also be great if you can try the patch in bug9998,#c76 on the
> 2.6.27-rc5 kernel.
>     Thanks.

I tried both, your 4 patches and the patch from bug9998 and both do work fine 
with my laptop so far.
Comment 35 Len Brown 2008-09-25 12:15:38 UTC
Created attachment 18045 [details]
patch vs 2.6.27-rc7

This version of Alexey's patch has been applied to the acpi test tree.
If it doesn't work for you, please let us know.

thanks,
-Len
Comment 36 Maximilian Engelhardt 2008-09-29 17:06:55 UTC
I tried the patch and it does work fine.

Thanks for the help to everybody.
Comment 37 Meelis Roos 2008-10-16 00:33:13 UTC
FYI: It also cures EC GPE storm on Thinkpad X20 that appeared in 2.6.27 (was OK in 2.6.26 and is OK in 2.6.27+ the patch).
Comment 38 Len Brown 2008-10-24 23:24:09 UTC
shipped in linux-2.6.28-rc1
closed

commit 7c6db4e050601f359081fde418ca6dc4fc2d0011
Author: Alexey Starikovskiy <astarikovskiy@suse.de>
Date:   Thu Sep 25 21:00:31 2008 +0400

    ACPI: EC: do transaction from interrupt context