Bug 10932

Created attachment 16527 [details]
Output of acpidump, dmesg, dmidecode, /proc/interrupts, lspci and kernel config

The file contains the output of acpidump, dmidecode and the kernel config.
For dmesg and lspci the output before and after suspend is provided.
The contents of /proc/interrupts are supplied before suspend, after waking up and after pressing a key combination (which should trigger an ACPI event) after waking up.

do you have any kernel that s3 used to work?
if yes, what's the latest work kernel version?
could you please re-attach the dmesg output that with acpi.debug_level=0x0f?

Created attachment 16550 [details]
Output of dmesg on 2.6.22.19 with acpi.debug_level 0x0f and 0x1f.

I can not remember any kernel whose S3 worked properly on my Thinkpad T22.

Yesterday I tried out the vanilla kernels 2.6.25 and 2.6.25.7 with the same 
results.

I just tested vanilla kernel 2.6.22.19. This one does not receive any ACPI 
events at all, but using acpi.debug_level=0x1f I can see that <Fn><F4> (the 
sleep-combination) triggers something but then again this is not recognized by 
acpid. Pressing <Fn><F7> (combination to switch between TFT and external VGA) 
twice turns off my TFT because then the Notebook runs on external VGA only. This 
is very interesting. In any other kernel that I tested this combination did only 
generate ACPI events (well, as long as events could be received) but did not do 
anything to the hardware. I attached the output of dmesg using kernel 2.6.22.19 
with both acpi.debug_level=0x1f and acpi.debug_level=0x0f.

I will test another kernel and attach its dmesg later.

Created attachment 16551 [details]
dmesg of  2.6.24-tuxonice-r9 with acpi.debug_level=0x0f

Zhang Rui, here is the dmesg output you requested.

I rebootet the machine, pressed <Fn><F3> (to turn of TFT), removed AC adapter, 
turned the TFT on again, logged in and did 'echo mem > /sys/power/state'. 
I wrote this message while the machine was sleeping and after waking it up 
pressing <Fn><F3> did create an event which was received by acpid!

But after resuming from S3 the second time the events are dead again.

are there any ACPI interrupts when pressing Fn+F3?

When the events still works, you can run "grep . /sys/firmware/acpi/interrupts/*" before and after pressing Fn+F3 to see which GPE the hotkey fires.
And when the events stop to work, do the same test and check if the corresponding GPE is fired.

BTW: when you got the dmesg output in comment #4, the events still work, right?
Please attach the dmesg AFTER the events are dead.

The output of dmesg in comment #4 has been created after I resumed for the 
second time, ACPI was dead at this time. I did the following:

<Fn><F3> # worked
removed AC adapter # got events
<Fn><F3> # worked
echo mem > /sys/power/state
<Fn><F3> # this worked, too
echo mem > /sys/power/state
<Fn><F3> # this did not work, ACPI is dead again
dmesg > 2.6.24-tuxonice-r9_0x0f_dmesg.txt

Created attachment 16560 [details]
Vanilla 2.6.23.17 — config, dmesg with acpi.debug_level 0x0f and 0x1f

Using 2.6.23.17 S3 seems to works without problems. I attached the output of 
dmesg with acpi.debug_level=0x1f and acpi.debug_level=0x0f and the kernel 
config. The outputs have been created after I pressed <Fn><F3> a few times and 
going to sleep (by pressing <Fn><F4> which I mapped to 'echo mem > /sys/power/state' 
using acpid) and waking up a few times.

In /sys/firmware/acpi using kernel 2.6.23.17 I only have the directory 'tables' 
which contains the files 'BOOT', 'DSDT', 'FACP' and 'FACS'. Using 
2.6.24-tuxonice-r9 the directory 'interrupts' is not there either.

So this is a regression, it would be great if you can run git bisect to narrow down the problem to a specific commit.

would you please rebuild the kernel with kernel config CONFIG_X86_UP_APIC and CONFIG_X86_UP_IOAPIC set?
could you do the test in comment#5 in the latest kernel release?

Well, I was busy the last weeks, but I had enough time to clone the git 
repository and run some tests. I wanted to run bisect, and so I checked the 
kernel versions 2.6.23, 2.6.24 and 2.6.25 from the repository again. Before 
booting a kernel, I detached the AC adapter and did not plug it in until all 
tests were finished. Then I booted the kernel, pressed FnF4 and resumed a few 
times.

With 2.6.23 the events were dead only few times (about 1 test of 6), using 
2.6.24 about half of the tests failed and with 2.6.25 the machine did never 
receive events after resuming.

Now I don't know how I could bisect the problem, since the issue is also 
existing in 2.6.23 and just became severe in 2.6.24. I may try to bisect the 
problem down to a specific commit, after which the failure rate of 1/6 turns 
into the 1/2 one, but I hardly believe that I will find origin of that 
behaviour.

With CONFIG_X86_UP_APIC and CONFIG_X86_UP_IOAPIC in 2.6.26-rc9 enabled I can see 
that FnF4 fires gpe0B, after resuming the events are dead and pressing FnF4 does 
not fire a gpe0B anymore. FnF3 does not seem to fire a GPE. The value for gpe09 
is constantly rising, both with working acpi and with acpi being dead. 

Created attachment 17044 [details]
 Output of `git bisect view` and `git bisect log`

I just finished bisect'ing. It was quite difficult sometimes to determine if 
the kernel was a "good" or "bad" one, since the bug also appears in 2.6.23 but 
less frequently. Especially the last kernel seemed to fail more often than the 
other "good" ones, but it did not fail as often as the "bad" ones. (I marked it 
as good). The one before the last one seemed to work perfectly.

Running the last kernel from git-bisect, /proc/acpi/bat and the appropriate menu 
item in make menuconfig are missing. The item for the AC Adapter is missing, too.

hi, andreas,
sorry for the delay, please attach the output of "lspci -vxxx" both before and after S3.

(In reply to comment #10)
> With CONFIG_X86_UP_APIC and CONFIG_X86_UP_IOAPIC in 2.6.26-rc9 enabled I can
> see 
> that FnF4 fires gpe0B, after resuming the events are dead and pressing FnF4
> does 
> not fire a gpe0B anymore. 
> FnF3 does not seem to fire a GPE.
which GPE does FnF3 fire before suspending?

Hi, Andreas
    Will you please try the latest kernel(2.6.27-rc5) and confirm whether the problem still exists? Of course please confirm whether the ACPI event can be reported correctly before S3.
    Do you mean that there is no ACPI event when unplugging AC adapter or pressing LID state after the system is resumed from S3? Right?
    thanks.

Created attachment 17866 [details]
Output of lspci -vxxx

I can not boot the system with neither 2.6.27-rc5 nor 2.6.27-rc6. I get 
a message that the root filesystem could not be mounted read/write.

FnF3 fires a GPE09 before suspending, so does the LID and unplugging the AC 
adapter.

I just recognized, that, using 2.6.26-rc9, ACPI dies also when the AC adapter 
is removed, even without going to S3.

The output of lspci -vxxx is the same before (ACPI working) and after S3 (ACPI 
dead).

>     Do you mean that there is no ACPI event when unplugging AC adapter or
> pressing LID state after the system is resumed from S3? Right?
>     thanks.

Yes, there are no events at all after resuming from S3. Depending on the kernel 
version, ACPI sometimes survives a sleep-resume cycle or detaching the AC 
adapter.
Even when ACPI is dead, /sys/firmware/acpi/interrupts/gpe09 increases every few 
seconds by a value between about ten and some hundreds.

Do you have opportunity to confirm whether the ACPI event can be reported correctly after resuming on windows? (Please check whether the FnF3/LID can work well).
   Thanks.

ping andreas, any updates?

Any updates?
   Will you please try the patch in http://marc.info/?l=linux-acpi&m=122307130419749&w=2 on the 2.6.27 stable kernel or the latest kernel and see whether the problem still exists?
   thanks.

Sorry, I'm quite busy these days.

I have no installation of windows and no unused harddisks, but probably I could try it out with a windows live-cd next week. Never done that before.

The patch applied to 2.6.27 does not solve the issue. After resuming, the output of dmesg contained the following message: "evxfevnt-0079 [00] enable                : System is already in ACPI mode".

Hi, Andreas
    Will you please try the debug patch in http://bugzilla.kernel.org/show_bug.cgi?id=11255#C49and see whether the problem still exists?
    
    Thanks. 

ping andreas

Using the patch above on linux-2.6.28-rc5, the machine is not able to resume at all. The screen turns on but remains blank, the LEDs for battery (even though there is no AC adapter attached), sleep and power are turned on. Even SysRq keys stop working when trying to resume.

Hi, Andreas
    Thanks for the test. The patch makes it more serious. Please ignore it.
    From the test in comment #7, the system can work well on the kernel of 2.6.23.17. And from the bisect log  it seems that the regression is realted with the following commit:
    >commit bbc615b16d64643a3d22ab4890fde1a685e86d83
    >Author: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
    >Date:   Sat Oct 20 00:32:36 2007 +0200
       >ide: use __ide_end_request() in ide_end_dequeued_request()
    Why is no ACPI event related with the IDE patch? It is diffictul to understand.
    Will you please try the latest kernel(2.6.28-rc6) and see whether the problem still exists?
    Thanks.

    
  

Created attachment 19112 [details]
use the attached tool to read I/O port

From the test it seems that the AC event is related with the GPE09 and some Fn hotkey is related with GPE0B. But after the system is resumed from S3, there is no ACPI event. 
    Will you please confirm whether ACPI event can be reported if pressing the power button when the problem happens?
    Will you please use the attached tool to read the following I/O port when no ACPI event is reported? (How to use the attached tool can be found in the Readme)
    a. ./ior --addr 0x1028 --width 16
    b. ./ior --addr 0x100e --width 16
    c. ./ior --addr 0x1004 --width 16
    
    Thanks.

> From the test in comment #7, the system can work well on the kernel of
> 2.6.23.17

It does not always work with 2.6.23.17, the problem does only occurr less often
with 2.6.23.17 than with other kernel verions I tested. (see comment #10)

> Why is no ACPI event related with the IDE patch? It is diffictul to
> understand.

Sorry, I don't know why the kernel or the machine behaves this way. This was
just what I found when I tried to find the bad change. In comment #11 I wrote
that I could not always clearly determine whether a set of changes was good or
bad, since the problem occurrs with all versions I tried but with different
frequency.

> Will you please try the latest kernel(2.6.28-rc6) and see whether the
> problem still exists?

Events for Fn keys are dead every time the machine resumes from S3.

> Will you please confirm whether ACPI event can be reported if pressing the
> power button when the problem happens?

After resuming with 2.6.28-rc6, there is still an event when pressing the power
button, even when the Fn keys are dead.

> Will you please use the attached tool to read the following I/O port when
> no ACPI event is reported?

With 2.6.28-rc6 I get the following:

err io_op # ./ior --addr 0x1028 --width 16

 the value of IO port 0x1028 is 1f01
err io_op # ./ior --addr 0x100e --width 16

 the value of IO port 0x100e is a00
err io_op # ./ior --addr 0x1004 --width 16

 the value of IO port 0x1004 is 1403


Regards,
Andreas

Hi,Andreas
   Thanks for the test. From the info in comment #24 it seems that the GPE09&GPE0B  is still enabled after resuming. As the power button event can be generated, it indicates that the ACPI SCI interrupt is still enabled. But it is very strange that there is no ACPI event(For example: LID, Fn + hotkey).
   From the acpidump it seems that the ACPI event of LID/HKEY device is triggered by GPE09(EC device).
   Now there are a bunch of EC fix patches.
   Will you please try the following debug patch on the latest kernel(2.6.28-rc7) and see whether the ACPI event can be reported after resuming.
   http://bugzilla.kernel.org/show_bug.cgi?id=11884#C62
   In the above debug patch the EC is initialized after _INI object.
   Thanks.

Hi Yakui,

I applied the patch to the kernel versions 2.6.28-rc7, 2.6.28-rc8 and
2.6.28-rc9. Each version has been tested three times, but after resuming, Fn
keys and LID were always dead. The power button did always trigger an event,
though.

Regards, Andreas

Hi, Andreas
     Sorry for the late response.
     Will you please not load the thinkpad_acpi driver and see whether the ACPI event can be reported after resume?(clear CONFIG_THINKPAD_ACPI in kernel configuration)
     Thanks.

Hi Yakui,

Sorry for keeping you waiting.

Without thinkpad_acpi the system does not trigger events for Fn+F3 and Fn+F12 at
all. Closing the lid or pressing Fn+F4, Fn+F7 or the power button triggers
events which have other names than the events generated by pressing the same
buttons when thinkpad_acpi is loaded. The events did neither die when the AC
adapter was removed nor when resumed from S3 without AC adapter -- I tested it
about 20 times and it worked always.

The tests above have been performed with linux-2.6.23.17 after removing
thinkpad-acpi from /etc/modules.autoload.d/ and rebooting. Simply removing
thinkpad_acpi from the running system did not help.

I have not yet tested whether the above is true for newer kernel versions since
I could not build any of them (make oldconfig fails since something somehow
pukes out 64bit object files).


Regards,
Andreas

Hi, Andreas
    Sorry for the late response.
    Do you mean that the ACPI event(LID, Fn+F4, Fn+F7, power button) can be reported correctly after suspend/resume if the thinkpad_acpi driver is not loaded? If the thinkpad_acpi driver is loaded, the ACPI event can't be reported correctly while running on battery. But it can work well while running on the AC. Right?
    If so, maybe this issue will be related with the thinkpad_acpi driver.
    cc: hmh@hmh.eng.br
    
Hi, Henrique
    Any idea about this issue?
    thanks.
    

Yes, I do...

It looks like EC interrupt or polling mode is losing events.  ThinkPad ECs seem to have an internal queue (because part of the GPE process is done in SMI, ARGH!), if you don't drain it, it will eventually stop issuing events.

So, all it would take would be Linux not always emptying the EC queue when we poll or service an EC interrupt, and eventually, all stops working.

*** Bug 12859 has been marked as a duplicate of this bug. ***

Guys, there is nothing I can do about this bug.

I will try to bring in the people working with the EC and GPE stuff into this, because AFAIK, our EC control HAS been fixed to attempt to drain the EC at every interrupt or poll cycle, in case it has internal queues like the ThinkPad ECs do.

Switching to EC component, reassigning, and:
Cc: Alexey Starikovskiy <aystarik@gmail.com>
Cc: Lin Ming <ming.m.lin@intel.com>

Hmm, Lin Ming was in error, not added to Cc.

Just thought of something. The T22 is old (ACPI 1.0 kind of old).  Is it possible that using the kernel parameters for the old ordering of calls might fix it?

i.e. "acpi_sleep=old_ordering"

acpi_sleep=s4_nonvs could also be worth a try.

Andreas, can you test that?

When booted with "acpi_sleep=old_ordering", the screen does only contain some
garbage after resuming from S3. When pressing some keys, the garbage disappears
and the screen remains blank. The machine can be rebooted using sysrq.

"acpi_sleep=s4_nonvs" seems to not affect the machine at all.

Tests were performed on 2.6.28-rc9.


Regards,
Andreas

still a problem with 2.6.30.stable?

Yes, using v2.6.30, the problem seems to still occurr every wakeup if running on battery.

Please check the fix for bug #14858 : ACPI events on T20 thinkpad stop being reported.  Maybe it also fixes your problem?

I think the easiest way to test that is to temporarily boot the latest 2.6.32 stable kernel with the patch in #14858 applied...

Sorry, I can't check that. My machine has died some weeks ago.

I can't comment on Andreas' machine, but having exactly same problem (on Thinkpad T21) I have to say that patch from bug #14858 (ACPI: EC: Accelerate query execution) applied on top of 2.6.32.5 solves this issue for me.

Alexey,

Given the answer from Karol Lewandowski on comment #40, should we mark this bug as a duplicate of bug #14858 ?

*** This bug has been marked as a duplicate of bug 14858 ***