Bug 12257

Summary: Resume from suspend to RAM broken on Asus L5D
Product: ACPI Reporter: Rafael J. Wysocki (rjw)
Component: Power-Sleep-WakeAssignee: Len Brown (lenb)
Status: CLOSED PATCH_ALREADY_AVAILABLE    
Severity: normal CC: acpi-bugzilla
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: up to and including 2.6.28-rc9 Tree: Mainline
Regression: ---
Bug Depends on:    
Bug Blocks: 7216    
Attachments: ACPI: PCI Interrupt Links -- disable when unused
Output of 'lspci -vv' from Asus L5D
Output of 'acpidump' from Asus L5D
Boot log from Asus L5D (without 'noapic')
/proc/interrupts from Asus L5D (booted without 'noapic'
dmesg log from Asus L5D containing one suspend-resume cycle
Boot log from Asus L5D (with 'noapic')
/proc/interrupts from Asus L5D (booted with 'noapic')
Debug patch for ACPI pci_link
dmesg log from Asus L5D containing one suspend-resume cycle
skge suspend patch
.config for Asus L5D
config that works on compaq presario r3000 nforce3
Boot log from Asus L5D (without 'noapic' and with 'acpi.debug_layer=0x400000')
patch vs 2.6.28-rc9 to use PIC mode on systems such as the Asus L5D
Boot log from Asus L5D with acpi=noirq
patch vs 2.6.28-rc9 to add irqrouter_suspend()

Description Rafael J. Wysocki 2008-12-19 14:10:13 UTC
Latest working kernel version: N/A
Earliest failing kernel version: 2.6.12 or so
Distribution: openSUSE 11.0
Hardware Environment: Asus L5D

Problem Description: The box doesn't resume from suspend to RAM

Steps to reproduce: Suspend to RAM and try to resume
Comment 1 Rafael J. Wysocki 2008-12-19 14:12:57 UTC
Created attachment 19383 [details]
ACPI: PCI Interrupt Links -- disable when unused

With this patch applied, the box resumes from suspend to RAM (unless it's been booted with 'noapic', in which case it doesn't resume), but after the resume some PCI devices don't work (most noticeably Ethernet, USB, wireless)
Comment 2 Rafael J. Wysocki 2008-12-19 14:14:12 UTC
Created attachment 19384 [details]
Output of 'lspci -vv' from Asus L5D
Comment 3 Rafael J. Wysocki 2008-12-19 14:14:46 UTC
Created attachment 19385 [details]
Output of 'acpidump' from Asus L5D
Comment 4 Rafael J. Wysocki 2008-12-19 14:15:26 UTC
Created attachment 19386 [details]
Boot log from Asus L5D (without 'noapic')
Comment 5 Rafael J. Wysocki 2008-12-19 14:16:02 UTC
Created attachment 19387 [details]
/proc/interrupts from Asus L5D (booted without 'noapic'
Comment 6 Rafael J. Wysocki 2008-12-19 14:16:35 UTC
Created attachment 19388 [details]
dmesg log from Asus L5D containing one suspend-resume cycle
Comment 7 Rafael J. Wysocki 2008-12-19 14:17:02 UTC
Created attachment 19389 [details]
Boot log from Asus L5D (with 'noapic')
Comment 8 Rafael J. Wysocki 2008-12-19 14:17:41 UTC
Created attachment 19390 [details]
/proc/interrupts from Asus L5D (booted with 'noapic')
Comment 9 Rafael J. Wysocki 2008-12-19 14:40:04 UTC
Created attachment 19391 [details]
Debug patch for ACPI pci_link
Comment 10 Rafael J. Wysocki 2008-12-19 14:43:23 UTC
Created attachment 19392 [details]
dmesg log from Asus L5D containing one suspend-resume cycle

The previous suspend log was from a wrong kernel.  Also, this one contains debug information generated by the patch from the previous comment.
Comment 11 Rafael J. Wysocki 2008-12-21 15:21:18 UTC
Additional information:

If skge, ohci-hcd and ehci-hcd are unloaded before suspend and reloaded after the subsequent resume, everything works.  If these modules are loaded during suspend, eth0 and USB don't work after the resume (in that case unloading and reloading skge doesn't bright eth0 to life).

This appears to be 100% reproducible.
Comment 12 Rafael J. Wysocki 2008-12-21 16:18:35 UTC
Interestingly enough, skge, ohci_hcd, ehci_hcd don't free IRQs on suspend.  However, making them do that doesn't seem to help.
Comment 13 Rafael J. Wysocki 2008-12-21 16:24:37 UTC
Created attachment 19410 [details]
skge suspend patch

In particular, I did a suspend-resume cycle with the attached patch making skge free IRQ during suspend.  Both ohci_hcd and ehci_hcd were unloaded before suspend.

skge didn't work after the resume and the audio adapter, which _doesn't_ share the interrupt with the skge, didn't work either.  Moreover, unloading and reloading skge after the resume didn't help.
Comment 14 Rafael J. Wysocki 2008-12-21 16:30:34 UTC
FWIW, booting with acpi_sleep=old_ordering doesn't change anything.
Comment 15 Len Brown 2008-12-22 12:32:01 UTC
Interrupts are not properly configured on this box,
even before suspend/resume is attempted.

Device assignment to IRQ is identical in PIC
and IOAPIC mode, and in IOAPIC mode, interrupts that are
shared are configured as IO-APIC-edge

  9:         27   IO-APIC-edge      acpi, yenta

Further, the existence of this interrupt:

  2:          0    XT-PIC-XT        cascade

suggests that MPS is configuring interrupts rather than ACPI.
Please attach the .config
Comment 16 Rafael J. Wysocki 2008-12-22 12:44:52 UTC
Created attachment 19434 [details]
.config for Asus L5D
Comment 17 Len Brown 2008-12-22 14:26:13 UTC
Created attachment 19438 [details]
config that works on compaq presario r3000 nforce3
Comment 18 Rafael J. Wysocki 2008-12-22 15:22:50 UTC
Created attachment 19440 [details]
Boot log from Asus L5D (without 'noapic' and with 'acpi.debug_layer=0x400000')
Comment 19 Rafael J. Wysocki 2008-12-22 15:39:59 UTC
Clarification: Comment #11 doesn't apply to the kernel booted with 'noapic' (ie. with 'noapic' resume doesn't work no matter what).
Comment 20 Len Brown 2008-12-22 23:09:36 UTC
Created attachment 19446 [details]
patch vs 2.6.28-rc9 to use PIC mode on systems such as the Asus L5D

This patch should give the same results as if the Asus L5D
were booted with "noapic" using previous kernels.

ie. the MPS Table information for the IOAPIC
should be ignored, and the system should register
IRQS in XT-PIC mode, consistent with how ACPI is
configuring the Interrupt Links.

Of course this fixes only the 1st layer of the onion --
getting the Asus L5D booted into a sane and consistent
interrupt configuration.

Assuming this works, the next layer of the onion shall
be to poke at the clue above that the resume failure is
interrupt related, and perhaps interrupt link related.

Please compare a patched kernel vs an older kernel booted
with "noapic" and report any dmesg and /proc/interrupts differences.
Comment 21 Rafael J. Wysocki 2008-12-23 06:04:10 UTC
Created attachment 19454 [details]
Boot log from Asus L5D with acpi=noirq

When booted with acpi=noirq, the contents of /proc/interrupts looks the same as in  attachment #19387 [details].  However, resume from suspend to RAM doesn't work at all in this configuration.
Comment 22 Rafael J. Wysocki 2008-12-23 06:32:56 UTC
With the patch from comment #20 applied, the kernel behaves like with 'noapic'.
Comment 23 Len Brown 2008-12-23 13:06:21 UTC
Created attachment 19460 [details]
patch vs 2.6.28-rc9 to add irqrouter_suspend()

Please apply this patch on top of Linus' tree,
in addition to the patch in commend #20.

This patch will disable all Links on suspend,
and will disable any unreferenced links on resume.
Comment 24 Rafael J. Wysocki 2008-12-23 14:20:56 UTC
Resume doesn't work with these two patches applied.

It seems to me that on this box we have to restore the IOAPIC registers during resume for things to work.
Comment 25 Len Brown 2009-01-09 14:10:41 UTC
note that patch in comment #20 shipped in 2.6.28-git14 (2.6.29-rc1)

patch in comment #23 looks correct, but needs a customer,
and this box doesn't appear to be it...