Bug 216647 - first hibernate to disk fails on HP EliteBook 845 G8 when Intel 7360 WWAN enabled in BIOS
Summary: first hibernate to disk fails on HP EliteBook 845 G8 when Intel 7360 WWAN ena...
Status: ASSIGNED
Alias: None
Product: Networking
Classification: Unclassified
Component: Wireless (show other bugs)
Hardware: AMD Linux
: P1 normal
Assignee: networking_wireless@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-11-01 11:40 UTC by kolAflash
Modified: 2024-04-12 11:27 UTC (History)
1 user (show)

See Also:
Kernel Version: 6.0.6
Subsystem:
Regression: No
Bisected commit-id:


Attachments
dmesg log (kernel.org 6.0.6) (172.46 KB, text/plain)
2022-11-01 11:40 UTC, kolAflash
Details
kernel logs for hibernation with platform and with shutdown (Linux 6.0.9) (370.00 KB, text/plain)
2022-11-21 15:52 UTC, kolAflash
Details
kernel log for hibernation with platform (Linux 6.0.11) (93.68 KB, text/plain)
2022-12-05 09:59 UTC, kolAflash
Details
kernel log for hibernation with platform (pm_debug_messages pm_wakeup_irq gpio Linux-6.0.11) (130.40 KB, text/plain)
2022-12-08 16:00 UTC, kolAflash
Details
dmesg and acpidump for hibernation with platform (bug 216647 c11 patch pinctrl-amd-dynamic-debug Linux-6.0.12) (1.07 MB, text/plain)
2022-12-12 14:38 UTC, kolAflash
Details
dmesg and acpidump for hibernation with platform (bug-216647-c11-patch with "pinctrl-amd.c +p" on Linux-6.0.12) (1.11 MB, text/plain)
2022-12-12 16:09 UTC, kolAflash
Details
dmesg with shutdown (iosm loaded Linux-6.0.12) (95.42 KB, text/plain)
2022-12-13 10:19 UTC, kolAflash
Details
Linux kernel 6.8.1 with disk=shutdown reboots instead of halts (152.85 KB, text/plain)
2024-04-10 21:12 UTC, kolAflash
Details
crash after subsequent s2disk and s2idle (55.62 KB, text/plain)
2024-04-12 09:23 UTC, kolAflash
Details
Linux-6.8.5: crash after subsequent s2disk and s2idle (30.75 KB, text/plain)
2024-04-12 11:27 UTC, kolAflash
Details

Description kolAflash 2022-11-01 11:40:31 UTC
Created attachment 303115 [details]
dmesg log (kernel.org 6.0.6)

On my HP EliteBook 845 G8 the first hibernate to disk after a fresh boot often fails.

See attached dmesg log at position 141.752276.
The second hibernate to disk at position 161.315363 is successful.



= System =

Model: HP EliteBook 845 G8 (notebook)
CPU+GPU: Ryzen 5650U incl. Radeon GPU
Kernel: 6.0.6 compiled from kernel.org
OS: openSUSE-15.4
Memory: 64 GB
SWAP: 64 GB in a file: /swap

Partitions:
NAME                MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1             259:0    0  1,8T  0 disk  
├─nvme0n1p1         259:1    0  512M  0 part  /boot/efi
├─nvme0n1p2         259:2    0   16M  0 part  
├─nvme0n1p3         259:3    0    2G  0 part  /boot
└─nvme0n1p4         259:4    0  1,8T  0 part  
  └─nvme0n1p4_crypt 254:0    0  1,8T  0 crypt /
(nvme0n1p2 is a "BIOS boot" partition)



Maybe related:
Bug 216516 - s2ram freezes screen (Ryzen-5650U incl. Radeon GPU)
Bug 216574 - Hybrid System Suspend broken HP EliteBook 845 G8 (a.k.a. Hybrid Sleep / s2both) (s2idle Notebook)
Comment 1 Mario Limonciello (AMD) 2022-11-01 13:20:42 UTC
2022-10-31T14:01:20.603318+01:00 myhost kernel: [  150.051133][ T5939] amdgpu 0000:05:00.0: amdgpu: 00000000604fe5dc pin failed
2022-10-31T14:01:20.603321+01:00 myhost kernel: [  150.051137][ T5939] [drm:dm_plane_helper_prepare_fb [amdgpu]] *ERROR* Failed to pin framebuffer with error -19

This is probably at least partially:

https://gitlab.freedesktop.org/drm/amd/-/issues/2213

You should keep an eye for when the solution for that comes around if it improves your problem.
Comment 2 Mario Limonciello (AMD) 2022-11-04 00:40:43 UTC
Please try the stuff listed in this comment

https://gitlab.freedesktop.org/drm/amd/-/issues/2213#note_1618476
Comment 3 Mario Limonciello (AMD) 2022-11-07 13:36:28 UTC
Two of those three patches are in 6.0.7, would be good to know if that helps your problem.
Comment 5 kolAflash 2022-11-21 15:52:11 UTC
Created attachment 303251 [details]
kernel logs for hibernation with platform and with shutdown (Linux 6.0.9)

@Mario

(In reply to Mario Limonciello (AMD) from comment #4)
> OK 6.0.7 +
> https://cgit.freedesktop.org/drm/drm-misc/commit/?h=drm-misc-
> fixes&id=e0b26b9482461e9528552f54fa662c2269f75b3f please.

I've just tested 6.0.9.
Looks like that commit is already included in 6.0.9. So I didn't apply it.

In the meantime I figured out, that
  /sys/power/disk = shutdown
seems to work better than the default
  /sys/power/disk = platform

So I did a test with "shutdown" and one with "platform".
Result:
"shutdown" worked fine!
With "platform" the first hibernation still fails.
I've attached logs with the test results.

Before I used 6.0.8 which also had good results with "shutdown", but bad behavior with "platform".
Note:
For everyday use I changed /etc/systemd/sleep.conf -> HibernateMode=shutdown
Comment 6 Mario Limonciello (AMD) 2022-12-01 04:50:29 UTC
Can you please better describe the symptoms of the failure you're seeing with "platform".  I don't see anything immediately obvious jumping out.

You can try to take these two commits and see if it helps.
https://github.com/torvalds/linux/commit/4f2bea62cf3874c5a58e987b0b472f9fb57117a2
https://patchwork.freedesktop.org/patch/512917/
Comment 7 kolAflash 2022-12-05 09:59:26 UTC
Created attachment 303358 [details]
kernel log for hibernation with platform (Linux 6.0.11)

Hi Mario,

thank you very much for your patience!


(In reply to Mario Limonciello (AMD) from comment #6)
> Can you please better describe the symptoms of the failure you're seeing
> with "platform".  I don't see anything immediately obvious jumping out.

With "platform" the system doesn't really enter hibernation. So the notebook stays on and doesn't turn off as expected for hibernation. (power led stays on)

The screen just becomes black for a few seconds. But then the system continues normally. (not crashed, still usable, but not powered off as expected for hibernation)


You can see that there are just 10 seconds between
  PM: hibernation: hibernation entry
and
  PM: hibernation: hibernation exit
in the
  hibernate_broken_via_platform_v6.0.9.txt
(see attachment 303251 [details] in comment 5)
In this time the screen was black.



> You can try to take these two commits and see if it helps.
> https://github.com/torvalds/linux/commit/
> 4f2bea62cf3874c5a58e987b0b472f9fb57117a2
> https://patchwork.freedesktop.org/patch/512917/

I've just tested 6.0.11.
Looks like both commits are already included in 6.0.11. So I didn't apply them.

Problem persists. See attached log.
Comment 8 Mario Limonciello (AMD) 2022-12-05 22:30:28 UTC
Oh I must have missed this the first time:

> Wakeup pending. Abort CPU freeze

So we need to figure out what event is causing the aborted hibernate when it's set to platform.

Can you please share a log with /sys/power/pm_debug_messages set?
Also when the failure happens what does /sys/power/pm_wakeup_irq indicate the IRQ was?
Comment 9 Mario Limonciello (AMD) 2022-12-05 22:51:19 UTC
Also, pre-emptively - if /sys/power/pm_wakeup_irq shows the GPIO controller, please attach /sys/kernel/debug/gpio so we can better understand it.
Comment 10 kolAflash 2022-12-08 16:00:36 UTC
Created attachment 303382 [details]
kernel log for hibernation with platform (pm_debug_messages pm_wakeup_irq gpio Linux-6.0.11)

(In reply to Mario Limonciello (AMD) from comment #8)
> [...]
> Can you please share a log with /sys/power/pm_debug_messages set?
> Also when the failure happens what does /sys/power/pm_wakeup_irq indicate
> the IRQ was?

(In reply to Mario Limonciello (AMD) from comment #9)
> Also, pre-emptively - if /sys/power/pm_wakeup_irq shows the GPIO controller,
> please attach /sys/kernel/debug/gpio so we can better understand it.

Sure, here it is :-)
Comment 11 Mario Limonciello (AMD) 2022-12-08 16:06:48 UTC
OK so we're getting closer.  It's an active GPIO that prevented going to sleep.

Can you please add this patch to your kernel https://lore.kernel.org/linux-gpio/20221013134729.5592-1-mario.limonciello@amd.com/T/#t and then turn on dynamic debugging for the pinctrl-amd driver and repeat?

Along with the dmesg log showing which GPIO it was, please also add an acpidump so perhaps we can at least get a guess which device it's connected to.
Comment 12 kolAflash 2022-12-12 14:38:22 UTC
Created attachment 303394 [details]
dmesg and acpidump for hibernation with platform (bug 216647 c11 patch pinctrl-amd-dynamic-debug Linux-6.0.12)

@Mario
This patch for pinctrl-amd.c?
Adding dev_dbg if regval & PIN_IRQ_PENDING.
https://lore.kernel.org/linux-gpio/20221013134729.5592-1-mario.limonciello@amd.com/T/#iZ2e.:..:20221013134729.5592-2-mario.limonciello::40amd.com:1drivers:pinctrl:pinctrl-amd.c

I compiled and started v6.0.12 with that patch.
Then I did:
echo 1 > /sys/power/pm_debug_messages

Is there anything else I need to do, to turn on dynamic debugging?
If not, please see the attached dmesg & acpidump.
Comment 13 Mario Limonciello (AMD) 2022-12-12 14:41:13 UTC
Something like:

echo "file drivers/pinctrl/pinctrl-amd.c +p" | sudo tee /sys/kernel/debug/dynamic_debug/control
Comment 14 kolAflash 2022-12-12 16:09:21 UTC
Created attachment 303395 [details]
dmesg and acpidump for hibernation with platform (bug-216647-c11-patch with "pinctrl-amd.c +p" on Linux-6.0.12)

@Mario
Same as comment 12, but with "pinctrl-amd.c +p".

In dmesg there's one
  [   92.829818] amd_gpio AMDI0030:00: GPIO 24 is active: 0x10057a00
while trying to enter hibernation.

After the hibernation failed dmesg is being spammed with
  [  106.935331] amd_gpio AMDI0030:00: GPIO 18 is active: 0x10041b00

0x10041b00 "spamming" lasted until [  307.646436].
(log cut before that)
Comment 15 Mario Limonciello (AMD) 2022-12-12 16:37:36 UTC
OK so GPIO 24 will notify a device wake to GP12 on your system.
                Case (0x18)
                {
                    MSTP (0x3924)
                    Notify (\_SB.PCI0.GP12, 0x02) // Device Wake
                }

This points to a device "PWAN" under PCI bridge 0000:00:01.2 which is at address zero:

It's this device:
[    0.332686] pci 0000:01:00.0: [8086:7360] type 00 class 0x0d4000

That's a WWAN device, but I don't see any driver in your kernel to handle it.

What it seems to me that is going on is that when the system goes into hibernate that it is sending a notification to the WWAN device to do something.  No driver is there to pick up the notification and so it causes this problem.

You should either:
1) Physically remove the device.
2) Disable it in BIOS setup.
or
3) Load a kernel and userspace stack that can support it.  The "iosm" driver I *think* is what you need for this, but I have zero experience with WWAN.
Comment 16 kolAflash 2022-12-13 10:19:27 UTC
Created attachment 303399 [details]
dmesg with shutdown (iosm loaded Linux-6.0.12)

(In reply to Mario Limonciello (AMD) from comment #15)
> [...]
> What it seems to me that is going on is that when the system goes into
> hibernate that it is sending a notification to the WWAN device to do
> something.  No driver is there to pick up the notification and so it causes
> this problem.
> 
> You should either:
> 1) Physically remove the device.
> 2) Disable it in BIOS setup.
> or
> 3) Load a kernel and userspace stack that can support it.  The "iosm" driver
> I *think* is what you need for this, but I have zero experience with WWAN.

Thank you for your help!
Maybe someone else can help further with the WWAN.


2)
Disabling WWNA in the BIOS solves the problem :-)


3)
Loading "iosm" makes the wwan module show up as "wwan0" in "ip link".

BUT when hibernating with "platform" while iosm is loaded the system now forcefully reboots!
Unfortunately there's nothing in the logs. The last line is simply
  PM: hibernation: hibernation entry
When booting again there's no resume from hibernation. So it's like a system crash.

When hibernating with "shutdown" while iosm is loaded the system also wrongly reboots. But the hibernation seems to have worked fine. So it resumes the hinternated system.
-> I attached a dmesg log saved after resuming.


Personally I don't need to pursue this further. I know how I can workaround it.
(workaround either by disabling WWAN in BIOS or not loading the "iosm" module and using "shutdown" instead of "platform")
But I like to help to find a solution for upstream so the "HP EliteBook 845 G8" notebook can be properly supported.
Comment 17 Mario Limonciello (AMD) 2022-12-13 12:51:18 UTC
I'v re-assigned the component to wireless, hopefully someone who works on "iosm" can help you with this bug.  If you don't get further responses you might reach out to the appropriate mailing list or to the maintainer from MAINTAINERS for this driver.
Comment 18 kolAflash 2024-04-10 21:12:14 UTC
Created attachment 306124 [details]
Linux kernel 6.8.1 with disk=shutdown reboots instead of halts

I'm still having that problem. And I started using the WWAN modem, so turning it off in the BIOS isn't an option anymore.

I recently changed the OS.
OS: Debian-12
Kernel: 6.6.25 and 6.8.1 tested
Memory: 32 GB
SWAP: 33 GB in a file: /swap

Also the bug changed a little.
When setting
  /sys/power/disk = platform
the system reboots without resuming. (fresh reboot)
When setting
  /sys/power/disk = shutdown
the system reboots and resumes the snapshot.

Kernel log output attached for "shutdown" setting.



WORKAROUND

Due to the reboot on disk=shutdown the workaround got a little more difficult.

# file: /etc/systemd/sleep.conf
# Sets /sys/power/disk = shutdown
HibernateMode=shutdown

# file: /etc/grub.d/40_custom
menuentry "Halt" {
	halt
}

systemctl edit --full --force elitebook-hibernate-halt.service
# insert:
[Unit]
Description=Halt system after unintended reboot on s2disk. See https://bugzilla.kernel.org/show_bug.cgi?id=216647
Before=hibernate.target
[Service]
ExecStart=/usr/sbin/grub-reboot Halt
User=root
[Install]
WantedBy=hibernate.target

systemctl enable elitebook-hibernate-halt.service
Comment 19 kolAflash 2024-04-12 09:23:18 UTC
Created attachment 306137 [details]
crash after subsequent s2disk and s2idle

With the workaround from comment 12 applied, the kernel sometimes crashes when doing a subsequent s2idle after the hibernation (s2disk).

See attached log, recovered from pstore.
The kernel seems to panic when waking up from s2idle.
Comment 20 kolAflash 2024-04-12 11:27:20 UTC
Created attachment 306138 [details]
Linux-6.8.5: crash after subsequent s2disk and s2idle

Same problems with Linux-6.8.5. See attached dmesg log recovered from pstore after subsequent s2disk (disk=shutdown) and s2idle. Crashed on wake up from s2idle.




Someone (Martin) is also having trouble with XMM7360 in recent HP EliteBook laptops between Linux 6.2 to 6.8 (current kernel).

after upgrading to Kernel 6.2 WWAN Intel XMM7560 LTE module is not working anymore
https://bugzilla.kernel.org/show_bug.cgi?id=217200

Intel 7560 LTE Modem stops working after resuming from standby
https://bugzilla.kernel.org/show_bug.cgi?id=217996

WWAN Intel XMM7560 LTE wakes system from s2idle
https://bugzilla.kernel.org/show_bug.cgi?id=218383

Note You need to log in before you can comment on or make changes to this bug.