Bug 200595 - watchdog fires after s2idle - Lenovo X1 Yoga 3rd generation
Summary: watchdog fires after s2idle - Lenovo X1 Yoga 3rd generation
Alias: None
Product: Power Management
Classification: Unclassified
Component: Hibernation/Suspend (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Chen Yu
Depends on:
Reported: 2018-07-18 23:06 UTC by wavexx
Modified: 2019-07-22 09:08 UTC (History)
6 users (show)

See Also:
Kernel Version: 4.17.6
Tree: Mainline
Regression: No

dmesg output (81.17 KB, text/plain)
2018-11-20 10:40 UTC, wavexx
dmesg for HP ENVY x360 15-dr0xxx (65.91 KB, text/plain)
2019-07-22 09:07 UTC, vojtechkral
acpidump for HP ENVY x360 15-dr0xxx (1.78 MB, text/plain)
2019-07-22 09:08 UTC, vojtechkral

Description wavexx 2018-07-18 23:06:34 UTC
The Lenovo X1 Yoga 3rd generation enters s2idle sleep mode by default. In this mode, CLOCK_MONOTONIC incorrectly includes the slept time when waking up, causing systemd's watchdog to kill innocent processes [0]. The laptop otherwise enters/exits s2idle correctly.

s2idle is used because the latest bios of the Lenovo X1 Yoga 3rd edition and X1 Carbon 6th edition do not advertise S3 anymore [1]. However, by patching the ACPI DSDT tables (also [1]), S3 can still be used. In fact, in S3 mode, CLOCK_MONOTONIC is correctly frozen during sleep.

I was able to verify this issue from kernel 4.16 up to 4.18-rc4 under debian unstable.

[0] https://github.com/systemd/systemd/issues/9538
[1] https://forums.lenovo.com/t5/Linux-Discussion/X1-Carbon-Gen-6-cannot-enter-deep-sleep-S3-state-aka-Suspend-to/td-p/3998182
Comment 1 Chen Yu 2018-07-25 05:51:01 UTC
s2idle would call timekeeping_suspend() to stop the tick, but the system might be woken  up by suspicious interrupt and loop in a context with all user space process been frozen, that is to say, I'm thinking of the following scenario in which the watchdog might be triggered:

1. freeze all user space tasks(including systemd)
2. stop the time keeping system
3. system falls into s2idle
4. system been woken up by one interrupt
5. time keeping system has been resumed
6.  if the  interrupt is an invalid wake up source,
    jump to step 2, otherwise jump to step 7
7.  resume the user space tasks(including systemd)
8.  since in step 5 the time keeping system is running,
    and if there;s invalid interrupt triggered again and again,
     the CLOCK_MONOTONIC is increasing and cause systemd to
     detect a false timeout.
Comment 2 wavexx 2018-07-25 10:08:43 UTC
With this particular laptop the clock increases on par with the slept time. The best I could do was to use a stop-watch manually, so +/- 1-2 seconds of accuracy considering how long it takes the display to light up. I tested a sleep time of 60 seconds, which increased MONOTONIC by 60 seconds as well.
Comment 3 Chen Yu 2018-08-20 04:49:52 UTC
It looks like the default clocksource tsc does not have any 'pause' support for suspend to idle scenario, thus the xtime would increase anyway.
@Rafael, do you think if it is applicable/over killed to add logic to save/restore TSC for suspend to idle? In that way during resume the TSC are restore the previous one before suspend and sync among all CPUs, similar to check_tsc_sync_source().
Comment 4 Zhang Rui 2018-11-20 02:49:38 UTC
please attach the output of "grep . /sys/devices/system/clocksource/*/*"
Comment 5 Zhang Rui 2018-11-20 03:10:05 UTC
please attach the output of lscpu, and full dmesg output after boot.
Comment 6 wavexx 2018-11-20 10:39:13 UTC
Note that I'm forcing s3 sleep in the current state (mem_sleep_default=deep) and patching the DSDT tables to restore s3 sleep as well.

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               142
Model name:          Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
Stepping:            10
CPU MHz:             745.785
CPU max MHz:         3400.0000
CPU min MHz:         400.0000
BogoMIPS:            3600.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d

# grep . /sys/devices/system/clocksource/*/* 2>/dev/null
/sys/devices/system/clocksource/clocksource0/available_clocksource:tsc hpet acpi_pm
Comment 7 wavexx 2018-11-20 10:40:51 UTC
Created attachment 279551 [details]
dmesg output
Comment 8 Len Brown 2018-11-29 01:17:34 UTC
[   14.734083] iTCO_vendor_support: vendor-support=0
[   14.738262] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   14.738355] iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
[   14.738707] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[   14.743214] EFI Variables Facility v0.08 2004-May-17

But looking at iTCO_wdt.c, it has its own hardware timer,
and it suspend/resume on s2idle, which stops the watchdog while suspended.
So it should be immune to anything to do with timekeeping....

Is this system's /dev/watchdog using a different watchdog timer?
(perhaps wdctl can output the watchdog status?)
Or perhaps systemd isn't using /dev/watchdog?
Comment 9 Len Brown 2018-11-29 16:40:04 UTC
> Linux version 4.18.0-2-amd64

Please re-test with the latest upstream RC kernel
Comment 10 Chen Yu 2018-12-12 15:46:06 UTC
ping @wavexx@thregr.org
Comment 11 Chen Yu 2018-12-27 04:59:43 UTC
Closing as no response for 1 month.. please feel free to reopen if it can be reproduced on latest kernel.
Comment 12 vojtechkral 2019-07-22 09:06:32 UTC
  I am facing a very similar issue on a HP ENVY x360 15-dr0xxx laptop. The symptoms wrt. systemd are the same. Unlike wavexx I didn't patch my ACPI tables to force S3, because on this laptop that doesn't seem to be possible (but maybe I tried wrong). In any case, I'm attempting s2idle.

My kernel is: 5.2.1-arch1-1-ARCH

00:00.0 Host bridge: Intel Corporation Device 3e34 (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (Whiskey Lake)
00:04.0 Signal processing controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem (rev 0b)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Point-LP Thermal Controller (rev 30)
00:13.0 Serial controller: Intel Corporation Cannon Point-LP Integrated Sensor Hub (rev 30)
00:14.0 USB controller: Intel Corporation Cannon Point-LP USB 3.1 xHCI Controller (rev 30)
00:14.2 RAM memory: Intel Corporation Cannon Point-LP Shared SRAM (rev 30)
00:14.3 Network controller: Intel Corporation Cannon Point-LP CNVi [Wireless-AC] (rev 30)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Point-LP Serial IO I2C Controller #0 (rev 30)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Point-LP Serial IO I2C Controller #1 (rev 30)
00:16.0 Communication controller: Intel Corporation Cannon Point-LP MEI Controller #1 (rev 30)
00:17.0 RAID bus controller: Intel Corporation 82801 Mobile SATA Controller [RAID mode] (rev 30)
00:1c.0 PCI bridge: Intel Corporation Cannon Point-LP PCI Express Root Port #1 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Cannon Point-LP PCI Express Root Port #5 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Point-LP PCI Express Root Port #9 (rev f0)
00:1d.4 PCI bridge: Intel Corporation Cannon Point-LP PCI Express Root Port #13 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 30)
00:1f.3 Multimedia audio controller: Intel Corporation Cannon Point-LP High Definition Audio Controller (rev 30)
00:1f.4 SMBus: Intel Corporation Cannon Point-LP SMBus Controller (rev 30)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Point-LP SPI Controller (rev 30)
02:00.0 3D controller: NVIDIA Corporation GP108M [GeForce MX250] (rev a1)
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader (rev 01)
04:00.0 Non-Volatile memory controller: SK hynix Device 1327

I am also attaching a post-boot dmesg and an acpidump.

In my case the iTCO watchdog doesn't properly initialize (as you can see in the demsg) and I have no /dev/watchdog at all.
However it seems to me the watchdog/clocksource problem is secondary and the main issue is problems with ACPI.

In my case the nvidia dGPU seems to be one component that is problematic: I don't think I ever got it to work with bbswitch and without bbswitch X11 needs to be told to not autoload it, otherwise it goes blank-screen on start. I was also getting errors in dmesg from the dGPU when attempting to s2idle along the lines of 'the device refused to change state, currently in D3'. This lead me to (try to) disable the dGPU altogether by writing 'auto' to /sys/bus/pci/devices/0000:02:00.0/power/control on boot and telling X11 to just use the intel GPU.

After that I no longer get those errors while trying to s2idle, but the s2idle still doesn't work properly - this is evidenced by the CLOCK_MONOTONIC issue and also by power consumption being quite high during sleep.

I also read somewhere that people were having PM problems with the hynix nvme device. I tried the patch from this thread: https://lore.kernel.org/patchwork/patch/1007283/ but to no avail. (I did change the device ID in that patch to the one present on my laptop.) However, this was before disabling the dGPU, so maybe _both_ of those are problematic and I might want to re-test with the hynix patch and dGPU disabled.

It might also be worth noting that quite often the system doesn't power-cycle correctly when rebooting from Linux. It will power-cycle (judging from the power LED having blinked and the fan having gone up briefly) and then it will hang up with a blank screen for about 30s or so. Then it will power-cycle again (the LED and the fan again), this time to a regular POST/BIOS splash screen. This doesn't always happen and it didn't happen with mswindows (before I erased it).

Thanks for any advice and let me know if you need some additional data.
Comment 13 vojtechkral 2019-07-22 09:07:34 UTC
Created attachment 283899 [details]
dmesg for HP ENVY x360 15-dr0xxx
Comment 14 vojtechkral 2019-07-22 09:08:05 UTC
Created attachment 283901 [details]
acpidump for HP ENVY x360 15-dr0xxx

Note You need to log in before you can comment on or make changes to this bug.