Bug 219364

I recorded a video to show the strange pause while moving cursor.
https://people.canonical.com/~acelan/bugs/bz-219364/

The strange pause also happens when typing on the built-in the keyboard. When encountered the pause, it might repeat the final character many times after the pause, or nothing happened after the pause.

Did this happen with earlier kernels like 6.10 or 6.11 as well?

Reproduce this issue with v6.11 kernel.
This is a new SoC, so didn't try older kernel.

1. Force the CPU to stay in C0 works
sudo su - ; cat >/dev/cpu_dma_latency <(echo -e -n "\x0\x0\x0\x0" ; sleep inf)

2. Disable C state works
“cpuidle.off=1 intel_idle.max_cstate=0“

Please share the C-states on this machine out of the box:

grep . /sys/devices/system/cpu/cpu0/cpuidle/*/*

Can you discover the deepest state that results in no symptoms
by booting with descending numbers in intel_idle.max_cstate=N
(or disable states via /sys/devices/system/cpu/cpu*/cpuidle/state*/disable)

Do you see the same symptoms if you simply boot with maxcpus=1 ?

Do the boot-time delays also go away with C-states off,
(please show dmesg for boot with C-states off)
or does disabling C-states only help with the interactive issues?

Question on delay in using the system:

Are these noticeable delays in using is only DC mode issue?

Did you try upstream latest 6.12-rc* kernel?

Is setting makes any difference in DC mode
/sys/devices/system/cpu/cpu*/power/energy_perf_bias = 7

Created attachment 307012 [details]
cpuidle/state* log

I can see pretty minor pause with `intel_idle.max_cstate=2`, and with `intel_idle.max_cstate=3` you can feel and see the pause while moving the cursor very clear.

And the boot up time also affects by the max_cstate, looks like it takes more time to boot up with the bigger cstate number.

The delay is easier to observe with DC, but I can observe it with AC with some reboot.
/sys/devices/system/cpu/cpu*/power/energy_perf_bias were 8, and set them to 7 helps, the pause becomes very minor.

Created attachment 307013 [details]
dmesg max_cstate=1

Created attachment 307014 [details]
dmesg max_cstate=2

(In reply to AceLan Kao from comment #8)
> Created attachment 307012 [details]
> cpuidle/state* log
> 
> I can see pretty minor pause with `intel_idle.max_cstate=2`, and with
> `intel_idle.max_cstate=3` you can feel and see the pause while moving the
> cursor very clear.
> 
> And the boot up time also affects by the max_cstate, looks like it takes
> more time to boot up with the bigger cstate number.
> 
> The delay is easier to observe with DC, but I can observe it with AC with
> some reboot.
> /sys/devices/system/cpu/cpu*/power/energy_perf_bias were 8, and set them to
> 7 helps, the pause becomes very minor.

Who set this value to 8? Is Dell BIOS is doing this? Unless BIOS is changing the value should be 6.

I think two sentence were separate, from my observe, EPB is 8 in DC, and 6 in AC.

But who set those EPBs? OS didn't.

I got the answer. This is done by Linux power profile daemon.

Please check EPB=7 fixes the issue.

No, EPB=7 doesn't fix the issue.
You still may observe the behavior from time to time, especially let the system idle for a while, the strange pause while moving the cursor or the abnormal keyboard behavior.

also set EPB=7 on my side:
echo 7 | tee /sys/devices/system/cpu/cpu*/power/energy_perf_bias

I can still observe the lag

To summarize all symptoms, I observer system lag when:
- using touchpad
- using onboard keyboard
- typing when using ssh
- playing music (wav, mp3) via apps (Rythmbox, audacious, video...)

Please check the following command can fix the issue or not.
1. Disable ACPI C3
for i in {0..7}; do echo 1 > /sys/devices/system/cpu/cpu$i/cpuidle/state3/disable;done

2. Offline E Core
for i in {4..7}; do echo 0 > /sys/devices/system/cpu/cpu$i/online;done

3. Change the Power Mode in Settings -> Power -> Power Mode to Performance.

I tried on touchpad + playing music at the same time as:

1. Make sure I observer the issue 
1.1 Disable ACPI C3 --> It works  
for i in {0..7}; do echo 1 > /sys/devices/system/cpu/cpu$i/cpuidle/state3/disable;done

2. Reboot and make sure I observer the issue 
2.2 Offline E Core --> It works
for i in {4..7}; do echo 0 > /sys/devices/system/cpu/cpu$i/online;done

3. Reboot and make sure I observer the issue 
3.3 Power Mode to Performance. --> this is not works

Verified below tests with AC is connected(EPB = 6)

> 1. Disable ACPI C3
> for i in {0..7}; do echo 1 > 
> /sys/devices/system/cpu/cpu$i/cpuidle/state3/disable;done
This works
echo 0 back to those files and reproduce the issue quickly.

> 2. Offline E Core
> for i in {4..7}; do echo 0 > /sys/devices/system/cpu/cpu$i/online;done
This works, too.
echo 1 back to those files and reproduce the issue quickly.

> 3. Change the Power Mode in Settings -> Power -> Power Mode to Performance.
(EPB = 0)
Encountered the issue, it doesn't imporve the situation, sometimes the pause is the same serious as DC power.

Can you try Linux 6.11.0-8-generic, even in #3 you mentioned 6.11 is reproduced. But just like to double confirm if this specified version can reproduce.

Yes, 6.11.0-8 is hard to reproduce the issue. But you still can observe the pause while moving the cursor sometimes.
The situation is much lighter than other kernels, but I won't say there is no issue with this kernel.

Can you help to check if system could enter C3 or S0ix ?

BTW, where I can get mainline v6.12-rc2 kernel ? Seems only update till 6.11 : https://kernel.ubuntu.com/mainline/

Yes, the Package C2, C6, and C10 counters increase in /sys/kernel/debug/pmc_core/package_cstate_show.

And the slp_s0_residency_usec counter increases when suspended.

We're moving our servers, so you may have to build the kernel by yourself.

Created attachment 307180 [details]
test X86_BUG_MONITOR patch

please try this patch

Created attachment 307184 [details]
test X86_BUG_MONITOR patch for Linux 6.10 and earlier

Same logical patch as above,
but this version applies to the syntax of Linux 6.10 and earlier,
while the version above applies to the syntax of 6.11 and later.

*** Bug 219477 has been marked as a duplicate of this bug. ***

Created attachment 307185 [details]
standalone program to detect migration stalls on LNL

The stand-alone program can detect stalls.

It uses sched_setaffinity(2) to migrate itself from cpu(4-7) to cpu(0-3),
using rdtsc to measure how long the migration takes.

The program will complain about migrations that take more than 1ms.
Sometimes it will detect stalls over 1000ms.

By default it migrates 1000 times, once ever 250ms, running for 250 sec.

Created attachment 307208 [details]
test X86_BUG_MONITOR patch

This patch applies to Linux-6.11 and later

Verified with the mainline v6.12 and Ubuntu 6.11 kernels, the patch fixes the issue.
Thanks.

(In reply to Len Brown from comment #31)
> Created attachment 307208 [details]
> test X86_BUG_MONITOR patch
> 
> This patch applies to Linux-6.11 and later

Fix verified on an Asus Vivobook S 2024 OLED. Thanks!