218245 – default_enter_idle leaked IRQ state

Bug 218245 - default_enter_idle leaked IRQ state

Summary: default_enter_idle leaked IRQ state

Status:	RESOLVED CODE_FIX

Alias:	None

Product:	Platform Specific/Hardware
Classification:	Unclassified
Component:	x86-64 (show other bugs)
Hardware:	All Linux

Importance:	P3 normal
Assignee:	platform_x86_64@kernel-bugs.osdl.org

URL:
Keywords:

Depends on:
Blocks:

Reported:	2023-12-09 11:41 UTC by Forza
Modified:	2024-01-02 15:41 UTC (History)
CC List:	1 user (show)

See Also:
Kernel Version:
Subsystem:
Regression:	No
Bisected commit-id:

Attachments
dmesg (64.95 KB, text/plain) 2023-12-09 11:59 UTC, Forza	Details
Kernel .config (117.03 KB, text/plain) 2023-12-09 12:00 UTC, Forza	Details
mark __cpuidle (674 bytes, patch) 2023-12-09 15:15 UTC, Borislav Petkov	Details \| Diff
Do not enable IRQs either (775 bytes, patch) 2023-12-09 18:55 UTC, Borislav Petkov	Details \| Diff
Show Obsolete (1) Add an attachment (proposed patch, testcase, etc.)

Description Forza 2023-12-09 11:41:14 UTC

Since kernels 6.2 or maybe 6.3 up until current 6.6.5 there is a spurious warning of leaked IRQ state in my Vms rubbning under KVM QEMU on a AMD Ryzen host.

The issue does not seem to cause any real problems, but it probably shouldn't happen. 

[   22.773887][    T0] ------------[ cut here ]------------
[   22.773897][    T0] default_enter_idle leaked IRQ state
[   22.773981][    T0] WARNING: CPU: 1 PID: 0 at drivers/cpuidle/cpuidle.c:269 cpuidle_enter_state+0x2bb/0x2c0
[   22.774001][    T0] Modules linked in:
[   22.774004][    T0] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.6.5-forza-virt #3
[   22.774007][    T0] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.0-20220807_005459-localhost 04/01/2014
[   22.774009][    T0] RIP: 0010:cpuidle_enter_state+0x2bb/0x2c0
[   22.774012][    T0] Code: c6 fe ff ff e9 7f fe ff ff 41 be 18 00 00 00 31 db eb 82 49 8b 76 50 48 c7 c7 b3 b0 5f a4 c6 05 06 4f f9 00 01 e8 55 bb 31 ff <0f> 0b eb c0 cc eb 0e 48 89 f8 48 89 d1 f3 a4 c3 0f 1f 44 00 00 48
[   22.774015][    T0] RSP: 0018:ffffac8b800a7e90 EFLAGS: 00010246
[   22.774017][    T0] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[   22.774018][    T0] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[   22.774020][    T0] RBP: ffff8bcd5fca8b40 R08: 0000000000000000 R09: 0000000000000000
[   22.774021][    T0] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa4d93620
[   22.774022][    T0] R13: 0000000000000001 R14: ffffffffa4d936a0 R15: 0000000000000000
[   22.774025][    T0] FS:  0000000000000000(0000) GS:ffff8bcd5fc80000(0000) knlGS:0000000000000000
[   22.774028][    T0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   22.774029][    T0] CR2: 00007f54f3488020 CR3: 000000001ec16000 CR4: 00000000003506e0
[   22.774031][    T0] Call Trace:
[   22.774042][    T0]  <TASK>
[   22.774047][    T0]  ? __warn+0x6b/0xd0
[   22.774055][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[   22.774057][    T0]  ? report_bug+0x140/0x180
[   22.774062][    T0]  ? handle_bug+0x3c/0x70
[   22.774064][    T0]  ? exc_invalid_op+0x13/0x70
[   22.774066][    T0]  ? asm_exc_invalid_op+0x16/0x20
[   22.774071][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[   22.774074][    T0]  cpuidle_enter+0x24/0x40
[   22.774087][    T0]  do_idle+0x1c5/0x220
[   22.774099][    T0]  cpu_startup_entry+0x21/0x30
[   22.774102][    T0]  start_secondary+0x100/0x120
[   22.774111][    T0]  secondary_startup_64_no_verify+0x178/0x17b
[   22.774122][    T0]  </TASK>
[   22.774122][    T0] ---[ end trace 0000000000000000 ]---

Comment 1 Forza 2023-12-09 11:59:32 UTC

Created attachment 305569 [details]
dmesg

Comment 2 Forza 2023-12-09 12:00:49 UTC

Created attachment 305570 [details]
Kernel .config

Comment 3 Forza 2023-12-09 13:34:17 UTC

I am currently on QEMU 8.1.3 but have tried most versions since 7.2.

Is there any specific debug options I can enable to help debug this issue?

Comment 4 Borislav Petkov 2023-12-09 15:15:16 UTC

Created attachment 305573 [details]
mark __cpuidle

See if this fixes it.

Comment 5 Forza 2023-12-09 18:07:19 UTC

(In reply to Borislav Petkov from comment #4)
> Created attachment 305573 [details]
> mark __cpuidle
> 
> See if this fixes it.

Thanks. Unfortunately same error.


[    2.267349][    T1] Freeing unused kernel image (initmem) memory: 1432K
[    2.348136][    T1] Write protecting the kernel read-only data: 28672k
[    2.350132][    T1] Freeing unused kernel image (rodata/data gap) memory: 680K
[    2.352000][    T1] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    2.352494][    T1] Run /sbin/init as init process
[    2.352929][    T1]   with arguments:
[    2.352931][    T1]     /sbin/init
[    2.352932][    T1]   with environment:
[    2.352933][    T1]     HOME=/
[    2.352934][    T1]     TERM=linux
[    3.130816][ T1382] udevd[1382]: starting version 3.2.14
[    3.200760][ T1382] udevd[1382]: starting eudev-3.2.14
[    3.683869][    T0] ------------[ cut here ]------------
[    3.683877][    T0] default_enter_idle leaked IRQ state
[    3.683966][    T0] WARNING: CPU: 3 PID: 0 at drivers/cpuidle/cpuidle.c:269 cpuidle_enter_state+0x2bb/0x2c0
[    3.683971][    T0] Modules linked in:
[    3.683973][    T0] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.6.5-forza-virt #4
[    3.683976][    T0] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.0-20220807_005459-localhost 04/01/2014
[    3.683977][    T0] RIP: 0010:cpuidle_enter_state+0x2bb/0x2c0
[    3.683980][    T0] Code: c6 fe ff ff e9 7f fe ff ff 41 be 18 00 00 00 31 db eb 82 49 8b 76 50 48 c7 c7 b3 b0 5f 96 c6 05 36 4f f9 00 01 e8 85 bb 31 ff <0f> 0b eb c0 cc eb 0e 48 89 f8 48 89 d1 f3 a4 c3 0f 1f 44 00 00 48
[    3.683982][    T0] RSP: 0018:ffffae6e800b7e90 EFLAGS: 00010246
[    3.683984][    T0] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[    3.683985][    T0] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    3.683986][    T0] RBP: ffff98f61e5a83d0 R08: 0000000000000000 R09: 0000000000000000
[    3.683987][    T0] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff96d93620
[    3.683988][    T0] R13: 0000000000000001 R14: ffffffff96d936a0 R15: 0000000000000000
[    3.683990][    T0] FS:  0000000000000000(0000) GS:ffff98f61e580000(0000) knlGS:0000000000000000
[    3.683995][    T0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.683998][    T0] CR2: 00007ffda7c10bb8 CR3: 0000000009c16000 CR4: 00000000003506e0
[    3.684000][    T0] Call Trace:
[    3.684009][    T0]  <TASK>
[    3.684011][    T0]  ? __warn+0x6b/0xd0
[    3.684027][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[    3.684030][    T0]  ? report_bug+0x140/0x180
[    3.684034][    T0]  ? handle_bug+0x3c/0x70
[    3.684036][    T0]  ? exc_invalid_op+0x13/0x70
[    3.684038][    T0]  ? asm_exc_invalid_op+0x16/0x20
[    3.684042][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[    3.684044][    T0]  cpuidle_enter+0x24/0x40
[    3.684064][    T0]  do_idle+0x1c5/0x220
[    3.684075][    T0]  cpu_startup_entry+0x21/0x30
[    3.684077][    T0]  start_secondary+0x100/0x120
[    3.684088][    T0]  secondary_startup_64_no_verify+0x178/0x17b
[    3.684098][    T0]  </TASK>
[    3.684099][    T0] ---[ end trace 0000000000000000 ]---

Comment 6 Borislav Petkov 2023-12-09 18:54:32 UTC

Bah, that was only half the truth. The cpuidle governors must return with IRQs disabled now. And haltpoll doesn't. Yet. See if this next patch fixes it...

It might not but I don't know because I can't reproduce your issue. How do you boot your guest? Any cmdline args selecting haltpoll as a cpuidle governor?

Pls send kernel .config also.

Thx.

Comment 7 Borislav Petkov 2023-12-09 18:55:39 UTC

Created attachment 305574 [details]
Do not enable IRQs either

Needs to not enable IRQs.

Comment 8 Forza 2023-12-09 21:14:33 UTC

(In reply to Borislav Petkov from comment #7)
> Created attachment 305574 [details]
> Do not enable IRQs either
> 
> Needs to not enable IRQs.

Initial tests seems good. I'll test more over the next few days.

Comment 9 Forza 2023-12-15 21:23:01 UTC

So far there has been no further occurrence, so I'd say the patch has fixed the issue.

Thanks

Comment 10 Forza 2023-12-22 19:39:50 UTC

(In reply to Borislav Petkov from comment #6)

> It might not but I don't know because I can't reproduce your issue. How do
> you boot your guest? Any cmdline args selecting haltpoll as a cpuidle
> governor?

Forgot to answer this question. I do not choose any specific cpuidle governor. Actually, I think the kernel forces it when it detects it is running in a KVM guest.

The kernel menuconfig says:

Prompt: KVM Guest support (including kvmclock)
Depends on: HYPERVISOR_GUEST [=y] && PARAVIRT [=y]
Location:
	-> Processor type and features
		-> Linux guest support (HYPERVISOR_GUEST [=y])
			-> KVM Guest support (including kvmclock) (KVM_GUEST [=y])
Selects: PARAVIRT_CLOCK [=y] && ARCH_CPUIDLE_HALTPOLL [=y] && X86_HV_CALLBACK_VECTOR [=y]



I boot most VMs with the following libvirt XML:

  <os>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
    <kernel>/mnt/virtiofs/volume/scsi-tgt/root/boot/vmlinuz-virt</kernel>
    <initrd>/mnt/virtiofs/volume/scsi-tgt/root/boot/initramfs-virt</initrd>
    <cmdline>rootfstype=virtiofs root=root rw iommu=pt</cmdline>
  </os>

> 
> Pls send kernel .config also.
> 

Kernel .config: https://bugzilla.kernel.org/attachment.cgi?id=305570
Kernel dmesg: https://bugzilla.kernel.org/attachment.cgi?id=305569

Comment 11 Borislav Petkov 2024-01-02 15:41:28 UTC

Queued, closing.

https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/commit/?h=bleeding-edge&id=c8f5caec3df84a02b937d6d9cda1f7ffa8dc443f

Note You need to log in before you can comment on or make changes to this bug.