Bug 218245

Summary: default_enter_idle leaked IRQ state
Product: Platform Specific/Hardware Reporter: Forza (forza)
Component: x86-64Assignee: platform_x86_64 (platform_x86_64)
Status: RESOLVED CODE_FIX    
Severity: normal CC: bp
Priority: P3    
Hardware: All   
OS: Linux   
Kernel Version: Subsystem:
Regression: No Bisected commit-id:
Attachments: dmesg
Kernel .config
mark __cpuidle
Do not enable IRQs either

Description Forza 2023-12-09 11:41:14 UTC
Since kernels 6.2 or maybe 6.3 up until current 6.6.5 there is a spurious warning of leaked IRQ state in my Vms rubbning under KVM QEMU on a AMD Ryzen host.

The issue does not seem to cause any real problems, but it probably shouldn't happen. 

[   22.773887][    T0] ------------[ cut here ]------------
[   22.773897][    T0] default_enter_idle leaked IRQ state
[   22.773981][    T0] WARNING: CPU: 1 PID: 0 at drivers/cpuidle/cpuidle.c:269 cpuidle_enter_state+0x2bb/0x2c0
[   22.774001][    T0] Modules linked in:
[   22.774004][    T0] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.6.5-forza-virt #3
[   22.774007][    T0] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.0-20220807_005459-localhost 04/01/2014
[   22.774009][    T0] RIP: 0010:cpuidle_enter_state+0x2bb/0x2c0
[   22.774012][    T0] Code: c6 fe ff ff e9 7f fe ff ff 41 be 18 00 00 00 31 db eb 82 49 8b 76 50 48 c7 c7 b3 b0 5f a4 c6 05 06 4f f9 00 01 e8 55 bb 31 ff <0f> 0b eb c0 cc eb 0e 48 89 f8 48 89 d1 f3 a4 c3 0f 1f 44 00 00 48
[   22.774015][    T0] RSP: 0018:ffffac8b800a7e90 EFLAGS: 00010246
[   22.774017][    T0] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[   22.774018][    T0] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[   22.774020][    T0] RBP: ffff8bcd5fca8b40 R08: 0000000000000000 R09: 0000000000000000
[   22.774021][    T0] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa4d93620
[   22.774022][    T0] R13: 0000000000000001 R14: ffffffffa4d936a0 R15: 0000000000000000
[   22.774025][    T0] FS:  0000000000000000(0000) GS:ffff8bcd5fc80000(0000) knlGS:0000000000000000
[   22.774028][    T0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   22.774029][    T0] CR2: 00007f54f3488020 CR3: 000000001ec16000 CR4: 00000000003506e0
[   22.774031][    T0] Call Trace:
[   22.774042][    T0]  <TASK>
[   22.774047][    T0]  ? __warn+0x6b/0xd0
[   22.774055][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[   22.774057][    T0]  ? report_bug+0x140/0x180
[   22.774062][    T0]  ? handle_bug+0x3c/0x70
[   22.774064][    T0]  ? exc_invalid_op+0x13/0x70
[   22.774066][    T0]  ? asm_exc_invalid_op+0x16/0x20
[   22.774071][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[   22.774074][    T0]  cpuidle_enter+0x24/0x40
[   22.774087][    T0]  do_idle+0x1c5/0x220
[   22.774099][    T0]  cpu_startup_entry+0x21/0x30
[   22.774102][    T0]  start_secondary+0x100/0x120
[   22.774111][    T0]  secondary_startup_64_no_verify+0x178/0x17b
[   22.774122][    T0]  </TASK>
[   22.774122][    T0] ---[ end trace 0000000000000000 ]---
Comment 1 Forza 2023-12-09 11:59:32 UTC
Created attachment 305569 [details]
dmesg
Comment 2 Forza 2023-12-09 12:00:49 UTC
Created attachment 305570 [details]
Kernel .config
Comment 3 Forza 2023-12-09 13:34:17 UTC
I am currently on QEMU 8.1.3 but have tried most versions since 7.2.

Is there any specific debug options I can enable to help debug this issue?
Comment 4 Borislav Petkov 2023-12-09 15:15:16 UTC
Created attachment 305573 [details]
mark __cpuidle

See if this fixes it.
Comment 5 Forza 2023-12-09 18:07:19 UTC
(In reply to Borislav Petkov from comment #4)
> Created attachment 305573 [details]
> mark __cpuidle
> 
> See if this fixes it.

Thanks. Unfortunately same error.


[    2.267349][    T1] Freeing unused kernel image (initmem) memory: 1432K
[    2.348136][    T1] Write protecting the kernel read-only data: 28672k
[    2.350132][    T1] Freeing unused kernel image (rodata/data gap) memory: 680K
[    2.352000][    T1] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    2.352494][    T1] Run /sbin/init as init process
[    2.352929][    T1]   with arguments:
[    2.352931][    T1]     /sbin/init
[    2.352932][    T1]   with environment:
[    2.352933][    T1]     HOME=/
[    2.352934][    T1]     TERM=linux
[    3.130816][ T1382] udevd[1382]: starting version 3.2.14
[    3.200760][ T1382] udevd[1382]: starting eudev-3.2.14
[    3.683869][    T0] ------------[ cut here ]------------
[    3.683877][    T0] default_enter_idle leaked IRQ state
[    3.683966][    T0] WARNING: CPU: 3 PID: 0 at drivers/cpuidle/cpuidle.c:269 cpuidle_enter_state+0x2bb/0x2c0
[    3.683971][    T0] Modules linked in:
[    3.683973][    T0] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.6.5-forza-virt #4
[    3.683976][    T0] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.0-20220807_005459-localhost 04/01/2014
[    3.683977][    T0] RIP: 0010:cpuidle_enter_state+0x2bb/0x2c0
[    3.683980][    T0] Code: c6 fe ff ff e9 7f fe ff ff 41 be 18 00 00 00 31 db eb 82 49 8b 76 50 48 c7 c7 b3 b0 5f 96 c6 05 36 4f f9 00 01 e8 85 bb 31 ff <0f> 0b eb c0 cc eb 0e 48 89 f8 48 89 d1 f3 a4 c3 0f 1f 44 00 00 48
[    3.683982][    T0] RSP: 0018:ffffae6e800b7e90 EFLAGS: 00010246
[    3.683984][    T0] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[    3.683985][    T0] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    3.683986][    T0] RBP: ffff98f61e5a83d0 R08: 0000000000000000 R09: 0000000000000000
[    3.683987][    T0] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff96d93620
[    3.683988][    T0] R13: 0000000000000001 R14: ffffffff96d936a0 R15: 0000000000000000
[    3.683990][    T0] FS:  0000000000000000(0000) GS:ffff98f61e580000(0000) knlGS:0000000000000000
[    3.683995][    T0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.683998][    T0] CR2: 00007ffda7c10bb8 CR3: 0000000009c16000 CR4: 00000000003506e0
[    3.684000][    T0] Call Trace:
[    3.684009][    T0]  <TASK>
[    3.684011][    T0]  ? __warn+0x6b/0xd0
[    3.684027][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[    3.684030][    T0]  ? report_bug+0x140/0x180
[    3.684034][    T0]  ? handle_bug+0x3c/0x70
[    3.684036][    T0]  ? exc_invalid_op+0x13/0x70
[    3.684038][    T0]  ? asm_exc_invalid_op+0x16/0x20
[    3.684042][    T0]  ? cpuidle_enter_state+0x2bb/0x2c0
[    3.684044][    T0]  cpuidle_enter+0x24/0x40
[    3.684064][    T0]  do_idle+0x1c5/0x220
[    3.684075][    T0]  cpu_startup_entry+0x21/0x30
[    3.684077][    T0]  start_secondary+0x100/0x120
[    3.684088][    T0]  secondary_startup_64_no_verify+0x178/0x17b
[    3.684098][    T0]  </TASK>
[    3.684099][    T0] ---[ end trace 0000000000000000 ]---
Comment 6 Borislav Petkov 2023-12-09 18:54:32 UTC
Bah, that was only half the truth. The cpuidle governors must return with IRQs disabled now. And haltpoll doesn't. Yet. See if this next patch fixes it...

It might not but I don't know because I can't reproduce your issue. How do you boot your guest? Any cmdline args selecting haltpoll as a cpuidle governor?

Pls send kernel .config also.

Thx.
Comment 7 Borislav Petkov 2023-12-09 18:55:39 UTC
Created attachment 305574 [details]
Do not enable IRQs either

Needs to not enable IRQs.
Comment 8 Forza 2023-12-09 21:14:33 UTC
(In reply to Borislav Petkov from comment #7)
> Created attachment 305574 [details]
> Do not enable IRQs either
> 
> Needs to not enable IRQs.

Initial tests seems good. I'll test more over the next few days.
Comment 9 Forza 2023-12-15 21:23:01 UTC
So far there has been no further occurrence, so I'd say the patch has fixed the issue.

Thanks
Comment 10 Forza 2023-12-22 19:39:50 UTC
(In reply to Borislav Petkov from comment #6)

> It might not but I don't know because I can't reproduce your issue. How do
> you boot your guest? Any cmdline args selecting haltpoll as a cpuidle
> governor?

Forgot to answer this question. I do not choose any specific cpuidle governor. Actually, I think the kernel forces it when it detects it is running in a KVM guest.

The kernel menuconfig says:

Prompt: KVM Guest support (including kvmclock)
Depends on: HYPERVISOR_GUEST [=y] && PARAVIRT [=y]
Location:
	-> Processor type and features
		-> Linux guest support (HYPERVISOR_GUEST [=y])
			-> KVM Guest support (including kvmclock) (KVM_GUEST [=y])
Selects: PARAVIRT_CLOCK [=y] && ARCH_CPUIDLE_HALTPOLL [=y] && X86_HV_CALLBACK_VECTOR [=y]



I boot most VMs with the following libvirt XML:

  <os>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
    <kernel>/mnt/virtiofs/volume/scsi-tgt/root/boot/vmlinuz-virt</kernel>
    <initrd>/mnt/virtiofs/volume/scsi-tgt/root/boot/initramfs-virt</initrd>
    <cmdline>rootfstype=virtiofs root=root rw iommu=pt</cmdline>
  </os>

> 
> Pls send kernel .config also.
> 

Kernel .config: https://bugzilla.kernel.org/attachment.cgi?id=305570
Kernel dmesg: https://bugzilla.kernel.org/attachment.cgi?id=305569