Bug 218288
| Summary: | [REGRESSION] WARNING: CPU: 0 ... at kernel/workqueue.c:1638 __queue_work+0x329/0x440 followed by a BUG | ||
|---|---|---|---|
| Product: | Drivers | Reporter: | michallinuxstuff |
| Component: | Network | Assignee: | drivers_network (drivers_network) |
| Status: | NEW --- | ||
| Severity: | normal | ||
| Priority: | P3 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | Subsystem: | ||
| Regression: | No | Bisected commit-id: | |
|
Description
michallinuxstuff
2023-12-18 21:50:37 UTC
(In reply to michallinuxstuff from comment #0) > Any hints would be appreciated. If this is a somewhat recent regression (see https://docs.kernel.org/admin-guide/reporting-regressions.html ) it should be fixed, but at least to me it's unclear if this is really a regression. Hence allow me to ask: Did this work earlier on the particular machine? Side note: the developers you are trying to contact might not see this report. Search for bugzilla in https://docs.kernel.org/admin-guide/reporting-issues.html and https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kernel-bug-reports-are-ignored/ to understand why; yes, I wish it would be different, but that's outside of my control. (In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #1) > (In reply to michallinuxstuff from comment #0) > > > Any hints would be appreciated. > > If this is a somewhat recent regression (see > https://docs.kernel.org/admin-guide/reporting-regressions.html ) it should > be fixed, but at least to me it's unclear if this is really a regression. > Hence allow me to ask: Did this work earlier on the particular machine? > > Side note: the developers you are trying to contact might not see this > report. Search for bugzilla in > https://docs.kernel.org/admin-guide/reporting-issues.html and > https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux- > kernel-bug-reports-are-ignored/ to understand why; yes, I wish it would be > different, but that's outside of my control. The whole issue is intermittent in nature, it's not 100% reproducible - it takes couple of spins to trigger it. But I do believe that soon after this particular test was introduced we started seeing this issue popping up (both under 6.1.14 and 6.5.12). Is there anything I could do to move this forward? :) I can collect kernel's core and fetch some data from it if needed. My c-fu is not that great so I would need some hints on what info would be useful for this case - I can basically provide anything that can be extracted from the vmcore via the crash util for instance. Marking as [REGRESSION] since my gut tells me it's too closely tied to the code path touched by the https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d1c1379d71777ddeda3e54f8fc26e9ecbfd1009. Under 6.5.12, sometimes, instead of an Oops this WARN is caught: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/kernel/workqueue.c?h=v6.5.12#n1638 and kernel continues (panic_on_warn == 0). But sooner or later, after spinning the test multiple times, we end up with the actual BUG again. Here's a userspace process which actively spins during the test and which holds on to the mlx code path:
crash> foreach bt | grep reactor
WARNING: possibly bogus exception frame
PID: 60215 TASK: ffff9d211e8f0000 CPU: 0 COMMAND: "reactor_0"
PID: 60219 TASK: ffff9d2107388000 CPU: 1 COMMAND: "reactor_1"
PID: 60257 TASK: ffff9d2107564000 CPU: 2 COMMAND: "reactor_2"
crash> set 60215
PID: 60215
COMMAND: "reactor_0"
TASK: ffff9d211e8f0000 [THREAD_INFO: ffff9d211e8f0000]
CPU: 0
STATE: TASK_WAKING
crash> bt
PID: 60215 TASK: ffff9d211e8f0000 CPU: 0 COMMAND: "reactor_0"
#0 [ffffab7b471b3650] __schedule at ffffffffb6fd322e
#1 [ffffab7b471b3708] schedule at ffffffffb6fd436e
#2 [ffffab7b471b3720] schedule_timeout at ffffffffb6fdab98
#3 [ffffab7b471b3770] wait_for_completion_timeout at ffffffffb6fd5293
#4 [ffffab7b471b37d0] cmd_exec at ffffffffc084c58d [mlx5_core]
#5 [ffffab7b471b3850] mlx5_cmd_do at ffffffffc084cab2 [mlx5_core]
#6 [ffffab7b471b3880] mlx5_cmd_exec at ffffffffc084cb0b [mlx5_core]
#7 [ffffab7b471b38a0] mlx5_query_nic_vport_min_inline at ffffffffc085e6d2 [mlx5_core]
#8 [ffffab7b471b39d8] mlx5_query_min_inline at ffffffffc085e766 [mlx5_core]
#9 [ffffab7b471b39e8] set_ucontext_resp at ffffffffc0e9f723 [mlx5_ib]
#10 [ffffab7b471b3a08] mlx5_ib_alloc_ucontext at ffffffffc0ea277f [mlx5_ib]
#11 [ffffab7b471b3ac0] ib_init_ucontext at ffffffffc0d97513 [ib_uverbs]
#12 [ffffab7b471b3b00] ib_uverbs_handler_UVERBS_METHOD_GET_CONTEXT at ffffffffc0d9ef4d [ib_uverbs]
#13 [ffffab7b471b3b30] ib_uverbs_cmd_verbs at ffffffffc0d9bef0 [ib_uverbs]
#14 [ffffab7b471b3dc8] ib_uverbs_ioctl at ffffffffc0d9c068 [ib_uverbs]
#15 [ffffab7b471b3e08] __x64_sys_ioctl at ffffffffb64787a4
#16 [ffffab7b471b3e38] do_syscall_64 at ffffffffb6fbe46d
#17 [ffffab7b471b3f50] entry_SYSCALL_64_after_hwframe at ffffffffb70000ea
RIP: 00007f0e910c1ecd RSP: 00007ffcf14908a0 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007ffcf14909e0 RCX: 00007f0e910c1ecd
RDX: 00007ffcf1490a00 RSI: 00000000c0181b01 RDI: 00000000000000f3
RBP: 00007ffcf14908f0 R8: 0000000001000010 R9: 0000000000000002
R10: 0000000000000050 R11: 0000000000000246 R12: 00007ffcf1490a18
R13: 00007ffcf1490a18 R14: 0000000000decc00 R15: 0000000000000001
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
crash>
Here's latest, "enhanced" stack dump (passed through decode_stacktrace.sh) that I got: [ 632.895084] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 632.902404] #PF: supervisor read access in kernel mode [ 632.907784] #PF: error_code(0x0000) - not-present page [ 632.913138] PGD 800000011f3a9067 P4D 800000011f3a9067 PUD 2d75bd067 PMD 0 [ 632.920226] Oops: 0000 [#1] PREEMPT SMP PTI [ 632.934829] Hardware name: Intel Corporation S2600WFQ/S2600WFQ, BIOS SE5C620.86B.00.01.0014.070920180847 07/09/2018 [ 632.945476] RIP: 0010:__queue_work (kernel/workqueue.c:1659 (discriminator 1)) [ 632.950052] Code: 8b 33 40 f6 c6 04 75 cf 48 c1 ee 05 81 fe ff ff ff 7f 0f 84 ae 00 00 00 48 63 f6 48 c7 c7 a0 cb 25 b8 e8 44 16 e6 00 49 89 c7 <48> 8b 7d 00 4d 85 ff 0f 84 93 00 00 00 49 39 ff 0f 84 8a 00 00 00 All code ======== 0: 8b 33 mov (%rbx),%esi 2: 40 f6 c6 04 test $0x4,%sil 6: 75 cf jne 0xffffffffffffffd7 8: 48 c1 ee 05 shr $0x5,%rsi c: 81 fe ff ff ff 7f cmp $0x7fffffff,%esi 12: 0f 84 ae 00 00 00 je 0xc6 18: 48 63 f6 movslq %esi,%rsi 1b: 48 c7 c7 a0 cb 25 b8 mov $0xffffffffb825cba0,%rdi 22: e8 44 16 e6 00 call 0xe6166b 27: 49 89 c7 mov %rax,%r15 2a:* 48 8b 7d 00 mov 0x0(%rbp),%rdi <-- trapping instruction 2e: 4d 85 ff test %r15,%r15 31: 0f 84 93 00 00 00 je 0xca 37: 49 39 ff cmp %rdi,%r15 3a: 0f 84 8a 00 00 00 je 0xca Code starting with the faulting instruction =========================================== 0: 48 8b 7d 00 mov 0x0(%rbp),%rdi 4: 4d 85 ff test %r15,%r15 7: 0f 84 93 00 00 00 je 0xa0 d: 49 39 ff cmp %rdi,%r15 10: 0f 84 8a 00 00 00 je 0xa0 [ 632.969245] RSP: 0018:ffffab7b40003e78 EFLAGS: 00010046 [ 632.974697] RAX: ffff9d2100c59c00 RBX: ffff9d2885a58c48 RCX: 0000000000000000 [ 632.982052] RDX: ffff9d2100c59c00 RSI: ffff9d2101001b60 RDI: 00000000000000e0 [ 632.989407] RBP: 0000000000000000 R08: ffff9d2101001c88 R09: ffffffffb825cba0 [ 632.996758] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9d28c5680800 [ 633.004107] R13: 0000000000002000 R14: 0000000000000000 R15: ffff9d2100c59c00 [ 633.011452] FS: 0000000000000000(0000) GS:ffff9d285f400000(0000) knlGS:0000000000000000 [ 633.019755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 633.025712] CR2: 0000000000000000 CR3: 0000000115b56002 CR4: 00000000007706f0 [ 633.033055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 633.040400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 633.047738] PKRU: 55555554 [ 633.050653] Call Trace: [ 633.053302] <IRQ> [ 633.055518] ? __die (arch/x86/kernel/dumpstack.c:421 arch/x86/kernel/dumpstack.c:434) [ 633.058779] ? page_fault_oops (arch/x86/mm/fault.c:707 (discriminator 1)) [ 633.063076] ? exc_page_fault (./arch/x86/include/asm/paravirt.h:695 arch/x86/mm/fault.c:1494 arch/x86/mm/fault.c:1542) [ 633.067198] ? asm_exc_page_fault (./arch/x86/include/asm/idtentry.h:570) [ 633.071585] ? __queue_work (kernel/workqueue.c:1659 (discriminator 1)) [ 633.075534] ? __queue_work (kernel/workqueue.c:787 kernel/workqueue.c:1658) [ 633.079479] ? __pfx_delayed_work_timer_fn (kernel/workqueue.c:1841) [ 633.084644] call_timer_fn (kernel/time/timer.c:1700) [ 633.088510] ? __pfx_delayed_work_timer_fn (kernel/workqueue.c:1841) [ 633.093679] __run_timers (kernel/time/timer.c:1747 kernel/time/timer.c:2022) [ 633.097547] run_timer_softirq (kernel/time/timer.c:2037 (discriminator 1)) [ 633.101681] __do_softirq (kernel/softirq.c:553) [ 633.105470] __irq_exit_rcu (kernel/softirq.c:427 kernel/softirq.c:632) [ 633.109345] sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1111 (discriminator 47)) [ 633.114347] </IRQ> [ 633.116658] <TASK> [ 633.118974] asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:645) [ 633.124335] RIP: 0010:cpuidle_enter_state (drivers/cpuidle/cpuidle.c:291) [ 633.129520] Code: aa 2c 1c ff e8 c5 f3 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 93 32 1b ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d All code ======== 0: aa stos %al,%es:(%rdi) 1: 2c 1c sub $0x1c,%al 3: ff (bad) 4: e8 c5 f3 ff ff call 0xfffffffffffff3ce 9: 8b 53 04 mov 0x4(%rbx),%edx c: 49 89 c5 mov %rax,%r13 f: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 14: 31 ff xor %edi,%edi 16: e8 93 32 1b ff call 0xffffffffff1b32ae 1b: 45 84 ff test %r15b,%r15b 1e: 0f 85 56 02 00 00 jne 0x27a 24: fb sti 25: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 2a:* 45 85 f6 test %r14d,%r14d <-- trapping instruction 2d: 0f 88 85 01 00 00 js 0x1b8 33: 49 63 d6 movslq %r14d,%rdx 36: 48 8d 04 52 lea (%rdx,%rdx,2),%rax 3a: 48 8d 04 82 lea (%rdx,%rax,4),%rax 3e: 49 rex.WB 3f: 8d .byte 0x8d Code starting with the faulting instruction =========================================== 0: 45 85 f6 test %r14d,%r14d 3: 0f 88 85 01 00 00 js 0x18e 9: 49 63 d6 movslq %r14d,%rdx c: 48 8d 04 52 lea (%rdx,%rdx,2),%rax 10: 48 8d 04 82 lea (%rdx,%rax,4),%rax 14: 49 rex.WB 15: 8d .byte 0x8d [ 633.148726] RSP: 0018:ffffffffb8203e28 EFLAGS: 00000246 [ 633.154181] RAX: ffff9d285f433e00 RBX: ffffcb7322c01388 RCX: 000000000000001f [ 633.161554] RDX: 0000000000000000 RSI: 000000003351fed6 RDI: 0000000000000000 [ 633.168926] RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000018 [ 633.176288] R10: ffff9d285f4327c4 R11: 0000000000000016 R12: ffffffffb843b0a0 [ 633.183654] R13: 000000935b7c7cf4 R14: 0000000000000002 R15: 0000000000000000 [ 633.191025] ? cpuidle_enter_state (drivers/cpuidle/cpuidle.c:285) [ 633.195611] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 (discriminator 2)) [ 633.199424] do_idle (kernel/sched/idle.c:219 kernel/sched/idle.c:282) [ 633.202883] cpu_startup_entry (kernel/sched/idle.c:379) [ 633.207029] rest_init (usercopy_64.c:?) [ 633.210474] arch_call_rest_init+0xe/0x30 [ 633.214696] start_kernel (init/main.c:833 (discriminator 1) init/main.c:909 (discriminator 1)) [ 633.218561] x86_64_start_reservations (arch/x86/kernel/head64.c:544) [ 633.223377] x86_64_start_kernel (arch/x86/kernel/head64.c:486 (discriminator 5)) [ 633.227674] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:441) [ 633.233100] </TASK> [ 633.235478] Modules linked in: nvme_rdma nvme_fabrics vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd rdma_ucm rdma_cm iw_cm ib_umad ib_cm usdm_drv(OE) rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common sunrpc skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel binfmt_misc kvm ipmi_ssif irqbypass crct10dif_pclmul qat_c62x(OE) crc32_pclmul crc32c_intel polyval_clmulni polyval_generic intel_qat(OE) mlx5_ib ghash_clmulni_intel sha512_ssse3 rapl ib_uverbs iTCO_wdt intel_pmc_bxt iTCO_vendor_support intel_cstate acpi_ipmi mei_me ast ipmi_si i2c_i801 ioatdma ib_core dax_pmem intel_uncore pcspkr mei i2c_smbus i2c_algo_bit ipmi_devintf uio lpc_ich intel_pch_thermal wmi dca ipmi_msghandler acpi_power_meter acpi_pad joydev ip6_tables ip_tables fuse zram bpf_preload loop overlay squashfs netconsole nd_pmem nd_btt nd_e820 libnvdimm virtio_blk virtio_net net_failover failover mlx5_core mlxfw psample tls pci_hyperv_intf nvme nvme_core nvme_common [ 633.235550] ice(OE) gnss i40e [last unloaded: nvme_fabrics] [ 633.332894] CR2: 0000000000000000 (In reply to michallinuxstuff from comment #3) > Is there anything I could do to move this forward? :) As mentioned earlier: you are unlikely to reach the developers here which you apparently try to contact. You should follow https://docs.kernel.org/admin-guide/reporting-issues.html 6.5 is EOL now anyway, so developers most likely want to know if current kernels (e.g. 6.7 and 6.8-rc1 once it's out) are still affected. Sorry, I wish things were different, but that's how it is. /me unCCs himself Fair enough - will try to test it out under 6.7 then and report my findings under proper mailing list instead. Thanks! |