Hello. (I apologize if I chose the wrong "Product" and "Component".) On two of my systems, I see strange "bug" when running 6+ kernels (below is a recent one): ``` May 14 14:48:07 smoon7.bkoty.ru kernel: RIP: 0010:__filemap_get_folio+0xbf/0x6a0 May 14 14:48:07 smoon7.bkoty.ru kernel: Code: ef e8 c5 60 c3 00 48 89 c7 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 0f 84 6a 04 00 00 a8 01 0f 85 6c 04 00 00 <8b> 40 34 85 c0 74 c4 8d 50 01 4c 8d 47 34 f0 0f b1 57 34 75 ee 48 May 14 14:48:07 smoon7.bkoty.ru kernel: RSP: 0000:ffffa7800b1dfbf8 EFLAGS: 00010246 May 14 14:48:07 smoon7.bkoty.ru kernel: RAX: 0000000000000002 RBX: 0000000000000000 RCX: 0000000000000004 May 14 14:48:07 smoon7.bkoty.ru kernel: RDX: ffffa7800b1dfc50 RSI: ffff9a2413646910 RDI: 0000000000000002 May 14 14:48:07 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 00007f862b600000 May 14 14:48:07 smoon7.bkoty.ru kernel: R10: 00007f8659246f48 R11: ffff9a21c1494a0c R12: 000000000002dc46 May 14 14:48:07 smoon7.bkoty.ru kernel: R13: ffffa7800b1dfc50 R14: ffff9a21e2cb82b0 R15: 00007f8659246f48 May 14 14:48:07 smoon7.bkoty.ru kernel: FS: 00007f87fcff96c0(0000) GS:ffff9a295e280000(0000) knlGS:0000000000000000 May 14 14:48:07 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 14 14:48:07 smoon7.bkoty.ru kernel: CR2: 0000000000000036 CR3: 0000000105b2c003 CR4: 00000000003706e0 May 14 14:48:07 smoon7.bkoty.ru kernel: Call Trace: May 14 14:48:07 smoon7.bkoty.ru kernel: <TASK> May 14 14:48:07 smoon7.bkoty.ru kernel: ? psi_group_change+0x274/0x430 May 14 14:48:07 smoon7.bkoty.ru kernel: filemap_fault+0x6f/0xfd0 May 14 14:48:07 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x15f/0x640 May 14 14:48:07 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 May 14 14:48:07 smoon7.bkoty.ru kernel: do_fault+0x1d7/0x400 May 14 14:48:07 smoon7.bkoty.ru kernel: handle_mm_fault+0xb48/0x1450 May 14 14:48:07 smoon7.bkoty.ru kernel: do_user_addr_fault+0x1c7/0x740 May 14 14:48:07 smoon7.bkoty.ru kernel: exc_page_fault+0x7c/0x180 May 14 14:48:07 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 May 14 14:48:07 smoon7.bkoty.ru kernel: RIP: 0033:0x7f881a56cb0d May 14 14:48:07 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe May 14 14:48:07 smoon7.bkoty.ru kernel: RSP: 002b:00007f87fcff72c8 EFLAGS: 00010202 May 14 14:48:07 smoon7.bkoty.ru kernel: RAX: 00007f87dc02a700 RBX: 00007f87fcff8308 RCX: 00007f87fcff7500 May 14 14:48:07 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f8659246f48 RDI: 00007f87dc02a700 May 14 14:48:07 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 May 14 14:48:07 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 May 14 14:48:07 smoon7.bkoty.ru kernel: R13: 00007f87dc001370 R14: 0000000000000009 R15: 00005645d0719a70 May 14 14:48:07 smoon7.bkoty.ru kernel: </TASK> ``` I've seen these errors since the very first kernel of the 6 series, while I see no problem with 5.15 on the same hardware. These two systems have the same CPU (Intel(R) Core(TM) i5-10500 CPU @ 3.10GHz) but slightly different motherboards, same amount of memory (same manufacturer, I tested it when plugged in). The hosts in question don't show this "bug" immediately, but after some time while having "heavy" disk load (torrents). The "bug" shows up whether I use `mitigations=off` or not (at first I thought the "bug" might be related to `mitigations=off`, but I got the above output when I removed that setting from the kernel command line). What puzzles me is that I don't see these errors on the other hosts (but they don't have "heavy" disk loads), they work just fine. On the other hand, they have different CPUs (not i5-10500). Sometimes (less often than this error) I saw the following in the kernel log (dmesg): ``` May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: Machine check events logged May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 0: 9000004000010005 May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: TSC 95596a63008b May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: PROCESSOR 0:a0653 TIME 1684022949 SOCKET 0 APIC 0 microcode f6 May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: Machine check events logged May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: CPU 5: Machine Check: 0 Bank 0: 9000004000010005 May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: TSC 95c56b82abf0 May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: PROCESSOR 0:a0653 TIME 1684023099 SOCKET 0 APIC a microcode f6 ``` So now I'm thinking of buying a new CPU (same socket) and see if I will see the same error.
(In reply to Vladimir Lomov from comment #0) > Hello. > > (I apologize if I chose the wrong "Product" and "Component".) > > On two of my systems, I see strange "bug" when running 6+ kernels (below is > a recent one): > > ``` > May 14 14:48:07 smoon7.bkoty.ru kernel: RIP: > 0010:__filemap_get_folio+0xbf/0x6a0 > May 14 14:48:07 smoon7.bkoty.ru kernel: Code: ef e8 c5 60 c3 00 48 89 c7 48 > 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 0f 84 6a 04 00 00 a8 > 01 0f 85 6c 04 00 00 <8b> 40 34 85 c0 74 c4 8d 50 01 4c 8d 47 34 f0 0f b1 57 > 34 75 ee 48 > May 14 14:48:07 smoon7.bkoty.ru kernel: RSP: 0000:ffffa7800b1dfbf8 EFLAGS: > 00010246 > May 14 14:48:07 smoon7.bkoty.ru kernel: RAX: 0000000000000002 RBX: > 0000000000000000 RCX: 0000000000000004 > May 14 14:48:07 smoon7.bkoty.ru kernel: RDX: ffffa7800b1dfc50 RSI: > ffff9a2413646910 RDI: 0000000000000002 > May 14 14:48:07 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: > ffffffffffffffc0 R09: 00007f862b600000 > May 14 14:48:07 smoon7.bkoty.ru kernel: R10: 00007f8659246f48 R11: > ffff9a21c1494a0c R12: 000000000002dc46 > May 14 14:48:07 smoon7.bkoty.ru kernel: R13: ffffa7800b1dfc50 R14: > ffff9a21e2cb82b0 R15: 00007f8659246f48 > May 14 14:48:07 smoon7.bkoty.ru kernel: FS: 00007f87fcff96c0(0000) > GS:ffff9a295e280000(0000) knlGS:0000000000000000 > May 14 14:48:07 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > May 14 14:48:07 smoon7.bkoty.ru kernel: CR2: 0000000000000036 CR3: > 0000000105b2c003 CR4: 00000000003706e0 > May 14 14:48:07 smoon7.bkoty.ru kernel: Call Trace: > May 14 14:48:07 smoon7.bkoty.ru kernel: <TASK> > May 14 14:48:07 smoon7.bkoty.ru kernel: ? psi_group_change+0x274/0x430 > May 14 14:48:07 smoon7.bkoty.ru kernel: filemap_fault+0x6f/0xfd0 > May 14 14:48:07 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x15f/0x640 > May 14 14:48:07 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 > May 14 14:48:07 smoon7.bkoty.ru kernel: do_fault+0x1d7/0x400 > May 14 14:48:07 smoon7.bkoty.ru kernel: handle_mm_fault+0xb48/0x1450 > May 14 14:48:07 smoon7.bkoty.ru kernel: do_user_addr_fault+0x1c7/0x740 > May 14 14:48:07 smoon7.bkoty.ru kernel: exc_page_fault+0x7c/0x180 > May 14 14:48:07 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 > May 14 14:48:07 smoon7.bkoty.ru kernel: RIP: 0033:0x7f881a56cb0d > May 14 14:48:07 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f > 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 > f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c > 16 e0 c5 fe > May 14 14:48:07 smoon7.bkoty.ru kernel: RSP: 002b:00007f87fcff72c8 EFLAGS: > 00010202 > May 14 14:48:07 smoon7.bkoty.ru kernel: RAX: 00007f87dc02a700 RBX: > 00007f87fcff8308 RCX: 00007f87fcff7500 > May 14 14:48:07 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: > 00007f8659246f48 RDI: 00007f87dc02a700 > May 14 14:48:07 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: > 0000000000000000 R09: 0000000000000000 > May 14 14:48:07 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: > 0000000000000246 R12: 0000000000000000 > May 14 14:48:07 smoon7.bkoty.ru kernel: R13: 00007f87dc001370 R14: > 0000000000000009 R15: 00005645d0719a70 > May 14 14:48:07 smoon7.bkoty.ru kernel: </TASK> > ``` > > I've seen these errors since the very first kernel of the 6 series, while I > see no problem with 5.15 on the same hardware. > > These two systems have the same CPU (Intel(R) Core(TM) i5-10500 CPU @ > 3.10GHz) but slightly different motherboards, same amount of memory (same > manufacturer, I tested it when plugged in). > > The hosts in question don't show this "bug" immediately, but after some time > while having "heavy" disk load (torrents). The "bug" shows up whether I use > `mitigations=off` or not (at first I thought the "bug" might be related to > `mitigations=off`, but I got the above output when I removed that setting > from the kernel command line). > > What puzzles me is that I don't see these errors on the other hosts (but > they don't have "heavy" disk loads), they work just fine. On the other hand, > they have different CPUs (not i5-10500). Sometimes (less often than this > error) I saw the following in the kernel log (dmesg): > > ``` > May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: Machine check > events logged > May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: CPU 0: > Machine Check: 0 Bank 0: 9000004000010005 > May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: TSC > 95596a63008b > May 14 08:09:09 smoon7.bkoty.ru kernel: mce: [Hardware Error]: PROCESSOR > 0:a0653 TIME 1684022949 SOCKET 0 APIC 0 microcode f6 > May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: Machine check > events logged > May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: CPU 5: > Machine Check: 0 Bank 0: 9000004000010005 > May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: TSC > 95c56b82abf0 > May 14 08:11:39 smoon7.bkoty.ru kernel: mce: [Hardware Error]: PROCESSOR > 0:a0653 TIME 1684023099 SOCKET 0 APIC a microcode f6 > ``` > > So now I'm thinking of buying a new CPU (same socket) and see if I will see > the same error. Can you do a bisection between v5.15..v6.0?
Hello. Bagas Sanjaya: > Can you do a bisection between v5.15..v6.0? I have never done this before. I'll try to follow https://wiki.archlinux.org/title/Bisecting_bugs_with_Git and build kernel packages to test. Does the kernel have a way to report which commit it was built from? Also, please note, that it takes a day or day and a half to "catch" this "bug", it would take some time (as well as building the kernel).
I'm sorry, but I ran into an error while bisecting. I did the following steps: 1. (Archlinux specific) download the linux-mainline-git package (from AUR); 2. Built it (OK); 3. cd src/linux-torvalds 4. git bisect start 5. git bisect good v5.15 6. git bisect bad v6.0 7. cd ../../ 8. makepkg -ef 9. Answered some questions from `make olddefconfig`; 10. but the kernel (package) build was stopped at ``` ... GEN .version CHK include/generated/compile.h UPD include/generated/compile.h CC init/version.o AR init/built-in.a LD vmlinux.o MODPOST vmlinux.symvers MODINFO modules.builtin.modinfo GEN modules.builtin LD .tmp_vmlinux.btf ld: warning: .tmp_vmlinux.btf has a LOAD segment with RWX permissions BTF .btf.vmlinux.bin.o LD .tmp_vmlinux.kallsyms1 ld: warning: .tmp_vmlinux.kallsyms1 has a LOAD segment with RWX permissions KSYMS .tmp_vmlinux.kallsyms1.S AS .tmp_vmlinux.kallsyms1.S LD .tmp_vmlinux.kallsyms2 ld: warning: .tmp_vmlinux.kallsyms2 has a LOAD segment with RWX permissions KSYMS .tmp_vmlinux.kallsyms2.S AS .tmp_vmlinux.kallsyms2.S LD vmlinux ld: warning: vmlinux has a LOAD segment with RWX permissions BTFIDS vmlinux FAILED: load BTF from vmlinux: Invalid argument make: *** [Makefile:1155: vmlinux] Error 255 make: *** Deleting file 'vmlinux' ``` It may be related to the kernel configuration, but then I don't know what to do.
On 5/16/23 14:27, bugzilla-daemon@kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > > --- Comment #3 from Vladimir Lomov (lomov.vl@bkoty.ru) --- > I'm sorry, but I ran into an error while bisecting. > > I did the following steps: > 1. (Archlinux specific) download the linux-mainline-git package (from AUR); > 2. Built it (OK); > 3. cd src/linux-torvalds > 4. git bisect start > 5. git bisect good v5.15 > 6. git bisect bad v6.0 > 7. cd ../../ > 8. makepkg -ef > 9. Answered some questions from `make olddefconfig`; > 10. but the kernel (package) build was stopped at > ``` > ... > GEN .version > CHK include/generated/compile.h > UPD include/generated/compile.h > CC init/version.o > AR init/built-in.a > LD vmlinux.o > MODPOST vmlinux.symvers > MODINFO modules.builtin.modinfo > GEN modules.builtin > LD .tmp_vmlinux.btf > ld: warning: .tmp_vmlinux.btf has a LOAD segment with RWX permissions > BTF .btf.vmlinux.bin.o > LD .tmp_vmlinux.kallsyms1 > ld: warning: .tmp_vmlinux.kallsyms1 has a LOAD segment with RWX permissions > KSYMS .tmp_vmlinux.kallsyms1.S > AS .tmp_vmlinux.kallsyms1.S > LD .tmp_vmlinux.kallsyms2 > ld: warning: .tmp_vmlinux.kallsyms2 has a LOAD segment with RWX permissions > KSYMS .tmp_vmlinux.kallsyms2.S > AS .tmp_vmlinux.kallsyms2.S > LD vmlinux > ld: warning: vmlinux has a LOAD segment with RWX permissions > BTFIDS vmlinux > FAILED: load BTF from vmlinux: Invalid argument > make: *** [Makefile:1155: vmlinux] Error 255 > make: *** Deleting file 'vmlinux' > ``` > > It may be related to the kernel configuration, but then I don't know what to > do. > Try disabling DEBUG_INFO_BTF. Anyway, after bisecting, can you also check latest mainline (currently 6.4-rc2)?
(In reply to Bagas Sanjaya from comment #4) > Anyway, after bisecting, can you also check latest mainline (currently > 6.4-rc2)? You might want to do it the other way around, as then you avoid a lot of work if the issue was already fixed. You also might want to check -next and look at various threads that look related, some of them with fixes: https://lore.kernel.org/all/?q=__filemap_get_folio
Hello. Bagas Sanjaya: > ... can you also check latest mainline (currently 6.4-rc2)? Compiled, installed (the Archlinux package) and booted: ``` $ uname -a Linux smoon7.bkoty.ru 6.4.0-rc2-1-mainline-git #1 SMP PREEMPT_DYNAMIC Tue, 16 May 2023 10:09:04 +0000 x86_64 GNU/Linux $ pacman -Qs linux-mainline local/linux-mainline-git v6.4.rc2.r0.f1fcbaa18b28-1 The Linus Torvalds' Mainline Linux kernel and modules $ uptime -s 2023-05-16 19:58:37 ``` (uptime reports local time, it is UTC+08). > Try disabling DEBUG_INFO_BTF. Thank you, I'll try that next time. The Linux kernel's regression tracker (Thorsten Leemhuis): > You also might want to check -next and look at various threads that look > related, some of them with fixes: > https://lore.kernel.org/all/?q=__filemap_get_folio I saw a recent thread about __filemap_get_folio, so if mainline shows the same problem, I'll try -next. P.S. I'll email next time for answers.
Created attachment 304355 [details] signature.asc Hello. ** bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> [2023-05-16 14:28:59 +0000]: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > --- Comment #5 from The Linux kernel's regression tracker (Thorsten Leemhuis) > (regressions@leemhuis.info) --- > (In reply to Bagas Sanjaya from comment #4) > >> Anyway, after bisecting, can you also check latest mainline (currently >> 6.4-rc2)? > You might want to do it the other way around, as then you avoid a lot of work > if the issue was already fixed. You also might want to check -next and look > at > various threads that look related, some of them with fixes: > https://lore.kernel.org/all/?q=__filemap_get_folio Some results with rc and -next. Host B (smoon7): - linux-mainline-6.4.0-rc2: ``` May 17 19:28:50 smoon7.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 May 17 19:28:50 smoon7.bkoty.ru kernel: Linux version 6.4.0-rc2-2-mainline-git (linux-mainline-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Wed, 17 May 2023 09:37:33 +0000 May 17 19:28:50 smoon7.bkoty.ru kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-mainline-git.img root=PARTUUID=b0dc9b9f-ce16-724d-97be-2140eed159f6 rw systemd.unified_cgroup_hierarchy=1 scsi_mod.use_blk_mq=1 systemd.hostname=smoon7.bkoty.ru mitigations =off ``` has the problem. Booted: May 17, "crashed": May 18; from kernel log (journalctl -k): ``` May 18 15:29:19 smoon7.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 0000000000000036 May 18 15:29:19 smoon7.bkoty.ru kernel: #PF: supervisor read access in kernel mode May 18 15:29:19 smoon7.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page May 18 15:29:19 smoon7.bkoty.ru kernel: PGD 0 P4D 0 May 18 15:29:19 smoon7.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI May 18 15:29:19 smoon7.bkoty.ru kernel: CPU: 7 PID: 338864 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc2-2-mainline-git #1 679366a558ac82b2b49e79e65b050d993161fbef May 18 15:29:19 smoon7.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F4b 06/22/2020 May 18 15:29:19 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 18 15:29:19 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 d2 5a a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 18 15:29:19 smoon7.bkoty.ru kernel: RSP: 0000:ffffbd4e4aa23c70 EFLAGS: 00010246 May 18 15:29:19 smoon7.bkoty.ru kernel: RAX: 0000000000000002 RBX: 0000000000000002 RCX: 0000000000000002 May 18 15:29:19 smoon7.bkoty.ru kernel: RDX: 0000000000000010 RSI: ffff950fdbb02da0 RDI: ffffbd4e4aa23c70 May 18 15:29:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: R10: ffff95094adcaa50 R11: ffff95094adcaa0c R12: ffff95096d905eb0 May 18 15:29:19 smoon7.bkoty.ru kernel: R13: 0000000000003614 R14: 0000000000000000 R15: ffff95091152b8d0 May 18 15:29:19 smoon7.bkoty.ru kernel: FS: 00007f7a2bfff6c0(0000) GS:ffff95109e3c0000(0000) knlGS:0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 18 15:29:19 smoon7.bkoty.ru kernel: CR2: 0000000000000036 CR3: 000000010484c005 CR4: 00000000003706e0 May 18 15:29:19 smoon7.bkoty.ru kernel: Call Trace: May 18 15:29:19 smoon7.bkoty.ru kernel: <TASK> May 18 15:29:19 smoon7.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 May 18 15:29:19 smoon7.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 May 18 15:29:19 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x162/0x640 May 18 15:29:19 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 May 18 15:29:19 smoon7.bkoty.ru kernel: do_fault+0x1e6/0x420 May 18 15:29:19 smoon7.bkoty.ru kernel: __handle_mm_fault+0x68b/0xc80 May 18 15:29:19 smoon7.bkoty.ru kernel: handle_mm_fault+0x155/0x350 May 18 15:29:19 smoon7.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 May 18 15:29:19 smoon7.bkoty.ru kernel: exc_page_fault+0x7f/0x180 May 18 15:29:19 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 May 18 15:29:19 smoon7.bkoty.ru kernel: RIP: 0033:0x7f7a4b96cb0d May 18 15:29:19 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe May 18 15:29:19 smoon7.bkoty.ru kernel: RSP: 002b:00007f7a2bffd2c8 EFLAGS: 00010202 May 18 15:29:19 smoon7.bkoty.ru kernel: RAX: 00007f1d1c0063e0 RBX: 00007f7a2bffe308 RCX: 00007f7a2bffd500 May 18 15:29:19 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f46884141ca RDI: 00007f1d1c0063e0 May 18 15:29:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: R13: 00007f1d1c00af50 R14: 0000000000000003 R15: 000055603f04d010 May 18 15:29:19 smoon7.bkoty.ru kernel: </TASK> May 18 15:29:19 smoon7.bkoty.ru kernel: Modules linked in: tls veth tun sit tunnel4 ip_tunnel overlay snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm wireguard ib_core curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat pppoe nf_conntrack pppox nf_defrag_ipv6 nf_defrag_ipv4 nf_tables ppp_generic slhc nfnetlink mousedev joydev nls_iso8859_1 vfat fat snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils soundwire_bus intel_rapl_msr snd_soc_skl snd_hda_codec_hdmi intel_rapl_common snd_soc_hdac_hda snd_hda_ext_core intel_tcc_cooling snd_soc_sst_ipc x86_pkg_temp_thermal snd_soc_sst_dsp intel_powerclamp snd_soc_acpi_intel_match coretemp snd_soc_acpi May 18 15:29:19 smoon7.bkoty.ru kernel: snd_hda_codec_realtek uas kvm_intel snd_soc_core snd_hda_codec_generic usb_storage usbhid ledtrig_audio snd_compress kvm ppdev irqbypass crct10dif_pclmul crc32_pclmul ac97_bus snd_pcm_dmaengine polyval_clmulni polyval_generic snd_hda_intel gf128mul ghash_clmulni_intel snd_intel_dspcfg sha512_ssse3 snd_intel_sdw_acpi aesni_intel iTCO_wdt snd_hda_codec mei_hdcp mei_pxp crypto_simd intel_pmc_bxt ee1004 iTCO_vendor_support snd_hda_core cryptd rapl snd_hwdep snd_pcm intel_cstate snd_timer ixgbe intel_uncore spi_nor snd i2c_i801 gigabyte_wmi mdio mei_me wmi_bmof e1000e mxm_wmi intel_wmi_thunderbolt mtd soundcore mei intel_pch_thermal dca i2c_smbus parport_pc parport serial_multi_instantiate acpi_pad acpi_tad mac_hid nfsd r8169 realtek mdio_devres auth_rpcgss libphy nfs_acl lockd grace fuse loop dm_mod sunrpc ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci spi_intel_pci crc32c_intel spi_intel nvme_common xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi drm_display_helper cec ttm May 18 15:29:19 smoon7.bkoty.ru kernel: intel_agp intel_gtt May 18 15:29:19 smoon7.bkoty.ru kernel: CR2: 0000000000000036 May 18 15:29:19 smoon7.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- May 18 15:29:19 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 18 15:29:19 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 d2 5a a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 18 15:29:19 smoon7.bkoty.ru kernel: RSP: 0000:ffffbd4e4aa23c70 EFLAGS: 00010246 May 18 15:29:19 smoon7.bkoty.ru kernel: RAX: 0000000000000002 RBX: 0000000000000002 RCX: 0000000000000002 May 18 15:29:19 smoon7.bkoty.ru kernel: RDX: 0000000000000010 RSI: ffff950fdbb02da0 RDI: ffffbd4e4aa23c70 May 18 15:29:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: R10: ffff95094adcaa50 R11: ffff95094adcaa0c R12: ffff95096d905eb0 May 18 15:29:19 smoon7.bkoty.ru kernel: R13: 0000000000003614 R14: 0000000000000000 R15: ffff95091152b8d0 May 18 15:29:19 smoon7.bkoty.ru kernel: FS: 00007f7a2bfff6c0(0000) GS:ffff95109e3c0000(0000) knlGS:0000000000000000 May 18 15:29:19 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 18 15:29:19 smoon7.bkoty.ru kernel: CR2: 0000000000000036 CR3: 000000010484c005 CR4: 00000000003706e0 May 18 15:29:19 smoon7.bkoty.ru kernel: note: qbittorrent-nox[338864] exited with irqs disabled ``` - linux-next: ``` May 22 15:55:56 smoon7.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 May 22 15:55:56 smoon7.bkoty.ru kernel: Linux version 6.4.0-rc2-next-20230519-1-next-git-03527-gdbd91ef4e91c (linux-next-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Sat, 20 May 2023 01:35:07 +0000 May 22 15:55:56 smoon7.bkoty.ru kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-next-git.img root=PARTUUID=b0dc9b9f-ce16-724d-97be-2140eed159f6 rw systemd.unified_cgroup_hierarchy=1 scsi_mod.use_blk_mq=1 systemd.hostname=smoon7.bkoty.ru mitigations=off ``` has the problem. Booted: May 22, "crashed": May 26; kernel log: ``` May 26 18:05:03 smoon7.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 00000000000000d6 May 26 18:05:03 smoon7.bkoty.ru kernel: #PF: supervisor read access in kernel mode May 26 18:05:03 smoon7.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page May 26 18:05:03 smoon7.bkoty.ru kernel: PGD 0 P4D 0 May 26 18:05:03 smoon7.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI May 26 18:05:03 smoon7.bkoty.ru kernel: CPU: 1 PID: 621184 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc2-next-20230519-1-next-git-03527-gdbd91ef4e91c #1 75ae43a4bc6c46979c203ee52ed6368c0bb3658c May 26 18:05:03 smoon7.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F4b 06/22/2020 May 26 18:05:03 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 26 18:05:03 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 e2 ba a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 26 18:05:03 smoon7.bkoty.ru kernel: RSP: 0000:ffffbb46600bfc70 EFLAGS: 00010246 May 26 18:05:03 smoon7.bkoty.ru kernel: RAX: 00000000000000a2 RBX: 00000000000000a2 RCX: 0000000000000002 May 26 18:05:03 smoon7.bkoty.ru kernel: RDX: 000000000000002c RSI: ffff91670ffb96d0 RDI: ffffbb46600bfc70 May 26 18:05:03 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: R10: ffff9166dd214450 R11: ffff9166dd21440c R12: ffff9166f9e9e2b0 May 26 18:05:03 smoon7.bkoty.ru kernel: R13: 00000000000008ef R14: 0000000000000000 R15: ffff9166e4b02070 May 26 18:05:03 smoon7.bkoty.ru kernel: FS: 00007f21cbfff6c0(0000) GS:ffff916e5e240000(0000) knlGS:0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 26 18:05:03 smoon7.bkoty.ru kernel: CR2: 00000000000000d6 CR3: 000000010e252006 CR4: 00000000003706e0 May 26 18:05:03 smoon7.bkoty.ru kernel: Call Trace: May 26 18:05:03 smoon7.bkoty.ru kernel: <TASK> May 26 18:05:03 smoon7.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 May 26 18:05:03 smoon7.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 May 26 18:05:03 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x162/0x640 May 26 18:05:03 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 May 26 18:05:03 smoon7.bkoty.ru kernel: do_fault+0x1e6/0x420 May 26 18:05:03 smoon7.bkoty.ru kernel: __handle_mm_fault+0x68b/0xc80 May 26 18:05:03 smoon7.bkoty.ru kernel: handle_mm_fault+0x155/0x350 May 26 18:05:03 smoon7.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 May 26 18:05:03 smoon7.bkoty.ru kernel: exc_page_fault+0x7f/0x180 May 26 18:05:03 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 May 26 18:05:03 smoon7.bkoty.ru kernel: RIP: 0033:0x7f21ecf6cb0d May 26 18:05:03 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe May 26 18:05:03 smoon7.bkoty.ru kernel: RSP: 002b:00007f21cbffd2c8 EFLAGS: 00010202 May 26 18:05:03 smoon7.bkoty.ru kernel: RAX: 00007f21b00550e0 RBX: 00007f21cbffe308 RCX: 00007f21cbffd500 May 26 18:05:03 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f03852efffb RDI: 00007f21b00550e0 May 26 18:05:03 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 000000000000000b R09: 0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: R13: 00007f21b006da00 R14: 000000000000000b R15: 000055a00161b270 May 26 18:05:03 smoon7.bkoty.ru kernel: </TASK> May 26 18:05:03 smoon7.bkoty.ru kernel: Modules linked in: tls sctp veth tun sit tunnel4 ip_tunnel overlay snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm ib_core wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat pppoe nf_conntrack pppox nf_defrag_ipv6 nf_defrag_ipv4 nf_tables ppp_generic slhc nfnetlink mousedev joydev nls_iso8859_1 vfat fat uas snd_sof_pci_intel_cnl usbhid usb_storage snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils intel_rapl_msr soundwire_bus intel_rapl_common intel_tcc_cooling x86_pkg_temp_thermal snd_soc_skl intel_powerclamp snd_soc_hdac_hda snd_hda_codec_hdmi coretemp snd_hda_ext_core snd_soc_sst_ipc snd_soc_sst_dsp kvm_intel May 26 18:05:03 smoon7.bkoty.ru kernel: snd_soc_acpi_intel_match snd_soc_acpi snd_hda_codec_realtek kvm snd_soc_core snd_hda_codec_generic ledtrig_audio irqbypass snd_compress crct10dif_pclmul ac97_bus crc32_pclmul polyval_clmulni snd_pcm_dmaengine polyval_generic gf128mul snd_hda_intel ppdev ghash_clmulni_intel snd_intel_dspcfg sha512_ssse3 snd_intel_sdw_acpi snd_hda_codec aesni_intel crypto_simd iTCO_wdt cryptd snd_hda_core intel_pmc_bxt snd_hwdep iTCO_vendor_support rapl ee1004 snd_pcm mei_pxp intel_cstate snd_timer mei_hdcp i2c_i801 spi_nor snd intel_uncore e1000e intel_wmi_thunderbolt mtd gigabyte_wmi i2c_smbus mxm_wmi soundcore wmi_bmof ixgbe mei_me parport_pc mdio mei intel_pch_thermal dca parport acpi_tad serial_multi_instantiate acpi_pad mac_hid nfsd auth_rpcgss nfs_acl r8169 lockd realtek grace mdio_devres libphy loop fuse dm_mod sunrpc ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core spi_intel_pci xhci_pci crc32c_intel nvme_common spi_intel xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi drm_display_helper cec May 26 18:05:03 smoon7.bkoty.ru kernel: ttm intel_agp intel_gtt May 26 18:05:03 smoon7.bkoty.ru kernel: CR2: 00000000000000d6 May 26 18:05:03 smoon7.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- May 26 18:05:03 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 26 18:05:03 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 e2 ba a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 26 18:05:03 smoon7.bkoty.ru kernel: RSP: 0000:ffffbb46600bfc70 EFLAGS: 00010246 May 26 18:05:03 smoon7.bkoty.ru kernel: RAX: 00000000000000a2 RBX: 00000000000000a2 RCX: 0000000000000002 May 26 18:05:03 smoon7.bkoty.ru kernel: RDX: 000000000000002c RSI: ffff91670ffb96d0 RDI: ffffbb46600bfc70 May 26 18:05:03 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: R10: ffff9166dd214450 R11: ffff9166dd21440c R12: ffff9166f9e9e2b0 May 26 18:05:03 smoon7.bkoty.ru kernel: R13: 00000000000008ef R14: 0000000000000000 R15: ffff9166e4b02070 May 26 18:05:03 smoon7.bkoty.ru kernel: FS: 00007f21cbfff6c0(0000) GS:ffff916e5e240000(0000) knlGS:0000000000000000 May 26 18:05:03 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 26 18:05:03 smoon7.bkoty.ru kernel: CR2: 00000000000000d6 CR3: 000000010e252006 CR4: 00000000003706e0 May 26 18:05:03 smoon7.bkoty.ru kernel: note: qbittorrent-nox[621184] exited with irqs disabled ``` Host A (smoon): - linux-mainline-6.4.0-rc3: ``` May 22 16:28:33 smoon.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 May 22 16:28:33 smoon.bkoty.ru kernel: Linux version 6.4.0-rc3-1-mainline-git (linux-mainline-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Sun, 21 May 2023 23:48:36 +0000 May 22 16:28:33 smoon.bkoty.ru kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-mainline-git.img root=PARTUUID=54f7c057-d29f-9b4c-aef1-01a7cfc6003c rw systemd.unified_cgroup_hierarchy=1 scsi_mod.use_blk_mq=1 systemd.hostname=smoon.bkoty.ru mitigations=off ``` has the problem. Booted: May 22, "crashed": May 24; kernel log: ``` May 24 17:37:57 smoon.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 0000000000000076 May 24 17:37:57 smoon.bkoty.ru kernel: #PF: supervisor read access in kernel mode May 24 17:37:57 smoon.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page May 24 17:37:57 smoon.bkoty.ru kernel: PGD 0 P4D 0 May 24 17:37:57 smoon.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI May 24 17:37:57 smoon.bkoty.ru kernel: CPU: 11 PID: 24962 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc3-1-mainline-git #1 20eee7089988637ffc19c8dd5e18ef2d2afcec08 May 24 17:37:57 smoon.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. B460HD3/B460 HD3, BIOS F1 04/15/2020 May 24 17:37:57 smoon.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 24 17:37:57 smoon.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 42 57 a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 24 17:37:57 smoon.bkoty.ru kernel: RSP: 0000:ffff9939cca8fc70 EFLAGS: 00010246 May 24 17:37:57 smoon.bkoty.ru kernel: RAX: 0000000000000042 RBX: 0000000000000042 RCX: 0000000000000002 May 24 17:37:57 smoon.bkoty.ru kernel: RDX: 0000000000000014 RSI: ffff8b68de3e8490 RDI: ffff9939cca8fc70 May 24 17:37:57 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: R10: ffff8b62ee1b0450 R11: ffff8b62ee1b040c R12: ffff8b6317b9e6b0 May 24 17:37:57 smoon.bkoty.ru kernel: R13: 00000000000095d5 R14: 0000000000000000 R15: ffff8b62c96523d0 May 24 17:37:57 smoon.bkoty.ru kernel: FS: 00007f33bf1ff6c0(0000) GS:ffff8b69ff4c0000(0000) knlGS:0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 24 17:37:57 smoon.bkoty.ru kernel: CR2: 0000000000000076 CR3: 000000010aee0003 CR4: 00000000003706e0 May 24 17:37:57 smoon.bkoty.ru kernel: Call Trace: May 24 17:37:57 smoon.bkoty.ru kernel: <TASK> May 24 17:37:57 smoon.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 May 24 17:37:57 smoon.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 May 24 17:37:57 smoon.bkoty.ru kernel: ? filemap_map_pages+0x162/0x640 May 24 17:37:57 smoon.bkoty.ru kernel: __do_fault+0x30/0x130 May 24 17:37:57 smoon.bkoty.ru kernel: do_fault+0x1e6/0x420 May 24 17:37:57 smoon.bkoty.ru kernel: __handle_mm_fault+0x68b/0xc80 May 24 17:37:57 smoon.bkoty.ru kernel: handle_mm_fault+0x155/0x350 May 24 17:37:57 smoon.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 May 24 17:37:57 smoon.bkoty.ru kernel: exc_page_fault+0x7f/0x180 May 24 17:37:57 smoon.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 May 24 17:37:57 smoon.bkoty.ru kernel: RIP: 0033:0x7f356b16cb0d May 24 17:37:57 smoon.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe May 24 17:37:57 smoon.bkoty.ru kernel: RSP: 002b:00007f33bf1fd2c8 EFLAGS: 00010202 May 24 17:37:57 smoon.bkoty.ru kernel: RAX: 00007f1adc018ba0 RBX: 00007f33bf1fe308 RCX: 00007f33bf1fd500 May 24 17:37:57 smoon.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f1eb51d5a3f RDI: 00007f1adc018ba0 May 24 17:37:57 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000004 R09: 0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: R13: 00007f1adc0011f0 R14: 0000000000000004 R15: 00005562fcfc0190 May 24 17:37:57 smoon.bkoty.ru kernel: </TASK> May 24 17:37:57 smoon.bkoty.ru kernel: Modules linked in: tls nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs nft_fib_ipv6 overlay nft_nat rpcrdma nft_fib_ipv4 rdma_cm nft_fib iw_cm wireguard curve25519_x86_64 libchacha20poly1305 ib_cm chacha_x86_64 poly1305_x86_64 ib_core libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel snd_seq_dummy snd_hrtimer snd_seq snd_seq_device sit tunnel4 ip_tunnel bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink nls_iso8859_1 vfat fat snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus intel_rapl_msr snd_soc_avs intel_rapl_common intel_tcc_cooling snd_hda_codec_hdmi snd_soc_hda_codec snd_hda_ext_core x86_pkg_temp_thermal intel_powerclamp snd_soc_core coretemp snd_hda_codec_realtek snd_compress kvm_intel May 24 17:37:57 smoon.bkoty.ru kernel: snd_hda_codec_generic ac97_bus ledtrig_audio snd_pcm_dmaengine kvm snd_hda_intel snd_intel_dspcfg irqbypass snd_intel_sdw_acpi crct10dif_pclmul crc32_pclmul ppdev snd_hda_codec polyval_clmulni snd_hda_core polyval_generic gf128mul snd_hwdep ghash_clmulni_intel ee1004 sha512_ssse3 iTCO_wdt aesni_intel snd_pcm intel_pmc_bxt crypto_simd iTCO_vendor_support uas r8169 cryptd mei_hdcp mei_pxp ixgbe rapl snd_timer realtek intel_cstate usb_storage usbhid snd i2c_i801 mdio_devres gigabyte_wmi soundcore mei_me i2c_smbus intel_uncore libphy mdio wmi_bmof dca mei serial_multi_instantiate parport_pc parport mac_hid acpi_pad acpi_tad nfsd e1000e auth_rpcgss 8139too nfs_acl mii lockd grace dm_multipath crypto_user sunrpc fuse dm_mod loop ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci crc32c_intel nvme_common xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi drm_display_helper cec ttm intel_agp intel_gtt May 24 17:37:57 smoon.bkoty.ru kernel: CR2: 0000000000000076 May 24 17:37:57 smoon.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- May 24 17:37:57 smoon.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 24 17:37:57 smoon.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 42 57 a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 24 17:37:57 smoon.bkoty.ru kernel: RSP: 0000:ffff9939cca8fc70 EFLAGS: 00010246 May 24 17:37:57 smoon.bkoty.ru kernel: RAX: 0000000000000042 RBX: 0000000000000042 RCX: 0000000000000002 May 24 17:37:57 smoon.bkoty.ru kernel: RDX: 0000000000000014 RSI: ffff8b68de3e8490 RDI: ffff9939cca8fc70 May 24 17:37:57 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: R10: ffff8b62ee1b0450 R11: ffff8b62ee1b040c R12: ffff8b6317b9e6b0 May 24 17:37:57 smoon.bkoty.ru kernel: R13: 00000000000095d5 R14: 0000000000000000 R15: ffff8b62c96523d0 May 24 17:37:57 smoon.bkoty.ru kernel: FS: 00007f33bf1ff6c0(0000) GS:ffff8b69ff4c0000(0000) knlGS:0000000000000000 May 24 17:37:57 smoon.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 24 17:37:57 smoon.bkoty.ru kernel: CR2: 0000000000000076 CR3: 000000010aee0003 CR4: 00000000003706e0 May 24 17:37:57 smoon.bkoty.ru kernel: note: qbittorrent-nox[24962] exited with irqs disabled ``` - linux-next: ``` May 24 20:44:46 smoon.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 May 24 20:44:46 smoon.bkoty.ru kernel: Linux version 6.4.0-rc2-next-20230519-1-next-git-03527-gdbd91ef4e91c (linux-next-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Sat, 20 May 2023 01:35:07 +0000 May 24 20:44:46 smoon.bkoty.ru kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-next-git.img root=PARTUUID=7c9a5053-216d-2b4e-8c73-22d16a87ae6b rw systemd.unified_cgroup_hierarchy=1 scsi_mod.use_blk_mq=1 systemd.hostname=smoon.bkoty.ru mitigations=off ``` has the problem. Booted: May 24, "crashed": May 26; kernel log: ``` May 26 23:58:23 smoon.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 0000000000000076 May 26 23:58:23 smoon.bkoty.ru kernel: #PF: supervisor read access in kernel mode May 26 23:58:23 smoon.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page May 26 23:58:23 smoon.bkoty.ru kernel: PGD 0 P4D 0 May 26 23:58:23 smoon.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI May 26 23:58:23 smoon.bkoty.ru kernel: CPU: 10 PID: 19041 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc2-next-20230519-1-next-git-03527-gdbd91ef4e91c #1 75ae43a4bc6c46979c203ee52ed6368c0bb3658c May 26 23:58:23 smoon.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. B460HD3/B460 HD3, BIOS F1 04/15/2020 May 26 23:58:23 smoon.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 26 23:58:23 smoon.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 e2 ba a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 26 23:58:23 smoon.bkoty.ru kernel: RSP: 0000:ffffad18c829fc70 EFLAGS: 00010246 May 26 23:58:23 smoon.bkoty.ru kernel: RAX: 0000000000000042 RBX: 0000000000000042 RCX: 0000000000000002 May 26 23:58:23 smoon.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff947508096fe8 RDI: ffffad18c829fc70 May 26 23:58:23 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: R10: ffff946ed0146150 R11: ffff946ed014610c R12: ffff946ecbc762b0 May 26 23:58:23 smoon.bkoty.ru kernel: R13: 00000000000b8d5e R14: 0000000000000000 R15: ffff946eb1ac16d8 May 26 23:58:23 smoon.bkoty.ru kernel: FS: 00007f6edb5ff6c0(0000) GS:ffff9475bf480000(0000) knlGS:0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 26 23:58:23 smoon.bkoty.ru kernel: CR2: 0000000000000076 CR3: 0000000104a4e002 CR4: 00000000003706e0 May 26 23:58:23 smoon.bkoty.ru kernel: Call Trace: May 26 23:58:23 smoon.bkoty.ru kernel: <TASK> May 26 23:58:23 smoon.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 May 26 23:58:23 smoon.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 May 26 23:58:23 smoon.bkoty.ru kernel: ? filemap_map_pages+0x162/0x640 May 26 23:58:23 smoon.bkoty.ru kernel: __do_fault+0x30/0x130 May 26 23:58:23 smoon.bkoty.ru kernel: do_fault+0x1e6/0x420 May 26 23:58:23 smoon.bkoty.ru kernel: __handle_mm_fault+0x68b/0xc80 May 26 23:58:23 smoon.bkoty.ru kernel: handle_mm_fault+0x155/0x350 May 26 23:58:23 smoon.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 May 26 23:58:23 smoon.bkoty.ru kernel: exc_page_fault+0x7f/0x180 May 26 23:58:23 smoon.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 May 26 23:58:23 smoon.bkoty.ru kernel: RIP: 0033:0x7fbcc556cb0d May 26 23:58:23 smoon.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe May 26 23:58:23 smoon.bkoty.ru kernel: RSP: 002b:00007f6edb5fd2c8 EFLAGS: 00010202 May 26 23:58:23 smoon.bkoty.ru kernel: RAX: 00007fbc8c04c250 RBX: 00007f6edb5fe308 RCX: 00007f6edb5fd500 May 26 23:58:23 smoon.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007fb6dfd5e558 RDI: 00007fbc8c04c250 May 26 23:58:23 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: 00000000000005f3 R09: 0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: R13: 00007fbc8c007940 R14: 00000000000005f3 R15: 000055e4ec77cc10 May 26 23:58:23 smoon.bkoty.ru kernel: </TASK> May 26 23:58:23 smoon.bkoty.ru kernel: Modules linked in: tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs nft_fib_ipv6 rpcrdma overlay rdma_cm iw_cm nft_nat ib_cm ib_core nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel snd_seq_dummy snd_hrtimer snd_seq snd_seq_device sit tunnel4 ip_tunnel bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence intel_rapl_msr snd_sof_intel_hda_mlink snd_sof_intel_hda intel_rapl_common intel_tcc_cooling snd_sof_pci snd_sof_xtensa_dsp x86_pkg_temp_thermal intel_powerclamp snd_sof coretemp snd_sof_utils snd_soc_hdac_hda kvm_intel snd_soc_acpi_intel_match snd_hda_codec_hdmi nls_iso8859_1 joydev snd_soc_acpi vfat soundwire_bus fat kvm snd_soc_avs snd_hda_codec_realtek snd_hda_codec_generic irqbypass ledtrig_audio May 26 23:58:23 smoon.bkoty.ru kernel: crct10dif_pclmul snd_soc_hda_codec crc32_pclmul snd_hda_ext_core polyval_clmulni ppdev polyval_generic snd_soc_core gf128mul ghash_clmulni_intel snd_compress sha512_ssse3 mei_hdcp ac97_bus iTCO_wdt aesni_intel mei_pxp intel_pmc_bxt snd_pcm_dmaengine crypto_simd ee1004 cryptd snd_hda_intel iTCO_vendor_support rapl snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec intel_cstate snd_hda_core gigabyte_wmi intel_uncore usbhid snd_hwdep wmi_bmof ixgbe snd_pcm parport_pc snd_timer r8169 mdio parport realtek mac_hid mei_me mdio_devres snd i2c_i801 dca serial_multi_instantiate acpi_pad mei soundcore acpi_tad libphy i2c_smbus nfsd e1000e auth_rpcgss nfs_acl 8139too lockd mii dm_multipath grace crypto_user loop fuse dm_mod sunrpc ip_tables x_tables uas usb_storage xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci crc32c_intel nvme_common xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi drm_display_helper cec ttm intel_agp intel_gtt May 26 23:58:23 smoon.bkoty.ru kernel: CR2: 0000000000000076 May 26 23:58:23 smoon.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- May 26 23:58:23 smoon.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 May 26 23:58:23 smoon.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 e2 ba a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 May 26 23:58:23 smoon.bkoty.ru kernel: RSP: 0000:ffffad18c829fc70 EFLAGS: 00010246 May 26 23:58:23 smoon.bkoty.ru kernel: RAX: 0000000000000042 RBX: 0000000000000042 RCX: 0000000000000002 May 26 23:58:23 smoon.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff947508096fe8 RDI: ffffad18c829fc70 May 26 23:58:23 smoon.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: R10: ffff946ed0146150 R11: ffff946ed014610c R12: ffff946ecbc762b0 May 26 23:58:23 smoon.bkoty.ru kernel: R13: 00000000000b8d5e R14: 0000000000000000 R15: ffff946eb1ac16d8 May 26 23:58:23 smoon.bkoty.ru kernel: FS: 00007f6edb5ff6c0(0000) GS:ffff9475bf480000(0000) knlGS:0000000000000000 May 26 23:58:23 smoon.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 26 23:58:23 smoon.bkoty.ru kernel: CR2: 0000000000000076 CR3: 0000000104a4e002 CR4: 00000000003706e0 May 26 23:58:23 smoon.bkoty.ru kernel: note: qbittorrent-nox[19041] exited with irqs disabled ``` P.S. I forgot to mention that both hosts use XFS. P.S.S. I wanted to try the next "linux-next", but compilation failed (same as https://lkml.org/lkml/2023/5/24/576).
Created attachment 304379 [details] signature.asc Hello. I tried the linux-next-20230602 kernel and the problem persists: on the second host: ``` Jun 02 17:55:05 smoon7.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 Jun 02 17:55:05 smoon7.bkoty.ru kernel: Linux version 6.4.0-rc4-next-20230602-1-next-git-06849-gbc708bbd8260 (linux-next-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Fri, 02 Jun 2023 06:54:02 +0000 ... Jun 05 22:27:19 smoon7.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 00000000000000b6 Jun 05 22:27:19 smoon7.bkoty.ru kernel: #PF: supervisor read access in kernel mode Jun 05 22:27:19 smoon7.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page Jun 05 22:27:19 smoon7.bkoty.ru kernel: PGD 0 P4D 0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Jun 05 22:27:19 smoon7.bkoty.ru kernel: CPU: 11 PID: 196246 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc4-next-20230602-1-next-git-06849-gbc708bbd8260 #1 e2bc2c7c17ec9449d00023ecb23f332188dc6bfc Jun 05 22:27:19 smoon7.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F4b 06/22/2020 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jun 05 22:27:19 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 a2 d7 a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RSP: 0000:ffffb02c8a12fc70 EFLAGS: 00010246 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RAX: 0000000000000082 RBX: 0000000000000082 RCX: 0000000000000002 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RDX: 0000000000000028 RSI: ffff918a40460920 RDI: ffffb02c8a12fc70 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R10: ffff9189a06c9c50 R11: ffff9189a06c9c0c R12: ffff9189a614c2b0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R13: 0000000000021629 R14: 0000000000000000 R15: ffff9189a0de1170 Jun 05 22:27:19 smoon7.bkoty.ru kernel: FS: 00007f21fffff6c0(0000) GS:ffff91911e4c0000(0000) knlGS:0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 05 22:27:19 smoon7.bkoty.ru kernel: CR2: 00000000000000b6 CR3: 00000001206c4006 CR4: 00000000003706e0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: Call Trace: Jun 05 22:27:19 smoon7.bkoty.ru kernel: <TASK> Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? __die+0x23/0x70 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? page_fault_oops+0x171/0x4e0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? psi_group_change+0x213/0x3c0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? exc_page_fault+0x7f/0x180 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? asm_exc_page_fault+0x26/0x30 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? filemap_get_entry+0x8a/0x130 Jun 05 22:27:19 smoon7.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 Jun 05 22:27:19 smoon7.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x162/0x640 Jun 05 22:27:19 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 Jun 05 22:27:19 smoon7.bkoty.ru kernel: do_fault+0x1e6/0x420 Jun 05 22:27:19 smoon7.bkoty.ru kernel: __handle_mm_fault+0x68b/0xc80 Jun 05 22:27:19 smoon7.bkoty.ru kernel: handle_mm_fault+0x155/0x350 Jun 05 22:27:19 smoon7.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 Jun 05 22:27:19 smoon7.bkoty.ru kernel: exc_page_fault+0x7f/0x180 Jun 05 22:27:19 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RIP: 0033:0x7f334996cb0d Jun 05 22:27:19 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe Jun 05 22:27:19 smoon7.bkoty.ru kernel: RSP: 002b:00007f21ffffd2d8 EFLAGS: 00010202 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RAX: 00007f330005e4f0 RBX: 00007f21ffffe300 RCX: 00007f21ffffd500 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f0b88429d84 RDI: 00007f330005e4f0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R13: 00007f330002cf80 R14: 0000000000000001 R15: 0000000021629d84 Jun 05 22:27:19 smoon7.bkoty.ru kernel: </TASK> Jun 05 22:27:19 smoon7.bkoty.ru kernel: Modules linked in: tls veth tun sit tunnel4 ip_tunnel overlay snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm ib_core wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat nf_conntrack pppoe nf_defrag_ipv6 pppox nf_defrag_ipv4 nf_tables ppp_generic slhc nfnetlink mousedev joydev nls_iso8859_1 vfat fat uas usbhid usb_storage snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils soundwire_bus snd_soc_skl snd_soc_hdac_hda intel_rapl_msr intel_rapl_common snd_hda_ext_core intel_tcc_cooling snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp snd_soc_sst_ipc snd_soc_sst_dsp coretemp Jun 05 22:27:19 smoon7.bkoty.ru kernel: snd_soc_acpi_intel_match kvm_intel snd_hda_codec_realtek snd_soc_acpi snd_soc_core snd_hda_codec_generic kvm ledtrig_audio snd_compress ac97_bus snd_pcm_dmaengine irqbypass crct10dif_pclmul crc32_pclmul snd_hda_intel polyval_clmulni polyval_generic snd_intel_dspcfg gf128mul ghash_clmulni_intel snd_intel_sdw_acpi sha512_ssse3 ppdev snd_hda_codec aesni_intel crypto_simd snd_hda_core mei_hdcp mei_pxp iTCO_wdt cryptd snd_hwdep intel_pmc_bxt ee1004 iTCO_vendor_support rapl snd_pcm intel_cstate snd_timer mei_me i2c_i801 ixgbe snd intel_uncore spi_nor e1000e gigabyte_wmi intel_wmi_thunderbolt mdio soundcore wmi_bmof mei i2c_smbus mtd mxm_wmi intel_pch_thermal dca serial_multi_instantiate parport_pc parport acpi_tad acpi_pad mac_hid nfsd auth_rpcgss nfs_acl r8169 lockd realtek grace mdio_devres libphy loop fuse dm_mod sunrpc ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci spi_intel_pci crc32c_intel nvme_common spi_intel xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi Jun 05 22:27:19 smoon7.bkoty.ru kernel: drm_display_helper cec ttm intel_agp intel_gtt Jun 05 22:27:19 smoon7.bkoty.ru kernel: CR2: 00000000000000b6 Jun 05 22:27:19 smoon7.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- Jun 05 22:27:19 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jun 05 22:27:19 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 a2 d7 a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RSP: 0000:ffffb02c8a12fc70 EFLAGS: 00010246 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RAX: 0000000000000082 RBX: 0000000000000082 RCX: 0000000000000002 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RDX: 0000000000000028 RSI: ffff918a40460920 RDI: ffffb02c8a12fc70 Jun 05 22:27:19 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R10: ffff9189a06c9c50 R11: ffff9189a06c9c0c R12: ffff9189a614c2b0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: R13: 0000000000021629 R14: 0000000000000000 R15: ffff9189a0de1170 Jun 05 22:27:19 smoon7.bkoty.ru kernel: FS: 00007f21fffff6c0(0000) GS:ffff91911e4c0000(0000) knlGS:0000000000000000 Jun 05 22:27:19 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 05 22:27:19 smoon7.bkoty.ru kernel: CR2: 00000000000000b6 CR3: 00000001206c4006 CR4: 00000000003706e0 Jun 05 22:27:19 smoon7.bkoty.ru kernel: note: qbittorrent-nox[196246] exited with irqs disabled ``` On the first host, the kernel is still running, but I expect it to "crash" in the next few days. Should I enable DEBUG options in the kernel configuration to get more details or will it be useless? P.S. I have compiled the linux-next-20230605 and will boot it later today. --- Vladimir Lomov
Hello. The linux-next 20230605 and linux-next 20230609 show the same behavior. The only difference I didn't catch in the previous report is that the kernel shows extra items on the stack. Both linux-next kernels have this (see attached files). I'm beginning to think that the problem might be NFS related. When the linux-next 20230605 "crashed" on the second system, it became almost "unresponsive" (huge delay when launching `sudo` or other programs reading from a disk). I see something similar on the host when it loses the network connection to the NFS server (e.g., when it takes a long time to reboot the NFS server). I also see a lot of `kernel: svc: svc_tcp_read_marker lockd RPC fragment too large:`. Also, it's almost impossible to reboot the system remotely, if the kernel "crashes", the reboot hangs, I have to forcibly reboot the system (by pressing the reset button). I could provide more information about my network configuration, if desired. I have attached the dmesg output for both hosts: for linux-next 20230605 and linux-next 20230609.
Created attachment 304416 [details] kernel log from host A
Created attachment 304417 [details] kernel log from host B, 20230605
Created attachment 304418 [details] kernel log from host B, 20230609
Not m area of expertise, but I wonder if this is a duplicate of https://bugzilla.kernel.org/show_bug.cgi?id=216646 ; there disabling THP (see https://bugzilla.kernel.org/show_bug.cgi?id=216646#c5 helped). Might want to try it?
Created attachment 304435 [details] signature.asc Hello. ** bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> [2023-06-14 08:19:49 +0000]: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > > --- Comment #13 from The Linux kernel's regression tracker (Thorsten > Leemhuis) (regressions@leemhuis.info) --- > Not m area of expertise, but I wonder if this is a duplicate of > https://bugzilla.kernel.org/show_bug.cgi?id=216646 ; there disabling THP (see > https://bugzilla.kernel.org/show_bug.cgi?id=216646#c5 helped). Might want to > try it? At first I thought it could be very similar to my case: something similar in the kernel stack, somewhat similar conditions causing the problem (the "heavy" disk load, even the use of qbittorrent and XFS). I compiled linux-next 20230614 and disabled transparent HP (as suggested on the Archlinux wiki page), but the kernel "crashed" again: host B: ``` Jun 14 19:29:05 smoon7.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 Jun 14 19:29:05 smoon7.bkoty.ru kernel: Linux version 6.4.0-rc6-next-20230614-1-next-git-09793-gb16049b21162 (linux-next-git@archlinux) (gcc (GCC) 13.1.1 20230429, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Wed, 14 Jun 2023 09:29:02 +0000 Jun 14 19:29:05 smoon7.bkoty.ru kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-next-git.img root=PARTUUID=b0dc9b9f-ce16-724d-97be-2140eed159f6 rw systemd.unified_cgroup_hierarchy=1 scsi_mod.use_blk_mq=1 systemd.hostname=smoon7.bkoty.ru mitigations=off ... Jun 16 01:49:09 smoon7.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 0000000000000096 Jun 16 01:49:09 smoon7.bkoty.ru kernel: #PF: supervisor read access in kernel mode Jun 16 01:49:09 smoon7.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page Jun 16 01:49:09 smoon7.bkoty.ru kernel: PGD 0 P4D 0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Jun 16 01:49:09 smoon7.bkoty.ru kernel: CPU: 6 PID: 338959 Comm: qbittorrent-nox Tainted: G U 6.4.0-rc6-next-20230614-1-next-git-09793-gb16049b21162 #1 d5bd20bc96dd08c4fc545a97f87341495fcd17a4 Jun 16 01:49:09 smoon7.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F4b 06/22/2020 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jun 16 01:49:09 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 c2 d9 a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RSP: 0000:ffffa6b7c7f4bc68 EFLAGS: 00010246 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RAX: 0000000000000062 RBX: 0000000000000062 RCX: 0000000000000002 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff991e021cd918 RDI: ffffa6b7c7f4bc68 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000012 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R10: ffff99178114db50 R11: ffff99178114db0c R12: ffff9917bb7adab0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R13: 000000000002bd1f R14: 0000000000000000 R15: ffff9917a2552648 Jun 16 01:49:09 smoon7.bkoty.ru kernel: FS: 00007fe5567fc6c0(0000) GS:ffff991f1e380000(0000) knlGS:0000000000000000 Jun 16 01:49:09 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 01:49:09 smoon7.bkoty.ru kernel: CR2: 0000000000000096 CR3: 000000012276e001 CR4: 00000000003706e0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: Call Trace: Jun 16 01:49:09 smoon7.bkoty.ru kernel: <TASK> Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? __die+0x23/0x70 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? page_fault_oops+0x171/0x4e0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? exc_page_fault+0x7f/0x180 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? asm_exc_page_fault+0x26/0x30 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? filemap_get_entry+0x8a/0x130 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? filemap_get_entry+0x6e/0x130 Jun 16 01:49:09 smoon7.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 Jun 16 01:49:09 smoon7.bkoty.ru kernel: filemap_fault+0x6b/0x9e0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x2d7/0x550 Jun 16 01:49:09 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 Jun 16 01:49:09 smoon7.bkoty.ru kernel: do_fault+0x248/0x3d0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: __handle_mm_fault+0x658/0xba0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: handle_mm_fault+0x155/0x350 Jun 16 01:49:09 smoon7.bkoty.ru kernel: do_user_addr_fault+0x236/0x740 Jun 16 01:49:09 smoon7.bkoty.ru kernel: exc_page_fault+0x7f/0x180 Jun 16 01:49:09 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RIP: 0033:0x7fe57276cb0d Jun 16 01:49:09 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe Jun 16 01:49:09 smoon7.bkoty.ru kernel: RSP: 002b:00007fe5567fa2d8 EFLAGS: 00010202 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RAX: 00007fe548015e70 RBX: 00007fe5567fb300 RCX: 00007fe5567fa500 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007fb26771f2d1 RDI: 00007fe548015e70 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000007 R09: 0000000000000000 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R13: 00007fe5480024d0 R14: 0000000000000007 R15: 000000002bd1f2d1 Jun 16 01:49:09 smoon7.bkoty.ru kernel: </TASK> Jun 16 01:49:09 smoon7.bkoty.ru kernel: Modules linked in: tls veth tun sit tunnel4 ip_tunnel overlay snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm ib_core wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat nf_nat nf_conntrack pppoe nf_defrag_ipv6 nf_defrag_ipv4 pppox nf_tables ppp_generic slhc nfnetlink joydev mousedev nls_iso8859_1 vfat fat usbhid uas usb_storage snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils soundwire_bus snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_sst_ipc snd_hda_codec_hdmi snd_soc_sst_dsp intel_rapl_msr snd_soc_acpi_intel_match intel_rapl_common snd_soc_acpi intel_tcc_cooling x86_pkg_temp_thermal Jun 16 01:49:09 smoon7.bkoty.ru kernel: snd_soc_core intel_powerclamp snd_hda_codec_realtek coretemp snd_compress kvm_intel snd_hda_codec_generic ac97_bus ledtrig_audio snd_pcm_dmaengine snd_hda_intel kvm irqbypass snd_intel_dspcfg crct10dif_pclmul crc32_pclmul ppdev snd_intel_sdw_acpi polyval_clmulni snd_hda_codec polyval_generic gf128mul snd_hda_core ghash_clmulni_intel snd_hwdep sha512_ssse3 iTCO_wdt aesni_intel intel_pmc_bxt crypto_simd iTCO_vendor_support ee1004 cryptd snd_pcm mei_pxp rapl mei_hdcp snd_timer intel_cstate ixgbe spi_nor snd mdio i2c_i801 intel_uncore serial_multi_instantiate soundcore dca mtd parport_pc mei_me i2c_smbus acpi_tad intel_pch_thermal mxm_wmi intel_wmi_thunderbolt e1000e mei acpi_pad parport gigabyte_wmi mac_hid wmi_bmof nfsd auth_rpcgss nfs_acl r8169 lockd realtek grace mdio_devres libphy loop fuse dm_mod sunrpc ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci spi_intel_pci crc32c_intel nvme_common spi_intel xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi drm_display_helper cec ttm Jun 16 01:49:09 smoon7.bkoty.ru kernel: intel_agp intel_gtt Jun 16 01:49:09 smoon7.bkoty.ru kernel: CR2: 0000000000000096 Jun 16 01:49:09 smoon7.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- Jun 16 01:49:09 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jun 16 01:49:09 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 c2 d9 a5 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RSP: 0000:ffffa6b7c7f4bc68 EFLAGS: 00010246 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RAX: 0000000000000062 RBX: 0000000000000062 RCX: 0000000000000002 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff991e021cd918 RDI: ffffa6b7c7f4bc68 Jun 16 01:49:09 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000012 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R10: ffff99178114db50 R11: ffff99178114db0c R12: ffff9917bb7adab0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: R13: 000000000002bd1f R14: 0000000000000000 R15: ffff9917a2552648 Jun 16 01:49:09 smoon7.bkoty.ru kernel: FS: 00007fe5567fc6c0(0000) GS:ffff991f1e380000(0000) knlGS:0000000000000000 Jun 16 01:49:09 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 16 01:49:09 smoon7.bkoty.ru kernel: CR2: 0000000000000096 CR3: 000000012276e001 CR4: 00000000003706e0 Jun 16 01:49:09 smoon7.bkoty.ru kernel: note: qbittorrent-nox[338959] exited with irqs disabled ``` I'll try linux-next again later, but for now I'll keep running 5.15 because it works. --- WBR, Vladimir Lomov
Seems this regression (and the one reported in Bug 216646, which look somewhat similar) is not really handled appropriately by the developers, hence I plan to slightly escalate this. But before I do this, could anyone still affected by this please check if 6.5-rc1 still shows the problem?
Created attachment 304605 [details] signature.asc Hello, ** bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> [2023-07-10 13:18:11 +0000]: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > --- Comment #15 from The Linux kernel's regression tracker (Thorsten > Leemhuis) (regressions@leemhuis.info) --- > Seems this regression (and the one reported in Bug 216646, which look > somewhat > similar) is not really handled appropriately by the developers, hence I plan > to > slightly escalate this. But before I do this, could anyone still affected by > this please check if 6.5-rc1 still shows the problem? I would be glad to test, but I'll be away for a week, so I'll test only after July 18. P.S. Is it crucial to test exactly 6.5-rc or cat I try linux-next? --- Vladimir Lomov
(In reply to Vladimir Lomov from comment #16) > I would be glad to test, but I'll be away for a week, so I'll test only after > July 18. No worries, enjoy your time away. > P.S. Is it crucial to test exactly 6.5-rc or cat I try linux-next? Hmmm. Not sure. In a ideal world testing both would be ideal, but that's obviously some work; but testing 6.5-rc is more important to know if the fix is in there (and not only in a patch that only will make it into 6.6 or later)
Prior to 6.5.0-rc1-1-git-00259-g831fe284d827 I was able to reproduce this bug by using the same app and getting (I believe) the same backtrace. After updating to 6.5.0-rc1 it has been stable for 24h while before it broke every 5/10 hours or so. This is the backtrace I used to get, pasting it just in case I was hitting a different bug and I am misinterpreting the bt. Jul 15 14:44:28 Hostname kernel: BUG: kernel NULL pointer dereference, address: 0000000000000056 Jul 15 14:44:28 Hostname kernel: #PF: supervisor read access in kernel mode Jul 15 14:44:28 Hostname kernel: #PF: error_code(0x0000) - not-present page Jul 15 14:44:28 Hostname kernel: PGD 0 P4D 0 Jul 15 14:44:28 Hostname kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Jul 15 14:44:28 Hostname kernel: CPU: 5 PID: 105158 Comm: qbittorrent Tainted: G U 6.4.3-arch1-1 #1 13c144d261447e0acbf2632534d4009bddc4c3ab Jul 15 14:44:28 Hostname kernel: Hardware name: SLIMBOOK EXECUTIVE-14/EXECUTIVE-14, BIOS N.1.09GRU00 01/28/2022 Jul 15 14:44:28 Hostname kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jul 15 14:44:28 Hostname kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 f2 62 a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jul 15 14:44:28 Hostname kernel: RSP: 0000:ffff9bcd62c3bc70 EFLAGS: 00010246 Jul 15 14:44:28 Hostname kernel: RAX: 0000000000000022 RBX: 0000000000000022 RCX: 0000000000000002 Jul 15 14:44:28 Hostname kernel: RDX: 000000000000000c RSI: ffff88bb9c923d98 RDI: ffff9bcd62c3bc70 Jul 15 14:44:28 Hostname kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jul 15 14:44:28 Hostname kernel: R10: 0000000055555554 R11: ffff88bd9259a30c R12: ffff88bd6fd952b0 Jul 15 14:44:28 Hostname kernel: R13: 000000000002828e R14: 0000000000000000 R15: ffff88bb0400d058 Jul 15 14:44:28 Hostname kernel: FS: 0000768d1b6b06c0(0000) GS:ffff88be9fb40000(0000) knlGS:0000000000000000 Jul 15 14:44:28 Hostname kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 15 14:44:28 Hostname kernel: CR2: 0000000000000056 CR3: 0000000288756006 CR4: 0000000000f70ee0 Jul 15 14:44:28 Hostname kernel: PKRU: 55555554 Jul 15 14:44:28 Hostname kernel: Call Trace: Jul 15 14:44:28 Hostname kernel: <TASK> Jul 15 14:44:28 Hostname kernel: ? __die+0x23/0x70 Jul 15 14:44:28 Hostname kernel: ? page_fault_oops+0x171/0x4e0 Jul 15 14:44:28 Hostname kernel: ? exc_page_fault+0x7f/0x180 Jul 15 14:44:28 Hostname kernel: ? asm_exc_page_fault+0x26/0x30 Jul 15 14:44:28 Hostname kernel: ? filemap_get_entry+0x8a/0x130 Jul 15 14:44:28 Hostname kernel: __filemap_get_folio+0x2b/0x230 Jul 15 14:44:28 Hostname kernel: filemap_fault+0x6b/0x9f0 Jul 15 14:44:28 Hostname kernel: ? filemap_map_pages+0x162/0x640 Jul 15 14:44:28 Hostname kernel: __do_fault+0x30/0x130 Jul 15 14:44:28 Hostname kernel: do_fault+0x1e6/0x420 Jul 15 14:44:28 Hostname kernel: __handle_mm_fault+0x68b/0xc80 Jul 15 14:44:28 Hostname kernel: handle_mm_fault+0x155/0x350 Jul 15 14:44:28 Hostname kernel: do_user_addr_fault+0x203/0x640 Jul 15 14:44:28 Hostname kernel: exc_page_fault+0x7f/0x180 Jul 15 14:44:28 Hostname kernel: asm_exc_page_fault+0x26/0x30 Jul 15 14:44:28 Hostname kernel: RIP: 0033:0x768d3f97c7cd Jul 15 14:44:28 Hostname kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 33 <62> e1 fe 28 6f 06 48 83 fa 40 0f 87 a3 00 00 00 62 e1 fe 28 6f 4c Jul 15 14:44:28 Hostname kernel: RSP: 002b:0000768d1b6ae2d8 EFLAGS: 00010202 Jul 15 14:44:28 Hostname kernel: RAX: 0000768c980131d0 RBX: 0000768d1b6af300 RCX: 0000768d1b6ae500 Jul 15 14:44:28 Hostname kernel: RDX: 0000000000004000 RSI: 00007682faa8eaa9 RDI: 0000768c980131d0 Jul 15 14:44:28 Hostname kernel: RBP: 0000000000000000 R08: 000000000000000a R09: 0000000000000000 Jul 15 14:44:28 Hostname kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 Jul 15 14:44:28 Hostname kernel: R13: 0000768c98002560 R14: 000000000000000a R15: 000000002828eaa9 Jul 15 14:44:28 Hostname kernel: </TASK> Jul 15 14:44:28 Hostname kernel: Modules linked in: ext4 mbcache jbd2 vxlan ip6_udp_tunnel udp_tunnel xt_policy iptable_mangle xt_mark xt_bpf xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter br_ne> Jul 15 14:44:28 Hostname kernel: iTCO_wdt snd_intel_sdw_acpi irqbypass hid_multitouch intel_pmc_bxt r8153_ecm mei_pxp btrtl uvcvideo mei_hdcp snd_hda_codec btbcm rapl mac80211 snd_usb_audio btintel cdc_ether btmtk snd_hda_core iTCO_vendor_support videobuf2_vmalloc usbnet intel_cstate asus_wmi uvc libarc4 videobuf2> Jul 15 14:44:28 Hostname kernel: libps2 i2c_algo_bit polyval_clmulni polyval_generic gf128mul vivaldi_fmap drm_buddy ghash_clmulni_intel nvme intel_gtt sha512_ssse3 aesni_intel crypto_simd drm_display_helper cryptd spi_intel_pci nvme_core cec spi_intel xhci_pci vmd xhci_pci_renesas ttm nvme_common i8042 video seri> Jul 15 14:44:28 Hostname kernel: CR2: 0000000000000056 Jul 15 14:44:28 Hostname kernel: ---[ end trace 0000000000000000 ]--- Jul 15 14:44:28 Hostname kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jul 15 14:44:28 Hostname kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 f2 62 a7 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jul 15 14:44:28 Hostname kernel: RSP: 0000:ffff9bcd62c3bc70 EFLAGS: 00010246 Jul 15 14:44:28 Hostname kernel: RAX: 0000000000000022 RBX: 0000000000000022 RCX: 0000000000000002 Jul 15 14:44:28 Hostname kernel: RDX: 000000000000000c RSI: ffff88bb9c923d98 RDI: ffff9bcd62c3bc70 Jul 15 14:44:28 Hostname kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jul 15 14:44:28 Hostname kernel: R10: 0000000055555554 R11: ffff88bd9259a30c R12: ffff88bd6fd952b0 Jul 15 14:44:28 Hostname kernel: R13: 000000000002828e R14: 0000000000000000 R15: ffff88bb0400d058 Jul 15 14:44:28 Hostname kernel: FS: 0000768d1b6b06c0(0000) GS:ffff88be9fb40000(0000) knlGS:0000000000000000 Jul 15 14:44:28 Hostname kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 15 14:44:28 Hostname kernel: CR2: 0000000000000056 CR3: 0000000288756006 CR4: 0000000000f70ee0 Jul 15 14:44:28 Hostname kernel: PKRU: 55555554
After little more than 48h running I could reproduce the bug again with 6.5.0-rc1-1-git-00259-g831fe284d827
Created attachment 304675 [details] signature.asc Hello. ** bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> [2023-07-10 13:18:11 +0000]: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > --- Comment #15 from The Linux kernel's regression tracker (Thorsten > Leemhuis) (regressions@leemhuis.info) --- > Seems this regression (and the one reported in Bug 216646, which look > somewhat similar) is not really handled appropriately by the developers, > hence I plan to slightly escalate this. But before I do this, could anyone > still affected by this please check if 6.5-rc1 still shows the problem? I tested linux-next-20230718 and on one host I see the same (?) behaviour: ``` Jul 19 07:58:24 smoon7.bkoty.ru kernel: microcode: updated early: 0xcc -> 0xf6, date = 2023-01-01 Jul 19 07:58:24 smoon7.bkoty.ru kernel: Linux version 6.5.0-rc2-next-20230718-1-next-git-03113-gaeba456828b4 (linux-next-git@archlinux) (gcc (GCC) 13.1.1 20230714, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Tue, 18 Jul 2023 22:16:16 +0000 ... Jul 20 19:03:24 smoon7.bkoty.ru kernel: BUG: kernel NULL pointer dereference, address: 0000000000000096 Jul 20 19:03:24 smoon7.bkoty.ru kernel: #PF: supervisor read access in kernel mode Jul 20 19:03:24 smoon7.bkoty.ru kernel: #PF: error_code(0x0000) - not-present page Jul 20 19:03:24 smoon7.bkoty.ru kernel: PGD 0 P4D 0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Jul 20 19:03:24 smoon7.bkoty.ru kernel: CPU: 4 PID: 305164 Comm: qbittorrent-nox Tainted: G U 6.5.0-rc2-next-20230718-1-next-git-03113-gaeba456828b4 #1 8d98cf92e1199e734fba1ef76a33030687665b92 Jul 20 19:03:24 smoon7.bkoty.ru kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F4b 06/22/2020 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jul 20 19:03:24 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 32 54 a6 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RSP: 0000:ffffb7d6c4e1bc70 EFLAGS: 00010246 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RAX: 0000000000000062 RBX: 0000000000000062 RCX: 0000000000000002 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff996fdd57c490 RDI: ffffb7d6c4e1bc70 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R10: ffff99695e549d50 R11: ffff99695e549d0c R12: ffff99699ffe2fc0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R13: 000000000001eb9d R14: 0000000000000000 R15: ffff9969a06cfef8 Jul 20 19:03:24 smoon7.bkoty.ru kernel: FS: 00007f9077fff6c0(0000) GS:ffff9970de300000(0000) knlGS:0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 20 19:03:24 smoon7.bkoty.ru kernel: CR2: 0000000000000096 CR3: 0000000344754004 CR4: 00000000003706e0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: Call Trace: Jul 20 19:03:24 smoon7.bkoty.ru kernel: <TASK> Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? __die+0x23/0x70 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? page_fault_oops+0x171/0x4e0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? psi_group_change+0x213/0x3c0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? exc_page_fault+0x7f/0x180 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? asm_exc_page_fault+0x26/0x30 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? filemap_get_entry+0x8a/0x130 Jul 20 19:03:24 smoon7.bkoty.ru kernel: __filemap_get_folio+0x2b/0x230 Jul 20 19:03:24 smoon7.bkoty.ru kernel: filemap_fault+0x6b/0x9f0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ? filemap_map_pages+0x2dc/0x560 Jul 20 19:03:24 smoon7.bkoty.ru kernel: __do_fault+0x30/0x130 Jul 20 19:03:24 smoon7.bkoty.ru kernel: do_fault+0x26c/0x430 Jul 20 19:03:24 smoon7.bkoty.ru kernel: __handle_mm_fault+0x73f/0xbb0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: handle_mm_fault+0x17f/0x360 Jul 20 19:03:24 smoon7.bkoty.ru kernel: do_user_addr_fault+0x1e6/0x640 Jul 20 19:03:24 smoon7.bkoty.ru kernel: exc_page_fault+0x7f/0x180 Jul 20 19:03:24 smoon7.bkoty.ru kernel: asm_exc_page_fault+0x26/0x30 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RIP: 0033:0x7f909336cb0d Jul 20 19:03:24 smoon7.bkoty.ru kernel: Code: 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 23 <c5> fe 6f 06 48 83 fa 40 0f 87 a5 00 00 00 c5 fe 6f 4c 16 e0 c5 fe Jul 20 19:03:24 smoon7.bkoty.ru kernel: RSP: 002b:00007f9077ffd298 EFLAGS: 00010202 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RAX: 00007f9050012770 RBX: 00007f9077ffe300 RCX: 00007f9077ffd4c0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RDX: 0000000000004000 RSI: 00007f77e759d10f RDI: 00007f9050012770 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: 0000000000000007 R09: 0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R13: 00007f9050000bf0 R14: 0000000000000007 R15: 000000001eb9d10f Jul 20 19:03:24 smoon7.bkoty.ru kernel: </TASK> Jul 20 19:03:24 smoon7.bkoty.ru kernel: Modules linked in: tls veth tun sit tunnel4 ip_tunnel overlay rpcsec_gss_krb5 snd_seq_dummy snd_hrtimer snd_seq snd_seq_device nfsv4 dns_resolver nfs fscache netfs nft_fib_ipv6 nft_nat rpcrdma rdma_cm iw_cm ib_cm ib_core wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_fib_ipv4 nft_fib bridge stp llc cfg80211 rfkill nft_masq nft_chain_nat pppoe nf_nat pppox nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 joydev mousedev ppp_generic slhc nf_tables nfnetlink snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation snd_sof_intel_hda_mlink uas soundwire_cadence usbhid usb_storage snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils soundwire_bus snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_sst_ipc snd_soc_sst_dsp intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi intel_tcc_cooling snd_soc_acpi_intel_match x86_pkg_temp_thermal snd_soc_acpi intel_powerclamp Jul 20 19:03:24 smoon7.bkoty.ru kernel: snd_soc_core coretemp snd_hda_codec_realtek snd_compress kvm_intel snd_hda_codec_generic ac97_bus ledtrig_audio snd_pcm_dmaengine nls_iso8859_1 snd_hda_intel kvm snd_intel_dspcfg irqbypass snd_intel_sdw_acpi crct10dif_pclmul snd_hda_codec ppdev crc32_pclmul polyval_clmulni polyval_generic gf128mul iTCO_wdt ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd snd_hda_core cryptd snd_hwdep mei_hdcp mei_pxp intel_pmc_bxt ee1004 snd_pcm iTCO_vendor_support rapl snd_timer intel_cstate intel_uncore wmi_bmof gigabyte_wmi snd ixgbe mei_me intel_wmi_thunderbolt parport_pc intel_pch_thermal soundcore e1000e spi_nor serial_multi_instantiate mxm_wmi mei parport i2c_i801 acpi_pad mdio mtd acpi_tad dca mac_hid i2c_smbus nfsd auth_rpcgss nfs_acl lockd r8169 realtek grace mdio_devres libphy fuse loop dm_mod sunrpc vfat fat ip_tables x_tables xfs libcrc32c crc32c_generic nvme nvme_core xhci_pci spi_intel_pci crc32c_intel spi_intel nvme_common xhci_pci_renesas i915 i2c_algo_bit drm_buddy video wmi ttm Jul 20 19:03:24 smoon7.bkoty.ru kernel: drm_display_helper cec intel_agp intel_gtt Jul 20 19:03:24 smoon7.bkoty.ru kernel: CR2: 0000000000000096 Jul 20 19:03:24 smoon7.bkoty.ru kernel: ---[ end trace 0000000000000000 ]--- Jul 20 19:03:24 smoon7.bkoty.ru kernel: RIP: 0010:filemap_get_entry+0x8a/0x130 Jul 20 19:03:24 smoon7.bkoty.ru kernel: Code: 24 18 03 00 00 00 48 89 e7 e8 32 54 a6 00 48 89 c3 48 3d 02 04 00 00 74 e4 48 3d 06 04 00 00 74 dc 48 85 c0 74 4f a8 01 75 4b <8b> 40 34 85 c0 74 cc 8d 50 01 f0 0f b1 53 34 75 f2 48 8b 54 24 18 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RSP: 0000:ffffb7d6c4e1bc70 EFLAGS: 00010246 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RAX: 0000000000000062 RBX: 0000000000000062 RCX: 0000000000000002 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RDX: 000000000000001c RSI: ffff996fdd57c490 RDI: ffffb7d6c4e1bc70 Jul 20 19:03:24 smoon7.bkoty.ru kernel: RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R10: ffff99695e549d50 R11: ffff99695e549d0c R12: ffff99699ffe2fc0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: R13: 000000000001eb9d R14: 0000000000000000 R15: ffff9969a06cfef8 Jul 20 19:03:24 smoon7.bkoty.ru kernel: FS: 00007f9077fff6c0(0000) GS:ffff9970de300000(0000) knlGS:0000000000000000 Jul 20 19:03:24 smoon7.bkoty.ru kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 20 19:03:24 smoon7.bkoty.ru kernel: CR2: 0000000000000096 CR3: 0000000344754004 CR4: 00000000003706e0 Jul 20 19:03:24 smoon7.bkoty.ru kernel: note: qbittorrent-nox[305164] exited with irqs disabled ``` The second host (smoon) still works.
Created attachment 304705 [details] proposed fix + testcase I have managed to create a test-case that reproduces the symptoms, and also a fix for that test-case. Maybe you're producing the symptoms in some excitingly different way, so this may not solve your problems.
(In reply to Matthew Wilcox from comment #21) > Created attachment 304705 [details] > proposed fix + testcase Did anybody try to check if this helps? That would be really really helpful.
Created attachment 304788 [details] signature.asc Hello, ** bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> [2023-08-07 08:08:25 +0000]: > https://bugzilla.kernel.org/show_bug.cgi?id=217441 > --- Comment #22 from The Linux kernel's regression tracker (Thorsten > Leemhuis) (regressions@leemhuis.info) --- > (In reply to Matthew Wilcox from comment #21) >> Created attachment 304705 [details] >> proposed fix + testcase > Did anybody try to check if this helps? That would be really really helpful. I'm currently running linux-next 20230802 (6.5.0-rc4-next-20230802-1-next-git-06443-g626c67169f99) with this patch (had to reboot a few times, not related to the issue) and so far all is good (uptime 4 days and 9 hours). --- WBR, Vladimir Lomov
(In reply to Matthew Wilcox from comment #21) > Created attachment 304705 [details] > proposed fix + testcase > Have not reproduced with this fix.
The patch for xarray.c has been added to the 6.5.5 kernel, and the hosts in question have been running it for 10 days now without any issue, so I think my problem is solved.