Created attachment 285367 [details] dmesg (kernel 5.4-rc1, PowerMac G4 DP) With lots of KASAN ppc fixes included in kernel 5.4-rc1 now, I am revisiting the issues of bug #204479. raid6 pq still reliably oopses: [...] [ 19.186966] BUG: Unable to handle kernel data access at 0x00f0fd0d [ 19.196913] Faulting instruction address: 0xf1692560 [ 19.235490] ohci-pci: OHCI PCI platform driver [ 19.236988] Oops: Kernel access of bad area, sig: 11 [#1] [ 19.255518] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac [ 19.265311] Modules linked in: ohci_pci(+) raid6_pq(+) zlib_inflate sungem hwmon sungem_phy i2c_algo_bit sr_mod drm_kms_helper cdrom syscopyarea sysfillrect firewire_ohci sysimgblt ohci_hcd ehci_pci fb_sys_fops ttm firewire_core snd_aoa_i2sbus ehci_hcd crc_itu_t snd_aoa_soundbus snd_pcm snd_timer drm snd soundcore usbcore ssb uninorth_agp drm_panel_orientation_quirks pcmcia usb_common pcmcia_core agpgart [ 19.287432] CPU: 0 PID: 127 Comm: modprobe Tainted: G W 5.4.0-rc1 #10 [ 19.298613] NIP: f1692560 LR: f16924e8 CTR: c023bcf4 [ 19.309692] REGS: e8d13730 TRAP: 0300 Tainted: G W (5.4.0-rc1) [ 19.320847] MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 28228822 XER: 00000000 [ 19.332102] DAR: 00f0fd0d DSISR: 40000000 GPR00: e8d13988 e8d137e8 ee01b4a0 f16a5070 00000010 00000000 f16924e8 00000000 GPR08: f16a5070 00f0fd0d e7171aea e8d13a70 c023bcf4 0050fff4 00000060 00000050 GPR16: 00000040 f16a5000 00000012 eb28d000 eb28c000 00000070 00000060 00000050 GPR24: 00000040 00000030 00000020 00000010 e8d13a38 eb28d010 eb28c010 00000000 [ 19.388060] NIP [f1692560] raid6_altivec8_gen_syndrome_real+0x3c8/0x488 [raid6_pq] [ 19.399417] LR [f16924e8] raid6_altivec8_gen_syndrome_real+0x350/0x488 [raid6_pq] [ 19.405058] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 19.410774] Call Trace: [ 19.410786] [e8d137e8] [00001032] 0x1032 (unreliable) [ 19.410824] [e8d139d8] [f1692654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq] [ 19.410857] [e8d13a08] [f15c83c0] init_module+0x3c0/0x518 [raid6_pq] [ 19.460942] Apple USB OHCI 0001:10:18.0 disabled by firmware [ 19.468239] [e8d13af8] [c0005820] do_one_initcall+0xb8/0x36c [ 19.468264] [e8d13bc8] [c0109480] do_init_module+0xa8/0x2c4 [ 19.502777] [e8d13bf8] [c010c2e4] load_module+0x2b68/0x2d24 [ 19.514192] [e8d13df8] [c010c704] sys_finit_module+0x100/0x140 [ 19.525556] [e8d13f38] [c001a274] ret_from_syscall+0x0/0x34 [ 19.536910] --- interrupt: c01 at 0x35e2c4 LR = 0x4e67c4 [ 19.559299] Instruction dump: [ 19.570504] 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 [ 19.581997] 7d11a8ce 115f5b06 116b5800 8141008c <7c0048ce> 392100a0 7d8048ce 392100b0 [ 19.593571] ---[ end trace 5e7216580d039a39 ]---
Created attachment 285369 [details] kernel .config (5.4-rc1, PowerMac G4 DP)
As a side effect of this bug the btrfs module fails to load. If btrfs is not built as a module, but built into the kernel the machine fails to boot. Btrfs behaves normal again when I set KASAN_SANITIZE := n in lib/raid6/Makefile.
Created attachment 286249 [details] kernel .config (5.5-rc1, PowerMac G4 DP)
Created attachment 286251 [details] dmesg (kernel 5.5-rc1, PowerMac G4 DP)
Unchanged in 5.5-rc1. [..] [ 19.425679] BUG: Unable to handle kernel data access on read at 0x00f0fd0d [ 19.425693] Faulting instruction address: 0xf165a560 [ 19.426731] Oops: Kernel access of bad area, sig: 11 [#1] [ 19.426745] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 DEBUG_PAGEALLOC PowerMac [ 19.426757] Modules linked in: raid6_pq(+) zlib_inflate ehci_pci(+) ohci_hcd hwmon ehci_hcd i2c_algo_bit drm_kms_helper sungem syscopyarea sungem_phy sysfillrect firewire_ohci firewire_core sysimgblt crc_itu_t sr_mod fb_sys_fops usbcore ttm cdrom snd_aoa_i2sbus snd_aoa_soundbus usb_common snd_pcm snd_timer snd soundcore drm ssb pcmcia drm_panel_orientation_quirks pcmcia_core uninorth_agp agpgart [ 19.426986] CPU: 1 PID: 127 Comm: modprobe Tainted: G W 5.5.0-rc1-PowerMacG4 #3 [ 19.426997] NIP: f165a560 LR: f165a4e8 CTR: c0247f54 [ 19.427009] REGS: e86d9740 TRAP: 0300 Tainted: G W (5.5.0-rc1-PowerMacG4) [ 19.427014] MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 22228828 XER: 00000000 [ 19.427048] DAR: 00f0fd0d DSISR: 40000000 GPR00: e86d9998 e86d97f8 ee26c720 f166d070 00000010 00000000 f165a4e8 00000000 GPR08: 00000000 00f0fd0d e7171aea fffffff0 c0247f54 00b25ff4 00000060 00000050 GPR16: f166d000 ec077000 ec076000 00000070 00000060 00000050 00000040 00000030 GPR24: 00000020 00000010 00000012 e86d9a84 e86d9a48 ec077010 ec076010 00000000 [ 19.427198] NIP [f165a560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq] [ 19.427228] LR [f165a4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq] [ 19.427234] Call Trace: [ 19.427242] [e86d97f8] [0000000a] 0xa (unreliable) [ 19.427277] [e86d99e8] [f165a654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq] [ 19.427309] [e86d9a08] [f15d83d8] init_module+0x3d8/0x528 [raid6_pq] [ 19.427334] [e86d9b18] [c0005828] do_one_initcall+0xb8/0x36c [ 19.427355] [e86d9be8] [c010e5e0] do_init_module+0xa8/0x2c4 [ 19.427369] [e86d9c18] [c01114c0] load_module+0x2be4/0x2d68 [ 19.427383] [e86d9e18] [c01118b8] sys_finit_module+0x100/0x138 [ 19.427397] [e86d9f38] [c001a274] ret_from_syscall+0x0/0x34 [ 19.427410] --- interrupt: c01 at 0x96ff78 LR = 0xafda14 [ 19.427416] Instruction dump: [ 19.427429] 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 [ 19.427463] 7d1098ce 115f5b06 116b5800 8141008c <7c0048ce> 392100a0 7d8048ce 392100b0 [ 19.427505] ---[ end trace 161d8d283bc9b7b8 ]---
Obviously, r9 is wrong 2538: 13 04 c4 c4 vxor v24,v4,v24 253c: 7d 20 48 ce lvx v9,0,r9 2540: 39 21 00 90 addi r9,r1,144 2544: 13 25 cc c4 vxor v25,v5,v25 2548: 7d 60 48 ce lvx v11,0,r9 254c: 13 46 d4 c4 vxor v26,v6,v26 2550: 81 21 00 88 lwz r9,136(r1) <== r9 is loaded here 2554: 13 67 dc c4 vxor v27,v7,v27 2558: 7d 11 a8 ce lvx v8,r17,r21 255c: 11 5f 5b 06 vcmpgtsb v10,v31,v11 2560: 11 6b 58 00 vaddubm v11,v11,v11 2564: 81 41 00 8c lwz r10,140(r1) ==> 2568: 7c 00 48 ce lvx v0,0,r9 256c: 39 21 00 a0 addi r9,r1,160 2570: 7d 80 48 ce lvx v12,0,r9 2574: 39 21 00 b0 addi r9,r1,176 So the stack must be clobbered somewhere
I enabled DEBUG_STACKOVERFLOW in the kernel .config but I got no additional hints... Is there any other debugging option that could help?
Found out this is probably a cause for bug #205283. For doing some btrfs tests with 5.5-rc1 kernel I needed to disable KASAN in lib/raid6/Makefile (KASAN_SANITIZE := n) as btrfs pulls in raid6_pq. btrfs gets modprobed seemingly ok the 1st time, but removing and reloading it provokes bug #205283. # modprobe -v btrfs # modprobe -r -v btrfs rmmod btrfs rmmod zlib_inflate rmmod libcrc32c rmmod raid6_pq rmmod zlib_deflate rmmod lzo_decompress rmmod lzo_compress rmmod zstd_compress rmmod zstd_decompress rmmod xor # modprobe -v btrfs insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_inflate/zlib_inflate.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/libcrc32c.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/raid6/raid6_pq.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_deflate/zlib_deflate.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_decompress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_compress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_compress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_decompress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/crypto/xor.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/fs/btrfs/btrfs.ko
Created attachment 286395 [details] dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) (In reply to Christophe Leroy from comment #10, bug #205283) > So it must be something more. By the way, now that KASAN_VMALLOC exists and > handle module loading better, I think we should analyse the impact of > activating KASAN_VMALLOC at all time. > > Regarding #205099, I believe there's a real stack related bug in raid6_pq > that needs to be identified. Did you try with INLINE KASAN ? With INLINE KASAN and KASAN_VMALLOC this hit does not show up at all. It does show up with OUTLINE KASAN and KASAN_VMALLOC.
Created attachment 286397 [details] dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + OUTLINE KASAN, PowerMac G4 DP)
Applied your patch series on top of 5.5-rc6. CONFIG_KASAN_VMALLOC is not non-selectable but forced on by default. Current situation is that the hit does not show up with INLINE KASAN in raid6_pq.
Created attachment 286869 [details] dmesg (kernel 5.5-rc6+, INLINE KASAN, PowerMac G4 DP) However I might have gotten a hint where the stack is clobbered. The IRQ stack overflows sooner or later (CONFIG_DEBUG_SHIRQ=y might have brought that up?), which causes all sorts of problems. I'll attach the dmesg here but I can open a new bug if this is more suitable.
Created attachment 286871 [details] kernel .config (5.5-rc6, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP))
Created attachment 286873 [details] dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) w. IRQ stack overflow
Interesting that stack overflows. Maybe we should increase THREAD_SHIFT by 1 when KASAN is selected. ARM64 does that. Would be interesting to try and see if the VMAP_STACK series detects the stack overflow. Series at https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973
(In reply to Christophe Leroy from comment #15) > Would be interesting to try and see if the VMAP_STACK series detects the > stack overflow. > Series at > https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973 I just tried that, but in this case the G4 won't boot at all. The kernel loads but booting gets stuck at a white display with only 2 lines visible where it tells me it will open the display of the G4s Radeons offb display. I applied your CONFIG_KASAN_VMALLOC on top of 5.5-rc6 and applied your VMAP_STACK series afterwards. # grep -i kasan .config CONFIG_HAVE_ARCH_KASAN=y CONFIG_HAVE_ARCH_KASAN_VMALLOC=y CONFIG_CC_HAS_KASAN_GENERIC=y CONFIG_KASAN=y CONFIG_KASAN_GENERIC=y # CONFIG_KASAN_OUTLINE is not set CONFIG_KASAN_INLINE=y CONFIG_KASAN_STACK=1 CONFIG_KASAN_VMALLOC=y # CONFIG_TEST_KASAN is not set CONFIG_KASAN_SHADOW_OFFSET=0xe0000000 # grep -i vmap .config CONFIG_HAVE_ARCH_VMAP_STACK=y CONFIG_VMAP_STACK=y
Created attachment 286907 [details] Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK Please try the attached patch, it fixes the setup of the kasan early hash table when VMAP_STACK is enabled.
(In reply to Christophe Leroy from comment #17) > Created attachment 286907 [details] > Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK > > Please try the attached patch, it fixes the setup of the kasan early hash > table when VMAP_STACK is enabled. Sorry, but still the same situation with this patch applied.
Can you tell exactly where it stops during the boot ? Or take a photo of the screen ? In parallele, could you try (without VMAP_STACK) increasing CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks.
Created attachment 286925 [details] kernel .config (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Created attachment 286927 [details] dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP)
(In reply to Christophe Leroy from comment #19) > Can you tell exactly where it stops during the boot ? Or take a photo of the > screen ? I'll attach a photo shortly. > In parallele, could you try (without VMAP_STACK) increasing > CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks. I can confirm that with CONFIG_THREAD_SHIFT=14 the G4 runs good again with INLINE KASAN + KASAN_VMALLOC enabled (as seen in attached dmesg). No KASAN hit at raid6_pq, no poblem with btrfs. Though with CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN + KASAN_VMALLOC I still get this KASAN hit at raid6_pq.
Created attachment 286929 [details] dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN, PowerMac G4 DP)
Created attachment 286931 [details] screenshot (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Created attachment 287621 [details] dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP) Gave 5.6-rc3 + KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN a test ride and it's still in: [...] Feb 26 16:29:30 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0 Feb 26 16:29:30 T600 kernel: Faulting instruction address: 0xf168b560 Feb 26 16:29:30 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1] Feb 26 16:29:30 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac Feb 26 16:29:30 T600 kernel: Modules linked in: raid6_pq(+) libcrc32c zlib_inflate snd_aoa_i2sbus snd_aoa_soundbus snd_pcm ohci_hcd ehci_pci(+) ssb snd_timer ehci_hcd pcmcia snd pcmcia_core soundcore uninorth_agp usbcore agpgart usb_common Feb 26 16:29:30 T600 kernel: CPU: 1 PID: 118 Comm: modprobe Tainted: G W 5.6.0-rc3-PowerMacG4+ #1 Feb 26 16:29:30 T600 kernel: NIP: f168b560 LR: f168b4e8 CTR: c024b2e0 Feb 26 16:29:30 T600 kernel: REGS: f10bd780 TRAP: 0300 Tainted: G W (5.6.0-rc3-PowerMacG4+) Feb 26 16:29:30 T600 kernel: MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 28228828 XER: 00000000 Feb 26 16:29:30 T600 kernel: DAR: 0050a0f0 DSISR: 40000000 GPR00: f10bd9d8 f10bd838 ebeca380 ec56c070 00000010 00000000 f168b4e8 00000000 GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c024b2e0 007efff4 00000060 00000050 GPR16: ec56c000 ec56f000 ec56e000 00000070 00000060 00000050 00000040 00000030 GPR24: 00000020 00000010 00000008 f10bda8c f10bda78 ec56f010 ec56e010 00000000 Feb 26 16:29:30 T600 kernel: NIP [f168b560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq] Feb 26 16:29:30 T600 kernel: LR [f168b4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq] Feb 26 16:29:30 T600 kernel: Call Trace: Feb 26 16:29:30 T600 kernel: [f10bd838] [0000000a] 0xa (unreliable) Feb 26 16:29:30 T600 kernel: [f10bda28] [f168b654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq] Feb 26 16:29:30 T600 kernel: [f10bda48] [f145b3cc] init_module+0x3cc/0x530 [raid6_pq] Feb 26 16:29:30 T600 kernel: [f10bdb18] [c00058e4] do_one_initcall+0xb8/0x36c Feb 26 16:29:30 T600 kernel: [f10bdbe8] [c010f760] do_init_module+0xa8/0x2c4 Feb 26 16:29:30 T600 kernel: [f10bdc18] [c0112630] load_module+0x2bd4/0x2d58 Feb 26 16:29:30 T600 kernel: [f10bde18] [c0112a28] sys_finit_module+0x100/0x138 Feb 26 16:29:30 T600 kernel: [f10bdf38] [c001a234] ret_from_syscall+0x0/0x34 Feb 26 16:29:30 T600 kernel: --- interrupt: c01 at 0x639988
Created attachment 287623 [details] kernel .config (5.6-rc3, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP)
Created attachment 287625 [details] dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP) And still the same phenomenon that I only get this hit with OUTLINE KASAN but not with INLINE KASAN (+ CONFIG_THREAD_SHIFT=14).
Created attachment 288411 [details] dmesg (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP) Re-tested with v5.7-rc1 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal. [...] Apr 13 16:01:11 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0 Apr 13 16:01:11 T600 kernel: Faulting instruction address: 0xf1aa1564 Apr 13 16:01:11 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1] Apr 13 16:01:11 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac Apr 13 16:01:11 T600 kernel: Modules linked in: raid6_pq(+) zlib_inflate firewire_ohci firewire_core ehci_pci(+) ohci_hcd crc_itu_t sr_mod cdrom ehci_hcd snd_aoa_i2sbus snd_aoa_soundbus snd_pcm snd_timer snd ss> Apr 13 16:01:11 T600 kernel: CPU: 0 PID: 149 Comm: modprobe Tainted: G W 5.7.0-rc1-PowerMacG4+ #1 Apr 13 16:01:11 T600 kernel: NIP: f1aa1564 LR: f1aa14e8 CTR: c0255b78 Apr 13 16:01:11 T600 kernel: REGS: f1963780 TRAP: 0300 Tainted: G W (5.7.0-rc1-PowerMacG4+) Apr 13 16:01:11 T600 kernel: MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 28228828 XER: 00000000 Apr 13 16:01:11 T600 kernel: DAR: 0050a0f0 DSISR: 40000000 GPR00: f19639d8 f1963838 ebd1de60 eb1dc070 00001000 00000003 f1aa14e8 00000000 GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c0255b78 00a48ff4 00000060 00000050 GPR16: eb1dc000 eb1df000 eb1de000 00000070 00000060 00000050 00000040 00000030 GPR24: 00000020 00000010 00000008 f1963a8c f1963a78 eb1df010 eb1de010 00000000 Apr 13 16:01:11 T600 kernel: NIP [f1aa1564] raid6_altivec8_gen_syndrome_real+0x3c4/0x480 [raid6_pq] Apr 13 16:01:11 T600 kernel: LR [f1aa14e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq] Apr 13 16:01:11 T600 kernel: Call Trace: Apr 13 16:01:11 T600 kernel: [f1963838] [0000000a] 0xa (unreliable) Apr 13 16:01:11 T600 kernel: [f1963a28] [f1aa1654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq] Apr 13 16:01:11 T600 kernel: [f1963a48] [f12603cc] init_module+0x3cc/0x530 [raid6_pq] Apr 13 16:01:11 T600 kernel: [f1963b18] [c00058e4] do_one_initcall+0xb8/0x36c Apr 13 16:01:11 T600 kernel: [f1963be8] [c0117f64] do_init_module+0xa8/0x2c4 Apr 13 16:01:11 T600 kernel: [f1963c18] [c011ae38] load_module+0x2bd8/0x2d5c Apr 13 16:01:11 T600 kernel: [f1963e18] [c011b230] sys_finit_module+0x100/0x138 Apr 13 16:01:11 T600 kernel: [f1963f38] [c001a224] ret_from_syscall+0x0/0x34 Apr 13 16:01:11 T600 kernel: --- interrupt: c01 at 0x892988 LR = 0xa20a14 Apr 13 16:01:11 T600 kernel: Instruction dump: Apr 13 16:01:11 T600 kernel: 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 Apr 13 16:01:11 T600 kernel: 8141008c 11284cc4 115f5b06 116b5800 <7c0048ce> 392100a0 7d8048ce 392100b0 Apr 13 16:01:11 T600 kernel: ---[ end trace ebe3b589d509d4f6 ]--- [...]
Created attachment 288413 [details] kernel .config (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP)
Created attachment 292345 [details] dmesg (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP) Re-tested with v5.9-rc3 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal: ================================================================== BUG: KASAN: user-memory-access in raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq] Read of size 4 at addr 5764b118 by task modprobe/126 CPU: 1 PID: 126 Comm: modprobe Tainted: G W 5.9.0-rc3-PowerMacG4 #2 Call Trace: [e32cb7b8] [c0517aac] dump_stack+0xc4/0xf8 (unreliable) [e32cb7e8] [c026e73c] kasan_report+0x16c/0x170 [e32cb828] [b02004e0] raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq] [e32cba18] [b02006fc] raid6_altivec8_gen_syndrome+0x4c/0x88 [raid6_pq] [e32cba38] [b021a42c] init_module+0x42c/0x590 [raid6_pq] [e32cbb08] [c00058a0] do_one_initcall+0xb8/0x3dc [e32cbbd8] [c011c0fc] do_init_module+0xa8/0x2c4 [e32cbc08] [c011f02c] load_module+0x2b98/0x2d4c [e32cbe18] [c011f448] sys_finit_module+0x100/0x138 [e32cbf38] [c001a1cc] ret_from_syscall+0x0/0x34 --- interrupt: c01 at 0x3d2068 LR = 0x506104 ================================================================== BUG: Unable to handle kernel data access on read at 0x5764b118 Faulting instruction address: 0xb02004e0 Oops: Kernel access of bad area, sig: 11 [#1]
Created attachment 292347 [details] kernel .config (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP) Does happen even if RAID support is not actively selected in the config as btrfs pulls in RAID6_PQ on its own. # CONFIG_DM_RAID is not set CONFIG_RAID6_PQ=m
Is this problem still there with 5.13 ?
I can tell you in the next few weeks, as soon as my G4 MDD gets its' overhaul with the serviced PSU.
Created attachment 299711 [details] dmesg (5.15.5, OUTLINE KASAN, PowerMac G4 DP) Finally my G4 DP got its' fixed & overhauled PSU so I am able to continue here. Tested kernel 5.15.5 and the original KASAN hit at raid6_pq did not show up. Instead I am getting this now: [...] Running code patching self-tests ... vmap allocation for size 33562624 failed: use vmalloc=<size> to increase size swapper/0: vmalloc error: size 33558528, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null) CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 5.15.5-gentoo-PowerMacG4 #5 Call Trace: [f1033bb0] [c0d933c0] dump_stack_lvl+0x80/0xb0 (unreliable) [f1033bd0] [c0516128] warn_alloc+0x11c/0x2b4 [f1033cb0] [c0508c40] __vmalloc_node_range+0xd8/0x64c [f1033d70] [c0508a58] __vmalloc_node+0xec/0xf4 [f1033db0] [c1c0ecb0] test_code_patching+0x72c/0xd50 [f1033df0] [c0008908] do_one_initcall+0x284/0x574 [f1033ec0] [c1c03f78] kernel_init_freeable+0x510/0x51c [f1033f10] [c000934c] kernel_init+0x24/0x140 [f1033f30] [c0033148] ret_from_kernel_thread+0x14/0x1c Mem-Info: active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 slab_reclaimable:1306 slab_unreclaimable:4214 mapped:0 shmem:0 pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:474459 free_pcp:592 free_cma:0 Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:840kB pagetables:0kB all_unreclaimable? no DMA free:588612kB min:3144kB low:3928kB high:4712kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:618264kB mlocked:0kB bounce:0kB free_pcp:1964kB local_pcp:1568kB free_cma:0kB lowmem_reserve[]: 0 0 1280 1280 HighMem free:1309224kB min:512kB low:2176kB high:3840kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:404kB local_pcp:116kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 DMA: 5*4kB (ME) 2*8kB (UM) 4*16kB (M) 5*32kB (ME) 1*64kB (M) 4*128kB (UME) 4*256kB (ME) 4*512kB (UM) 3*1024kB (M) 4*2048kB (ME) 140*4096kB (M) = 588612kB HighMem: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 2*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (U) 319*4096kB (M) = 1309224kB 0 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB 524288 pages RAM 327680 pages HighMem/MovableOnly 42042 pages reserved code-patching: test failed at line 598 [...] This makes me wonder whether KASAN is working correctly when not enough vmap size is available. I did set it to vmalloc=64M via Kernel command line but this does not seem to be reckognized. Same situation with INLINE KASAN. In both cases with VMAP_STACK and CONFIG_THREAD_SHIFT=14.
Created attachment 299713 [details] dmesg (5.15.5, INLINE KASAN, PowerMac G4 DP)
Created attachment 299715 [details] kernel .config (5.15.5, PowerMac G4 DP)
I see no obvious reason for a 32Mb allocation to fail while you have 588612kB free memory. And that happens early at boot, before user processes are started so the vmap area, allthough not very big, should still have 32M space available.
Looks like only x86 are arm implement this vmalloc= parameter: [chleroy@PO20335 linux-powerpc]$ git grep 'early_param("vmalloc"' arch/ arch/arm/mm/mmu.c:early_param("vmalloc", early_vmalloc); arch/x86/mm/pgtable_32.c:early_param("vmalloc", parse_vmalloc); However, your vmalloc area has a size of 65M: Kernel virtual memory layout: * 0xf6000000..0xfec00000 : kasan shadow mem * 0xf5bbf000..0xf5fff000 : fixmap * 0xf5400000..0xf5800000 : highmem PTEs * 0xf5115000..0xf5400000 : early ioremap * 0xf1000000..0xf5110000 : vmalloc & ioremap * 0xb0000000..0xc0000000 : modules Memory: 1928984K/2097152K available (22288K kernel code, 2616K rwdata, 4868K rodata, 1408K init, 8981K bss, 168168K reserved, 0K cma-reserved, 1310720K highmem)
Can you retry with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000 ?
Would also be great if you can activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables
Created attachment 299759 [details] dmesg (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x28000000, PowerMac G4 DP) Ok, finally getting somewhere.. With CONFIG_LOWMEM_SIZE=0x28000000 I don't get this "vmap allocation for size 33562624 failed" error and also OUTLINE KASAN does not show the originally reported "BUG: Unable to handle kernel data access at 0x00f0fd0d" for raid6_pq! For INLINE KASAN I still get "vmap allocation for size 33562624 failed", even with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000.
Created attachment 299761 [details] dmesg (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Created attachment 299763 [details] kernel .config (5.15.5, PowerMac G4 DP)
Interesting. I wonder why it works with OUTLINE KASAN but not with INLINE KASAN. Can you activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables for the cases than don't work (OUTILINE KASAN + 0x30000000 LOWMEM_SIZE, INLINE KASAN + 0x20000000 LOWMEM SIZE).
Created attachment 299773 [details] kernel_page_tables (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP) Ah yes, I forgot about including the /sys/kernel/debug/kernel_page_tables.. Sorry! Here you are.
Created attachment 299775 [details] kernel_page_tables (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Created attachment 299793 [details] kernel_page_tables (5.15.5 + patch, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
Created attachment 299795 [details] kernel_page_tables (5.15.5 + patch, INLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
As the other KASAN inconsistencies mentioned here got ironed out by Christophe (patches available but not in stable yet) and this "KASAN hit at raid6_pq: BUG: Unable to handle kernel data access at 0x00f0fd0d" does no longer show up with OUTLINE/INLINE KASAN on 5.15.5 or 5.16-rc3 I think it is safe to close this bug as obsolete.