Bug 205099
|
Description
Erhard F.
2019-10-06 17:49:07 UTC
Created attachment 285369 [details]
kernel .config (5.4-rc1, PowerMac G4 DP)
As a side effect of this bug the btrfs module fails to load. If btrfs is not built as a module, but built into the kernel the machine fails to boot. Btrfs behaves normal again when I set KASAN_SANITIZE := n in lib/raid6/Makefile. Created attachment 286249 [details]
kernel .config (5.5-rc1, PowerMac G4 DP)
Created attachment 286251 [details]
dmesg (kernel 5.5-rc1, PowerMac G4 DP)
Unchanged in 5.5-rc1.
[..]
[ 19.425679] BUG: Unable to handle kernel data access on read at 0x00f0fd0d
[ 19.425693] Faulting instruction address: 0xf165a560
[ 19.426731] Oops: Kernel access of bad area, sig: 11 [#1]
[ 19.426745] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 DEBUG_PAGEALLOC PowerMac
[ 19.426757] Modules linked in: raid6_pq(+) zlib_inflate ehci_pci(+) ohci_hcd hwmon ehci_hcd i2c_algo_bit drm_kms_helper sungem syscopyarea sungem_phy sysfillrect firewire_ohci firewire_core sysimgblt crc_itu_t sr_mod fb_sys_fops usbcore ttm cdrom snd_aoa_i2sbus snd_aoa_soundbus usb_common snd_pcm snd_timer snd soundcore drm ssb pcmcia drm_panel_orientation_quirks pcmcia_core uninorth_agp agpgart
[ 19.426986] CPU: 1 PID: 127 Comm: modprobe Tainted: G W 5.5.0-rc1-PowerMacG4 #3
[ 19.426997] NIP: f165a560 LR: f165a4e8 CTR: c0247f54
[ 19.427009] REGS: e86d9740 TRAP: 0300 Tainted: G W (5.5.0-rc1-PowerMacG4)
[ 19.427014] MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 22228828 XER: 00000000
[ 19.427048] DAR: 00f0fd0d DSISR: 40000000
GPR00: e86d9998 e86d97f8 ee26c720 f166d070 00000010 00000000 f165a4e8 00000000
GPR08: 00000000 00f0fd0d e7171aea fffffff0 c0247f54 00b25ff4 00000060 00000050
GPR16: f166d000 ec077000 ec076000 00000070 00000060 00000050 00000040 00000030
GPR24: 00000020 00000010 00000012 e86d9a84 e86d9a48 ec077010 ec076010 00000000
[ 19.427198] NIP [f165a560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq]
[ 19.427228] LR [f165a4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
[ 19.427234] Call Trace:
[ 19.427242] [e86d97f8] [0000000a] 0xa (unreliable)
[ 19.427277] [e86d99e8] [f165a654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
[ 19.427309] [e86d9a08] [f15d83d8] init_module+0x3d8/0x528 [raid6_pq]
[ 19.427334] [e86d9b18] [c0005828] do_one_initcall+0xb8/0x36c
[ 19.427355] [e86d9be8] [c010e5e0] do_init_module+0xa8/0x2c4
[ 19.427369] [e86d9c18] [c01114c0] load_module+0x2be4/0x2d68
[ 19.427383] [e86d9e18] [c01118b8] sys_finit_module+0x100/0x138
[ 19.427397] [e86d9f38] [c001a274] ret_from_syscall+0x0/0x34
[ 19.427410] --- interrupt: c01 at 0x96ff78
LR = 0xafda14
[ 19.427416] Instruction dump:
[ 19.427429] 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4
[ 19.427463] 7d1098ce 115f5b06 116b5800 8141008c <7c0048ce> 392100a0 7d8048ce 392100b0
[ 19.427505] ---[ end trace 161d8d283bc9b7b8 ]---
Obviously, r9 is wrong
2538: 13 04 c4 c4 vxor v24,v4,v24
253c: 7d 20 48 ce lvx v9,0,r9
2540: 39 21 00 90 addi r9,r1,144
2544: 13 25 cc c4 vxor v25,v5,v25
2548: 7d 60 48 ce lvx v11,0,r9
254c: 13 46 d4 c4 vxor v26,v6,v26
2550: 81 21 00 88 lwz r9,136(r1) <== r9 is loaded here
2554: 13 67 dc c4 vxor v27,v7,v27
2558: 7d 11 a8 ce lvx v8,r17,r21
255c: 11 5f 5b 06 vcmpgtsb v10,v31,v11
2560: 11 6b 58 00 vaddubm v11,v11,v11
2564: 81 41 00 8c lwz r10,140(r1)
==> 2568: 7c 00 48 ce lvx v0,0,r9
256c: 39 21 00 a0 addi r9,r1,160
2570: 7d 80 48 ce lvx v12,0,r9
2574: 39 21 00 b0 addi r9,r1,176
So the stack must be clobbered somewhere
I enabled DEBUG_STACKOVERFLOW in the kernel .config but I got no additional hints... Is there any other debugging option that could help? Found out this is probably a cause for bug #205283. For doing some btrfs tests with 5.5-rc1 kernel I needed to disable KASAN in lib/raid6/Makefile (KASAN_SANITIZE := n) as btrfs pulls in raid6_pq. btrfs gets modprobed seemingly ok the 1st time, but removing and reloading it provokes bug #205283. # modprobe -v btrfs # modprobe -r -v btrfs rmmod btrfs rmmod zlib_inflate rmmod libcrc32c rmmod raid6_pq rmmod zlib_deflate rmmod lzo_decompress rmmod lzo_compress rmmod zstd_compress rmmod zstd_decompress rmmod xor # modprobe -v btrfs insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_inflate/zlib_inflate.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/libcrc32c.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/raid6/raid6_pq.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_deflate/zlib_deflate.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_decompress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_compress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_compress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_decompress.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/crypto/xor.ko insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/fs/btrfs/btrfs.ko Created attachment 286395 [details] dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) (In reply to Christophe Leroy from comment #10, bug #205283) > So it must be something more. By the way, now that KASAN_VMALLOC exists and > handle module loading better, I think we should analyse the impact of > activating KASAN_VMALLOC at all time. > > Regarding #205099, I believe there's a real stack related bug in raid6_pq > that needs to be identified. Did you try with INLINE KASAN ? With INLINE KASAN and KASAN_VMALLOC this hit does not show up at all. It does show up with OUTLINE KASAN and KASAN_VMALLOC. Created attachment 286397 [details]
dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + OUTLINE KASAN, PowerMac G4 DP)
Applied your patch series on top of 5.5-rc6. CONFIG_KASAN_VMALLOC is not non-selectable but forced on by default. Current situation is that the hit does not show up with INLINE KASAN in raid6_pq. Created attachment 286869 [details]
dmesg (kernel 5.5-rc6+, INLINE KASAN, PowerMac G4 DP)
However I might have gotten a hint where the stack is clobbered. The IRQ stack overflows sooner or later (CONFIG_DEBUG_SHIRQ=y might have brought that up?), which causes all sorts of problems.
I'll attach the dmesg here but I can open a new bug if this is more suitable.
Created attachment 286871 [details]
kernel .config (5.5-rc6, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP))
Created attachment 286873 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) w. IRQ stack overflow
Interesting that stack overflows. Maybe we should increase THREAD_SHIFT by 1 when KASAN is selected. ARM64 does that. Would be interesting to try and see if the VMAP_STACK series detects the stack overflow. Series at https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973 (In reply to Christophe Leroy from comment #15) > Would be interesting to try and see if the VMAP_STACK series detects the > stack overflow. > Series at > https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973 I just tried that, but in this case the G4 won't boot at all. The kernel loads but booting gets stuck at a white display with only 2 lines visible where it tells me it will open the display of the G4s Radeons offb display. I applied your CONFIG_KASAN_VMALLOC on top of 5.5-rc6 and applied your VMAP_STACK series afterwards. # grep -i kasan .config CONFIG_HAVE_ARCH_KASAN=y CONFIG_HAVE_ARCH_KASAN_VMALLOC=y CONFIG_CC_HAS_KASAN_GENERIC=y CONFIG_KASAN=y CONFIG_KASAN_GENERIC=y # CONFIG_KASAN_OUTLINE is not set CONFIG_KASAN_INLINE=y CONFIG_KASAN_STACK=1 CONFIG_KASAN_VMALLOC=y # CONFIG_TEST_KASAN is not set CONFIG_KASAN_SHADOW_OFFSET=0xe0000000 # grep -i vmap .config CONFIG_HAVE_ARCH_VMAP_STACK=y CONFIG_VMAP_STACK=y Created attachment 286907 [details]
Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK
Please try the attached patch, it fixes the setup of the kasan early hash table when VMAP_STACK is enabled.
(In reply to Christophe Leroy from comment #17) > Created attachment 286907 [details] > Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK > > Please try the attached patch, it fixes the setup of the kasan early hash > table when VMAP_STACK is enabled. Sorry, but still the same situation with this patch applied. Can you tell exactly where it stops during the boot ? Or take a photo of the screen ? In parallele, could you try (without VMAP_STACK) increasing CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks. Created attachment 286925 [details]
kernel .config (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Created attachment 286927 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP)
(In reply to Christophe Leroy from comment #19) > Can you tell exactly where it stops during the boot ? Or take a photo of the > screen ? I'll attach a photo shortly. > In parallele, could you try (without VMAP_STACK) increasing > CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks. I can confirm that with CONFIG_THREAD_SHIFT=14 the G4 runs good again with INLINE KASAN + KASAN_VMALLOC enabled (as seen in attached dmesg). No KASAN hit at raid6_pq, no poblem with btrfs. Though with CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN + KASAN_VMALLOC I still get this KASAN hit at raid6_pq. Created attachment 286929 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN, PowerMac G4 DP)
Created attachment 286931 [details]
screenshot (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Created attachment 287621 [details]
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP)
Gave 5.6-rc3 + KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN a test ride and it's still in:
[...]
Feb 26 16:29:30 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0
Feb 26 16:29:30 T600 kernel: Faulting instruction address: 0xf168b560
Feb 26 16:29:30 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Feb 26 16:29:30 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Feb 26 16:29:30 T600 kernel: Modules linked in: raid6_pq(+) libcrc32c zlib_inflate snd_aoa_i2sbus snd_aoa_soundbus snd_pcm ohci_hcd ehci_pci(+) ssb snd_timer ehci_hcd pcmcia snd pcmcia_core soundcore uninorth_agp usbcore agpgart usb_common
Feb 26 16:29:30 T600 kernel: CPU: 1 PID: 118 Comm: modprobe Tainted: G W 5.6.0-rc3-PowerMacG4+ #1
Feb 26 16:29:30 T600 kernel: NIP: f168b560 LR: f168b4e8 CTR: c024b2e0
Feb 26 16:29:30 T600 kernel: REGS: f10bd780 TRAP: 0300 Tainted: G W (5.6.0-rc3-PowerMacG4+)
Feb 26 16:29:30 T600 kernel: MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 28228828 XER: 00000000
Feb 26 16:29:30 T600 kernel: DAR: 0050a0f0 DSISR: 40000000
GPR00: f10bd9d8 f10bd838 ebeca380 ec56c070 00000010 00000000 f168b4e8 00000000
GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c024b2e0 007efff4 00000060 00000050
GPR16: ec56c000 ec56f000 ec56e000 00000070 00000060 00000050 00000040 00000030
GPR24: 00000020 00000010 00000008 f10bda8c f10bda78 ec56f010 ec56e010 00000000
Feb 26 16:29:30 T600 kernel: NIP [f168b560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq]
Feb 26 16:29:30 T600 kernel: LR [f168b4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
Feb 26 16:29:30 T600 kernel: Call Trace:
Feb 26 16:29:30 T600 kernel: [f10bd838] [0000000a] 0xa (unreliable)
Feb 26 16:29:30 T600 kernel: [f10bda28] [f168b654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
Feb 26 16:29:30 T600 kernel: [f10bda48] [f145b3cc] init_module+0x3cc/0x530 [raid6_pq]
Feb 26 16:29:30 T600 kernel: [f10bdb18] [c00058e4] do_one_initcall+0xb8/0x36c
Feb 26 16:29:30 T600 kernel: [f10bdbe8] [c010f760] do_init_module+0xa8/0x2c4
Feb 26 16:29:30 T600 kernel: [f10bdc18] [c0112630] load_module+0x2bd4/0x2d58
Feb 26 16:29:30 T600 kernel: [f10bde18] [c0112a28] sys_finit_module+0x100/0x138
Feb 26 16:29:30 T600 kernel: [f10bdf38] [c001a234] ret_from_syscall+0x0/0x34
Feb 26 16:29:30 T600 kernel: --- interrupt: c01 at 0x639988
Created attachment 287623 [details]
kernel .config (5.6-rc3, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP)
Created attachment 287625 [details]
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP)
And still the same phenomenon that I only get this hit with OUTLINE KASAN but not with INLINE KASAN (+ CONFIG_THREAD_SHIFT=14).
Created attachment 288411 [details]
dmesg (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP)
Re-tested with v5.7-rc1 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal.
[...]
Apr 13 16:01:11 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0
Apr 13 16:01:11 T600 kernel: Faulting instruction address: 0xf1aa1564
Apr 13 16:01:11 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 13 16:01:11 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Apr 13 16:01:11 T600 kernel: Modules linked in: raid6_pq(+) zlib_inflate firewire_ohci firewire_core ehci_pci(+) ohci_hcd crc_itu_t sr_mod cdrom ehci_hcd snd_aoa_i2sbus snd_aoa_soundbus snd_pcm snd_timer snd ss>
Apr 13 16:01:11 T600 kernel: CPU: 0 PID: 149 Comm: modprobe Tainted: G W 5.7.0-rc1-PowerMacG4+ #1
Apr 13 16:01:11 T600 kernel: NIP: f1aa1564 LR: f1aa14e8 CTR: c0255b78
Apr 13 16:01:11 T600 kernel: REGS: f1963780 TRAP: 0300 Tainted: G W (5.7.0-rc1-PowerMacG4+)
Apr 13 16:01:11 T600 kernel: MSR: 02009032 <VEC,EE,ME,IR,DR,RI> CR: 28228828 XER: 00000000
Apr 13 16:01:11 T600 kernel: DAR: 0050a0f0 DSISR: 40000000
GPR00: f19639d8 f1963838 ebd1de60 eb1dc070 00001000 00000003 f1aa14e8 00000000
GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c0255b78 00a48ff4 00000060 00000050
GPR16: eb1dc000 eb1df000 eb1de000 00000070 00000060 00000050 00000040 00000030
GPR24: 00000020 00000010 00000008 f1963a8c f1963a78 eb1df010 eb1de010 00000000
Apr 13 16:01:11 T600 kernel: NIP [f1aa1564] raid6_altivec8_gen_syndrome_real+0x3c4/0x480 [raid6_pq]
Apr 13 16:01:11 T600 kernel: LR [f1aa14e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
Apr 13 16:01:11 T600 kernel: Call Trace:
Apr 13 16:01:11 T600 kernel: [f1963838] [0000000a] 0xa (unreliable)
Apr 13 16:01:11 T600 kernel: [f1963a28] [f1aa1654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
Apr 13 16:01:11 T600 kernel: [f1963a48] [f12603cc] init_module+0x3cc/0x530 [raid6_pq]
Apr 13 16:01:11 T600 kernel: [f1963b18] [c00058e4] do_one_initcall+0xb8/0x36c
Apr 13 16:01:11 T600 kernel: [f1963be8] [c0117f64] do_init_module+0xa8/0x2c4
Apr 13 16:01:11 T600 kernel: [f1963c18] [c011ae38] load_module+0x2bd8/0x2d5c
Apr 13 16:01:11 T600 kernel: [f1963e18] [c011b230] sys_finit_module+0x100/0x138
Apr 13 16:01:11 T600 kernel: [f1963f38] [c001a224] ret_from_syscall+0x0/0x34
Apr 13 16:01:11 T600 kernel: --- interrupt: c01 at 0x892988
LR = 0xa20a14
Apr 13 16:01:11 T600 kernel: Instruction dump:
Apr 13 16:01:11 T600 kernel: 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4
Apr 13 16:01:11 T600 kernel: 8141008c 11284cc4 115f5b06 116b5800 <7c0048ce> 392100a0 7d8048ce 392100b0
Apr 13 16:01:11 T600 kernel: ---[ end trace ebe3b589d509d4f6 ]---
[...]
Created attachment 288413 [details]
kernel .config (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP)
Created attachment 292345 [details]
dmesg (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP)
Re-tested with v5.9-rc3 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal:
==================================================================
BUG: KASAN: user-memory-access in raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq]
Read of size 4 at addr 5764b118 by task modprobe/126
CPU: 1 PID: 126 Comm: modprobe Tainted: G W 5.9.0-rc3-PowerMacG4 #2
Call Trace:
[e32cb7b8] [c0517aac] dump_stack+0xc4/0xf8 (unreliable)
[e32cb7e8] [c026e73c] kasan_report+0x16c/0x170
[e32cb828] [b02004e0] raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq]
[e32cba18] [b02006fc] raid6_altivec8_gen_syndrome+0x4c/0x88 [raid6_pq]
[e32cba38] [b021a42c] init_module+0x42c/0x590 [raid6_pq]
[e32cbb08] [c00058a0] do_one_initcall+0xb8/0x3dc
[e32cbbd8] [c011c0fc] do_init_module+0xa8/0x2c4
[e32cbc08] [c011f02c] load_module+0x2b98/0x2d4c
[e32cbe18] [c011f448] sys_finit_module+0x100/0x138
[e32cbf38] [c001a1cc] ret_from_syscall+0x0/0x34
--- interrupt: c01 at 0x3d2068
LR = 0x506104
==================================================================
BUG: Unable to handle kernel data access on read at 0x5764b118
Faulting instruction address: 0xb02004e0
Oops: Kernel access of bad area, sig: 11 [#1]
Created attachment 292347 [details]
kernel .config (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP)
Does happen even if RAID support is not actively selected in the config as btrfs pulls in RAID6_PQ on its own.
# CONFIG_DM_RAID is not set
CONFIG_RAID6_PQ=m
Is this problem still there with 5.13 ? I can tell you in the next few weeks, as soon as my G4 MDD gets its' overhaul with the serviced PSU. Created attachment 299711 [details]
dmesg (5.15.5, OUTLINE KASAN, PowerMac G4 DP)
Finally my G4 DP got its' fixed & overhauled PSU so I am able to continue here.
Tested kernel 5.15.5 and the original KASAN hit at raid6_pq did not show up. Instead I am getting this now:
[...]
Running code patching self-tests ...
vmap allocation for size 33562624 failed: use vmalloc=<size> to increase size
swapper/0: vmalloc error: size 33558528, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null)
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 5.15.5-gentoo-PowerMacG4 #5
Call Trace:
[f1033bb0] [c0d933c0] dump_stack_lvl+0x80/0xb0 (unreliable)
[f1033bd0] [c0516128] warn_alloc+0x11c/0x2b4
[f1033cb0] [c0508c40] __vmalloc_node_range+0xd8/0x64c
[f1033d70] [c0508a58] __vmalloc_node+0xec/0xf4
[f1033db0] [c1c0ecb0] test_code_patching+0x72c/0xd50
[f1033df0] [c0008908] do_one_initcall+0x284/0x574
[f1033ec0] [c1c03f78] kernel_init_freeable+0x510/0x51c
[f1033f10] [c000934c] kernel_init+0x24/0x140
[f1033f30] [c0033148] ret_from_kernel_thread+0x14/0x1c
Mem-Info:
active_anon:0 inactive_anon:0 isolated_anon:0
active_file:0 inactive_file:0 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:1306 slab_unreclaimable:4214
mapped:0 shmem:0 pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:474459 free_pcp:592 free_cma:0
Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:840kB pagetables:0kB all_unreclaimable? no
DMA free:588612kB min:3144kB low:3928kB high:4712kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:618264kB mlocked:0kB bounce:0kB free_pcp:1964kB local_pcp:1568kB free_cma:0kB
lowmem_reserve[]: 0 0 1280 1280
HighMem free:1309224kB min:512kB low:2176kB high:3840kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:404kB local_pcp:116kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
DMA: 5*4kB (ME) 2*8kB (UM) 4*16kB (M) 5*32kB (ME) 1*64kB (M) 4*128kB (UME) 4*256kB (ME) 4*512kB (UM) 3*1024kB (M) 4*2048kB (ME) 140*4096kB (M) = 588612kB
HighMem: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 2*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (U) 319*4096kB (M) = 1309224kB
0 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
524288 pages RAM
327680 pages HighMem/MovableOnly
42042 pages reserved
code-patching: test failed at line 598
[...]
This makes me wonder whether KASAN is working correctly when not enough vmap size is available. I did set it to vmalloc=64M via Kernel command line but this does not seem to be reckognized. Same situation with INLINE KASAN. In both cases with VMAP_STACK and CONFIG_THREAD_SHIFT=14.
Created attachment 299713 [details]
dmesg (5.15.5, INLINE KASAN, PowerMac G4 DP)
Created attachment 299715 [details]
kernel .config (5.15.5, PowerMac G4 DP)
I see no obvious reason for a 32Mb allocation to fail while you have 588612kB free memory. And that happens early at boot, before user processes are started so the vmap area, allthough not very big, should still have 32M space available. Looks like only x86 are arm implement this vmalloc= parameter:
[chleroy@PO20335 linux-powerpc]$ git grep 'early_param("vmalloc"' arch/
arch/arm/mm/mmu.c:early_param("vmalloc", early_vmalloc);
arch/x86/mm/pgtable_32.c:early_param("vmalloc", parse_vmalloc);
However, your vmalloc area has a size of 65M:
Kernel virtual memory layout:
* 0xf6000000..0xfec00000 : kasan shadow mem
* 0xf5bbf000..0xf5fff000 : fixmap
* 0xf5400000..0xf5800000 : highmem PTEs
* 0xf5115000..0xf5400000 : early ioremap
* 0xf1000000..0xf5110000 : vmalloc & ioremap
* 0xb0000000..0xc0000000 : modules
Memory: 1928984K/2097152K available (22288K kernel code, 2616K rwdata, 4868K rodata, 1408K init, 8981K bss, 168168K reserved, 0K cma-reserved, 1310720K highmem)
Can you retry with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000 ? Would also be great if you can activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables Created attachment 299759 [details]
dmesg (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x28000000, PowerMac G4 DP)
Ok, finally getting somewhere.. With CONFIG_LOWMEM_SIZE=0x28000000 I don't get this "vmap allocation for size 33562624 failed" error and also OUTLINE KASAN does not show the originally reported "BUG: Unable to handle kernel data access at 0x00f0fd0d" for raid6_pq!
For INLINE KASAN I still get "vmap allocation for size 33562624 failed", even with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000.
Created attachment 299761 [details]
dmesg (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Created attachment 299763 [details]
kernel .config (5.15.5, PowerMac G4 DP)
Interesting. I wonder why it works with OUTLINE KASAN but not with INLINE KASAN. Can you activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables for the cases than don't work (OUTILINE KASAN + 0x30000000 LOWMEM_SIZE, INLINE KASAN + 0x20000000 LOWMEM SIZE). Created attachment 299773 [details]
kernel_page_tables (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
Ah yes, I forgot about including the /sys/kernel/debug/kernel_page_tables.. Sorry! Here you are.
Created attachment 299775 [details]
kernel_page_tables (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Created attachment 299793 [details]
kernel_page_tables (5.15.5 + patch, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
Created attachment 299795 [details]
kernel_page_tables (5.15.5 + patch, INLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
As the other KASAN inconsistencies mentioned here got ironed out by Christophe (patches available but not in stable yet) and this "KASAN hit at raid6_pq: BUG: Unable to handle kernel data access at 0x00f0fd0d" does no longer show up with OUTLINE/INLINE KASAN on 5.15.5 or 5.16-rc3 I think it is safe to close this bug as obsolete. |