Bug 205099 - KASAN hit at raid6_pq: BUG: Unable to handle kernel data access at 0x00f0fd0d
Summary: KASAN hit at raid6_pq: BUG: Unable to handle kernel data access at 0x00f0fd0d
Status: RESOLVED OBSOLETE
Alias: None
Product: Platform Specific/Hardware
Classification: Unclassified
Component: PPC-32 (show other bugs)
Hardware: PPC-32 Linux
: P1 normal
Assignee: platform_ppc-32
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-06 17:49 UTC by Erhard F.
Modified: 2021-11-30 15:39 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.5-rc1
Subsystem:
Regression: No
Bisected commit-id:


Attachments
dmesg (kernel 5.4-rc1, PowerMac G4 DP) (40.72 KB, text/plain)
2019-10-06 17:49 UTC, Erhard F.
Details
kernel .config (5.4-rc1, PowerMac G4 DP) (94.52 KB, text/plain)
2019-10-06 17:49 UTC, Erhard F.
Details
kernel .config (5.5-rc1, PowerMac G4 DP) (95.64 KB, text/plain)
2019-12-10 23:51 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc1, PowerMac G4 DP) (42.44 KB, text/plain)
2019-12-10 23:52 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) (46.61 KB, text/plain)
2019-12-21 17:42 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + OUTLINE KASAN, PowerMac G4 DP) (47.49 KB, text/plain)
2019-12-21 17:43 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc6+, INLINE KASAN, PowerMac G4 DP) (73.29 KB, text/plain)
2020-01-17 21:19 UTC, Erhard F.
Details
kernel .config (5.5-rc6, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP)) (95.55 KB, text/plain)
2020-01-17 21:22 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) w. IRQ stack overflow (73.29 KB, text/plain)
2020-01-17 21:25 UTC, Erhard F.
Details
Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK (1.15 KB, application/mbox)
2020-01-20 12:40 UTC, Christophe Leroy
Details
kernel .config (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP) (95.60 KB, text/plain)
2020-01-21 12:20 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP) (53.93 KB, text/plain)
2020-01-21 12:24 UTC, Erhard F.
Details
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN, PowerMac G4 DP) (47.28 KB, text/plain)
2020-01-21 13:00 UTC, Erhard F.
Details
screenshot (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP) (2.27 MB, image/jpeg)
2020-01-21 13:31 UTC, Erhard F.
Details
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP) (60.25 KB, text/plain)
2020-02-26 15:49 UTC, Erhard F.
Details
kernel .config (5.6-rc3, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP) (95.93 KB, text/plain)
2020-02-26 15:52 UTC, Erhard F.
Details
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP) (66.24 KB, text/plain)
2020-02-26 16:17 UTC, Erhard F.
Details
dmesg (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP) (78.58 KB, text/plain)
2020-04-13 14:12 UTC, Erhard F.
Details
kernel .config (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP) (97.75 KB, text/plain)
2020-04-13 14:15 UTC, Erhard F.
Details
dmesg (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP) (81.64 KB, text/plain)
2020-09-04 22:50 UTC, Erhard F.
Details
kernel .config (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP) (98.28 KB, text/plain)
2020-09-04 22:54 UTC, Erhard F.
Details
dmesg (5.15.5, OUTLINE KASAN, PowerMac G4 DP) (52.28 KB, text/plain)
2021-11-26 01:47 UTC, Erhard F.
Details
dmesg (5.15.5, INLINE KASAN, PowerMac G4 DP) (57.37 KB, text/plain)
2021-11-26 01:48 UTC, Erhard F.
Details
kernel .config (5.15.5, PowerMac G4 DP) (108.08 KB, text/plain)
2021-11-26 01:49 UTC, Erhard F.
Details
dmesg (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x28000000, PowerMac G4 DP) (55.52 KB, text/plain)
2021-11-29 00:32 UTC, Erhard F.
Details
dmesg (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP) (57.41 KB, text/plain)
2021-11-29 00:32 UTC, Erhard F.
Details
kernel .config (5.15.5, PowerMac G4 DP) (108.41 KB, text/plain)
2021-11-29 00:34 UTC, Erhard F.
Details
kernel_page_tables (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP) (40.90 KB, text/plain)
2021-11-29 19:46 UTC, Erhard F.
Details
kernel_page_tables (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP) (40.90 KB, text/plain)
2021-11-29 19:46 UTC, Erhard F.
Details
kernel_page_tables (5.15.5 + patch, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP) (40.51 KB, text/plain)
2021-11-30 13:58 UTC, Erhard F.
Details
kernel_page_tables (5.15.5 + patch, INLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP) (39.62 KB, text/plain)
2021-11-30 13:58 UTC, Erhard F.
Details

Description Erhard F. 2019-10-06 17:49:07 UTC
Created attachment 285367 [details]
dmesg (kernel 5.4-rc1, PowerMac G4 DP)

With lots of KASAN ppc fixes included in kernel 5.4-rc1 now, I am revisiting the issues of bug #204479.

raid6 pq still reliably oopses:

[...]
[   19.186966] BUG: Unable to handle kernel data access at 0x00f0fd0d
[   19.196913] Faulting instruction address: 0xf1692560
[   19.235490] ohci-pci: OHCI PCI platform driver
[   19.236988] Oops: Kernel access of bad area, sig: 11 [#1]
[   19.255518] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
[   19.265311] Modules linked in: ohci_pci(+) raid6_pq(+) zlib_inflate sungem hwmon sungem_phy i2c_algo_bit sr_mod drm_kms_helper cdrom syscopyarea sysfillrect firewire_ohci sysimgblt ohci_hcd ehci_pci fb_sys_fops ttm firewire_core snd_aoa_i2sbus ehci_hcd crc_itu_t snd_aoa_soundbus snd_pcm snd_timer drm snd soundcore usbcore ssb uninorth_agp drm_panel_orientation_quirks pcmcia usb_common pcmcia_core agpgart
[   19.287432] CPU: 0 PID: 127 Comm: modprobe Tainted: G        W         5.4.0-rc1 #10
[   19.298613] NIP:  f1692560 LR: f16924e8 CTR: c023bcf4
[   19.309692] REGS: e8d13730 TRAP: 0300   Tainted: G        W          (5.4.0-rc1)
[   19.320847] MSR:  02009032 <VEC,EE,ME,IR,DR,RI>  CR: 28228822  XER: 00000000
[   19.332102] DAR: 00f0fd0d DSISR: 40000000 
               GPR00: e8d13988 e8d137e8 ee01b4a0 f16a5070 00000010 00000000 f16924e8 00000000 
               GPR08: f16a5070 00f0fd0d e7171aea e8d13a70 c023bcf4 0050fff4 00000060 00000050 
               GPR16: 00000040 f16a5000 00000012 eb28d000 eb28c000 00000070 00000060 00000050 
               GPR24: 00000040 00000030 00000020 00000010 e8d13a38 eb28d010 eb28c010 00000000 
[   19.388060] NIP [f1692560] raid6_altivec8_gen_syndrome_real+0x3c8/0x488 [raid6_pq]
[   19.399417] LR [f16924e8] raid6_altivec8_gen_syndrome_real+0x350/0x488 [raid6_pq]
[   19.405058] usb 1-1: new high-speed USB device number 2 using ehci-pci
[   19.410774] Call Trace:
[   19.410786] [e8d137e8] [00001032] 0x1032 (unreliable)
[   19.410824] [e8d139d8] [f1692654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
[   19.410857] [e8d13a08] [f15c83c0] init_module+0x3c0/0x518 [raid6_pq]
[   19.460942] Apple USB OHCI 0001:10:18.0 disabled by firmware
[   19.468239] [e8d13af8] [c0005820] do_one_initcall+0xb8/0x36c
[   19.468264] [e8d13bc8] [c0109480] do_init_module+0xa8/0x2c4
[   19.502777] [e8d13bf8] [c010c2e4] load_module+0x2b68/0x2d24
[   19.514192] [e8d13df8] [c010c704] sys_finit_module+0x100/0x140
[   19.525556] [e8d13f38] [c001a274] ret_from_syscall+0x0/0x34
[   19.536910] --- interrupt: c01 at 0x35e2c4
                   LR = 0x4e67c4
[   19.559299] Instruction dump:
[   19.570504] 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 
[   19.581997] 7d11a8ce 115f5b06 116b5800 8141008c <7c0048ce> 392100a0 7d8048ce 392100b0 
[   19.593571] ---[ end trace 5e7216580d039a39 ]---
Comment 1 Erhard F. 2019-10-06 17:49:59 UTC
Created attachment 285369 [details]
kernel .config (5.4-rc1, PowerMac G4 DP)
Comment 2 Erhard F. 2019-10-09 21:19:45 UTC
As a side effect of this bug the btrfs module fails to load. If btrfs is not built as a module, but built into the kernel the machine fails to boot.

Btrfs behaves normal again when I set KASAN_SANITIZE := n in lib/raid6/Makefile.
Comment 3 Erhard F. 2019-12-10 23:51:25 UTC
Created attachment 286249 [details]
kernel .config (5.5-rc1, PowerMac G4 DP)
Comment 4 Erhard F. 2019-12-10 23:52:02 UTC
Created attachment 286251 [details]
dmesg (kernel 5.5-rc1, PowerMac G4 DP)
Comment 5 Erhard F. 2019-12-10 23:54:54 UTC
Unchanged in 5.5-rc1.

[..]
[   19.425679] BUG: Unable to handle kernel data access on read at 0x00f0fd0d
[   19.425693] Faulting instruction address: 0xf165a560
[   19.426731] Oops: Kernel access of bad area, sig: 11 [#1]
[   19.426745] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 DEBUG_PAGEALLOC PowerMac
[   19.426757] Modules linked in: raid6_pq(+) zlib_inflate ehci_pci(+) ohci_hcd hwmon ehci_hcd i2c_algo_bit drm_kms_helper sungem syscopyarea sungem_phy sysfillrect firewire_ohci firewire_core sysimgblt crc_itu_t sr_mod fb_sys_fops usbcore ttm cdrom snd_aoa_i2sbus snd_aoa_soundbus usb_common snd_pcm snd_timer snd soundcore drm ssb pcmcia drm_panel_orientation_quirks pcmcia_core uninorth_agp agpgart
[   19.426986] CPU: 1 PID: 127 Comm: modprobe Tainted: G        W         5.5.0-rc1-PowerMacG4 #3
[   19.426997] NIP:  f165a560 LR: f165a4e8 CTR: c0247f54
[   19.427009] REGS: e86d9740 TRAP: 0300   Tainted: G        W          (5.5.0-rc1-PowerMacG4)
[   19.427014] MSR:  02009032 <VEC,EE,ME,IR,DR,RI>  CR: 22228828  XER: 00000000
[   19.427048] DAR: 00f0fd0d DSISR: 40000000 
               GPR00: e86d9998 e86d97f8 ee26c720 f166d070 00000010 00000000 f165a4e8 00000000 
               GPR08: 00000000 00f0fd0d e7171aea fffffff0 c0247f54 00b25ff4 00000060 00000050 
               GPR16: f166d000 ec077000 ec076000 00000070 00000060 00000050 00000040 00000030 
               GPR24: 00000020 00000010 00000012 e86d9a84 e86d9a48 ec077010 ec076010 00000000 
[   19.427198] NIP [f165a560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq]
[   19.427228] LR [f165a4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
[   19.427234] Call Trace:
[   19.427242] [e86d97f8] [0000000a] 0xa (unreliable)
[   19.427277] [e86d99e8] [f165a654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
[   19.427309] [e86d9a08] [f15d83d8] init_module+0x3d8/0x528 [raid6_pq]
[   19.427334] [e86d9b18] [c0005828] do_one_initcall+0xb8/0x36c
[   19.427355] [e86d9be8] [c010e5e0] do_init_module+0xa8/0x2c4
[   19.427369] [e86d9c18] [c01114c0] load_module+0x2be4/0x2d68
[   19.427383] [e86d9e18] [c01118b8] sys_finit_module+0x100/0x138
[   19.427397] [e86d9f38] [c001a274] ret_from_syscall+0x0/0x34
[   19.427410] --- interrupt: c01 at 0x96ff78
                   LR = 0xafda14
[   19.427416] Instruction dump:
[   19.427429] 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 
[   19.427463] 7d1098ce 115f5b06 116b5800 8141008c <7c0048ce> 392100a0 7d8048ce 392100b0 
[   19.427505] ---[ end trace 161d8d283bc9b7b8 ]---
Comment 6 Christophe Leroy 2019-12-11 19:22:31 UTC
Obviously, r9 is wrong

    2538:       13 04 c4 c4     vxor    v24,v4,v24
    253c:       7d 20 48 ce     lvx     v9,0,r9
    2540:       39 21 00 90     addi    r9,r1,144
    2544:       13 25 cc c4     vxor    v25,v5,v25
    2548:       7d 60 48 ce     lvx     v11,0,r9
    254c:       13 46 d4 c4     vxor    v26,v6,v26
    2550:       81 21 00 88     lwz     r9,136(r1)  <== r9 is loaded here
    2554:       13 67 dc c4     vxor    v27,v7,v27
    2558:       7d 11 a8 ce     lvx     v8,r17,r21
    255c:       11 5f 5b 06     vcmpgtsb v10,v31,v11
    2560:       11 6b 58 00     vaddubm v11,v11,v11
    2564:       81 41 00 8c     lwz     r10,140(r1)
==> 2568:       7c 00 48 ce     lvx     v0,0,r9
    256c:       39 21 00 a0     addi    r9,r1,160
    2570:       7d 80 48 ce     lvx     v12,0,r9
    2574:       39 21 00 b0     addi    r9,r1,176

So the stack must be clobbered somewhere
Comment 7 Erhard F. 2019-12-14 13:49:05 UTC
I enabled DEBUG_STACKOVERFLOW in the kernel .config but I got no additional hints...

Is there any other debugging option that could help?
Comment 8 Erhard F. 2019-12-15 16:01:27 UTC
Found out this is probably a cause for bug #205283. For doing some btrfs tests with 5.5-rc1 kernel I needed to disable KASAN in lib/raid6/Makefile (KASAN_SANITIZE := n) as btrfs pulls in raid6_pq.

btrfs gets modprobed seemingly ok the 1st time, but removing and reloading it provokes bug #205283.

# modprobe -v btrfs
# modprobe -r -v btrfs
rmmod btrfs
rmmod zlib_inflate
rmmod libcrc32c
rmmod raid6_pq
rmmod zlib_deflate
rmmod lzo_decompress
rmmod lzo_compress
rmmod zstd_compress
rmmod zstd_decompress
rmmod xor
# modprobe -v btrfs
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_inflate/zlib_inflate.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/libcrc32c.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/raid6/raid6_pq.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zlib_deflate/zlib_deflate.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_decompress.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/lzo/lzo_compress.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_compress.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/lib/zstd/zstd_decompress.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/crypto/xor.ko 
insmod /lib/modules/5.5.0-rc1-PowerMacG4+/kernel/fs/btrfs/btrfs.ko
Comment 9 Erhard F. 2019-12-21 17:42:41 UTC
Created attachment 286395 [details]
dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP)

(In reply to Christophe Leroy from comment #10, bug #205283)
> So it must be something more. By the way, now that KASAN_VMALLOC exists and
> handle module loading better, I think we should analyse the impact of
> activating KASAN_VMALLOC at all time.
> 
> Regarding #205099, I believe there's a real stack related bug in raid6_pq
> that needs to be identified. Did you try with INLINE KASAN ?
With INLINE KASAN and KASAN_VMALLOC this hit does not show up at all.

It does show up with OUTLINE KASAN and KASAN_VMALLOC.
Comment 10 Erhard F. 2019-12-21 17:43:13 UTC
Created attachment 286397 [details]
dmesg (kernel 5.5-rc2+, KASAN_VMALLOC + OUTLINE KASAN, PowerMac G4 DP)
Comment 11 Erhard F. 2020-01-17 21:16:45 UTC
Applied your patch series on top of 5.5-rc6. CONFIG_KASAN_VMALLOC is not non-selectable but forced on by default.

Current situation is that the hit does not show up with INLINE KASAN in raid6_pq.
Comment 12 Erhard F. 2020-01-17 21:19:35 UTC
Created attachment 286869 [details]
dmesg (kernel 5.5-rc6+, INLINE KASAN, PowerMac G4 DP)

However I might have gotten a hint where the stack is clobbered. The IRQ stack overflows sooner or later (CONFIG_DEBUG_SHIRQ=y might have brought that up?), which causes all sorts of problems.

I'll attach the dmesg here but I can open a new bug if this is more suitable.
Comment 13 Erhard F. 2020-01-17 21:22:29 UTC
Created attachment 286871 [details]
kernel .config (5.5-rc6, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP))
Comment 14 Erhard F. 2020-01-17 21:25:47 UTC
Created attachment 286873 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + INLINE KASAN, PowerMac G4 DP) w. IRQ stack overflow
Comment 15 Christophe Leroy 2020-01-18 08:47:32 UTC
Interesting that stack overflows. Maybe we should increase THREAD_SHIFT by 1 when KASAN is selected. ARM64 does that.

Would be interesting to try and see if the VMAP_STACK series detects the stack overflow.
Series at https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973
Comment 16 Erhard F. 2020-01-18 20:06:34 UTC
(In reply to Christophe Leroy from comment #15)
> Would be interesting to try and see if the VMAP_STACK series detects the
> stack overflow.
> Series at
> https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=149973
I just tried that, but in this case the G4 won't boot at all. The kernel loads but booting gets stuck at a white display with only 2 lines visible where it tells me it will open the display of the G4s Radeons offb display.

I applied your CONFIG_KASAN_VMALLOC on top of 5.5-rc6 and applied your VMAP_STACK series afterwards.

 # grep -i kasan .config
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_KASAN=y
CONFIG_KASAN_GENERIC=y
# CONFIG_KASAN_OUTLINE is not set
CONFIG_KASAN_INLINE=y
CONFIG_KASAN_STACK=1
CONFIG_KASAN_VMALLOC=y
# CONFIG_TEST_KASAN is not set
CONFIG_KASAN_SHADOW_OFFSET=0xe0000000

 # grep -i vmap .config
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
Comment 17 Christophe Leroy 2020-01-20 12:40:39 UTC
Created attachment 286907 [details]
Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK

Please try the attached patch, it fixes the setup of the kasan early hash table  when VMAP_STACK is enabled.
Comment 18 Erhard F. 2020-01-20 22:56:23 UTC
(In reply to Christophe Leroy from comment #17)
> Created attachment 286907 [details]
> Patch to fix kasan with KASAN_VMALLOC and VMAP_STACK
> 
> Please try the attached patch, it fixes the setup of the kasan early hash
> table  when VMAP_STACK is enabled.
Sorry, but still the same situation with this patch applied.
Comment 19 Christophe Leroy 2020-01-21 05:52:36 UTC
Can you tell exactly where it stops during the boot ? Or take a photo of the screen ?

In parallele, could you try (without VMAP_STACK) increasing CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks.
Comment 20 Erhard F. 2020-01-21 12:20:42 UTC
Created attachment 286925 [details]
kernel .config (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Comment 21 Erhard F. 2020-01-21 12:24:12 UTC
Created attachment 286927 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP)
Comment 22 Erhard F. 2020-01-21 13:00:12 UTC
(In reply to Christophe Leroy from comment #19)
> Can you tell exactly where it stops during the boot ? Or take a photo of the
> screen ?
I'll attach a photo shortly.
 
> In parallele, could you try (without VMAP_STACK) increasing
> CONFIG_THREAD_SHIFT to 14 ? It will double the size of the stacks.
I can confirm that with CONFIG_THREAD_SHIFT=14 the G4 runs good again with INLINE KASAN + KASAN_VMALLOC enabled (as seen in attached dmesg). No KASAN hit at raid6_pq, no poblem with btrfs.

Though with CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN + KASAN_VMALLOC I still get this KASAN hit at raid6_pq.
Comment 23 Erhard F. 2020-01-21 13:00:54 UTC
Created attachment 286929 [details]
dmesg (kernel 5.5-rc6+, KASAN_VMALLOC + CONFIG_THREAD_SHIFT=14 + OUTLINE KASAN, PowerMac G4 DP)
Comment 24 Erhard F. 2020-01-21 13:31:39 UTC
Created attachment 286931 [details]
screenshot (5.5-rc6+, KASAN_VMALLOC + VMAP_STACK + INLINE KASAN, PowerMac G4 DP)
Comment 25 Erhard F. 2020-02-26 15:49:50 UTC
Created attachment 287621 [details]
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP)

Gave 5.6-rc3 + KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN a test ride and it's still in:

[...]
Feb 26 16:29:30 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0
Feb 26 16:29:30 T600 kernel: Faulting instruction address: 0xf168b560
Feb 26 16:29:30 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Feb 26 16:29:30 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Feb 26 16:29:30 T600 kernel: Modules linked in: raid6_pq(+) libcrc32c zlib_inflate snd_aoa_i2sbus snd_aoa_soundbus snd_pcm ohci_hcd ehci_pci(+) ssb snd_timer ehci_hcd pcmcia snd pcmcia_core soundcore uninorth_agp usbcore agpgart usb_common
Feb 26 16:29:30 T600 kernel: CPU: 1 PID: 118 Comm: modprobe Tainted: G        W         5.6.0-rc3-PowerMacG4+ #1
Feb 26 16:29:30 T600 kernel: NIP:  f168b560 LR: f168b4e8 CTR: c024b2e0
Feb 26 16:29:30 T600 kernel: REGS: f10bd780 TRAP: 0300   Tainted: G        W          (5.6.0-rc3-PowerMacG4+)
Feb 26 16:29:30 T600 kernel: MSR:  02009032 <VEC,EE,ME,IR,DR,RI>  CR: 28228828  XER: 00000000
Feb 26 16:29:30 T600 kernel: DAR: 0050a0f0 DSISR: 40000000 
                             GPR00: f10bd9d8 f10bd838 ebeca380 ec56c070 00000010 00000000 f168b4e8 00000000 
                             GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c024b2e0 007efff4 00000060 00000050 
                             GPR16: ec56c000 ec56f000 ec56e000 00000070 00000060 00000050 00000040 00000030 
                             GPR24: 00000020 00000010 00000008 f10bda8c f10bda78 ec56f010 ec56e010 00000000 
Feb 26 16:29:30 T600 kernel: NIP [f168b560] raid6_altivec8_gen_syndrome_real+0x3c0/0x480 [raid6_pq]
Feb 26 16:29:30 T600 kernel: LR [f168b4e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
Feb 26 16:29:30 T600 kernel: Call Trace:
Feb 26 16:29:30 T600 kernel: [f10bd838] [0000000a] 0xa (unreliable)
Feb 26 16:29:30 T600 kernel: [f10bda28] [f168b654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
Feb 26 16:29:30 T600 kernel: [f10bda48] [f145b3cc] init_module+0x3cc/0x530 [raid6_pq]
Feb 26 16:29:30 T600 kernel: [f10bdb18] [c00058e4] do_one_initcall+0xb8/0x36c
Feb 26 16:29:30 T600 kernel: [f10bdbe8] [c010f760] do_init_module+0xa8/0x2c4
Feb 26 16:29:30 T600 kernel: [f10bdc18] [c0112630] load_module+0x2bd4/0x2d58
Feb 26 16:29:30 T600 kernel: [f10bde18] [c0112a28] sys_finit_module+0x100/0x138
Feb 26 16:29:30 T600 kernel: [f10bdf38] [c001a234] ret_from_syscall+0x0/0x34
Feb 26 16:29:30 T600 kernel: --- interrupt: c01 at 0x639988
Comment 26 Erhard F. 2020-02-26 15:52:42 UTC
Created attachment 287623 [details]
kernel .config (5.6-rc3, KASAN_VMALLOC + VMAP_STACK + OUTLINE KASAN, PowerMac G4 DP)
Comment 27 Erhard F. 2020-02-26 16:17:03 UTC
Created attachment 287625 [details]
dmesg (5.6-rc3+, KASAN_VMALLOC + VMAP_STACK + CONFIG_THREAD_SHIFT=14 + INLINE KASAN, PowerMac G4 DP)

And still the same phenomenon that I only get this hit with OUTLINE KASAN but not with INLINE KASAN (+ CONFIG_THREAD_SHIFT=14).
Comment 28 Erhard F. 2020-04-13 14:12:10 UTC
Created attachment 288411 [details]
dmesg (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP)

Re-tested with v5.7-rc1 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal.

[...]
Apr 13 16:01:11 T600 kernel: BUG: Unable to handle kernel data access on read at 0x0050a0f0
Apr 13 16:01:11 T600 kernel: Faulting instruction address: 0xf1aa1564
Apr 13 16:01:11 T600 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 13 16:01:11 T600 kernel: BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Apr 13 16:01:11 T600 kernel: Modules linked in: raid6_pq(+) zlib_inflate firewire_ohci firewire_core ehci_pci(+) ohci_hcd crc_itu_t sr_mod cdrom ehci_hcd snd_aoa_i2sbus snd_aoa_soundbus snd_pcm snd_timer snd ss>
Apr 13 16:01:11 T600 kernel: CPU: 0 PID: 149 Comm: modprobe Tainted: G        W         5.7.0-rc1-PowerMacG4+ #1
Apr 13 16:01:11 T600 kernel: NIP:  f1aa1564 LR: f1aa14e8 CTR: c0255b78
Apr 13 16:01:11 T600 kernel: REGS: f1963780 TRAP: 0300   Tainted: G        W          (5.7.0-rc1-PowerMacG4+)
Apr 13 16:01:11 T600 kernel: MSR:  02009032 <VEC,EE,ME,IR,DR,RI>  CR: 28228828  XER: 00000000
Apr 13 16:01:11 T600 kernel: DAR: 0050a0f0 DSISR: 40000000 
                             GPR00: f19639d8 f1963838 ebd1de60 eb1dc070 00001000 00000003 f1aa14e8 00000000 
                             GPR08: 00000000 0050a0f0 5d0dfdad fffffff0 c0255b78 00a48ff4 00000060 00000050 
                             GPR16: eb1dc000 eb1df000 eb1de000 00000070 00000060 00000050 00000040 00000030 
                             GPR24: 00000020 00000010 00000008 f1963a8c f1963a78 eb1df010 eb1de010 00000000 
Apr 13 16:01:11 T600 kernel: NIP [f1aa1564] raid6_altivec8_gen_syndrome_real+0x3c4/0x480 [raid6_pq]
Apr 13 16:01:11 T600 kernel: LR [f1aa14e8] raid6_altivec8_gen_syndrome_real+0x348/0x480 [raid6_pq]
Apr 13 16:01:11 T600 kernel: Call Trace:
Apr 13 16:01:11 T600 kernel: [f1963838] [0000000a] 0xa (unreliable)
Apr 13 16:01:11 T600 kernel: [f1963a28] [f1aa1654] raid6_altivec8_gen_syndrome+0x34/0x58 [raid6_pq]
Apr 13 16:01:11 T600 kernel: [f1963a48] [f12603cc] init_module+0x3cc/0x530 [raid6_pq]
Apr 13 16:01:11 T600 kernel: [f1963b18] [c00058e4] do_one_initcall+0xb8/0x36c
Apr 13 16:01:11 T600 kernel: [f1963be8] [c0117f64] do_init_module+0xa8/0x2c4
Apr 13 16:01:11 T600 kernel: [f1963c18] [c011ae38] load_module+0x2bd8/0x2d5c
Apr 13 16:01:11 T600 kernel: [f1963e18] [c011b230] sys_finit_module+0x100/0x138
Apr 13 16:01:11 T600 kernel: [f1963f38] [c001a224] ret_from_syscall+0x0/0x34
Apr 13 16:01:11 T600 kernel: --- interrupt: c01 at 0x892988
                                 LR = 0xa20a14
Apr 13 16:01:11 T600 kernel: Instruction dump:
Apr 13 16:01:11 T600 kernel: 1304c4c4 7d2048ce 39210090 1325ccc4 7d6048ce 1346d4c4 81210088 1367dcc4 
Apr 13 16:01:11 T600 kernel: 8141008c 11284cc4 115f5b06 116b5800 <7c0048ce> 392100a0 7d8048ce 392100b0 
Apr 13 16:01:11 T600 kernel: ---[ end trace ebe3b589d509d4f6 ]---
[...]
Comment 29 Erhard F. 2020-04-13 14:15:28 UTC
Created attachment 288413 [details]
kernel .config (5.7-rc1, OUTLINE KASAN, PowerMac G4 DP)
Comment 30 Erhard F. 2020-09-04 22:50:12 UTC
Created attachment 292345 [details]
dmesg (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP)

Re-tested with v5.9-rc3 out of curiosity. Not much change here, the bug shows up with OUTLINE KASAN but not with INLINE KASAN, everything else being equal:

==================================================================
BUG: KASAN: user-memory-access in raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq]
Read of size 4 at addr 5764b118 by task modprobe/126

CPU: 1 PID: 126 Comm: modprobe Tainted: G        W         5.9.0-rc3-PowerMacG4 #2
Call Trace:
[e32cb7b8] [c0517aac] dump_stack+0xc4/0xf8 (unreliable)
[e32cb7e8] [c026e73c] kasan_report+0x16c/0x170
[e32cb828] [b02004e0] raid6_altivec8_gen_syndrome_real+0x2b0/0x480 [raid6_pq]
[e32cba18] [b02006fc] raid6_altivec8_gen_syndrome+0x4c/0x88 [raid6_pq]
[e32cba38] [b021a42c] init_module+0x42c/0x590 [raid6_pq]
[e32cbb08] [c00058a0] do_one_initcall+0xb8/0x3dc
[e32cbbd8] [c011c0fc] do_init_module+0xa8/0x2c4
[e32cbc08] [c011f02c] load_module+0x2b98/0x2d4c
[e32cbe18] [c011f448] sys_finit_module+0x100/0x138
[e32cbf38] [c001a1cc] ret_from_syscall+0x0/0x34
--- interrupt: c01 at 0x3d2068
    LR = 0x506104
==================================================================
BUG: Unable to handle kernel data access on read at 0x5764b118
Faulting instruction address: 0xb02004e0
Oops: Kernel access of bad area, sig: 11 [#1]
Comment 31 Erhard F. 2020-09-04 22:54:13 UTC
Created attachment 292347 [details]
kernel .config (5.9-rc3, OUTLINE KASAN, PowerMac G4 DP)

Does happen even if RAID support is not actively selected in the config as btrfs pulls in RAID6_PQ on its own.

# CONFIG_DM_RAID is not set
CONFIG_RAID6_PQ=m
Comment 32 Christophe Leroy 2021-06-19 06:27:42 UTC
Is this problem still there with 5.13 ?
Comment 33 Erhard F. 2021-06-19 08:25:51 UTC
I can tell you in the next few weeks, as soon as my G4 MDD gets its' overhaul with the serviced PSU.
Comment 34 Erhard F. 2021-11-26 01:47:31 UTC
Created attachment 299711 [details]
dmesg (5.15.5, OUTLINE KASAN, PowerMac G4 DP)

Finally my G4 DP got its' fixed & overhauled PSU so I am able to continue here.

Tested kernel 5.15.5 and the original KASAN hit at raid6_pq did not show up. Instead I am getting this now:

[...]
Running code patching self-tests ...
vmap allocation for size 33562624 failed: use vmalloc=<size> to increase size
swapper/0: vmalloc error: size 33558528, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null)
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W         5.15.5-gentoo-PowerMacG4 #5
Call Trace:
[f1033bb0] [c0d933c0] dump_stack_lvl+0x80/0xb0 (unreliable)
[f1033bd0] [c0516128] warn_alloc+0x11c/0x2b4
[f1033cb0] [c0508c40] __vmalloc_node_range+0xd8/0x64c
[f1033d70] [c0508a58] __vmalloc_node+0xec/0xf4
[f1033db0] [c1c0ecb0] test_code_patching+0x72c/0xd50
[f1033df0] [c0008908] do_one_initcall+0x284/0x574
[f1033ec0] [c1c03f78] kernel_init_freeable+0x510/0x51c
[f1033f10] [c000934c] kernel_init+0x24/0x140
[f1033f30] [c0033148] ret_from_kernel_thread+0x14/0x1c
Mem-Info:
active_anon:0 inactive_anon:0 isolated_anon:0
 active_file:0 inactive_file:0 isolated_file:0
 unevictable:0 dirty:0 writeback:0
 slab_reclaimable:1306 slab_unreclaimable:4214
 mapped:0 shmem:0 pagetables:0 bounce:0
 kernel_misc_reclaimable:0
 free:474459 free_pcp:592 free_cma:0
Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:840kB pagetables:0kB all_unreclaimable? no
DMA free:588612kB min:3144kB low:3928kB high:4712kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:618264kB mlocked:0kB bounce:0kB free_pcp:1964kB local_pcp:1568kB free_cma:0kB
lowmem_reserve[]: 0 0 1280 1280
HighMem free:1309224kB min:512kB low:2176kB high:3840kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:404kB local_pcp:116kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
DMA: 5*4kB (ME) 2*8kB (UM) 4*16kB (M) 5*32kB (ME) 1*64kB (M) 4*128kB (UME) 4*256kB (ME) 4*512kB (UM) 3*1024kB (M) 4*2048kB (ME) 140*4096kB (M) = 588612kB
HighMem: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 2*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (U) 319*4096kB (M) = 1309224kB
0 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
524288 pages RAM
327680 pages HighMem/MovableOnly
42042 pages reserved
code-patching: test failed at line 598
[...]

This makes me wonder whether KASAN is working correctly when not enough vmap size is available. I did set it to vmalloc=64M via Kernel command line but this does not seem to be reckognized. Same situation with INLINE KASAN. In both cases with VMAP_STACK and CONFIG_THREAD_SHIFT=14.
Comment 35 Erhard F. 2021-11-26 01:48:08 UTC
Created attachment 299713 [details]
dmesg (5.15.5, INLINE KASAN, PowerMac G4 DP)
Comment 36 Erhard F. 2021-11-26 01:49:09 UTC
Created attachment 299715 [details]
kernel .config (5.15.5, PowerMac G4 DP)
Comment 37 Christophe Leroy 2021-11-26 09:03:18 UTC
I see no obvious reason for a 32Mb allocation to fail while you have 588612kB free memory.

And that happens early at boot, before user processes are started so the vmap area, allthough not very big, should still have 32M space available.
Comment 38 Christophe Leroy 2021-11-26 09:14:06 UTC
Looks like only x86 are arm implement this vmalloc= parameter:

[chleroy@PO20335 linux-powerpc]$ git grep 'early_param("vmalloc"' arch/
arch/arm/mm/mmu.c:early_param("vmalloc", early_vmalloc);
arch/x86/mm/pgtable_32.c:early_param("vmalloc", parse_vmalloc);

However, your vmalloc area has a size of 65M:

Kernel virtual memory layout:
  * 0xf6000000..0xfec00000  : kasan shadow mem
  * 0xf5bbf000..0xf5fff000  : fixmap
  * 0xf5400000..0xf5800000  : highmem PTEs
  * 0xf5115000..0xf5400000  : early ioremap
  * 0xf1000000..0xf5110000  : vmalloc & ioremap
  * 0xb0000000..0xc0000000  : modules
Memory: 1928984K/2097152K available (22288K kernel code, 2616K rwdata, 4868K rodata, 1408K init, 8981K bss, 168168K reserved, 0K cma-reserved, 1310720K highmem)
Comment 39 Christophe Leroy 2021-11-26 09:20:49 UTC
Can you retry with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000 ?
Comment 40 Christophe Leroy 2021-11-26 11:33:37 UTC
Would also be great if you can activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables
Comment 41 Erhard F. 2021-11-29 00:32:10 UTC
Created attachment 299759 [details]
dmesg (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x28000000, PowerMac G4 DP)

Ok, finally getting somewhere.. With CONFIG_LOWMEM_SIZE=0x28000000 I don't get this "vmap allocation for size 33562624 failed" error and also OUTLINE KASAN does not show the originally reported "BUG: Unable to handle kernel data access at 0x00f0fd0d" for raid6_pq!

For INLINE KASAN I still get "vmap allocation for size 33562624 failed", even with CONFIG_LOWMEM_SIZE=0x28000000 or CONFIG_LOWMEM_SIZE=0x20000000.
Comment 42 Erhard F. 2021-11-29 00:32:59 UTC
Created attachment 299761 [details]
dmesg (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Comment 43 Erhard F. 2021-11-29 00:34:24 UTC
Created attachment 299763 [details]
kernel .config (5.15.5, PowerMac G4 DP)
Comment 44 Christophe Leroy 2021-11-29 06:51:31 UTC
Interesting.

I wonder why it works with OUTLINE KASAN but not with INLINE KASAN.

Can you activate CONFIG_PTDUMP_DEBUGFS and provide the content of /sys/kernel/debug/kernel_page_tables for the cases than don't work (OUTILINE KASAN + 0x30000000 LOWMEM_SIZE, INLINE KASAN + 0x20000000 LOWMEM SIZE).
Comment 45 Erhard F. 2021-11-29 19:46:19 UTC
Created attachment 299773 [details]
kernel_page_tables (5.15.5, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)

Ah yes, I forgot about including the /sys/kernel/debug/kernel_page_tables.. Sorry! Here you are.
Comment 46 Erhard F. 2021-11-29 19:46:56 UTC
Created attachment 299775 [details]
kernel_page_tables (5.15.5, INLINE KASAN, LOWMEM_SIZE=0x20000000, PowerMac G4 DP)
Comment 47 Erhard F. 2021-11-30 13:58:03 UTC
Created attachment 299793 [details]
kernel_page_tables (5.15.5 + patch, OUTLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
Comment 48 Erhard F. 2021-11-30 13:58:38 UTC
Created attachment 299795 [details]
kernel_page_tables (5.15.5 + patch, INLINE KASAN, LOWMEM_SIZE=0x30000000, PowerMac G4 DP)
Comment 49 Erhard F. 2021-11-30 15:39:41 UTC
As the other KASAN inconsistencies mentioned here got ironed out by Christophe (patches available but not in stable yet) and this "KASAN hit at raid6_pq: BUG: Unable to handle kernel data access at 0x00f0fd0d" does no longer show up with OUTLINE/INLINE KASAN on 5.15.5 or 5.16-rc3 I think it is safe to close this bug as obsolete.

Note You need to log in before you can comment on or make changes to this bug.