Created attachment 300115 [details]
kernel .config (5.15.10, PowerMac G4 DP)

Probably hard to track.

Any chance to bisect the issue ?

Bisecting will take some time. I'll report back as soon as I have any findings.

I was able to easily reproduce this on 5.15.13, however not on 5.16-rc8.

But on 5.16-rc8 I got this the 3rd time I ran the glibc testsuite:

[...]
watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/u4:7:32566]
Modules linked in: auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc ghash_generic gf128mul gcm ccm algif_aead des_generic libdes ctr cbc ecb algif_skcipher aes_generic libaes cmac sha512_generic sha1_generic sha1_powerpc md5 md5_ppc md4 b43legacy mac80211 libarc4 snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill evdev mac_hid therm_windtunnel firewire_ohci firewire_core crc_itu_t sr_mod cdrom snd_aoa_i2sbus snd_aoa_soundbus snd_pcm snd_timer snd ohci_pci soundcore radeon ohci_hcd ehci_pci ehci_hcd hwmon i2c_algo_bit drm_ttm_helper ttm ssb drm_kms_helper pcmcia pcmcia_core usbcore 8250_pci syscopyarea sysfillrect sysimgblt usb_common 8250 8250_base serial_mctrl_gpio fb_sys_fops pkcs8_key_parser fuse drm drm_panel_orientation_quirks configfs
CPU: 1 PID: 32566 Comm: kworker/u4:7 Not tainted 5.16.0-rc8-PowerMacG4 #1
Workqueue: zswap1 compact_page_work
NIP:  c0078730 LR: c0078724 CTR: 00000000
REGS: f698dd40 TRAP: 0900   Not tainted  (5.16.0-rc8-PowerMacG4)
MSR:  00009032 <EE,ME,IR,DR,RI>  CR: 44008242  XER: 20000000

GPR00: c01856c8 f698de00 ca20b540 00000001 d4c73ffc 00000000 de0bd0bc aaaaaaaa 
GPR08: aaaaaaaa 00000000 ffffffff 00000004 84002242 00000000 c00553fc 00000001 
GPR16: 00000002 d4c73fc0 c0980000 002ec02c 00000040 d4c7300c d4c7302e c19c4bc0 
GPR24: c19c4bc0 c0185d74 ef0d0040 d4c73008 d4c74a4c 0000007f de0bd000 d4c74a54 
NIP [c0078730] arch_write_lock+0x28/0x3c
LR [c0078724] arch_write_lock+0x1c/0x3c
Call Trace:
[f698de00] [c0185d74] release_z3fold_page_locked+0x0/0x44 (unreliable)
[f698de20] [c01856c8] do_compact_page+0x334/0x508
[f698de80] [c004f354] process_one_work+0x1d4/0x288
[f698dec0] [c004f814] worker_thread+0x1b8/0x260
[f698df00] [c0055514] kthread+0x118/0x11c
[f698df30] [c0016268] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
39610020 4bfa7668 9421ffe0 7c0802a6 90010024 93e1001c 7c7f1b78 7fe3fb78 
4bffff0d 2c030000 41a20014 813f0000 <2c090000> 4182ffe8 4bfffff4 39610020 
Kernel panic - not syncing: softlockup: hung tasks
CPU: 1 PID: 32566 Comm: kworker/u4:7 Tainted: G             L    5.16.0-rc8-PowerMacG4 #1
Workqueue: zswap1 compact_page_work
Call Trace:
[f698dbb0] [c03e7f04] dump_stack_lvl+0x60/0x80 (unreliable)
[f698dbd0] [c0037734] panic+0x128/0x30c
[f698dc30] [c00c6334] watchdog_nmi_enable+0x0/0x10
[f698dc70] [c0097fc8] __hrtimer_run_queues+0xf0/0x154
[f698dcb0] [c0098b7c] hrtimer_interrupt+0xf8/0x25c
[f698dcf0] [c000d70c] timer_interrupt+0x20c/0x294
[f698dd30] [c0004a50] Decrementer_virt+0x100/0x104
--- interrupt: 900 at arch_write_lock+0x28/0x3c
NIP:  c0078730 LR: c0078724 CTR: 00000000
REGS: f698dd40 TRAP: 0900   Tainted: G             L     (5.16.0-rc8-PowerMacG4)
MSR:  00009032 <EE,ME,IR,DR,RI>  CR: 44008242  XER: 20000000

GPR00: c01856c8 f698de00 ca20b540 00000001 d4c73ffc 00000000 de0bd0bc aaaaaaaa 
GPR08: aaaaaaaa 00000000 ffffffff 00000004 84002242 00000000 c00553fc 00000001 
GPR16: 00000002 d4c73fc0 c0980000 002ec02c 00000040 d4c7300c d4c7302e c19c4bc0 
GPR24: c19c4bc0 c0185d74 ef0d0040 d4c73008 d4c74a4c 0000007f de0bd000 d4c74a54 
NIP [c0078730] arch_write_lock+0x28/0x3c
LR [c0078724] arch_write_lock+0x1c/0x3c
--- interrupt: 900
[f698de00] [c0185d74] release_z3fold_page_locked+0x0/0x44 (unreliable)
[f698de20] [c01856c8] do_compact_page+0x334/0x508
[f698de80] [c004f354] process_one_work+0x1d4/0x288
[f698dec0] [c004f814] worker_thread+0x1b8/0x260
[f698df00] [c0055514] kthread+0x118/0x11c
[f698df30] [c0016268] ret_from_kernel_thread+0x5c/0x64
Rebooting in 40 seconds..


Which is interesting because on bug #213837 my not yet finished bisect is also giving hints z3fold may be the problem...

I'll check out next whether the issue is reproduceable on 5.15.x when I use zbud or zmalloc for zswap instead of z3fold.

Ok, with zswap lzo/zbud I also get this memory corruption on 5.15.13. So most probably it's not lzo/z3pool but something else. I'll start a bisect then...

Created attachment 300318 [details]
bisect.log

Ok, finally got it. Interesting find:

 # git bisect bad
db972a3787d12b1ce9ba7a31ec376d8a79e04c47 is the first bad commit
commit db972a3787d12b1ce9ba7a31ec376d8a79e04c47
Author: Christophe Leroy <christophe.leroy@csgroup.eu>
Date:   Tue Dec 8 05:24:19 2020 +0000

    powerpc/powermac: Fix low_sleep_handler with CONFIG_VMAP_STACK
    
    low_sleep_handler() can't restore the context from standard
    stack because the stack can hardly be accessed with MMU OFF.
    
    Store everything in a global storage area instead of storing
    a pointer to the stack in that global storage area.
    
    To avoid a complete churn of the function, still use r1 as
    the pointer to the storage area during restore.
    
    Fixes: cd08f109e262 ("powerpc/32s: Enable CONFIG_VMAP_STACK")
    Reported-by: Giuseppe Sacco <giuseppe@sguazz.it>
    Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
    Tested-by: Giuseppe Sacco <giuseppe@sguazz.it>
    Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
    Link: https://lore.kernel.org/r/e3e0d8042a3ba75cb4a9546c19c408b5b5b28994.1607404931.git.christophe.leroy@csgroup.eu

 arch/powerpc/platforms/Kconfig.cputype  |   2 +-
 arch/powerpc/platforms/powermac/sleep.S | 132 ++++++++++++++------------------
 2 files changed, 60 insertions(+), 74 deletions(-)

Interesting ... Though confusing.

Looking closer, in fact that might be a false positive.

The huge difference with that bad commit is that:
- Before the commit, the kernel is built _without_ CONFIG_VMAP_STACK
- After the commit, the kernel is built _with_ CONFIG_VMAP_STACK

Would you be able to perform following tests:
- Disable VMAP_STACK and see if the problem still occurs.
- Disable ADB_PMU and see it the problem still occurs.

With the version which preceeds the bad commit, can you disable ADB_PMU and enable VMAP_STACK and see what happens ?

Created attachment 300354 [details]
dmesg (5.10-rc2 with ADB_PMU disabled, PowerMac G4 DP)

Took a little time but I double checked the results (one time using distcc '-j8 -l2', one time native '-j3') to be sure:

ADB_PMU disabled, VMAP_STACK disabled  ...  "neverending build"
ADB_PMU enabled,  VMAP_STACK disabled  ...  works ok
ADB_PMU disabled, VMAP_STACK enabled   ...  "neverending build"
ADB_PMU enabled,  VMAP_STACK enabled   ...  memory corruption

Version used was git db972a3787d12b1ce9ba7a31ec376d8a79e04c47, which is the one before a last 'git bisect bad' ends the git bisect.

The "neverending builds" happen when I run this kernel with ADB_PMU disabled. The G4 runs for several hours building (?) without reaching the glibc test stage. With ADB_PMU enabled I get a pass or memory corruption much earlier.

Also without ADB_PMU I get a kernel panic when rebooting or shutting down the G4. Also the G4 does not reboot/poweroff in this case, I need to switch it off manually.

Thanks for the tests.

I'm not surprised that the system doesn't poweroff or reboot without ADB_PMU because the PMU manages power.

The "neverending build" is maybe because the PMU also manages RTC clock and without it you get inconsistent time ?

Anyway, it looks like there is indeed something linked to VMAP_STACK.

I'm wondering whether you could be running out of vmalloc space. I initially thought you were using KASAN, but it seems not according to your .config.

Could you try reducing CONFIG_LOWMEM_SIZE to 0x28000000 for instance and see if the memory corruption still happens ?

To do this you'll need CONFIG_ADVANCED_OPTIONS and CONFIG_LOWMEM_SIZE_BOOL.

(In reply to Christophe Leroy from comment #10)
> I'm wondering whether you could be running out of vmalloc space. I initially
> thought you were using KASAN, but it seems not according to your .config.
Correct, I was not using KASAN. I use it only for testing -rc kernels or when I am particularly wary. This memory corruption I noticed during regular usage. Seems running the kernel with slub_debug=FZP page_poison=1 is a good thing. ;)

> Could you try reducing CONFIG_LOWMEM_SIZE to 0x28000000 for instance and see
> if the memory corruption still happens ?
Thanks, that did the trick! With CONFIG_LOWMEM_SIZE=0x28000000 the memory corruption is gone on VMAP_STACK enabled kernels. Tested it additionally on current 5.16.4 where this works too.

Created attachment 300774 [details]
dmesg (5.18-rc3, PowerMac G4 DP)

Another try with running glibc-2.34 testsuite on kernel 5.18-rc3. Looks like it's still a problem.

[...]
pagealloc: memory corruption
fffdfff0: 00 00 00 00                                      ....
CPU: 0 PID: 21222 Comm: install Not tainted 5.18.0-rc3-PMacG4 #5
Call Trace:
[f8085a70] [c06e8820] dump_stack_lvl+0x80/0xc0 (unreliable)
[f8085a90] [c02c0b2c] __kernel_unpoison_pages+0x1c0/0x204
[f8085ae0] [c02a4cb0] get_page_from_freelist+0xcb4/0xeb0
[f8085ba0] [c02a5754] __alloc_pages+0x184/0x11b4
[f8085c70] [c0230d50] __filemap_get_folio+0x224/0x598
[f8085cf0] [c0240ebc] pagecache_get_page+0x20/0x88
[f8085d10] [c04e2600] prepare_pages+0xf8/0x358
[f8085d60] [c04e4e54] btrfs_buffered_write+0x334/0x850
[f8085e20] [c04ea598] btrfs_do_write_iter+0x3a8/0x768
[f8085e80] [c02ee25c] vfs_write+0x364/0x488
[f8085f00] [c02ee52c] ksys_write+0x78/0x128
[f8085f30] [c001e1a8] ret_from_syscall+0x0/0x2c
--- interrupt: c00 at 0x5c5d08
NIP:  005c5d08 LR: 005c5ce0 CTR: c0289c9c
REGS: f8085f40 TRAP: 0c00   Not tainted  (5.18.0-rc3-PMacG4)
MSR:  0000f932 <EE,PR,FP,ME,IR,DR,RI>  CR: 28022464  XER: 20000000

GPR00: 00000004 af820720 a7ced760 00000006 a77aa000 00020000 00000000 00000000 
GPR08: 00120000 a77a9000 00000008 403d77ca 403d7497 0077fff4 00000000 00020000 
GPR16: 00000000 af8208c8 00020000 00000000 00000000 af821d2b a77aa000 00020000 
GPR24: 00000000 00000006 7ff00000 00000006 a77aa000 00020000 006c7ff4 00020000 
NIP [005c5d08] 0x5c5d08
LR [005c5ce0] 0x5c5ce0
--- interrupt: c00
page:ef4c4ec4 refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x31069
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page detail

Created attachment 300775 [details]
kernel .config (5.18-rc3, PowerMac G4 DP)

Do you mean it still happens with the default values, or it also happens with the reduced CONFIG_LOWMEM_SIZE ?

It definitively still happens with the default values. Can test with the reduced CONFIG_LOWMEM_SIZE next week and report back.

Created attachment 300929 [details]
dmesg (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, PowerMac G4 DP)

(In reply to Christophe Leroy from comment #14)
> Do you mean it still happens with the default values, or it also happens
> with the reduced CONFIG_LOWMEM_SIZE ?
Turns out the memory corruption also happens with the reduced CONFIG_LOWMEM_SIZE=0x28000000.

Tested again on v5.18-rc6, both with CONFIG_LOWMEM_SIZE=0x28000000 and without.

Created attachment 300930 [details]
kernel .config (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, PowerMac G4 DP)

Ok, and another problem during building via distcc on the G4, still LOWMEM_SIZE=0x28000000 (kernel v5.17.6).

[...]
Oops: Kernel stack overflow, sig: 11 [#1]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Modules linked in: auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc ghash_generic gf128mul gcm ccm algif_aead des_generic libdes ctr cbc ecb algif_skcipher aes_generic libaes cmac sha512_generic sha1_generic sha1_powerpc md5 md5_ppc md4 hid_generic b43legacy usbhid mac80211 hid libarc4 cfg80211 snd_aoa_codec_tas rfkill snd_aoa_fabric_layout snd_aoa evdev mac_hid therm_windtunnel firewire_ohci firewire_core crc_itu_t sr_mod cdrom ohci_pci 8250_pci radeon snd_aoa_i2sbus ohci_hcd snd_aoa_soundbus ssb snd_pcm ehci_pci snd_timer pcmcia snd soundcore pcmcia_core hwmon 8250 ehci_hcd i2c_algo_bit 8250_base drm_ttm_helper serial_mctrl_gpio ttm drm_kms_helper usbcore syscopyarea sysfillrect sysimgblt usb_common fb_sys_fops pkcs8_key_parser fuse drm drm_panel_orientation_quirks configfs
CPU: 0 PID: 24122 Comm: sh Not tainted 5.17.6-gentoo-PMacG4 #1
NIP:  c0018614 LR: 00000000 CTR: c103cbe0
REGS: e7fe9f50 TRAP: 0000   Not tainted  (5.17.6-gentoo-PMacG4)
MSR:  00001030 <ME,IR,DR>  CR: 00000001  XER: c000e234

GPR00: a78bfe90 80002288 00000000 d6a5e1a0 e991de60 0068c6c4 a7a3ff98 c1099000 
GPR08: 00000000 e991dec0 d6a5e1a0 80002288 005900d0 0068fff4 00000000 00000007 
GPR16: 00000029 00000007 00bc44b0 a7ddafe8 a78bfe90 fffff000 00000000 00000000 
GPR24: 005900d0 0068c6c4 c0dcc7a0 c1402b48 caa899c0 c103cbe0 c4ce9400 c08fe234 
NIP [c0018614] interrupt_return+0x17c/0x190
LR [00000000] 0x0
Call Trace:
Instruction dump:
40860018 7ccff120 80c10028 80010010 80210014 4c000064 7ccff120 7d3043a6 
392100c0 80c10028 80010010 80210014 <91210000> 7d3042a6 4c000064 7c000828 
---[ end trace 0000000000000000 ]---


@Christophe: Would it be helpful for these issues to try a KASAN build?

Yes KASAN can bring some additional inputs.
Maybe start with CONFIG_KFENCE, it is lighter than KASAN.


For the above problem, maybe CONFIG_DEBUG_STACKOVERFLOW can help.

DEBUG_STACKOVERFLOW and KFENCE have been enabled already in the builds I did here (see kernel attached kernel .config here).

However if I enable (inline) KASAN the kernel won't boot at all. I get dropped out in OpenFirmware console with:

[...]
Finalizing device tree... using OF tree (promptr=ff847240)

Invalid memory access at %SRR0: 40000000 %SRR1: 00000000

Increasing the stack size (CONFIG_THREAD_SHIFT) might avoid the stack overflows and allow you to debug the original issue in isolation.

Created attachment 300977 [details]
dmesg (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, outline KASAN, PowerMac G4 DP)

I increased THREAD_SHIFT to 14 and used outline KASAN still with CONFIG_LOWMEM_SIZE=0x28000000. The memory corruption output looks slightly different (but not much):

[...]
pagealloc: memory corruption
f5fcfff0: 00 00 00 00                                      ....
CPU: 1 PID: 29742 Comm: ld.so.1 Not tainted 5.18.0-rc6-PMacG4 #7
Call Trace:
[eea3ba90] [c09890d4] dump_stack_lvl+0x80/0xc0 (unreliable)
[eea3bab0] [c03cce40] __kernel_unpoison_pages+0x208/0x250
[eea3bb00] [c03a2e48] post_alloc_hook+0x108/0x144
[eea3bb30] [c03a66e0] get_page_from_freelist+0x9d4/0x12dc
[eea3bc70] [c03a7ad0] __alloc_pages+0x23c/0x1570
[eea3bde0] [c0379c8c] handle_mm_fault+0x610/0x1240
[eea3bed0] [c002e2d4] ___do_page_fault+0x19c/0x850
[eea3bf10] [c002ebbc] do_page_fault+0x28/0x5c
[eea3bf30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0x6fe0338c
NIP:  6fe0338c LR: 6fe032c4 CTR: 6fe033e0
REGS: eea3bf40 TRAP: 0300   Not tainted  (5.18.0-rc6-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 48002262  XER: 20000000
DAR: 046a5000 DSISR: 42000000 
GPR00: 6ffbcb94 afc45940 a7c95560 046a4fe4 8a000000 000127e0 03e59a8b 00000003 
GPR08: 046a5004 046a5000 04621cfc 6fe03170 6fe032c4 6ffece34 00000000 6ffef34d 
GPR16: 02dea020 04416750 00000003 01f8cbec 02de9fa0 01f8c660 00000000 00000000 
GPR24: afc45aa0 6ffef37c afc45a18 04678c7c 0007630c 04678c7c 6ff76ff4 045f5990 
NIP [6fe0338c] 0x6fe0338c
LR [6fe032c4] 0x6fe032c4
--- interrupt: 300
page:e739d6ec refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x290a3
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details
[...]

With THREAD_SHIFT=14 the stack issue does not show up.

A kernel with inline KASAN and same setup otherwise won't boot showing me this at the OpenFirmware prompt:

[...]
Finalizing device tree... using OF tree (promptr=ff847240)

Invalid memory access at %SRR0: 40000000 %SRR1: 00000000

Created attachment 300978 [details]
kernel .config (5.18-rc6, CONFIG_LOWMEM_SIZE=0x28000000, outline KASAN, PowerMac G4 DP)

Seems like with Inline KASAN your kernel is far too big compared to what we support at the time being:

c2468000 T __end_rodata
c2800000 T __init_begin
c2800000 T _sinittext

c2801644 T prom_init

The init text section is behind the 32Mbytes boundary, it means that prom_init and other functions are not called anymore directly but via a trampoline.

c000000c <__start>:
c000000c:       2c 05 00 00     cmpwi   r5,0
c0000010:       41 82 00 1c     beq     c000002c <__start+0x20>
c0000014:       42 9f 00 05     bcl     20,4*cr7+so,c0000018 <__start+0xc>
c0000018:       7d 08 02 a6     mflr    r8
c000001c:       3d 08 00 00     addis   r8,r8,0
c0000020:       39 08 ff e8     addi    r8,r8,-24
c0000024:       48 00 38 e5     bl      c0003908 <setup_disp_bat+0x30>
...
c0003908:       3d 80 c2 80     lis     r12,-15744
c000390c:       39 8c 16 44     addi    r12,r12,5700
c0003910:       7d 89 03 a6     mtctr   r12
c0003914:       4e 80 04 20     bctr


And it cannot work because at that time the kernel is not yet relocated to its final location.

There was the same problem with PPC64 and it was fix by 24d33ac5b8ff ("powerpc/64s: Make prom_init require RELOCATABLE").

Don't know if a similar approach could work.

The Kernel stack overflow looks odd.

Value of R1 is wrong and LR is NULL. Don't know how we ended up here, but probably not by a real stack overflow.

Note that THREAD_SHIFT is set to 14 when using KASAN:

config THREAD_SHIFT
	int "Thread shift" if EXPERT
	range 13 15
	default "15" if PPC_256K_PAGES
	default "14" if PPC64
	default "14" if KASAN
	default "13"
	help
	  Used to define the stack size. The default is almost always what you
	  want. Only change this if you know what you are doing.

I opened a new bug for the stack issue which contains a bit more data. Hopefully the output is of some help (see bug #216041).

Created attachment 301302 [details]
dmesg (5.19-rc4, PowerMac G4 DP)

Re-tried on v5.19-rc4 (without fadditional patches) + KFENCE.

My findings so far:
1. Memory corruption still persists.
2. Even without KASAN I need THREAD_SHIFT=14 or else I get the stack overflow from bug #216041.
3. Memory corruption also happens with CONFIG_LOWMEM_SIZE=0x28000000.
4. But the "neverending build" commit mentioned in comment #9 is gone (be it with default .config or CONFIG_LOWMEM_SIZE=0x28000000).

[...]
pagealloc: memory corruption
fffdfff0: 00 00 00 00                                      ....
CPU: 0 PID: 29136 Comm: localedef Not tainted 5.19.0-rc4-PMacG4 #3
Call Trace:
[f39b3c20] [c05eb9c0] dump_stack_lvl+0x60/0x90 (unreliable)
[f39b3c40] [c0232fb0] __kernel_unpoison_pages+0x1a8/0x1ec
[f39b3c90] [c02170dc] get_page_from_freelist+0xc20/0xe70
[f39b3d50] [c0217bdc] __alloc_pages+0x18c/0xe80
[f39b3e10] [c01f46b4] wp_page_copy+0x214/0xa1c
[f39b3e80] [c01fa0b8] handle_mm_fault+0x720/0xd64
[f39b3f00] [c00215dc] do_page_fault+0x1d4/0x830
[f39b3f30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0x669410
NIP:  00669410 LR: 006693e4 CTR: 00000000
REGS: f39b3f40 TRAP: 0300   Not tainted  (5.19.0-rc4-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 84002462  XER: 20000000
DAR: a7a3cce8 DSISR: 0a000000 
GPR00: 0066961c afd34060 a7bd3000 01a069bc 01b76d60 00000009 a4e0c05a 0005ccd8 
GPR08: 01b76140 a7a3cce8 a7a43e44 400a713a 44002862 0068fe34 01b8d730 00000001 
GPR16: 00000000 01a069bc 01a069f8 01a06990 01b8d170 01a06894 0000000f 00000009 
GPR24: 01b76d60 a4e0c05a 0000018d a7ad9f00 a79e0010 000041cb 00697cdc 01a069bc 
NIP [00669410] 0x669410
LR [006693e4] 0x6693e4
--- interrupt: 300
page:ef4bd80c refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310ab
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details

Created attachment 301303 [details]
kernel .config (5.19-rc4, PowerMac G4 DP)

It's a bit of a stab in the dark, but can you try turning preempt off?

ie. CONFIG_PREEMPT_NONE=y

(In reply to Michael Ellerman from comment #30)
> It's a bit of a stab in the dark, but can you try turning preempt off?
> 
> ie. CONFIG_PREEMPT_NONE=y
Just tested that. Backtrace looks a little different but not much.

[..]
pagealloc: memory corruption
fffdfff0: 00 00 00 00                                      ....
CPU: 0 PID: 29086 Comm: localedef Not tainted 5.19.0-rc4-PMacG4 #2
Call Trace:
[f397bc90] [c05eb280] dump_stack_lvl+0x60/0x90 (unreliable)
[f397bcb0] [c0233128] __kernel_unpoison_pages+0x1a8/0x1ec
[f397bd00] [c02172ec] get_page_from_freelist+0xc20/0xe70
[f397bdc0] [c0217de0] __alloc_pages+0x180/0xe98
[f397be80] [c01fa164] handle_mm_fault+0x450/0xd64
[f397bf00] [c00215d8] do_page_fault+0x1d0/0x82c
[f397bf30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0x83f1b8
NIP:  0083f1b8 LR: 0083e25c CTR: 00000000
REGS: f397bf40 TRAP: 0300   Not tainted  (5.19.0-rc4-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 88224462  XER: 00000000
DAR: 01232b3c DSISR: 42000000 
GPR00: 00840220 af9416c0 a7ca4000 01231b50 00000fe0 00000005 01232b38 00000000 
GPR08: 00000ff1 01231b48 0000f4c9 008422b0 01067408 00a2fe34 00000070 01231b50 
GPR16: 00000000 00000000 00000000 00000007 0000003f 009ba23c 01067010 009ba79c 
GPR24: 00000062 009bdac8 000000fe 009ba79c 00000fe0 009ba764 009b9ff4 00000ff0 
NIP [0083f1b8] 0x83f1b8
LR [0083e25c] 0x83e25c
--- interrupt: 300
page:ef4bd80c refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310ab
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details


Interesting thing is the memory corruption always seems to happen in the last stage of installing, after building is done at copying over the binaries from build directory to target directory:

[...]
if test -r /var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h && cmp -s /var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h /var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h; \
then echo 'stubs.h unchanged'; \
else /usr/lib/portage/python3.10/ebuild-helpers/xattr/install -c -m 644 /var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h /var/tmp/portage/sys-libs/glibc-2.34-r13/image//usr/include/gnu/stubs-32.h; fi
rm -f /var/tmp/portage/sys-libs/glibc-2.34-r13/work/build-ppc-powerpc-unknown-linux-gnu-nptl/stubs.h
make[1]: Leaving directory '/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34'
>>> Completed installing sys-libs/glibc-2.34-r13 into
>>> /var/tmp/portage/sys-libs/glibc-2.34-r13/image

 * Final size of build directory: 635640 KiB (620.7 MiB)
 * Final size of installed tree:  109892 KiB (107.3 MiB)

making executable: /usr/lib/libc.so
compressme           : 44.96%   (  3.80 KiB =>   1.71 KiB, compressme.zst)     
[...]
/var/tmp/portage/sys-libs/glibc-2.34-r13/image/usr/share/doc/glibc-2.34-r13/NEWS : 33.98%   (   315 KiB =>    107 KiB, /var/tmp/portage/sys-libs/glibc-2.34-r13/image/usr/share/doc/glibc-2.34-r13/NEWS.zst) 
strip: powerpc-unknown-linux-gnu-strip --strip-unneeded -N __gentoo_check_ldflags__ -R .comment -R .GCC.command.line -R .note.gnu.gold-version
   /usr/lib/crt1.o
   /usr/lib/Mcrt1.o
   /usr/lib/gcrt1.o
   /usr/lib/Scrt1.o
[...]
   /lib/ld.so.1
   /usr/lib/audit/sotruss-lib.so
   /usr/bin/pldd
installsources: rsyncing source files
rsync: [sender] link_stat "/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/iconv/charmap-kw.gperf" failed: No such file or directory (2)
rsync: [sender] link_stat "/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/locale/charmap-kw.gperf" failed: No such file or directory (2)
rsync: [sender] link_stat "/var/tmp/portage/sys-libs/glibc-2.34-r13/work/glibc-2.34/locale/locfile-kw.gperf" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1326) [sender=3.2.4]

>>> Installing (1 of 1) sys-libs/glibc-2.34-r13::gentoo
 * Defaulting /etc/host.conf:multi to on
 * Last-minute run tests with ./ld.so.1 in /lib ...
[...]

(In reply to Michael Ellerman from comment #30)
> It's a bit of a stab in the dark, but can you try turning preempt off?
> 
> ie. CONFIG_PREEMPT_NONE=y
Looks like your intuition was not bad at all. ;) CONFIG_PREEMPT_NONE=y had no effect but when I disable SMP at all '# CONFIG_SMP is not set' I get no memory corruption and also no stack overflow issues.

Also no special treatment with Advanced Options or setting THREAD_SHIFT manually was necessary. The G4 just does fine, albeit with 1 of it's 2 CPUs only with disabled SMP.

For testing I did 6 of this glibc testsuite builds in a row without getting issues. With SMP enabled I get memory corruption or stack overflow at the 1st build allmost all of the time.

Created attachment 301337 [details]
dmesg (5.19-rc5, outline KASAN, PowerMac G4 DP)

Re-tested on 5.19-rc5 + https://patchwork.ozlabs.org/project/linuxppc-dev/patch/2ee707512b8b212b079b877f4ceb525a1606a3fb.1656655567.git.christophe.leroy@csgroup.eu/

I can run the kernel with outline KASAN, default THREAD_SHIFT and without advanced options necessary. Also I don't get the stack issue (bug #216041) any longer.

However as long as CONFIG_SMP=y (CONFIG_NR_CPUS=2) is set I still get the memory corruption:

[...]
pagealloc: memory corruption
f5fcfff0: 00 00 00 00                                      ....
CPU: 1 PID: 27635 Comm: estrip Not tainted 5.19.0-rc5-PMacG4+ #1
Call Trace:
[f380b9b0] [c0829ebc] dump_stack_lvl+0x60/0x90 (unreliable)
[f380b9d0] [c0307528] __kernel_unpoison_pages+0x1d8/0x220
[f380ba20] [c02dd3bc] post_alloc_hook+0x108/0x144
[f380ba50] [c02e0a70] get_page_from_freelist+0x9e0/0x1278
[f380bb90] [c02e1e04] __alloc_pages+0x250/0x1078
[f380bcf0] [c02af098] wp_page_copy+0x128/0xdb8
[f380bde0] [c02b6fdc] handle_mm_fault+0x954/0x1138
[f380bed0] [c0029938] ___do_page_fault+0x250/0x84c
[f380bf10] [c002a168] do_page_fault+0x28/0x5c
[f380bf30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0x65b734
NIP:  0065b734 LR: 0065b708 CTR: 00354600
REGS: f380bf40 TRAP: 0300   Not tainted  (5.19.0-rc5-PMacG4+)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 82222420  XER: 00000000
DAR: 026fcea0 DSISR: 0a000000 
GPR00: 00000000 afbd5250 a7b0c560 026bb5f0 0269deac 026bb628 696e6f64 026fcea0 
GPR08: 00000000 00000000 00000000 00354600 42222420 0071fff4 026af620 0072243c 
GPR16: 00723b50 007223a4 026b1770 026ec8a0 007222e4 0269de70 02700920 00000001 
GPR24: 00721e9c 00721eb8 0072082c 00000000 afbd52ec 00000000 0072608c 00000000 
NIP [0065b734] 0x65b734
LR [0065b708] 0x65b708
--- interrupt: 300
page:ef4bd6ec refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x310a3
flags: 0x80000000(zone=2)
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details

Created attachment 301639 [details]
dmesg (6.0-rc2, outline KASAN, PowerMac G4 DP)

Getting a more interesting backtrace with v6.0.0-rc2 + outline KASAN:

[...]
BUG: KASAN: slab-out-of-bounds in handle_mm_fault+0x27c/0x10f4
Read of size 4 at addr c32edd48 by task cc1plus/1230

CPU: 1 PID: 1230 Comm: cc1plus Tainted: G                T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2bd40] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2bd60] [c032b8d8] print_report+0x30c/0x688
[f4d2bdb0] [c032befc] kasan_report+0xe4/0x214
[f4d2be00] [c02ce4d8] handle_mm_fault+0x27c/0x10f4
[f4d2bed0] [c002cc98] ___do_page_fault+0x25c/0x8d0
[f4d2bf10] [c002d560] do_page_fault+0x28/0x6c
[f4d2bf30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G                T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300

Allocated by task 1:
 __kasan_slab_alloc+0xd0/0x134
 kmem_cache_alloc+0x21c/0x66c
 __kernfs_new_node+0xe8/0x354
 kernfs_new_node+0x84/0xfc
 __kernfs_create_file+0x50/0x204
 sysfs_add_file_mode_ns+0xf4/0x1f0
 internal_create_group+0x1f0/0x620
 btrfs_init_sysfs+0x264/0x350
 init_btrfs_fs+0x24/0x280
 do_one_initcall+0xc0/0x34c
 kernel_init_freeable+0x2c0/0x400
 kernel_init+0x28/0x178
 ret_from_kernel_thread+0x5c/0x64

The buggy address belongs to the object at c32edd50
 which belongs to the cache kernfs_node_cache of size 88
The buggy address is located 8 bytes to the left of
 88-byte region [c32edd50, c32edda8)

The buggy address belongs to the physical page:
page:eee4a954 refcount:1 mapcount:0 mapping:00000000 index:0x0 pfn:0x32ed
flags: 0x200(slab|zone=0)
raw: 00000200 00000100 00000122 c1852520 00000000 001e003c ffffffff 00000001
raw: 00000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
 c32edc00: 00 00 fc fc fc fc fc fc 00 00 00 00 00 00 00 00
 c32edc80: 00 00 00 fc fc fc fc fc fc 00 00 00 00 00 00 00
>c32edd00: 00 00 00 00 fc fc fc fc fc fc 00 00 00 00 00 00
                                      ^
 c32edd80: 00 00 00 00 00 fc fc fc fc fc fc 00 00 00 00 00
 c32ede00: 00 00 00 00 00 00 fc fc fc fc fc fc 00 00 00 00
==================================================================
Disabling lock debugging due to kernel taint
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
[...]
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
get_swap_device: Bad swap file entry 64cccccc
_swap_info_get: Bad swap file entry 64cccccc
BUG: Bad page map in process cc1plus  pte:cccccccc pmd:032ed000
addr:9a352000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a352
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
_swap_info_get: Bad swap file entry 64cccccc
BUG: Bad page map in process cc1plus  pte:cccccccc pmd:032ed000
addr:9a353000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a353
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
BUG: Bad page map in process cc1plus  pte:00000001 pmd:032ed000
page:eedd8000 refcount:1 mapcount:-1 mapping:00000000 index:0x0 pfn:0x0
flags: 0x1000(reserved|zone=0)
raw: 00001000 eedd8004 eedd8004 00000000 00000000 00000000 fffffffe 00000001
raw: 00000000
page dumped because: bad pte
addr:9a354000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a354
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9974] unmap_page_range+0x69c/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
_swap_info_get: Bad swap file entry 14c32d92
BUG: Bad page map in process cc1plus  pte:c32d9228 pmd:032ed000
addr:9a356000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a356
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
_swap_info_get: Bad swap file entry 60c0e0fa
BUG: Bad page map in process cc1plus  pte:c0e0fac0 pmd:032ed000
addr:9a357000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a357
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
_swap_info_get: Bad swap file entry 50c32ed0
BUG: Bad page map in process cc1plus  pte:c32ed0a0 pmd:032ed000
addr:9a358000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a358
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
_swap_info_get: Bad swap file entry 4c2a45bd
BUG: Bad page map in process cc1plus  pte:2a45bd98 pmd:032ed000
addr:9a35c000 vm_flags:00100073 anon_vma:c5933ee8 mapping:00000000 index:9a35c
file:(null) fault:0x0 mmap:0x0 read_folio:0x0
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
Call Trace:
[f4d2b9b0] [c0864cc4] dump_stack_lvl+0x60/0xa4 (unreliable)
[f4d2b9d0] [c02c5bc4] print_bad_pte+0x2e8/0x364
[f4d2ba60] [c02c9c3c] unmap_page_range+0x964/0xb78
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
BUG: Unable to handle kernel data access on read at 0x09fcbaf8
Faulting instruction address: 0xc02c99a8
Oops: Kernel access of bad area, sig: 11 [#1]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Modules linked in: auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc hid_generic usbhid hid b43legacy mac80211 snd_aoa_codec_tas libarc4 snd_aoa_fabric_layout snd_aoa cfg80211 rfkill evdev mac_hid firewire_ohci therm_windtunnel firewire_core sr_mod cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus snd_pcm snd_timer snd 8250_pci soundcore ssb pcmcia pcmcia_core 8250 8250_base serial_mctrl_gpio ohci_pci radeon hwmon ohci_hcd ehci_pci i2c_algo_bit drm_ttm_helper ttm drm_display_helper ehci_hcd drm_kms_helper syscopyarea sysfillrect usbcore sysimgblt fb_sys_fops usb_common fuse drm drm_panel_orientation_quirks configfs
CPU: 0 PID: 1230 Comm: cc1plus Tainted: G    B   W       T  6.0.0-rc2-PMacG4 #5
NIP:  c02c99a8 LR: c02c99a8 CTR: 00000000
REGS: f4d2b9a0 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  00009032 <EE,ME,IR,DR,RI>  CR: 24d88838  XER: 20000000
DAR: 09fcbaf8 DSISR: 40000000 
GPR00: 00000000 f4d2ba60 c3ff0020 00000000 00000000 00000000 00000000 00000000 
GPR08: 00000000 00000000 00000000 00000000 00000000 11c3d4e0 f4d2bad0 00000000 
GPR16: c7d11008 fe9a5752 c0de15e0 f4d2bb50 f4d2bab0 c3626ac8 00000000 c16ed525 
GPR24: 9a400000 fffffffd 00000000 f4d2bc10 09fcbaf4 09fcbaf4 9a35f000 c32edd78 
NIP [c02c99a8] unmap_page_range+0x6d0/0xb78
LR [c02c99a8] unmap_page_range+0x6d0/0xb78
Call Trace:
[f4d2ba60] [c02c99a8] unmap_page_range+0x6d0/0xb78 (unreliable)
[f4d2bb20] [c02ca590] unmap_vmas+0x168/0x2d4
[f4d2bbd0] [c02d8af0] exit_mmap+0x11c/0x2dc
[f4d2bca0] [c005e8f4] mmput+0xa0/0x254
[f4d2bcd0] [c006e1b4] do_exit+0x430/0xe08
[f4d2bd50] [c006ed88] do_group_exit+0x68/0x11c
[f4d2bd80] [c0086818] get_signal+0xbfc/0xc50
[f4d2be30] [c000edf8] do_notify_resume+0xf0/0x540
[f4d2bf10] [c0019cfc] interrupt_exit_user_prepare_main+0x7c/0xd0
[f4d2bf30] [c00234ac] interrupt_return+0x14/0x190
--- interrupt: 300 at 0xfa9c0c0
NIP:  0fa9c0c0 LR: 1066b838 CTR: 0fa9bea4
REGS: f4d2bf40 TRAP: 0300   Tainted: G    B   W       T   (6.0.0-rc2-PMacG4)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 24022828  XER: 20000000
DAR: 9a352014 DSISR: 42000000 
GPR00: 1066b828 af869c10 a7dd1ba0 9a352000 00000000 00000018 9a352018 00000000 
GPR08: 11c30000 0fb89a88 099aec30 0fa9bea4 88022444 11c3d4e0 00000001 af869e78 
GPR16: 9afeeed0 9afedd60 10cef83c 115fdfd0 af869e80 11603030 9afeeed0 00000002 
GPR24: 9afeeed0 9b611f60 115fdfd0 a0c82c30 9afeeed0 00000005 0000006e 9a352000 
NIP [0fa9c0c0] 0xfa9c0c0
LR [1066b838] 0x1066b838
--- interrupt: 300
Instruction dump:
7ecfb378 82410014 82c10018 4bffff04 3d40c170 578901be 83aa5580 1d290024 
7fbd4a14 387d0004 7fbceb78 48063245 <813d0004> 712a0001 40820304 7f83e378 
---[ end trace 0000000000000000 ]---

Fixing recursive fault but reboot is needed!


I deleted about 120.000 lines of "get_swap_device: Bad swap file entry 64cccccc" in the kernel dmesg to make it more compact. swap partition is 8192 MiB large at /dev/sdb6.

Created attachment 301640 [details]
kernel .config (6.0-rc2, PowerMac G4 DP)

Would be nice to give it a new try with KCSAN enabled.

To get KCSAN on powerpc/32, apply following series: https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=354731

Created attachment 304308 [details]
dmesg (6.3.3, KCSAN, PowerMac G4 DP)

Thanks for taking another look into this Christophe!

Applied the patches on top of 6.3.3 and these are my findings so far:

1. KCSAN works fine on my G4 and passes self tests.
2. It does not generate any additional output when I hit the "pagealloc: memory corruption".
3. When setting CONFIG_KCSAN_WEAK_MEMORY=y my G4 won't finish booting. Early boot works, the screen shows some dmesg but booting gets stuck there never reaching console. I also don't get any netconsole output with CONFIG_KCSAN_WEAK_MEMORY=y.
4. As soon as I set CONFIG_KCSAN_EARLY_ENABLE=y dmesg shows plenty of data races!

netconsole output and kernel .config attached.

To provoke the memory corruption 'stress' is a good tool. stress -m2 --vm-bytes 915M provokes the corruption easily and --vm-bytes 915M is small enough to not provoke the OOM killer on my G4 DP with its' 2 CPUs and 2 GiB RAM.

Created attachment 304309 [details]
kernel .config (6.3.3, PowerMac G4 DP)

No change with 6.4-rc4, only additional data "page_type: 0xffffffff()" is shown:

[...]
pagealloc: memory corruption
06fe3258: 00 00 00 00                                      ....
CPU: 1 PID: 397 Comm: stress Tainted: G        W       T  6.4.0-rc3-PMacG4-dirty #1
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[f2a35c70] [c0eea17c] dump_stack_lvl+0x60/0xa4 (unreliable)
[f2a35c90] [c0eea1d8] dump_stack+0x18/0x30
[f2a35ca0] [c0360f90] __kernel_unpoison_pages+0x234/0x288
[f2a35ce0] [c033fdf4] get_page_from_freelist+0xd90/0x10d8
[f2a35d90] [c0340978] __alloc_pages+0x138/0xdd8
[f2a35e40] [c0315b80] handle_mm_fault+0xab8/0x15e0
[f2a35ed0] [c003a3d4] ___do_page_fault+0x320/0x8c4
[f2a35f10] [c003abe0] do_page_fault+0x28/0x80
[f2a35f30] [c000433c] DataAccess_virt+0x124/0x17c
--- interrupt: 300 at 0xaf30d8
NIP:  00af30d8 LR: 00af30b4 CTR: 00000000
REGS: f2a35f40 TRAP: 0300   Tainted: G        W       T   (6.4.0-rc3-PMacG4-dirty)
MSR:  0000d032 <EE,PR,ME,IR,DR,RI>  CR: 20882464  XER: 00000000
DAR: 8f7a3010 DSISR: 42000000 
GPR00: 00af30b4 af9c9cb0 a7cd2740 6e97d010 39300000 20224462 00000000 00a10264 
GPR08: 20e27000 20e26000 00000000 4062ceda 20882462 00b0fff4 00000000 00000000 
GPR16: 00000000 00000002 00000000 0000005a 40802462 80002462 40002462 00b100a4 
GPR24: ffffffff ffffffff 39300000 00000000 00000000 6e97d010 00b17d64 00001000 
NIP [00af30d8] 0xaf30d8
LR [00af30b4] 0xaf30b4
--- interrupt: 300
page:e314e657 refcount:1 mapcount:0 mapping:00000000 index:0x1 pfn:0x31065
flags: 0x80000000(zone=2)
page_type: 0xffffffff()
raw: 80000000 00000100 00000122 00000000 00000001 00000000 ffffffff 00000001
raw: 00000000
page dumped because: pagealloc: corrupted page details

Created attachment 305297 [details]
dmesg (5.5-rc5, PowerMac G4 DP)

Re-visiting this bug as it's reproducible on v6.6-rc7.

This time I tried the other way round. CONFIG_VMAP_STACK was added for ppc with commit cd08f109e26231b279bcc0388428afcac6408ec6 (at about kernel v5.5-rc5 time). So I did a git checkout cd08f109e26231b279bcc0388428afcac6408ec6 and started from there with a further reduced kernel .config.

I added two additional patches to get the G4 to boot with VMAP_STACK enabled: 4119622 "powerpc/32s: Fix kasan_early_hash_table() for CONFIG_VMAP_STACK" and 232ca1e "powerpc/32s: Fix DSI and ISI exceptions for CONFIG_VMAP_STACK".

Then I burdened the memory subsystem with "stress -c 2 --vm 2 --vm-bytes 896M" as before and hit the issue in less than 20 sec. Not hitting the issue means my G4 runs "stress -c 2 --vm 2 --vm-bytes 896M" for about half an hour without side effects.

So it looks like the issue was here from the start when CONFIG_VMAP_STACK was added for ppc. (see dmesg)

I don't hit the issue when:
   1. nr_cpus=1 is set + VMAP_STACK enabled
   2. VMAP_STACK disabled

Setting LOWMEM_SIZE to 0x28000000 does not seem to have an effect on it.


This bug really plays hard to get... T'll do further KCSAN checks in recent kernels and open separate issues if KCSAN digs up something useful.

Created attachment 305298 [details]
attachment-30247-0.html

I'm out of office until 06 Nov.

Created attachment 305299 [details]
kernel .config (5.5-rc5, PowerMac G4 DP)