Bug 208181

Created attachment 289661 [details]
kernel .config (5.8-rc1, PowerMac G4 DP)

Created attachment 289935 [details]
dmesg (5.8-rc3, INLINE KASAN, PowerMac G4 DP

No change with 5.8-rc3.

Created attachment 289937 [details]
kernel .config (5.8-rc3, PowerMac G4 DP)

Erm wait... there is some change.

In 5.8-rc1 stacktrace was:
BUG: KASAN: stack-out-of-bounds in strcmp+0x58/0xd8
Read of size 1 at addr c11c1a80 by task swapper/0

CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc1-PowerMacG4 #2
Call Trace:
[c1ae97d0] [c0a2069c] dump_stack+0xfc/0x158 (unreliable)
[c1ae9800] [c04ac5cc] print_address_description.isra.0+0x30/0x3fc
[c1ae9870] [c04acb28] kasan_report+0x110/0x170
[c1ae98b0] [c0a44c00] strcmp+0x58/0xd8
[c1ae98d0] [c0170790] register_lock_class+0xfa4/0x10a0
[c1ae9990] [c0170a34] __lock_acquire+0x1a8/0x382c
[c1ae9b40] [c016f398] lock_acquire+0x5e0/0x854
[c1ae9c00] [c1144014] _raw_spin_lock_irqsave+0x48/0x70
[c1ae9c20] [c0ccbe84] of_find_property+0x2c/0x5c
[c1ae9c50] [c0ccbec8] of_get_property+0x14/0x6c
[c1ae9c70] [c0cdbcd8] unflatten_dt_nodes+0xc4c/0xcdc
[c1ae9ec0] [c0cdbe90] __unflatten_device_tree+0x114/0x1e0
[c1ae9ef0] [c184a294] unflatten_device_tree+0x38/0x54
[c1ae9f10] [c1808600] setup_arch+0xc8/0x630
[c1ae9f50] [c1803268] start_kernel+0xcc/0x4cc
[c1ae9ff0] [000038a0] 0x38a0


In 5.8-rc3 stacktrace is:
BUG: KASAN: stack-out-of-bounds in vprintk_func+0x100/0x4b4
Read of size 4 at addr c1919e14 by task swapper/0

CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc3-PowerMacG4 #2
Call Trace:
[c1ae9c00] [c0a304dc] dump_stack+0xfc/0x158 (unreliable)
[c1ae9c30] [c04ac990] print_address_description.isra.0+0x30/0x3fc
[c1ae9ca0] [c04aceec] kasan_report+0x110/0x170
[c1ae9ce0] [c018c204] vprintk_func+0x100/0x4b4
[c1ae9d10] [c018afd4] printk+0xa8/0xd4
[c1ae9db0] [c003c8c4] __ioremap_caller+0x1c4/0x27c
[c1ae9df0] [c003c394] ioremap+0x20/0x30
[c1ae9e00] [c1813fe4] pmac_feature_init+0x288/0xd90
[c1ae9ed0] [c1812cb0] pmac_probe+0x13c/0x190
[c1ae9ef0] [c001d938] probe_machine+0xe8/0x13c
[c1ae9f10] [c1808614] setup_arch+0xdc/0x630
[c1ae9f50] [c1803268] start_kernel+0xcc/0x4cc
[c1ae9ff0] [000038a0] 0x38a0


What stays the same are the two "setbat: no BAT available" messages in both cases.

Can we get a dump of /sys/kernel/debug/powerpc/block_address_translation

Created attachment 289945 [details]
block_address_translation

Sure.

Created attachment 289947 [details]
segment_registers

block_address_translation contains funny sizes. But the adresses seems ok.
So it shows you have a 24 Mb text+rodata area. 8 BATs are used (16+8+8+32+64+128+256+256)
By increasing CONFIG_DATA_SHIFT to 25, you'll get a 32Mb alignment
So you will have only 6 bats used (32+32+64+128+256+256), so two additional data BATs will be available for KASAN.

But regardless of the BAT stuff, KASAN should work properly.

Ok, thanks for the clarification! So if KASAN works properly something else must cause this hit. I will start a bisect the next few days and see how that turns out...

I reproduced the problem and bisected it to commit https://github.com/torvalds/linux/commit/d2a91cef9bbdeb87b7449fdab1a6be6000930210

I'll investigate the issue

Can you confirm this commit is also the curprit from your side ?

The issue is that that commit moved more code than described into kasan_init():

Kasan Pages allocation have to be moved into kasan_init() but page tables allocation must remain before the switch to the final hash table.

Problem only occurs on book3s/32 having hash MMU.

See proposed fix at https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=187165 (2 patches).

Successfully applied your 2 patches. The kernel still complains about these 2 BATs, but the KASAN hit at early boot is gone with the patches. Thanks!

Thanks for testing.

Regarding the two BATs, I suggest you increase CONFIG_DATA_SHIFT as explained in a previous comment.

Ah, I've overlooked that...

To set CONFIG_DATA_SHIFT=25 I needed to set ADVANCED_OPTIONS=y, DATA_SHIFT_BOOL=y first.

But with CONFIG_DATA_SHIFT=25 this kernel won't boot at all. OpenFirmware shows for a brief moment that the kernel gets loaded and then I get dropped back to OF console.

OpenFirmware tells me:
Invalid memory access at %SRR0: a0000000  %SRR1: 00000000

Ah yes, having init_text above the 24 bits limit might be a problem for function calls. I'm surprised that the linker doesn't complain.

Anyway, it is not a problem in itself, and it's unrelated to this bug.

Created attachment 290285 [details]
kernel .config (5.8-rc5, PowerMac G4 DP)

Did some additional test-runs, seems there are still problems with stack usage when running (inline) KASAN:

5.8-rc3 + the 2 patches applied:
Instruction dump:
usercopy: Kernel memory overwrite attemp detected to kernel text (offset 5432, size 4)!
------------[ cut here ]------------
kernel BUG at mm/usercopy.c:99!
Oops: Exeption in kernel mode, sig:5 [#6]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Modules linked in: auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc b43legacy input_leds joydev mac80211 hid_generic g4_windtunnel sungem sungem_phy btrfs ohci_pci xor lzo_compress lzo_decompress zlib_deflate raid6_pq zlib_inflate firewire_ohci hcd soundcore ssb pcmcia usbcore uninorth_agp pcmcia_core agpart usb_common
CPU: 1 PID: 5250 Comm: mount.nfs Tainted: G       W        5.8.0-rc3-PowerMacG4+ #8
NIP: c04d654c LR: c04d654c CTR: 00000000
REGS: c0001198 TRAP: 0700  Tainted: G       W        (5.8.0-rc3-PowerMacG4+)
MSR:  00021031 <MR,IR,DR,RI> CR: 24028822 XER: 00000000

GPR00: c04d654c c0001498 e719b980 00000058 c01899f4 00000000 00000027 e8dc4e0c
GPR08: 00000023 00000000 00000000 c0001498 44028822 0061bff4 f80002s9 00000003
GPR16: c115a340 f80002d7 c00016b8 c00016c8 c04d654c c115a260 c04d651c f80002d5
GPR24: c00016ac 180002d5 e8dda024 c0000000 c000153c 00000000 00000004 c0001538
NIP [c04d654c] usercopy_abort+0x68/0x78
LR [c04d654c] usercopy_abort+0x68/0x78
Call Trace:
Instruction dump:
usercopy: Kernel memory overwrite attemp detected to kernel text (offset 4848, size 4)!
------------[ cut here ]------------
kernel BUG at mm/usercopy.c:99!
Oops: Exeption in kernel mode, sig:5 [#7]
BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
Modules linked in: auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc b43legacy input_leds joydev mac80211 hid_generic g4_windtunnel sungem sungem_phy btrfs ohci_pci xor lzo_compress lzo_decompress zlib_deflate raid6_pq zlib_inflate firewire_ohci hcd soundcore ssb pcmcia usbcore uninorth_agp pcmcia_core agpart usb_common
CPU: 1 PID: 5250 Comm: mount.nfs Tainted: G       W        5.8.0-rc3-PowerMacG4+ #8
NIP: c04d654c LR: c04d654c CTR: 00000000
REGS: c0001198 TRAP: 0700  Tainted: G       W        (5.8.0-rc3-PowerMacG4+)
MSR:  00021031 <MR,IR,DR,RI> CR: 24028822 XER: 00000000

GPR00: c04d654c c0001250 e719b980 00000058 c01899f4 00000000 00000027 e8dc4e0c
GPR08: 00000023 00000000 00000000 c0001250 44028822 0061bff4 f8000290 00000003
GPR16: c115a340 f800028e c0001470 c0001480 c04d654c c115a260 c04d651c f800028c
GPR24: c0001464 1800028c e8dda024 c0000000 c00012f4 00000000 00000004 c00012f0
NIP [c04d654c] usercopy_abort+0x68/0x78
Unrecoverable FP Unavailable Exception 801 at 908
LR [c04d654c] usercopy_abort+0x68/0x78
Call Trace:


5.8-rc5 + the 2 patches applied:
do_IRQ: stack overflow: 1984
CPU: 1 PID: 347 Comm: gzip Tainted: G       W        5.8.0-rc5-PowerMacG4+ #1
Call Trace:

Created attachment 292337 [details]
dmesg (5.9-rc3, INLINE KASAN, PowerMac G4 DP)

Re-tried with 5.9-rc3 (inline KASAN). The original problem (stack-out-of-bounds in strcmp+0x58/0xd8) is gone, but still problems with stack usage when doing larger build jobs:

[...]
[ 1929.683510] do_IRQ: stack overflow: 1696
[ 1929.690727] CPU: 1 PID: 735 Comm: mount.nfs Tainted: G        W         5.9.0-rc3-PowerMacG4 #1
[ 1929.697847] Call Trace:
[ 1929.704633] [d0ca4670] [c0a75518] dump_stack+0xfc/0x130 (unreliable)
[ 1929.711507] [d0ca46a0] [c000b094] do_IRQ+0x128/0x180
[ 1929.717998] [d0ca46d0] [c002e560] ret_from_except+0x0/0x14
[ 1929.724652] --- interrupt: 501 at _raw_spin_unlock_irqrestore+0x3c/0xa4
                   LR = _raw_spin_unlock_irqrestore+0x38/0xa4
[ 1929.738722] [d0ca47b8] [c0a6dc90] stack_depot_save+0x20c/0x390
[ 1929.746132] [d0ca4818] [c04d4b70] kasan_save_stack+0x40/0x48
[ 1929.753675] [d0ca4928] [c04d4b9c] kasan_set_track+0x24/0x30
[ 1929.761298] [d0ca4938] [c04d710c] kasan_set_free_info+0x28/0x3c
[ 1929.769073] [d0ca4948] [c04d4f74] __kasan_slab_free+0x104/0x118
[ 1929.776983] [d0ca4968] [c04ce800] slab_free_freelist_hook+0xec/0x17c
[ 1929.785111] [d0ca49a8] [c04d3468] kmem_cache_free+0x58/0x2a0
[ 1929.793391] [d0ca49f8] [c11b251c] packet_rcv+0xb9c/0xbb4
[ 1929.801797] [d0ca4a48] [c0dbfd98] dev_queue_xmit_nit+0x6e4/0x748
[ 1929.810434] [d0ca4ab8] [c0dcaf80] dev_hard_start_xmit+0xec/0x880
[ 1929.819207] [d0ca4b18] [c0ea4814] sch_direct_xmit+0x1f8/0x818
[ 1929.828111] [d0ca4bf8] [c0dcc884] __dev_queue_xmit+0xed4/0x136c
[ 1929.837202] [d0ca4d28] [c0f256dc] ip_finish_output2+0xfcc/0x1028
[ 1929.846472] [d0ca4d88] [c0f2d848] __ip_queue_xmit+0xde0/0x1018
[ 1929.855892] [d0ca4df8] [c0f929d8] __tcp_transmit_skb+0x2550/0x2cb8
[ 1929.865486] [d0ca4ee8] [c0f98470] tcp_write_xmit+0x1d28/0x3498
[ 1929.875216] [d0ca4f78] [c0f99c8c] __tcp_push_pending_frames+0xac/0x1c4
[ 1929.885189] [d0ca4f98] [c0f5a970] tcp_sendmsg_locked+0x1c50/0x2294
[ 1929.895338] [d0ca5098] [c0f5afe4] tcp_sendmsg+0x30/0x48
[ 1929.905564] [d0ca50b8] [c0d598b0] sock_sendmsg_nosec+0xf4/0x10c
[ 1929.916463] [d0ca50d8] [b0a31840] xprt_sock_sendmsg+0x2c0/0x6e8 [sunrpc]
[ 1929.927494] [d0ca51b8] [b0a34ce8] xs_tcp_send_request+0x360/0x580 [sunrpc]
[ 1929.938699] [d0ca52e8] [b0a2eae8] xprt_transmit+0x4f8/0xe30 [sunrpc]
[ 1929.950044] [d0ca5368] [b0a1dcd8] call_transmit+0x238/0x25c [sunrpc]
[ 1929.961450] [d0ca5388] [b0a6641c] __rpc_execute+0x35c/0xbf8 [sunrpc]
[ 1929.972996] [d0ca5448] [b0a21d18] rpc_run_task+0x790/0x79c [sunrpc]
[ 1929.984850] [d0ca5498] [b1282e50] nfs4_call_sync_custom+0x14/0x80 [nfsv4]
[ 1929.996821] [d0ca54b8] [b128302c] nfs4_do_call_sync+0x170/0x1a8 [nfsv4]
[ 1930.008922] [d0ca55a8] [b12b3570] nfs4_proc_lookup_common+0x314/0xc54 [nfsv4]
[ 1930.020820] [d0ca5758] [b12b4244] nfs4_proc_lookup+0x158/0x2f0 [nfsv4]
[ 1930.032753] [d0ca57f8] [b0b49544] nfs_lookup+0x2ac/0x9ac [nfs]
[ 1930.044062] [d0ca5838] [c052c984] __lookup_slow+0x278/0x2a8
[ 1930.055461] [d0ca5958] [c05340a0] walk_component+0x288/0x30c
[ 1930.066816] [d0ca5a08] [c0534e5c] path_lookupat.isra.0+0x1b8/0x438
[ 1930.078282] [d0ca5a48] [c05372a0] filename_lookup+0x144/0x1c4
[ 1930.089834] [d0ca5b98] [c05373fc] vfs_path_lookup+0x94/0xc0
[ 1930.101389] [d0ca5c18] [c05714b8] mount_subtree+0x1c4/0x250
[ 1930.113267] [d0ca5ca8] [b12e1b2c] do_nfs4_mount+0x570/0x7fc [nfsv4]
[ 1930.125298] [d0ca5d68] [b12e202c] nfs4_try_get_tree+0xfc/0x16c [nfsv4]
[ 1930.137200] [d0ca5d88] [c050e434] vfs_get_tree+0xf8/0x398
[ 1930.149133] [d0ca5db8] [c056f968] path_mount+0x1074/0x113c
[ 1930.161107] [d0ca5e78] [c056fad8] do_mount+0xa8/0xe4
[ 1930.173109] [d0ca5f08] [c0570054] sys_mount+0xa8/0xb8
[ 1930.185160] [d0ca5f38] [c002e1cc] ret_from_syscall+0x0/0x34
[ 1930.197313] --- interrupt: c01 at 0x8b5754
                   LR = 0xac0be0
[ 1930.222896] Kernel panic - not syncing: corrupted stack end detected inside scheduler


But feel free to close this bug if appropriate as the original issue is solved.

Created attachment 292339 [details]
kernel .config (5.9-rc3, PowerMac G4 DP)

I noticed that I covered the "do_IRQ: stack overflow: ...." problem already in bug #207129 so closing this one as suggested before.