Bug 204371 - BUG kmalloc-4k (Tainted: G W ): Object padding overwritten
Summary: BUG kmalloc-4k (Tainted: G W ): Object padding overwritten
Status: NEW
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Slab Allocator (show other bugs)
Hardware: PPC-32 Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-29 22:35 UTC by Erhard F.
Modified: 2019-08-22 13:39 UTC (History)
4 users (show)

See Also:
Kernel Version: 5.3.0-rc3
Tree: Mainline
Regression: No


Attachments
dmesg (PowerMac G4 DP, kernel 5.3-rc2) (58.58 KB, text/plain)
2019-07-29 22:35 UTC, Erhard F.
Details
kernel .config (PowerMac G4 DP, kernel 5.3-rc2) (90.89 KB, text/plain)
2019-07-29 22:36 UTC, Erhard F.
Details
config_524_g4 (90.84 KB, application/octet-stream)
2019-08-01 12:47 UTC, Erhard F.
Details
dmesg (PowerMac G5 11,2, kernel 5.3-rc3) (131.53 KB, text/plain)
2019-08-06 20:48 UTC, Erhard F.
Details
kernel .config (PowerMac G5 11,2, kernel 5.3-rc3) (95.02 KB, text/plain)
2019-08-06 20:51 UTC, Erhard F.
Details
kernel .config (PowerMac G4 DP, kernel 4.18.0-rc8+, final bisect) (83.84 KB, text/plain)
2019-08-13 09:19 UTC, Erhard F.
Details
Patch to trace misaligned destination in copy_page() on PPC32 (1.14 KB, patch)
2019-08-14 08:56 UTC, Christophe Leroy
Details | Diff
Patch to trace misaligned destination in copy_page() in asm on PPC32 (440 bytes, patch)
2019-08-14 16:10 UTC, Christophe Leroy
Details | Diff
dmesg (PowerMac G4 DP, kernel 5.3-rc4 + debug patch) (165.36 KB, text/plain)
2019-08-14 17:44 UTC, Erhard F.
Details
dmesg (PowerMac G4 DP, kernel 5.3-rc4 + debug patch) (133.08 KB, text/plain)
2019-08-14 17:45 UTC, Erhard F.
Details
kernel .config (PowerMac G4 DP, kernel 5.3-rc4) (90.81 KB, text/plain)
2019-08-14 17:48 UTC, Erhard F.
Details

Description Erhard F. 2019-07-29 22:35:48 UTC
Created attachment 284033 [details]
dmesg (PowerMac G4 DP, kernel 5.3-rc2)

Seeing this during boot with SLUB_DEBUG_ON enabled in the kernel. Happens on 5.3.0-rc2, 5.2.4 is also affected. I did not test earlier kernels.

Machine is a PowerMac G4 DP (3,6), ppc32 running Gentoo Linux.

[...]
[   17.499445] =============================================================================
[   17.508472] BUG kmalloc-4k (Tainted: G        W        ): Object padding overwritten
[   17.517521] -----------------------------------------------------------------------------

[   17.535804] INFO: 0x(ptrval)-0x(ptrval). First byte 0x0 instead of 0x5a
[   17.544986] INFO: Allocated in proc_cgroup_show+0x30/0x24c age=63 cpu=0 pid=1
[   17.554078] 	__slab_alloc.constprop.73+0x40/0x6c
[   17.563007] 	kmem_cache_alloc_trace+0x7c/0x1a0
[   17.571874] 	proc_cgroup_show+0x30/0x24c
[   17.580677] 	proc_single_show+0x54/0x74
[   17.589359] 	seq_read+0x27c/0x460
[   17.597919] 	__vfs_read+0x3c/0x10c
[   17.606352] 	vfs_read+0xa8/0xf8
[   17.614656] 	ksys_read+0x7c/0xd0
[   17.622875] 	ret_from_syscall+0x0/0x34
[   17.631064] INFO: Freed in proc_cgroup_show+0xbc/0x24c age=4294882542 cpu=0 pid=0
[   17.639423] 	kfree+0x264/0x29c
[   17.647698] 	proc_cgroup_show+0xbc/0x24c
[   17.655819] 	proc_single_show+0x54/0x74
[   17.663730] 	seq_read+0x27c/0x460
[   17.671542] 	__vfs_read+0x3c/0x10c
[   17.679290] 	vfs_read+0xa8/0xf8
[   17.686990] 	ksys_read+0x7c/0xd0
[   17.694683] 	ret_from_syscall+0x0/0x34
[   17.702331] INFO: Slab 0x(ptrval) objects=7 used=7 fp=0x(ptrval) flags=0x10200
[   17.710165] INFO: Object 0x(ptrval) @offset=21408 fp=0x(ptrval)

[   17.725690] Redzone (ptrval): bb bb bb bb bb bb bb bb                          ........
[   17.733495] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.741376] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.749151] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.756811] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.764402] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.771916] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.779354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.786790] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.794226] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.801579] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.808819] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.815940] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.822914] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.829760] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.836547] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.843231] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.849810] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.856317] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.862758] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.869038] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.875111] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.881062] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.886893] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.892602] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.898248] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.903705] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.908980] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.914129] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.919216] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.924171] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.929013] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.933772] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.938444] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.942999] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.947394] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.951620] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.955736] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.959744] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.963697] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.967459] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.971032] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.974419] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.977616] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.980689] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.983620] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.986408] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.989118] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.991759] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.994377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.996931] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   17.999437] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.001892] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.004302] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.006655] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.008848] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.010879] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.012846] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.014789] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.016669] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.018500] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.020282] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.022018] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.023696] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.025223] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.026609] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.027883] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.029062] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.030085] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.031108] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.032131] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.033154] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.034177] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.035200] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.036223] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.037246] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.038269] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.039292] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.040315] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.041337] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.042360] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.043383] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.044406] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.045429] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.046452] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.047475] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.048498] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.049521] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.050544] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.051567] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.052590] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.053612] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.054635] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.055658] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.056681] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.057704] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.058727] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.059750] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.060773] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.061796] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.062819] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.063841] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.064864] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.065887] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.066910] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.067933] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.068956] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.069979] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.071002] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.072024] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.073047] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.074070] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.075093] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.076116] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.077139] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.078162] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.079185] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.080208] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.081231] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.082254] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.083277] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.084299] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.085322] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.086345] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.087368] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.088391] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.089414] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.090437] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.091460] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.092483] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.093506] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.094529] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.095552] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.096575] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.097598] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.098621] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.099643] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.100666] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.101689] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.102712] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.103735] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.104758] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.105781] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.106804] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.107826] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.108849] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.109872] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.110895] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.111918] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.112941] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.113964] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.114987] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.116010] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.117033] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.118056] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.119079] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.120102] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.121124] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.122147] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.123170] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.124193] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.125216] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.126239] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.127262] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.128285] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.129308] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.130331] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.131354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.132377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.133399] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.134422] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.135445] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.136468] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.137491] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.138514] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.139537] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.140560] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.141583] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.142605] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.143628] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.144651] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.145674] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.146697] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.147720] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.148743] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.149766] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.150789] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.151812] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.152835] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.153858] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.154880] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.155903] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.156926] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.157949] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.158972] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.159995] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.161018] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.162041] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.163064] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.164087] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.165110] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.166133] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.167156] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.168179] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.169203] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.170226] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.171249] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.172272] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.173295] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.174318] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.175341] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.176364] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.177387] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.178410] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.179433] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.180456] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.181479] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.182502] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.183525] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.184548] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.185571] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.186594] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.187617] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.188640] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.189663] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.190686] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.191709] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.192732] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.193756] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.194778] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.195801] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.196825] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.197848] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.198871] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.199894] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.200917] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.201940] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.202963] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.203986] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.205009] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.206032] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.207055] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.208078] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.209101] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.210124] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.211147] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.212169] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.213192] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.214215] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.215239] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.216262] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.217285] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.218308] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.219331] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.220354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.221377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[   18.222400] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
[   18.223429] Redzone (ptrval): bb bb bb bb                                      ....
[   18.224584] Padding (ptrval): 00 00 00 00 00 00 00 00                          ........
[   18.225813] CPU: 0 PID: 140 Comm: (md-udevd) Tainted: G    B   W         5.3.0-rc2 #4
[   18.227171] Call Trace:
[   18.228478] [ed38bc88] [c063ec6c] dump_stack+0xa0/0xfc (unreliable)
[   18.230033] [ed38bcb8] [c019cc98] check_bytes_and_report+0xc8/0xf0
[   18.231675] [ed38bce8] [c019d794] check_object+0x10c/0x224
[   18.233364] [ed38bd18] [c019e210] alloc_debug_processing+0xc4/0x13c
[   18.235168] [ed38bd38] [c019e470] ___slab_alloc.constprop.74+0x1e8/0x380
[   18.237081] [ed38bdc8] [c019e648] __slab_alloc.constprop.73+0x40/0x6c
[   18.239080] [ed38bdf8] [c01a1328] __kmalloc_track_caller+0xd8/0x1d4
[   18.241162] [ed38be38] [c016013c] kmemdup+0x28/0x5c
[   18.243286] [ed38be58] [c054dfd8] bpf_prepare_filter+0x5a8/0x688
[   18.245533] [ed38bec8] [c054e254] bpf_prog_create_from_user+0xe8/0x114
[   18.247882] [ed38bef8] [c00df0e8] do_seccomp+0x30c/0x700
[   18.250288] [ed38bf38] [c0014274] ret_from_syscall+0x0/0x34
[   18.252772] --- interrupt: c00 at 0x5292c4
                   LR = 0x7521a4
[   18.257881] FIX kmalloc-4k: Restoring 0x(ptrval)-0x(ptrval)=0x5a
[...]
Comment 1 Erhard F. 2019-07-29 22:36:48 UTC
Created attachment 284035 [details]
kernel .config (PowerMac G4 DP, kernel 5.3-rc2)
Comment 2 Andrew Morton 2019-07-30 18:52:54 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).


On Mon, 29 Jul 2019 22:35:48 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
>             Bug ID: 204371
>            Summary: BUG kmalloc-4k (Tainted: G        W        ): Object
>                     padding overwritten
>            Product: Memory Management
>            Version: 2.5
>     Kernel Version: 5.3.0-rc2
>           Hardware: PPC-32
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Slab Allocator
>           Assignee: akpm@linux-foundation.org
>           Reporter: erhard_f@mailbox.org
>         Regression: No

cc'ing various people here.

I suspect proc_cgroup_show() is innocent and that perhaps
bpf_prepare_filter() had a memory scribble.  iirc there has been at
least one recent pretty serious bpf fix applied recently.  Can others
please take a look?

(Seriously - please don't modify this report via the bugzilla web interface!)

> Created attachment 284033 [details]
>   --> https://bugzilla.kernel.org/attachment.cgi?id=284033&action=edit
> dmesg (PowerMac G4 DP, kernel 5.3-rc2)
> 
> Seeing this during boot with SLUB_DEBUG_ON enabled in the kernel. Happens on
> 5.3.0-rc2, 5.2.4 is also affected. I did not test earlier kernels.
> 
> Machine is a PowerMac G4 DP (3,6), ppc32 running Gentoo Linux.
> 
> [...]
> [   17.499445]
> =============================================================================
> [   17.508472] BUG kmalloc-4k (Tainted: G        W        ): Object padding
> overwritten
> [   17.517521]
> -----------------------------------------------------------------------------
> 
> [   17.535804] INFO: 0x(ptrval)-0x(ptrval). First byte 0x0 instead of 0x5a
> [   17.544986] INFO: Allocated in proc_cgroup_show+0x30/0x24c age=63 cpu=0
> pid=1
> [   17.554078]  __slab_alloc.constprop.73+0x40/0x6c
> [   17.563007]  kmem_cache_alloc_trace+0x7c/0x1a0
> [   17.571874]  proc_cgroup_show+0x30/0x24c
> [   17.580677]  proc_single_show+0x54/0x74
> [   17.589359]  seq_read+0x27c/0x460
> [   17.597919]  __vfs_read+0x3c/0x10c
> [   17.606352]  vfs_read+0xa8/0xf8
> [   17.614656]  ksys_read+0x7c/0xd0
> [   17.622875]  ret_from_syscall+0x0/0x34
> [   17.631064] INFO: Freed in proc_cgroup_show+0xbc/0x24c age=4294882542
> cpu=0
> pid=0
> [   17.639423]  kfree+0x264/0x29c
> [   17.647698]  proc_cgroup_show+0xbc/0x24c
> [   17.655819]  proc_single_show+0x54/0x74
> [   17.663730]  seq_read+0x27c/0x460
> [   17.671542]  __vfs_read+0x3c/0x10c
> [   17.679290]  vfs_read+0xa8/0xf8
> [   17.686990]  ksys_read+0x7c/0xd0
> [   17.694683]  ret_from_syscall+0x0/0x34
> [   17.702331] INFO: Slab 0x(ptrval) objects=7 used=7 fp=0x(ptrval)
> flags=0x10200
> [   17.710165] INFO: Object 0x(ptrval) @offset=21408 fp=0x(ptrval)
> 
> [   17.725690] Redzone (ptrval): bb bb bb bb bb bb bb bb                      
>   ........
> [   17.733495] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.741376] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.749151] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.756811] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.764402] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.771916] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.779354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.786790] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.794226] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.801579] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.808819] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.815940] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.822914] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.829760] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.836547] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.843231] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.849810] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.856317] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.862758] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.869038] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.875111] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.881062] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.886893] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.892602] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.898248] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.903705] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.908980] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.914129] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.919216] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.924171] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.929013] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.933772] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.938444] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.942999] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.947394] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.951620] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.955736] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.959744] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.963697] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.967459] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.971032] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.974419] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.977616] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.980689] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.983620] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.986408] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.989118] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.991759] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.994377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.996931] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   17.999437] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.001892] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.004302] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.006655] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.008848] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.010879] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.012846] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.014789] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.016669] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.018500] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.020282] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.022018] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.023696] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.025223] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.026609] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.027883] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.029062] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.030085] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.031108] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.032131] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.033154] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.034177] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.035200] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.036223] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.037246] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.038269] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.039292] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.040315] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.041337] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.042360] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.043383] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.044406] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.045429] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.046452] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.047475] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.048498] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.049521] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.050544] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.051567] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.052590] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.053612] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.054635] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.055658] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.056681] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.057704] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.058727] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.059750] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.060773] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.061796] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.062819] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.063841] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.064864] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.065887] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.066910] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.067933] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.068956] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.069979] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.071002] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.072024] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.073047] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.074070] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.075093] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.076116] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.077139] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.078162] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.079185] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.080208] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.081231] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.082254] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.083277] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.084299] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.085322] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.086345] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.087368] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.088391] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.089414] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.090437] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.091460] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.092483] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.093506] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.094529] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.095552] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.096575] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.097598] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.098621] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.099643] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.100666] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.101689] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.102712] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.103735] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.104758] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.105781] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.106804] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.107826] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.108849] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.109872] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.110895] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.111918] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.112941] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.113964] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.114987] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.116010] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.117033] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.118056] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.119079] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.120102] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.121124] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.122147] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.123170] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.124193] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.125216] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.126239] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.127262] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.128285] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.129308] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.130331] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.131354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.132377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.133399] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.134422] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.135445] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.136468] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.137491] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.138514] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.139537] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.140560] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.141583] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.142605] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.143628] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.144651] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.145674] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.146697] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.147720] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.148743] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.149766] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.150789] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.151812] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.152835] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.153858] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.154880] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.155903] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.156926] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.157949] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.158972] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.159995] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.161018] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.162041] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.163064] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.164087] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.165110] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.166133] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.167156] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.168179] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.169203] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.170226] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.171249] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.172272] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.173295] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.174318] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.175341] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.176364] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.177387] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.178410] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.179433] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.180456] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.181479] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.182502] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.183525] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.184548] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.185571] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.186594] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.187617] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.188640] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.189663] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.190686] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.191709] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.192732] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.193756] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.194778] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.195801] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.196825] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.197848] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.198871] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.199894] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.200917] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.201940] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.202963] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.203986] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.205009] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.206032] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.207055] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.208078] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.209101] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.210124] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.211147] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.212169] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.213192] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.214215] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.215239] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.216262] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.217285] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.218308] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.219331] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.220354] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.221377] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b
>  kkkkkkkkkkkkkkkk
> [   18.222400] Object (ptrval): 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> a5
>  kkkkkkkkkkkkkkk.
> [   18.223429] Redzone (ptrval): bb bb bb bb                                  
>   ....
> [   18.224584] Padding (ptrval): 00 00 00 00 00 00 00 00                      
>   ........
> [   18.225813] CPU: 0 PID: 140 Comm: (md-udevd) Tainted: G    B   W        
> 5.3.0-rc2 #4
> [   18.227171] Call Trace:
> [   18.228478] [ed38bc88] [c063ec6c] dump_stack+0xa0/0xfc (unreliable)
> [   18.230033] [ed38bcb8] [c019cc98] check_bytes_and_report+0xc8/0xf0
> [   18.231675] [ed38bce8] [c019d794] check_object+0x10c/0x224
> [   18.233364] [ed38bd18] [c019e210] alloc_debug_processing+0xc4/0x13c
> [   18.235168] [ed38bd38] [c019e470] ___slab_alloc.constprop.74+0x1e8/0x380
> [   18.237081] [ed38bdc8] [c019e648] __slab_alloc.constprop.73+0x40/0x6c
> [   18.239080] [ed38bdf8] [c01a1328] __kmalloc_track_caller+0xd8/0x1d4
> [   18.241162] [ed38be38] [c016013c] kmemdup+0x28/0x5c
> [   18.243286] [ed38be58] [c054dfd8] bpf_prepare_filter+0x5a8/0x688
> [   18.245533] [ed38bec8] [c054e254] bpf_prog_create_from_user+0xe8/0x114
> [   18.247882] [ed38bef8] [c00df0e8] do_seccomp+0x30c/0x700
> [   18.250288] [ed38bf38] [c0014274] ret_from_syscall+0x0/0x34
> [   18.252772] --- interrupt: c00 at 0x5292c4
>                    LR = 0x7521a4
> [   18.257881] FIX kmalloc-4k: Restoring 0x(ptrval)-0x(ptrval)=0x5a
> [...]
> 
> -- 
> You are receiving this mail because:
> You are the assignee for the bug.
Comment 3 Erhard F. 2019-07-31 09:03:45 UTC
On Tue, 30 Jul 2019 11:52:44 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> 
> On Mon, 29 Jul 2019 22:35:48 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=204371
> > 
> >             Bug ID: 204371
> >            Summary: BUG kmalloc-4k (Tainted: G        W        ): Object
> >                     padding overwritten
> >            Product: Memory Management
> >            Version: 2.5
> >     Kernel Version: 5.3.0-rc2
> >           Hardware: PPC-32
> >                 OS: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Slab Allocator
> >           Assignee: akpm@linux-foundation.org
> >           Reporter: erhard_f@mailbox.org
> >         Regression: No  
> 
> cc'ing various people here.
> 
> I suspect proc_cgroup_show() is innocent and that perhaps
> bpf_prepare_filter() had a memory scribble.  iirc there has been at
> least one recent pretty serious bpf fix applied recently.  Can others
> please take a look?
> 
> (Seriously - please don't modify this report via the bugzilla web interface!)

Hm, don't know whether this is bpfs fault.. I am getting this for other things too:

[...]
Jul 31 10:46:53 T600 kernel: Object 442ee539: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Jul 31 10:46:53 T600 kernel: Object 41b83bb9: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Jul 31 10:46:53 T600 kernel: Redzone 720e193a: bb bb bb bb                                      ....
Jul 31 10:46:53 T600 kernel: Padding 0b116c89: 00 00 00 00 00 00 00 00                          ........
Jul 31 10:46:53 T600 kernel: CPU: 1 PID: 120 Comm: systemd-journal Tainted: G    B   W         5.2.4-gentoo #1
Jul 31 10:46:53 T600 kernel: Call Trace:
Jul 31 10:46:53 T600 kernel: [dd663b68] [c0628d80] dump_stack+0xa0/0xfc (unreliable)
Jul 31 10:46:53 T600 kernel: [dd663b98] [c01984ac] check_bytes_and_report+0xc8/0xf0
Jul 31 10:46:53 T600 kernel: [dd663bc8] [c0198fd0] check_object+0x10c/0x224
Jul 31 10:46:53 T600 kernel: [dd663bf8] [c0199964] alloc_debug_processing+0xc4/0x13c
Jul 31 10:46:53 T600 kernel: [dd663c18] [c0199bc4] ___slab_alloc.constprop.72+0x1e8/0x380
Jul 31 10:46:53 T600 kernel: [dd663ca8] [c0199d9c] __slab_alloc.constprop.71+0x40/0x6c
Jul 31 10:46:53 T600 kernel: [dd663cd8] [c019a014] kmem_cache_alloc_trace+0x7c/0x170
Jul 31 10:46:53 T600 kernel: [dd663d18] [c02d6a5c] btrfs_opendir+0x48/0x78
Jul 31 10:46:53 T600 kernel: [dd663d38] [c01a9320] do_dentry_open+0x25c/0x2f0
Jul 31 10:46:53 T600 kernel: [dd663d68] [c01bc284] path_openat+0x814/0xaf0
Jul 31 10:46:53 T600 kernel: [dd663e38] [c01bc5a4] do_filp_open+0x44/0xa0
Jul 31 10:46:53 T600 kernel: [dd663ee8] [c01aa178] do_sys_open+0x7c/0x108
Jul 31 10:46:53 T600 kernel: [dd663f38] [c0015274] ret_from_syscall+0x0/0x34
Jul 31 10:46:53 T600 kernel: --- interrupt: c00 at 0x7eae14
                                 LR = 0x7eadf8
Jul 31 10:46:53 T600 kernel: FIX kmalloc-4k: Restoring 0x0b116c89-0x85f2eca1=0x5a
[...]
Comment 4 mpe 2019-07-31 12:09:54 UTC
bugzilla-daemon@bugzilla.kernel.org writes:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
>
> --- Comment #2 from Andrew Morton (akpm@linux-foundation.org) ---
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
>
> On Mon, 29 Jul 2019 22:35:48 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=204371
>> 
>>             Bug ID: 204371
>>            Summary: BUG kmalloc-4k (Tainted: G        W        ): Object
>>                     padding overwritten
>>            Product: Memory Management
>>            Version: 2.5
>>     Kernel Version: 5.3.0-rc2
>>           Hardware: PPC-32
>>                 OS: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: Slab Allocator
>>           Assignee: akpm@linux-foundation.org
>>           Reporter: erhard_f@mailbox.org
>>         Regression: No
>
> cc'ing various people here.
>
> I suspect proc_cgroup_show() is innocent and that perhaps
> bpf_prepare_filter() had a memory scribble.  iirc there has been at
> least one recent pretty serious bpf fix applied recently.  Can others
> please take a look?

I haven't been able to reproduce this on a 64-bit or 32-bit powerpc
machine here. But I don't run gentoo userspace, so I suspect I'm not
tripping the same path at boot. I did run the seccomp selftest and that
didn't trip it either.

cheers
Comment 5 Erhard F. 2019-08-01 12:47:19 UTC
Created attachment 284071 [details]
config_524_g4

On Wed, 31 Jul 2019 12:09:54 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #4 from mpe@ellerman.id.au ---
>
> > I suspect proc_cgroup_show() is innocent and that perhaps
> > bpf_prepare_filter() had a memory scribble.  iirc there has been at
> > least one recent pretty serious bpf fix applied recently.  Can others
> > please take a look?  
> 
> I haven't been able to reproduce this on a 64-bit or 32-bit powerpc
> machine here. But I don't run gentoo userspace, so I suspect I'm not
> tripping the same path at boot. I did run the seccomp selftest and that
> didn't trip it either.
> 
> cheers

Doing some fiddling around on another bug (bug #204375), I noticed that I get this "kmalloc-4k (Tainted: G W ): Object padding overwritten" during boot only when I boot from my btrfs partition, but not from my other ext4 partition. The ext4 partition is not a clone, but pretty much the same stuff in the same versions. My btrfs root is mounted with 'lazytime,compress=zstd:1', systemd is 242.

I built a 5.2.5 kernel on the Talos II with CONFIG_SLUB_DEBUG=y but here I don't hit the bug, even if I boot from a btrfs partition with the same settings. Have to test it on the G5 yet (kernel .config more similar to the G4 one than the Talos II one).
Comment 6 Erhard F. 2019-08-06 20:47:37 UTC
On Wed, 31 Jul 2019 12:09:54 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #4 from mpe@ellerman.id.au ---
> bugzilla-daemon@bugzilla.kernel.org writes:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=204371
> >
> > --- Comment #2 from Andrew Morton (akpm@linux-foundation.org) ---
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> >
> >
> > On Mon, 29 Jul 2019 22:35:48 +0000 bugzilla-daemon@bugzilla.kernel.org
> wrote:
> >  
> >> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> >> 
> >>             Bug ID: 204371
> >>            Summary: BUG kmalloc-4k (Tainted: G        W        ): Object
> >>                     padding overwritten
> >>            Product: Memory Management
> >>            Version: 2.5
> >>     Kernel Version: 5.3.0-rc2
> >>           Hardware: PPC-32
> >>                 OS: Linux
> >>               Tree: Mainline
> >>             Status: NEW
> >>           Severity: normal
> >>           Priority: P1
> >>          Component: Slab Allocator
> >>           Assignee: akpm@linux-foundation.org
> >>           Reporter: erhard_f@mailbox.org
> >>         Regression: No  
> >
> > cc'ing various people here.
> >
> > I suspect proc_cgroup_show() is innocent and that perhaps
> > bpf_prepare_filter() had a memory scribble.  iirc there has been at
> > least one recent pretty serious bpf fix applied recently.  Can others
> > please take a look?  
> 
> I haven't been able to reproduce this on a 64-bit or 32-bit powerpc
> machine here. But I don't run gentoo userspace, so I suspect I'm not
> tripping the same path at boot. I did run the seccomp selftest and that
> didn't trip it either.

Had the time to test this on my G5 11,2. It's kernel 5.3-rc3 now, also booting from a zstd:1 compressed btrfs partition. Without SLUB_DEBUG_ON selected in the kernel, the machine boots seemingly fine, with SLUB_DEBUG_ON I get this:

[...]
Aug 06 22:26:35 T800 kernel: BTRFS info (device sda7): use zstd compression, level 1
Aug 06 22:26:35 T800 kernel: BTRFS info (device sda7): disk space caching is enabled
Aug 06 22:26:38 T800 kernel: =============================================================================
Aug 06 22:26:38 T800 kernel: BUG kmalloc-4k (Tainted: G        W        ): Object padding overwritten
Aug 06 22:26:38 T800 kernel: -----------------------------------------------------------------------------
Aug 06 22:26:38 T800 kernel: INFO: 0x0000000062cd4309-0x000000004edab9d1. First byte 0x0 instead of 0x5a
Aug 06 22:26:38 T800 kernel: INFO: Slab 0x0000000070aa589a objects=7 used=7 fp=0x0000000016708aa5 flags=0x7fe00000010200
Aug 06 22:26:38 T800 kernel: INFO: Object 0x000000007ed48057 @offset=17736 fp=0x00000000b4be3601
Aug 06 22:26:38 T800 kernel: Redzone 00000000f5b164d9: bb bb bb bb bb bb bb bb                          ........
Aug 06 22:26:38 T800 kernel: Object 000000007ed48057: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[...]
Aug 06 22:26:38 T800 kernel: Redzone 00000000bd6d4c8f: bb bb bb bb bb bb bb bb                          ........
Aug 06 22:26:38 T800 kernel: Padding 0000000062cd4309: 00 00 00 00 00 00 00 00                          ........
Aug 06 22:26:38 T800 kernel: CPU: 0 PID: 118 Comm: systemd-journal Tainted: G    B   W         5.3.0-rc3 #5
Aug 06 22:26:38 T800 kernel: Call Trace:
Aug 06 22:26:38 T800 kernel: [c00000045baa72a0] [c0000000009e1a74] .dump_stack+0xe0/0x15c (unreliable)
Aug 06 22:26:38 T800 kernel: [c00000045baa7340] [c0000000002d4640] .print_trailer+0x228/0x250
Aug 06 22:26:38 T800 kernel: [c00000045baa73e0] [c0000000002c81f8] .check_bytes_and_report+0x118/0x140
Aug 06 22:26:38 T800 kernel: [c00000045baa7490] [c0000000002ca9fc] .check_object+0xcc/0x3a0
Aug 06 22:26:38 T800 kernel: [c00000045baa7540] [c0000000002cc6b8] .alloc_debug_processing+0x158/0x210
Aug 06 22:26:38 T800 kernel: [c00000045baa75d0] [c0000000002cce28] .___slab_alloc+0x6b8/0x860
Aug 06 22:26:38 T800 kernel: [c00000045baa7710] [c0000000002cd024] .__slab_alloc+0x54/0xc0
Aug 06 22:26:38 T800 kernel: [c00000045baa7790] [c0000000002cd854] .kmem_cache_alloc_trace+0x3b4/0x410
Aug 06 22:26:38 T800 kernel: [c00000045baa7840] [c0000000004b9928] .alloc_log_tree+0x38/0x140
Aug 06 22:26:38 T800 kernel: [c00000045baa78d0] [c0000000004b9ad0] .btrfs_add_log_tree+0x30/0x130
Aug 06 22:26:38 T800 kernel: [c00000045baa7960] [c000000000525624] .btrfs_log_inode_parent+0x4a4/0xeb0
Aug 06 22:26:38 T800 kernel: [c00000045baa7ae0] [c00000000052737c] .btrfs_log_dentry_safe+0x6c/0xb0
Aug 06 22:26:38 T800 kernel: [c00000045baa7b80] [c0000000004e1e3c] .btrfs_sync_file+0x1ec/0x570
Aug 06 22:26:38 T800 kernel: [c00000045baa7c90] [c000000000355ac4] .vfs_fsync_range+0x64/0xe0
Aug 06 22:26:38 T800 kernel: [c00000045baa7d20] [c000000000355ba8] .do_fsync+0x48/0xc0
Aug 06 22:26:38 T800 kernel: [c00000045baa7db0] [c000000000356028] .__se_sys_fsync+0x18/0x30
Aug 06 22:26:38 T800 kernel: [c00000045baa7e20] [c00000000000a324] system_call+0x5c/0x70
Aug 06 22:26:38 T800 kernel: FIX kmalloc-4k: Restoring 0x0000000062cd4309-0x000000004edab9d1=0x5a
[...]

Also I get:

[...]
Aug 06 22:27:53 T800 kernel: =============================================================================
Aug 06 22:27:53 T800 kernel: BUG bfq_queue (Tainted: G    B   W        ): Poison overwritten
Aug 06 22:27:53 T800 kernel: -----------------------------------------------------------------------------
Aug 06 22:27:53 T800 kernel: INFO: 0x00000000c2bbc60e-0x00000000710e6222. First byte 0x0 instead of 0x6b
Aug 06 22:27:53 T800 kernel: INFO: Allocated in .bfq_get_queue+0x27c/0x600 age=22029 cpu=1 pid=155
Aug 06 22:27:53 T800 kernel:         .__slab_alloc+0x54/0xc0
Aug 06 22:27:53 T800 kernel:         .kmem_cache_alloc_node+0xf8/0x460
Aug 06 22:27:53 T800 kernel:         .bfq_get_queue+0x27c/0x600
Aug 06 22:27:53 T800 kernel:         .bfq_init_rq+0x720/0x940
Aug 06 22:27:53 T800 kernel:         .bfq_insert_requests+0x130/0x1120
Aug 06 22:27:53 T800 kernel:         .blk_mq_sched_insert_requests+0x138/0x420
Aug 06 22:27:53 T800 kernel:         .blk_mq_flush_plug_list+0x224/0x4e0
Aug 06 22:27:53 T800 kernel:         .blk_flush_plug_list+0x128/0x170
Aug 06 22:27:53 T800 kernel:         .blk_finish_plug+0x24/0x40
Aug 06 22:27:53 T800 kernel:         .read_pages+0xa0/0x240
Aug 06 22:27:53 T800 kernel:         .__do_page_cache_readahead+0x238/0x2b0
Aug 06 22:27:53 T800 kernel:         .force_page_cache_readahead+0xbc/0x1c0
Aug 06 22:27:53 T800 kernel:         .generic_file_read_iter+0x914/0xd80
Aug 06 22:27:53 T800 kernel:         .blkdev_read_iter+0x40/0x70
Aug 06 22:27:53 T800 kernel:         .new_sync_read+0x140/0x1c0
Aug 06 22:27:53 T800 kernel:         .vfs_read+0xb0/0x1b0
Aug 06 22:27:53 T800 kernel: INFO: Freed in .bfq_put_queue+0xc4/0x100 age=21892 cpu=0 pid=143
Aug 06 22:27:53 T800 kernel:         .kmem_cache_free+0x52c/0x530
Aug 06 22:27:53 T800 kernel:         .bfq_put_queue+0xc4/0x100
Aug 06 22:27:53 T800 kernel:         .bfq_put_idle_entity+0x74/0xc0
Aug 06 22:27:53 T800 kernel:         .bfq_bfqq_served+0xc4/0x120
Aug 06 22:27:53 T800 kernel:         .bfq_dispatch_request+0x344/0xbd0
Aug 06 22:27:53 T800 kernel:         .blk_mq_do_dispatch_sched+0x104/0x180
Aug 06 22:27:53 T800 kernel:         .blk_mq_sched_dispatch_requests+0x144/0x230
Aug 06 22:27:53 T800 kernel:         .__blk_mq_run_hw_queue+0xa4/0x140
Aug 06 22:27:53 T800 kernel:         .__blk_mq_delay_run_hw_queue+0x234/0x240
Aug 06 22:27:53 T800 kernel:         .blk_mq_run_hw_queue+0xac/0x130
Aug 06 22:27:53 T800 kernel:         .blk_mq_sched_insert_requests+0x190/0x420
Aug 06 22:27:53 T800 kernel:         .blk_mq_flush_plug_list+0x224/0x4e0
Aug 06 22:27:53 T800 kernel:         .blk_flush_plug_list+0x128/0x170
Aug 06 22:27:53 T800 kernel:         .blk_finish_plug+0x24/0x40
Aug 06 22:27:53 T800 kernel:         .read_pages+0xa0/0x240
Aug 06 22:27:53 T800 kernel:         .__do_page_cache_readahead+0x238/0x2b0
Aug 06 22:27:53 T800 kernel: INFO: Slab 0x00000000559e0a9c objects=19 used=19 fp=0x0000000016708aa5 flags=0x7fe00000010200
Aug 06 22:27:53 T800 kernel: INFO: Object 0x00000000d181f14b @offset=8 fp=0x0000000035f5f997
Aug 06 22:27:53 T800 kernel: Redzone 000000006c7b1db8: bb bb bb bb bb bb bb bb                          ........
Aug 06 22:27:53 T800 kernel: Object 00000000d181f14b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000f4600676: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000009ecde695: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000007dfb2519: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000004c46d89f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000b68dc230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000b6fcf14d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000f3752aca: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000007662c42e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 0000000086080f07: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000003df14b51: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000769dc0ba: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000006f036f9c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000005fbbe251: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000001c3da628: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000003535f2cc: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 000000006c4f0b17: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000aa181422: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000e632967b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 0000000083919b29: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000ae24557c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000dc2cc57d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000995c45ac: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000632e218e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000c0c20784: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000f48aad9c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000f5449c05: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000146f6d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000e78d4c0d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 0000000038d3f642: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Aug 06 22:27:53 T800 kernel: Object 00000000c9784ba9: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00  kkkkkkkk........
Aug 06 22:27:53 T800 kernel: Object 00000000d0fb292a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Aug 06 22:27:53 T800 kernel: Redzone 0000000033bfd673: bb bb bb bb bb bb bb bb                          ........
Aug 06 22:27:53 T800 kernel: Padding 00000000833b50bf: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Aug 06 22:27:53 T800 kernel: CPU: 0 PID: 284 Comm: (direxec) Tainted: G    B   W         5.3.0-rc3 #5
Aug 06 22:27:53 T800 kernel: Call Trace:
Aug 06 22:27:53 T800 kernel: [c00000045d93ea30] [c0000000009e1a74] .dump_stack+0xe0/0x15c (unreliable)
Aug 06 22:27:53 T800 kernel: [c00000045d93ead0] [c0000000002d4640] .print_trailer+0x228/0x250
Aug 06 22:27:53 T800 kernel: [c00000045d93eb70] [c0000000002c81f8] .check_bytes_and_report+0x118/0x140
Aug 06 22:27:53 T800 kernel: [c00000045d93ec20] [c0000000002cac48] .check_object+0x318/0x3a0
Aug 06 22:27:53 T800 kernel: [c00000045d93ecd0] [c0000000002cc6b8] .alloc_debug_processing+0x158/0x210
Aug 06 22:27:53 T800 kernel: [c00000045d93ed60] [c0000000002cce28] .___slab_alloc+0x6b8/0x860
Aug 06 22:27:53 T800 kernel: [c00000045d93eea0] [c0000000002cd024] .__slab_alloc+0x54/0xc0
Aug 06 22:27:53 T800 kernel: [c00000045d93ef20] [c0000000002cda98] .kmem_cache_alloc_node+0xf8/0x460
Aug 06 22:27:53 T800 kernel: [c00000045d93efd0] [c00000000062a53c] .bfq_get_queue+0x27c/0x600
Aug 06 22:27:53 T800 kernel: [c00000045d93f0a0] [c00000000062d80c] .bfq_init_rq+0x43c/0x940
Aug 06 22:27:53 T800 kernel: [c00000045d93f180] [c00000000062e0c0] .bfq_insert_requests+0x130/0x1120
Aug 06 22:27:53 T800 kernel: [c00000045d93f2e0] [c000000000606118] .blk_mq_sched_insert_requests+0x138/0x420
Aug 06 22:27:53 T800 kernel: [c00000045d93f390] [c0000000005ff2f4] .blk_mq_flush_plug_list+0x224/0x4e0
Aug 06 22:27:53 T800 kernel: [c00000045d93f490] [c0000000005ef978] .blk_flush_plug_list+0x128/0x170
Aug 06 22:27:53 T800 kernel: [c00000045d93f550] [c0000000005ef9e4] .blk_finish_plug+0x24/0x40
Aug 06 22:27:53 T800 kernel: [c00000045d93f5c0] [c000000000234fc0] .read_pages+0xa0/0x240
Aug 06 22:27:53 T800 kernel: [c00000045d93f6b0] [c000000000235398] .__do_page_cache_readahead+0x238/0x2b0
Aug 06 22:27:53 T800 kernel: [c00000045d93f7b0] [c0000000002356f8] .ondemand_readahead+0x2e8/0x640
Aug 06 22:27:53 T800 kernel: [c00000045d93f870] [c000000000224fb4] .generic_file_read_iter+0x914/0xd80
Aug 06 22:27:53 T800 kernel: [c00000045d93f9f0] [c0000000002fd7a0] .new_sync_read+0x140/0x1c0
Aug 06 22:27:53 T800 kernel: [c00000045d93fae0] [c000000000300490] .vfs_read+0xb0/0x1b0
Aug 06 22:27:53 T800 kernel: [c00000045d93fb80] [c0000000003005d8] .kernel_read+0x48/0x80
Aug 06 22:27:53 T800 kernel: [c00000045d93fc00] [c000000000309bc4] .prepare_binprm+0x194/0x210
Aug 06 22:27:53 T800 kernel: [c00000045d93fca0] [c00000000030b3d4] .__do_execve_file.isra.46+0x6c4/0xca0
Aug 06 22:27:53 T800 kernel: [c00000045d93fda0] [c00000000030c948] .__se_sys_execve+0x48/0x60
Aug 06 22:27:53 T800 kernel: [c00000045d93fe20] [c00000000000a324] system_call+0x5c/0x70
Aug 06 22:27:53 T800 kernel: FIX bfq_queue: Restoring 0x00000000c2bbc60e-0x00000000710e6222=0x6b
Aug 06 22:27:53 T800 kernel: FIX bfq_queue: Marking all objects used
[...]

On the G4 DP I use a SSD with kyber scheduler, on the G5 it's a HDD with bfq.
Comment 7 Erhard F. 2019-08-06 20:48:11 UTC
Created attachment 284241 [details]
dmesg (PowerMac G5 11,2, kernel 5.3-rc3)
Comment 8 Erhard F. 2019-08-06 20:51:50 UTC
Created attachment 284243 [details]
kernel .config (PowerMac G5 11,2, kernel 5.3-rc3)
Comment 9 David Sterba 2019-08-08 10:53:23 UTC
I've hit the same problem, on x86_64.
Comment 10 David Sterba 2019-08-08 11:26:58 UTC
In my case it happened on 5.3-rc3, with a strestest. The same machine has been running fstests periodically, with slab debug on, but there are no slab reports like that.

[ 8516.870046] BUG kmalloc-4k (Not tainted): Poison overwritten                                                                                                                                                 
[ 8516.875873] -----------------------------------------------------------------------------                                                                                                                    
                                                                                                                                                                                                                
[ 8516.885864] Disabling lock debugging due to kernel taint                                                                                                                                                     
[ 8516.891312] INFO: 0x000000001c70c8c9-0x000000003cd1e164. First byte 0x16 instead of 0x6b                                                                                                                     
[ 8516.899717] INFO: Allocated in btrfs_read_tree_root+0x46/0x120 [btrfs] age=1769 cpu=7 pid=8717                                                                                                               
[ 8516.908544]  __slab_alloc.isra.53+0x3e/0x70                                                                                                                                                                  
[ 8516.912861]  kmem_cache_alloc_trace+0x1b0/0x330                                                                                                                                                              
[ 8516.917581]  btrfs_read_tree_root+0x46/0x120 [btrfs]                                                                                                                                                         
[ 8516.922737]  btrfs_read_fs_root+0xe/0x40 [btrfs]                                                                                                                                                             
[ 8516.927552]  create_reloc_root+0x17f/0x2a0 [btrfs]                                                                                                                                                           
[ 8516.932536]  btrfs_init_reloc_root+0x72/0xe0 [btrfs]                                                                                                                                                         
[ 8516.937686]  record_root_in_trans+0xbb/0xf0 [btrfs]                                                                                                                                                          
[ 8516.942750]  btrfs_record_root_in_trans+0x50/0x70 [btrfs]                                                                                                                                                    
[ 8516.948340]  start_transaction+0xa1/0x550 [btrfs]                                                                                                                                                            
[ 8516.953237]  __btrfs_prealloc_file_range+0xca/0x490 [btrfs]                                                                                                                                                  
[ 8516.959003]  btrfs_prealloc_file_range+0x10/0x20 [btrfs]                                                                                                                                                     
[ 8516.964509]  prealloc_file_extent_cluster+0x13e/0x2b0 [btrfs]                                                                                                                                                
[ 8516.970447]  relocate_file_extent_cluster+0x8d/0x530 [btrfs]                                                                                                                                                 
[ 8516.976305]  relocate_data_extent+0x80/0x110 [btrfs]                                                                                                                                                         
[ 8516.981469]  relocate_block_group+0x473/0x720 [btrfs]                                                                                                                                                        
[ 8516.986711]  btrfs_relocate_block_group+0x15f/0x2c0 [btrfs]                                                                                                                                                  
[ 8516.992470] INFO: Freed in btrfs_drop_snapshot+0x832/0xbb0 [btrfs] age=331 cpu=5 pid=8717                                                                                                                    
[ 8517.000865]  kfree+0x29a/0x2d0                                                                                                                                                                               
[ 8517.004098]  btrfs_drop_snapshot+0x832/0xbb0 [btrfs]                                                                                                                                                         
[ 8517.009279]  clean_dirty_subvols+0xf7/0x120 [btrfs]                                                                                                                                                          
[ 8517.014369]  relocate_block_group+0x25a/0x720 [btrfs]                                                                                                                                                        
[ 8517.019616]  btrfs_relocate_block_group+0x15f/0x2c0 [btrfs]                                                                                                                                                  
[ 8517.025385]  btrfs_relocate_chunk+0x49/0x100 [btrfs]                                                                                                                                                         
[ 8517.030557]  __btrfs_balance+0xa00/0xdb0 [btrfs]                                                                                                                                                             
[ 8517.035365]  btrfs_balance+0x3b8/0xbb0 [btrfs]                                                                                                                                                               
[ 8517.040011]  btrfs_ioctl_balance+0x2d5/0x380 [btrfs]                                                                                                                                                         
[ 8517.045176]  btrfs_ioctl+0x16db/0x3460 [btrfs]                                                                                                                                                               
[ 8517.049772]  do_vfs_ioctl+0xa5/0x710                                                                                                                                                                         
[ 8517.053491]  ksys_ioctl+0x70/0x80                                                                                                                                                                            
[ 8517.056958]  __x64_sys_ioctl+0x16/0x20                                                                                                                                                                       
[ 8517.060845]  do_syscall_64+0x5c/0x1d0                                                                                                                                                                        
[ 8517.064650]  entry_SYSCALL_64_after_hwframe+0x49/0xbe

[ 8518.630509] INFO: 0x00000000088ac804-0x00000000600f3eff. First byte 0x17 instead of 0x6b                                                                                                                     
[ 8518.640015] Object 0000000064763fee: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.650047] INFO: Allocated in btrfs_read_tree_root+0x46/0x120 [btrfs] age=2298 cpu=4 pid=8634                                                                                                               
[ 8518.658240] Object 000000001d16ab39: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.667744]  __slab_alloc.isra.53+0x3e/0x70                                                                                                                                                                  
[ 8518.667751]  kmem_cache_alloc_trace+0x1b0/0x330                                                                                                                                                              
[ 8518.676569] Object 000000000f5b2c4b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.686125]  btrfs_read_tree_root+0x46/0x120 [btrfs]                                                                                                                                                         
[ 8518.686186]  btrfs_read_fs_root+0xe/0x40 [btrfs]                                                                                                                                                             
[ 8518.690444] Object 000000000e589530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.695159]  create_reloc_root+0x17f/0x2a0 [btrfs]                                                                                                                                                           
[ 8518.695226]  btrfs_init_reloc_root+0x72/0xe0 [btrfs]                                                                                                                                                         
[ 8518.704680] Object 00000000e3821ddd: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.709851]  record_root_in_trans+0xbb/0xf0 [btrfs]                                                                                                                                                          
[ 8518.709912]  btrfs_record_root_in_trans+0x50/0x70 [btrfs]                                                                                                                                                    
[ 8518.714606] Object 000000009552602b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.724164]  start_transaction+0xa1/0x550 [btrfs]                                                                                                                                                            
[ 8518.724225]  btrfs_start_transaction_fallback_global_rsv+0x34/0x1f0 [btrfs]                                                                                                                                  
[ 8518.729096] Object 00000000048bc005: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.734242]  btrfs_unlink+0x34/0xd0 [btrfs]                                                                                                                                                                  
[ 8518.734251]  vfs_unlink+0x106/0x1f0                                                                                                                                                                          
[ 8518.743763] Object 00000000e803d7b6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.748767]  do_unlinkat+0x2bf/0x330                                                                                                                                                                         
[ 8518.748775]  do_syscall_64+0x5c/0x1d0                                                                                                                                                                        
[ 8518.754301] Object 00000000774a30d7: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.763804]  entry_SYSCALL_64_after_hwframe+0x49/0xbe                                                                                                                                                        
[ 8518.763864] INFO: Freed in btrfs_drop_snapshot+0x832/0xbb0 [btrfs] age=746 cpu=5 pid=8717                                                                                                                    
[ 8518.768641] Object 000000007b92411f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.775730]  kfree+0x29a/0x2d0                                                                                                                                                                               
[ 8518.775789]  btrfs_drop_snapshot+0x832/0xbb0 [btrfs]                                                                                                                                                         
[ 8518.785253] Object 00000000ae532d5f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.789626]  clean_dirty_subvols+0xf7/0x120 [btrfs]                                                                                                                                                          
[ 8518.789693]  relocate_block_group+0x25a/0x720 [btrfs]                                                                                                                                                        
[ 8518.793253] Object 000000002df294e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.802820]  btrfs_relocate_block_group+0x15f/0x2c0 [btrfs]                                                                                                                                                  
[ 8518.802886]  btrfs_relocate_chunk+0x49/0x100 [btrfs]                                                                                                                                                         
[ 8518.806528] Object 00000000df2dd63a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.810370]  __btrfs_balance+0xa00/0xdb0 [btrfs]                                                                                                                                                             
[ 8518.810437]  btrfs_balance+0x3b8/0xbb0 [btrfs]                                                                                                                                                               
[ 8518.819894] Object 00000000682d1c71: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.825135]  btrfs_ioctl_balance+0x2d5/0x380 [btrfs]                                                                                                                                                         
[ 8518.825202]  btrfs_ioctl+0x16db/0x3460 [btrfs]                                                                                                                                                               
[ 8518.833522] Object 00000000eb8c2c61: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.843038]  do_vfs_ioctl+0xa5/0x710                                                                                                                                                                         
[ 8518.843044]  ksys_ioctl+0x70/0x80                                                                                                                                                                            
[ 8518.846228] Object 00000000574d97aa: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.851322]  __x64_sys_ioctl+0x16/0x20                                                                                                                                                                       
[ 8518.851329]  do_syscall_64+0x5c/0x1d0                                                                                                                                                                        
[ 8518.860844] Object 00000000a5c7d1b2: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.865870]  entry_SYSCALL_64_after_hwframe+0x49/0xbe                                                                                                                                                        
[ 8518.865876] INFO: Slab 0x000000001ef6adf1 objects=7 used=7 fp=0x00000000b9747429 flags=0x3ffff000010200                                                                                                      
[ 8518.871058] Object 000000009c9435a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk                                                                                                       
[ 8518.880559] INFO: Object 0x000000003bdbade7 @offset=8872 fp=0x00000000b9747429

[ 8522.364211] Redzone 00000000be2e5096: bb bb bb bb bb bb bb bb                          ........                                                                                                              
[ 8522.364214] Padding 000000005d4fac5d: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ                                                                                                              
[ 8522.364228] CPU: 3 PID: 2817 Comm: tmux Tainted: G    B             5.3.0-rc3-1.ge195904-vanilla+ #474                                                                                                       
[ 8522.429558] Hardware name: empty empty/S3993, BIOS PAQEX0-3 02/24/2008                                                                                                                                       
[ 8522.429561] Call Trace:                                                                                                                                                                                      
[ 8522.429581]  dump_stack+0x67/0x9b                                                                                                                                                                            
[ 8522.444139]  check_bytes_and_report+0xc9/0xf0                                                                                                                                                                
[ 8522.444149]  check_object+0x284/0x330                                                                                                                                                                        
[ 8522.444157]  ? __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                          
[ 8522.444163]  ? __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                          
[ 8522.444169]  alloc_debug_processing+0x197/0x200                                                                                                                                                              
[ 8522.444178]  ___slab_alloc+0x500/0x620                                                                                                                                                                       
[ 8522.470992]  ? __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                          
[ 8522.471005]  ? stack_trace_save+0x70/0x70                                                                                                                                                                    
[ 8522.480156]  ? __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                          
[ 8522.480162]  ? __slab_alloc.isra.53+0x3e/0x70                                                                                                                                                                
[ 8522.489651]  __slab_alloc.isra.53+0x3e/0x70                                                                                                                                                                  
[ 8522.489676]  ? __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                          
[ 8522.489680]  __kmalloc+0x25e/0x370                                                                                                                                                                           
[ 8522.489689]  __tty_buffer_request_room+0x94/0x1a0                                                                                                                                                            
[ 8522.507369]  tty_insert_flip_string_fixed_flag+0x57/0x130                                                                                                                                                    
[ 8522.507412]  pty_write+0x52/0x90                                                                                                                                                                             
[ 8522.507421]  n_tty_write+0x402/0x4f0                                                                                                                                                                         
[ 8522.507433]  ? do_wait_intr_irq+0xe0/0xe0                                                                                                                                                                    
[ 8522.507443]  tty_write+0x1a3/0x350                                                                                                                                                                           
[ 8522.507450]  ? process_echoes+0x60/0x60                                                                                                                                                                      
[ 8522.507459]  do_iter_write+0x182/0x1f0                                                                                                                                                                       
[ 8522.507466]  ? import_iovec+0x8b/0xb0                                                                                                                                                                        
[ 8522.507473]  vfs_writev+0x92/0x120                                                                                                                                                                           
[ 8522.507497]  ? do_writev+0xde/0x130                                                                                                                                                                          
[ 8522.546489]  do_writev+0xde/0x130                                                                                                                                                                            
[ 8522.546500]  do_syscall_64+0x5c/0x1d0                                                                                                                                                                        
[ 8522.553740]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
Comment 11 mpe 2019-08-09 12:31:26 UTC
bugzilla-daemon@bugzilla.kernel.org writes:
> https://bugzilla.kernel.org/show_bug.cgi?id=204371
>
> --- Comment #10 from David Sterba (dsterba@suse.com) ---
> In my case it happened on 5.3-rc3, with a strestest. The same machine has
> been
> running fstests periodically, with slab debug on, but there are no slab
> reports
> like that.
>
> [ 8516.870046] BUG kmalloc-4k (Not tainted): Poison overwritten               
> [ 8516.875873]
> ----------------------------------------------------------------------------- 
>
> [ 8516.885864] Disabling lock debugging due to kernel taint                   
> [ 8516.891312] INFO: 0x000000001c70c8c9-0x000000003cd1e164. First byte 0x16
> instead of 0x6b                                                               
> [ 8516.899717] INFO: Allocated in btrfs_read_tree_root+0x46/0x120 [btrfs]
> age=1769 cpu=7 pid=8717                                                       
> [ 8516.908544]  __slab_alloc.isra.53+0x3e/0x70                                
> [ 8516.912861]  kmem_cache_alloc_trace+0x1b0/0x330                            
> [ 8516.917581]  btrfs_read_tree_root+0x46/0x120 [btrfs]                       
> [ 8516.922737]  btrfs_read_fs_root+0xe/0x40 [btrfs]                           
> [ 8516.927552]  create_reloc_root+0x17f/0x2a0 [btrfs]                         
> [ 8516.932536]  btrfs_init_reloc_root+0x72/0xe0 [btrfs]                       
> [ 8516.937686]  record_root_in_trans+0xbb/0xf0 [btrfs]                        
> [ 8516.942750]  btrfs_record_root_in_trans+0x50/0x70 [btrfs]                  
> [ 8516.948340]  start_transaction+0xa1/0x550 [btrfs]                          
> [ 8516.953237]  __btrfs_prealloc_file_range+0xca/0x490 [btrfs]                
> [ 8516.959003]  btrfs_prealloc_file_range+0x10/0x20 [btrfs]                   
> [ 8516.964509]  prealloc_file_extent_cluster+0x13e/0x2b0 [btrfs]              
> [ 8516.970447]  relocate_file_extent_cluster+0x8d/0x530 [btrfs]               
> [ 8516.976305]  relocate_data_extent+0x80/0x110 [btrfs]                       
> [ 8516.981469]  relocate_block_group+0x473/0x720 [btrfs]                      
> [ 8516.986711]  btrfs_relocate_block_group+0x15f/0x2c0 [btrfs]                

So this is looking more like it could be a btrfs bug, given you've both
hit it using btrfs but on different platforms.

cheers
Comment 12 Erhard F. 2019-08-11 20:03:02 UTC
On Fri, 09 Aug 2019 12:31:26 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 

Tried a few LTS kernels on the G4 DP. Looks like 4.19.x is affected (tested 4.19.66) whereas 4.14.x (tested 4.14.138) is not.

Also found a way to trigger the bug without the need of a btrfs root partition:
btrfs still built into the kernel. Mount another btrfs partition via /etc/fstab at boot, e.g.
LABEL="tmp"		/var/tmp/portage	btrfs	compress=lzo,noatime	0 1

Mounting /var/tmp/portage in my case works without problems. But I reliably get the BUG kmalloc-4k at unmounting /var/tmp/portage.

I'll try to bisect the next few days and report back.
Comment 13 Erhard F. 2019-08-11 21:18:11 UTC
On Fri, 09 Aug 2019 12:31:26 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
[...]
[   22.809365] =============================================================================
[   22.809700] BUG kmalloc-4096 (Tainted: G        W        ): Redzone overwritten
[   22.809971] -----------------------------------------------------------------------------

[   22.810286] INFO: 0xbe1a5921-0xfbfc06cd. First byte 0x0 instead of 0xcc
[   22.810866] INFO: Allocated in __load_free_space_cache+0x588/0x780 [btrfs] age=22 cpu=0 pid=224
[   22.811193] 	__slab_alloc.constprop.26+0x44/0x70
[   22.811345] 	kmem_cache_alloc_trace+0xf0/0x2ec
[   22.811588] 	__load_free_space_cache+0x588/0x780 [btrfs]
[   22.811848] 	load_free_space_cache+0xf4/0x1b0 [btrfs]
[   22.812090] 	cache_block_group+0x1d0/0x3d0 [btrfs]
[   22.812321] 	find_free_extent+0x680/0x12a4 [btrfs]
[   22.812549] 	btrfs_reserve_extent+0xec/0x220 [btrfs]
[   22.812785] 	btrfs_alloc_tree_block+0x178/0x5f4 [btrfs]
[   22.813032] 	__btrfs_cow_block+0x150/0x5d4 [btrfs]
[   22.813262] 	btrfs_cow_block+0x194/0x298 [btrfs]
[   22.813484] 	commit_cowonly_roots+0x44/0x294 [btrfs]
[   22.813718] 	btrfs_commit_transaction+0x63c/0xc0c [btrfs]
[   22.813973] 	close_ctree+0xf8/0x2a4 [btrfs]
[   22.814107] 	generic_shutdown_super+0x80/0x110
[   22.814250] 	kill_anon_super+0x18/0x30
[   22.814437] 	btrfs_kill_super+0x18/0x90 [btrfs]
[   22.814590] INFO: Freed in proc_cgroup_show+0xc0/0x248 age=41 cpu=0 pid=83
[   22.814841] 	proc_cgroup_show+0xc0/0x248
[   22.814967] 	proc_single_show+0x54/0x98
[   22.815086] 	seq_read+0x278/0x45c
[   22.815190] 	__vfs_read+0x28/0x17c
[   22.815289] 	vfs_read+0xa8/0x14c
[   22.815381] 	ksys_read+0x50/0x94
[   22.815475] 	ret_from_syscall+0x0/0x38
[   22.815593] INFO: Slab 0x6b5768ec objects=7 used=7 fp=0x  (null) flags=0x8101
[   22.815854] INFO: Object 0x6eefea7d @offset=17128 fp=0x  (null)

[   22.816063] Redzone be1a5921: 00 00 00 00 00 00 00 00                          ........
[   22.816354] Object 6eefea7d: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[...]
[   23.715311] Object ea0b92e7: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[   23.718376] Redzone a1d8f890: cc cc cc cc                                      ....
[   23.721607] Padding d4007128: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[   23.724958] CPU: 0 PID: 224 Comm: umount Tainted: G    B   W         4.19.0 #1
[   23.728433] Call Trace:
[   23.731847] [ec525cc0] [c053ca68] dump_stack+0xa4/0x100 (unreliable)
[   23.735595] [ec525ce0] [c019b21c] check_bytes_and_report+0xc8/0xf0
[   23.739445] [ec525d10] [c019bf44] check_object+0x50/0x278
[   23.743339] [ec525d30] [c019e4c4] free_debug_processing+0x200/0x318
[   23.747341] [ec525d70] [c019e7b4] __slab_free+0x1d8/0x440
[   23.751591] [ec525df0] [f3c34854] free_bitmap+0x24/0x68 [btrfs]
[   23.755906] [ec525e00] [f3c35a28] __btrfs_remove_free_space_cache_locked+0x68/0x6c [btrfs]
[   23.760481] [ec525e20] [f3c38de8] btrfs_remove_free_space_cache+0x38/0x84 [btrfs]
[   23.765173] [ec525e40] [f3bc7408] btrfs_free_block_groups+0x218/0x2f0 [btrfs]
[   23.769993] [ec525e70] [f3bde164] close_ctree+0x200/0x2a4 [btrfs]
[   23.774824] [ec525eb0] [c01b6534] generic_shutdown_super+0x80/0x110
[   23.779750] [ec525ec0] [c01b678c] kill_anon_super+0x18/0x30
[   23.784852] [ec525ed0] [f3baec88] btrfs_kill_super+0x18/0x90 [btrfs]
[   23.790012] [ec525ee0] [c01b6cd8] deactivate_locked_super+0x54/0xa4
[   23.795258] [ec525ef0] [c01d5db8] cleanup_mnt+0x50/0x78
[   23.800575] [ec525f00] [c0055cac] task_work_run+0xa4/0xc4
[   23.805994] [ec525f30] [c000b658] do_notify_resume+0xcc/0x108
[   23.811478] [ec525f40] [c00146bc] do_user_signal+0x2c/0x34
[   23.817049] --- interrupt: c00 at 0x8d43d4
                   LR = 0x8d43b8
[   23.828287] FIX kmalloc-4096: Restoring 0xbe1a5921-0xfbfc06cd=0xcc

[   23.840295] FIX kmalloc-4096: Object at 0x6eefea7d not freed
[   23.846788] =============================================================================
[   23.852638] BUG kmalloc-4096 (Tainted: G    B   W        ): Redzone overwritten
[   23.858590] -----------------------------------------------------------------------------

[   23.870891] INFO: 0xad3f3ec9-0x8e4e748e. First byte 0x0 instead of 0xcc
[   23.877502] INFO: Allocated in __load_free_space_cache+0x588/0x780 [btrfs] age=333 cpu=0 pid=224
[   23.884297] 	__slab_alloc.constprop.26+0x44/0x70
[   23.891119] 	kmem_cache_alloc_trace+0xf0/0x2ec
[   23.898100] 	__load_free_space_cache+0x588/0x780 [btrfs]
[   23.905235] 	load_free_space_cache+0xf4/0x1b0 [btrfs]
[   23.912417] 	cache_block_group+0x1d0/0x3d0 [btrfs]
[   23.919721] 	find_free_extent+0x680/0x12a4 [btrfs]
[   23.927070] 	btrfs_reserve_extent+0xec/0x220 [btrfs]
[   23.934474] 	btrfs_alloc_tree_block+0x178/0x5f4 [btrfs]
[   23.942024] 	__btrfs_cow_block+0x150/0x5d4 [btrfs]
[   23.949627] 	btrfs_cow_block+0x194/0x298 [btrfs]
[   23.957351] 	commit_cowonly_roots+0x44/0x294 [btrfs]
[   23.965154] 	btrfs_commit_transaction+0x63c/0xc0c [btrfs]
[   23.973073] 	close_ctree+0xf8/0x2a4 [btrfs]
[   23.980977] 	generic_shutdown_super+0x80/0x110
[   23.988999] 	kill_anon_super+0x18/0x30
[   23.997063] 	btrfs_kill_super+0x18/0x90 [btrfs]
[   24.005191] INFO: Freed in seq_release+0x1c/0x38 age=352 cpu=1 pid=1
[   24.013500] 	seq_release+0x1c/0x38
[   24.021894] 	kernfs_fop_release+0x74/0x90
[   24.030337] 	__fput+0x104/0x1e4
[   24.038822] 	task_work_run+0xa4/0xc4
[   24.047320] 	do_notify_resume+0xcc/0x108
[   24.055936] 	do_user_signal+0x2c/0x34
[   24.064520] INFO: Slab 0x7ec9c2e3 objects=7 used=6 fp=0xbc375e23 flags=0x8101
[   24.073478] INFO: Object 0x8564a246 @offset=17128 fp=0x  (null)

[   24.091483] Redzone ad3f3ec9: 00 00 00 00 00 00 00 00                          ........
[   24.100772] Object 8564a246: f0 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00  ................
[...]
[   25.242900] Object 5560df93: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[   25.245595] Redzone 4cfc344b: cc cc cc cc                                      ....
[   25.248446] Padding 399de3f9: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[   25.251412] CPU: 0 PID: 224 Comm: umount Tainted: G    B   W         4.19.0 #1
[   25.254501] Call Trace:
[   25.257513] [ec525cc0] [c053ca68] dump_stack+0xa4/0x100 (unreliable)
[   25.260807] [ec525ce0] [c019b21c] check_bytes_and_report+0xc8/0xf0
[   25.264180] [ec525d10] [c019bf44] check_object+0x50/0x278
[   25.267620] [ec525d30] [c019e4c4] free_debug_processing+0x200/0x318
[   25.271174] [ec525d70] [c019e7b4] __slab_free+0x1d8/0x440
[   25.274931] [ec525df0] [f3c34854] free_bitmap+0x24/0x68 [btrfs]
[   25.278720] [ec525e00] [f3c35a28] __btrfs_remove_free_space_cache_locked+0x68/0x6c [btrfs]
[   25.282776] [ec525e20] [f3c38de8] btrfs_remove_free_space_cache+0x38/0x84 [btrfs]
[   25.286969] [ec525e40] [f3bc7408] btrfs_free_block_groups+0x218/0x2f0 [btrfs]
[   25.291230] [ec525e70] [f3bde164] close_ctree+0x200/0x2a4 [btrfs]
[   25.295473] [ec525eb0] [c01b6534] generic_shutdown_super+0x80/0x110
[   25.299835] [ec525ec0] [c01b678c] kill_anon_super+0x18/0x30
[   25.304360] [ec525ed0] [f3baec88] btrfs_kill_super+0x18/0x90 [btrfs]
[   25.308936] [ec525ee0] [c01b6cd8] deactivate_locked_super+0x54/0xa4
[   25.313590] [ec525ef0] [c01d5db8] cleanup_mnt+0x50/0x78
[   25.318277] [ec525f00] [c0055cac] task_work_run+0xa4/0xc4
[   25.323064] [ec525f30] [c000b658] do_notify_resume+0xcc/0x108
[   25.327903] [ec525f40] [c00146bc] do_user_signal+0x2c/0x34
[   25.332836] --- interrupt: c00 at 0x8d43d4
                   LR = 0x8d43b8
[   25.342792] FIX kmalloc-4096: Restoring 0xad3f3ec9-0x8e4e748e=0xcc

[   25.353647] FIX kmalloc-4096: Object at 0x8564a246 not freed
Comment 14 Erhard F. 2019-08-13 09:19:12 UTC
Created attachment 284353 [details]
kernel .config (PowerMac G4 DP, kernel 4.18.0-rc8+, final bisect)
Comment 15 Erhard F. 2019-08-13 09:20:03 UTC
On Fri, 09 Aug 2019 12:31:26 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
# cat ~/bisect01.log 
binäre Suche: danach noch 37903 Commits zum Testen übrig (ungefähr 15 Schritte)
[9abf8acea297b4c65f5fa3206e2b8e468e730e84] Merge tag 'tty-4.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
binäre Suche: danach noch 19051 Commits zum Testen übrig (ungefähr 14 Schritte)
[7c00e8ae041b349992047769af741b67379ce19a] Merge tag 'armsoc-soc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
binäre Suche: danach noch 9762 Commits zum Testen übrig (ungefähr 13 Schritte)
[dafa5f6577a9eecd2941add553d1672c30b02364] Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
binäre Suche: danach noch 4644 Commits zum Testen übrig (ungefähr 12 Schritte)
[2ed9db3074fcd8d12709fe40ff0e691d74229818] net: sched: cls_api: fix dead code in switch
binäre Suche: danach noch 2319 Commits zum Testen übrig (ungefähr 11 Schritte)
[b219a1d2de0c025318475e3bbf8e3215cf49d083] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/shli/md
binäre Suche: danach noch 1153 Commits zum Testen übrig (ungefähr 10 Schritte)
[85a0b791bc17f7a49280b33e2905d109c062a47b] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
binäre Suche: danach noch 629 Commits zum Testen übrig (ungefähr 9 Schritte)
[10f3e23f07cb0c20f9bcb77a5b5a7eb2a1b2a2fe] Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
binäre Suche: danach noch 273 Commits zum Testen übrig (ungefähr 8 Schritte)
[575b94386bd539a7d803aee9fd4a8d275844c40f] Merge tag 'locks-v4.19-1' of git://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux
binäre Suche: danach noch 136 Commits zum Testen übrig (ungefähr 7 Schritte)
[d7e8555b1dd493c809e56e359974eecabe7d3fde] btrfs: remove unused member async_submit_bio::fs_info
binäre Suche: danach noch 68 Commits zum Testen übrig (ungefähr 6 Schritte)
[389305b2aa68723c754f88d9dbd268a400e10664] btrfs: relocation: Only remove reloc rb_trees if reloc control has been initialized
binäre Suche: danach noch 34 Commits zum Testen übrig (ungefähr 5 Schritte)
[d814a49198eafa6163698bdd93961302f3a877a4] btrfs: use correct compare function of dirty_metadata_bytes
binäre Suche: danach noch 16 Commits zum Testen übrig (ungefähr 4 Schritte)
[c7b562c5480322ffaf591f45a4ff7ee089340ab4] btrfs: raid56: catch errors from full_stripe_write
binäre Suche: danach noch 8 Commits zum Testen übrig (ungefähr 3 Schritte)
[65ad010488a5cc0f123a9924f7ad26a1b3f6a4f6] btrfs: pass only eb to num_extent_pages
binäre Suche: danach noch 3 Commits zum Testen übrig (ungefähr 2 Schritte)
[37508515621551538addaf826ab4b8a9aaf0a382] btrfs: simplify some assignments of inode numbers
binäre Suche: danach noch 1 Commit zum Testen übrig (ungefähr 1 Schritt)
[69d2480456d1baf027a86e530989d7bedd698d5f] btrfs: use copy_page for copying pages instead of memcpy
binäre Suche: danach noch 0 Commits zum Testen übrig (ungefähr 0 Schritte)
[3ffbd68c48320730ef64ebfb5e639220f1f65483] btrfs: simplify pointer chasing of local fs_info variables
69d2480456d1baf027a86e530989d7bedd698d5f is the first bad commit
commit 69d2480456d1baf027a86e530989d7bedd698d5f
Author: David Sterba <dsterba@suse.com>
Date:   Fri Jun 29 10:56:44 2018 +0200

    btrfs: use copy_page for copying pages instead of memcpy
    
    Use the helper that's possibly optimized for full page copies.
    
    Signed-off-by: David Sterba <dsterba@suse.com>

:040000 040000 87de10a38618c1655c3266ff5a31358068fa1ca6 d0a2612d260215acaff66adaa5183ebd29a4b710 M	fs
Comment 16 Christophe Leroy 2019-08-13 15:38:17 UTC
Interesting.

I see in that commit that in fs/btrfs/free-space-cache.c, copy_page() is done using entry->bitmap.

entry->bitmap is allocated with kmalloc() so there is a possibility that entry->bitmap is not page aligned.

copy_page() in arch/powerpc/kernel/misc_32.S assumes that source and destination are aligned on cache lines at least.
Comment 17 Christophe Leroy 2019-08-14 08:56:34 UTC
Created attachment 284379 [details]
Patch to trace misaligned destination in copy_page() on PPC32

Can you try the attached patch to trace misaligned destination on copy_page() ?
Comment 18 Erhard F. 2019-08-14 15:12:13 UTC
On Wed, 14 Aug 2019 08:56:34 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #17 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> Created attachment 284379 [details]
>   --> https://bugzilla.kernel.org/attachment.cgi?id=284379&action=edit  
> Patch to trace misaligned destination in copy_page() on PPC32
> 
> Can you try the attached patch to trace misaligned destination on copy_page()
> ?
Sorry, the patched kernel does not build:

# LC_ALL=C git status
HEAD detached at v5.3-rc4
You are currently bisecting, started from branch 'master'.
  (use "git bisect reset" to get back to the original branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   arch/powerpc/include/asm/page_32.h
	modified:   arch/powerpc/kernel/misc_32.S

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	trace_misaligned_copy_page.diff

no changes added to commit (use "git add" and/or "git commit -a")
# LC_ALL=C make
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  CHK     include/generated/compile.h
  CALL    arch/powerpc/kernel/prom_init_check.sh
  CC      lib/generic-radix-tree.o
In file included from ./arch/powerpc/include/asm/page.h:244,
                 from ./include/linux/generic-radix-tree.h:39,
                 from lib/generic-radix-tree.c:3:
./arch/powerpc/include/asm/page_32.h: In Funktion »copy_page«:
./arch/powerpc/include/asm/page_32.h:58:2: Fehler: Implizite Deklaration der Funktion »WARN_ON«; meinten Sie »KERN_SOH«? [-Werror=implicit-function-declaration]
  WARN_ON((unsigned long)to & (L1_CACHE_BYTES - 1));
  ^~~~~~~
  KERN_SOH
cc1: Einige Warnungen werden als Fehler behandelt
make[1]: *** [scripts/Makefile.build:281: lib/generic-radix-tree.o] Fehler 1
make: *** [Makefile:1083: lib] Error 2
Comment 19 Christophe Leroy 2019-08-14 16:10:53 UTC
Created attachment 284389 [details]
Patch to trace misaligned destination in copy_page() in asm on PPC32

Oops.

Can you test with this new patch which implements the warning directly in assembly.? This time it only modifies misc_32.S and It builds ok.
Comment 20 Erhard F. 2019-08-14 17:44:32 UTC
Created attachment 284397 [details]
dmesg (PowerMac G4 DP, kernel 5.3-rc4 + debug patch)

/dev/sdb2 mounted after booting, dmesg after unmounting
Comment 21 Erhard F. 2019-08-14 17:45:48 UTC
Created attachment 284399 [details]
dmesg (PowerMac G4 DP, kernel 5.3-rc4 + debug patch)

/dev/sdb2 mounted at boot, dmesg after unmounting.
Comment 22 Erhard F. 2019-08-14 17:48:20 UTC
Created attachment 284401 [details]
kernel .config (PowerMac G4 DP, kernel 5.3-rc4)
Comment 23 Erhard F. 2019-08-14 17:53:06 UTC
On Wed, 14 Aug 2019 16:10:53 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #19 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> Created attachment 284389 [details]
>   --> https://bugzilla.kernel.org/attachment.cgi?id=284389&action=edit  
> Patch to trace misaligned destination in copy_page() in asm on PPC32
> 
> Oops.
> 
> Can you test with this new patch which implements the warning directly in
> assembly.? This time it only modifies misc_32.S and It builds ok.
Please find the full dmesg attatched at the kernel bugtracker.

[...]
Aug 14 19:32:52 T600 kernel: WARNING: CPU: 1 PID: 252 at arch/powerpc/kernel/misc_32.S:457 copy_page+0x4/0x98
Aug 14 19:32:52 T600 kernel: Modules linked in: b43legacy input_leds led_class mac80211 joydev hid_generic usbhid hid cfg80211 snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa rfkill libarc4 evdev ohci_pci btrfs xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate ehci_pci ohci_hcd therm_windtunnel ehci_hcd hwmon i2c_algo_bit firewire_ohci backlight firewire_core sr_mod sungem crc_itu_t drm_kms_helper cdrom sungem_phy usbcore syscopyarea sysfillrect usb_common sysimgblt fb_sys_fops ttm snd_aoa_i2sbus drm snd_aoa_soundbus snd_pcm snd_timer drm_panel_orientation_quirks ssb snd uninorth_agp soundcore agpgart lzo lzo_compress lzo_decompress zram zsmalloc
Aug 14 19:32:52 T600 kernel: CPU: 1 PID: 252 Comm: umount Tainted: G        W         5.3.0-rc4+ #1
Aug 14 19:32:52 T600 kernel: NIP:  c0011524 LR: f1a563f8 CTR: c0011520
Aug 14 19:32:52 T600 kernel: REGS: ed22b810 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
Aug 14 19:32:52 T600 kernel: MSR:  00029032 <EE,ME,IR,DR,RI>  CR: 22048222  XER: 20000000
Aug 14 19:32:52 T600 kernel: 
                             GPR00: f1a563e0 ed22b8c8 e7348020 e6b442e8 dae3e000 00000008 c0596c20 dae3effc 
                             GPR08: 00000000 b2209525 00000000 ed22b8c8 c0011520 00745ff4 00000000 e8dec1fc 
                             GPR16: 00000001 00000000 c07fe5f8 00000001 00000000 00000000 f1af07f0 c06fd6fc 
                             GPR24: e8dec178 00000000 ed22b8d8 f1af0000 ec13f1e8 00000000 ec13f1e8 e8a945e8 
Aug 14 19:32:52 T600 kernel: NIP [c0011524] copy_page+0x4/0x98
Aug 14 19:32:52 T600 kernel: LR [f1a563f8] __load_free_space_cache+0x540/0x61c [btrfs]
Aug 14 19:32:52 T600 kernel: Call Trace:
Aug 14 19:32:52 T600 kernel: [ed22b8c8] [f1a563e0] __load_free_space_cache+0x528/0x61c [btrfs] (unreliable)
Aug 14 19:32:52 T600 kernel: [ed22b958] [f1a565bc] load_free_space_cache+0xe8/0x1bc [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22b998] [f19e83f4] cache_block_group+0x1cc/0x3b4 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22b9f8] [f19f04c8] find_free_extent+0x56c/0xe70 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bad8] [f19f0eb8] btrfs_reserve_extent+0xec/0x220 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bb48] [f19f1130] btrfs_alloc_tree_block+0x144/0x35c [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bc38] [f19dc9c0] alloc_tree_block_no_bg_flush+0x88/0x98 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bc78] [f19dfce0] __btrfs_cow_block+0x140/0x4d0 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bce8] [f19e021c] btrfs_cow_block+0x144/0x23c [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bd18] [f1a039e4] commit_cowonly_roots+0x50/0x294 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bd68] [f1a062c4] btrfs_commit_transaction+0x5e4/0x994 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bdb8] [f1a01800] close_ctree+0xf4/0x2c4 [btrfs]
Aug 14 19:32:52 T600 kernel: [ed22bdf8] [c01ab508] generic_shutdown_super+0x80/0x110
Aug 14 19:32:52 T600 kernel: [ed22be18] [c01ab718] kill_anon_super+0x18/0x30
Aug 14 19:32:53 T600 kernel: [ed22be38] [f19d88b4] btrfs_kill_super+0x18/0x30 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22be58] [c01abdbc] deactivate_locked_super+0x54/0xa4
Aug 14 19:32:53 T600 kernel: [ed22be78] [c01cbcb4] cleanup_mnt+0x6c/0xe4
Aug 14 19:32:53 T600 kernel: [ed22bea8] [c0054f50] task_work_run+0xa0/0xc0
Aug 14 19:32:53 T600 kernel: [ed22bed8] [c000bc44] do_notify_resume+0x160/0x2c8
Aug 14 19:32:53 T600 kernel: [ed22bf38] [c0014800] do_user_signal+0x2c/0x34
Aug 14 19:32:53 T600 kernel: --- interrupt: c00 at 0x5a93d4
                                 LR = 0x5a93b8
Aug 14 19:32:53 T600 kernel: Instruction dump:
Aug 14 19:32:53 T600 kernel: 38630020 4200fff8 7c0004ac 7c8903a6 7c0037ac 38c60020 4200fff8 7c0004ac 
Aug 14 19:32:53 T600 kernel: 7d400124 4c00012c 4e800020 546506fe <0f050000> 3863fffc 3884fffc 38a00004 
Aug 14 19:32:53 T600 kernel: irq event stamp: 0
Aug 14 19:32:53 T600 kernel: hardirqs last  enabled at (0): [<00000000>] 0x0
Aug 14 19:32:53 T600 kernel: hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
Aug 14 19:32:53 T600 kernel: softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
Aug 14 19:32:53 T600 kernel: softirqs last disabled at (0): [<00000000>] 0x0
Aug 14 19:32:53 T600 kernel: ---[ end trace 419c4df4c0ad0128 ]---
Aug 14 19:32:53 T600 kernel: WARNING: CPU: 1 PID: 252 at arch/powerpc/kernel/misc_32.S:457 copy_page+0x4/0x98
Aug 14 19:32:53 T600 kernel: Modules linked in: b43legacy input_leds led_class mac80211 joydev hid_generic usbhid hid cfg80211 snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa rfkill libarc4 evdev ohci_pci btrfs xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate ehci_pci ohci_hcd therm_windtunnel ehci_hcd hwmon i2c_algo_bit firewire_ohci backlight firewire_core sr_mod sungem crc_itu_t drm_kms_helper cdrom sungem_phy usbcore syscopyarea sysfillrect usb_common sysimgblt fb_sys_fops ttm snd_aoa_i2sbus drm snd_aoa_soundbus snd_pcm snd_timer drm_panel_orientation_quirks ssb snd uninorth_agp soundcore agpgart lzo lzo_compress lzo_decompress zram zsmalloc
Aug 14 19:32:53 T600 kernel: CPU: 1 PID: 252 Comm: umount Tainted: P        W         5.3.0-rc4+ #1
Aug 14 19:32:53 T600 kernel: NIP:  c0011524 LR: f1a563f8 CTR: c0011520
Aug 14 19:32:53 T600 kernel: REGS: ed22b810 TRAP: 0700   Tainted: P        W          (5.3.0-rc4+)
Aug 14 19:32:53 T600 kernel: MSR:  00029032 <EE,ME,IR,DR,RI>  CR: 22048222  XER: 20000000
Aug 14 19:32:53 T600 kernel: 
                             GPR00: f1a563e0 ed22b8c8 e7348020 de3eb230 dae3f000 00000010 c0596c20 dae3fffc 
                             GPR08: 00000000 c9dc33ec 00000000 ed22b8c8 c0011520 00745ff4 00000000 e8dec1fc 
                             GPR16: 00000001 00000000 c07fe5f8 00000001 00000000 00000000 f1af07f0 c06fd6fc 
                             GPR24: e8dec178 00000000 ed22b8d8 f1af0000 ec13fb48 00000000 ec13fb48 e8a945e8 
Aug 14 19:32:53 T600 kernel: NIP [c0011524] copy_page+0x4/0x98
Aug 14 19:32:53 T600 kernel: LR [f1a563f8] __load_free_space_cache+0x540/0x61c [btrfs]
Aug 14 19:32:53 T600 kernel: Call Trace:
Aug 14 19:32:53 T600 kernel: [ed22b8c8] [f1a563e0] __load_free_space_cache+0x528/0x61c [btrfs] (unreliable)
Aug 14 19:32:53 T600 kernel: [ed22b958] [f1a565bc] load_free_space_cache+0xe8/0x1bc [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22b998] [f19e83f4] cache_block_group+0x1cc/0x3b4 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22b9f8] [f19f04c8] find_free_extent+0x56c/0xe70 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bad8] [f19f0eb8] btrfs_reserve_extent+0xec/0x220 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bb48] [f19f1130] btrfs_alloc_tree_block+0x144/0x35c [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bc38] [f19dc9c0] alloc_tree_block_no_bg_flush+0x88/0x98 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bc78] [f19dfce0] __btrfs_cow_block+0x140/0x4d0 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bce8] [f19e021c] btrfs_cow_block+0x144/0x23c [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bd18] [f1a039e4] commit_cowonly_roots+0x50/0x294 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bd68] [f1a062c4] btrfs_commit_transaction+0x5e4/0x994 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bdb8] [f1a01800] close_ctree+0xf4/0x2c4 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22bdf8] [c01ab508] generic_shutdown_super+0x80/0x110
Aug 14 19:32:53 T600 kernel: [ed22be18] [c01ab718] kill_anon_super+0x18/0x30
Aug 14 19:32:53 T600 kernel: [ed22be38] [f19d88b4] btrfs_kill_super+0x18/0x30 [btrfs]
Aug 14 19:32:53 T600 kernel: [ed22be58] [c01abdbc] deactivate_locked_super+0x54/0xa4
Aug 14 19:32:53 T600 kernel: [ed22be78] [c01cbcb4] cleanup_mnt+0x6c/0xe4
Aug 14 19:32:53 T600 kernel: [ed22bea8] [c0054f50] task_work_run+0xa0/0xc0
Aug 14 19:32:53 T600 kernel: [ed22bed8] [c000bc44] do_notify_resume+0x160/0x2c8
Aug 14 19:32:53 T600 kernel: [ed22bf38] [c0014800] do_user_signal+0x2c/0x34
Aug 14 19:32:53 T600 kernel: --- interrupt: c00 at 0x5a93d4
                                 LR = 0x5a93b8
Aug 14 19:32:53 T600 kernel: Instruction dump:
Aug 14 19:32:53 T600 kernel: 38630020 4200fff8 7c0004ac 7c8903a6 7c0037ac 38c60020 4200fff8 7c0004ac 
Aug 14 19:32:53 T600 kernel: 7d400124 4c00012c 4e800020 546506fe <0f050000> 3863fffc 3884fffc 38a00004 
Aug 14 19:32:53 T600 kernel: irq event stamp: 0
Aug 14 19:32:53 T600 kernel: hardirqs last  enabled at (0): [<00000000>] 0x0
Aug 14 19:32:53 T600 kernel: hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
Aug 14 19:32:53 T600 kernel: softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
Aug 14 19:32:53 T600 kernel: softirqs last disabled at (0): [<00000000>] 0x0
Aug 14 19:32:53 T600 kernel: ---[ end trace 419c4df4c0ad0129 ]---
Aug 14 19:32:55 T600 kernel: =============================================================================
Aug 14 19:32:55 T600 kernel: BUG kmalloc-4k (Tainted: P        W        ): Redzone overwritten
Aug 14 19:32:55 T600 kernel: -----------------------------------------------------------------------------
Aug 14 19:32:55 T600 kernel: INFO: 0xb1102a74-0x308f4f85. First byte 0x0 instead of 0xcc
Aug 14 19:32:55 T600 kernel: INFO: Allocated in __load_free_space_cache+0x420/0x61c [btrfs] age=264 cpu=1 pid=252
Aug 14 19:32:55 T600 kernel:         __slab_alloc.constprop.74+0x40/0x6c
Aug 14 19:32:55 T600 kernel:         kmem_cache_alloc_trace+0x7c/0x1a0
Aug 14 19:32:55 T600 kernel:         __load_free_space_cache+0x420/0x61c [btrfs]
Aug 14 19:32:55 T600 kernel:         load_free_space_cache+0xe8/0x1bc [btrfs]
Aug 14 19:32:55 T600 kernel:         cache_block_group+0x1cc/0x3b4 [btrfs]
Aug 14 19:32:55 T600 kernel:         find_free_extent+0x56c/0xe70 [btrfs]
Aug 14 19:32:55 T600 kernel:         btrfs_reserve_extent+0xec/0x220 [btrfs]
Aug 14 19:32:55 T600 kernel:         btrfs_alloc_tree_block+0x144/0x35c [btrfs]
Aug 14 19:32:55 T600 kernel:         alloc_tree_block_no_bg_flush+0x88/0x98 [btrfs]
Aug 14 19:32:55 T600 kernel:         __btrfs_cow_block+0x140/0x4d0 [btrfs]
Aug 14 19:32:55 T600 kernel:         btrfs_cow_block+0x144/0x23c [btrfs]
Aug 14 19:32:55 T600 kernel:         commit_cowonly_roots+0x50/0x294 [btrfs]
Aug 14 19:32:55 T600 kernel:         btrfs_commit_transaction+0x5e4/0x994 [btrfs]
Aug 14 19:32:55 T600 kernel:         close_ctree+0xf4/0x2c4 [btrfs]
Aug 14 19:32:55 T600 kernel:         generic_shutdown_super+0x80/0x110
Aug 14 19:32:55 T600 kernel:         kill_anon_super+0x18/0x30
Aug 14 19:32:55 T600 kernel: INFO: Freed in proc_cgroup_show+0xbc/0x24c age=381 cpu=0 pid=95
Aug 14 19:32:55 T600 kernel:         kfree+0x264/0x29c
Aug 14 19:32:55 T600 kernel:         proc_cgroup_show+0xbc/0x24c
Aug 14 19:32:55 T600 kernel:         proc_single_show+0x54/0x74
Aug 14 19:32:55 T600 kernel:         seq_read+0x27c/0x460
Aug 14 19:32:55 T600 kernel:         __vfs_read+0x3c/0x10c
Aug 14 19:32:55 T600 kernel:         vfs_read+0xa8/0xf8
Aug 14 19:32:55 T600 kernel:         ksys_read+0x7c/0xd0
Aug 14 19:32:55 T600 kernel:         ret_from_syscall+0x0/0x34
Aug 14 19:32:55 T600 kernel: INFO: Slab 0x5d010511 objects=7 used=7 fp=0xcc30daf2 flags=0x10201
Aug 14 19:32:55 T600 kernel: INFO: Object 0x6ef21f55 @offset=12848 fp=0xcc30daf2
Aug 14 19:32:55 T600 kernel: Redzone b1102a74: 00 00 00 00 00 00 00 00                          ........
[...]
Aug 14 19:32:56 T600 kernel: Redzone de5eb20d: cc cc cc cc                                      ....
Aug 14 19:32:56 T600 kernel: Padding 2d50102a: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Aug 14 19:32:56 T600 kernel: CPU: 1 PID: 252 Comm: umount Tainted: P    B   W         5.3.0-rc4+ #1
Aug 14 19:32:56 T600 kernel: Call Trace:
Aug 14 19:32:56 T600 kernel: [ed22bb68] [c0532654] dump_stack+0xa0/0xfc (unreliable)
Aug 14 19:32:56 T600 kernel: [ed22bb98] [c0195540] check_bytes_and_report+0xc8/0xf0
Aug 14 19:32:56 T600 kernel: [ed22bbc8] [c0195f80] check_object+0x50/0x224
Aug 14 19:32:56 T600 kernel: [ed22bbf8] [c0197d6c] free_debug_processing+0x17c/0x27c
Aug 14 19:32:56 T600 kernel: [ed22bc48] [c0197ff4] __slab_free+0x188/0x338
Aug 14 19:32:56 T600 kernel: [ed22bcc8] [c0198408] kfree+0x264/0x29c
Aug 14 19:32:56 T600 kernel: [ed22bd18] [f1a53444] free_bitmap+0x24/0x68 [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22bd38] [f1a534f0] __btrfs_remove_free_space_cache_locked+0x68/0x6c [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22bd58] [f1a566c8] btrfs_remove_free_space_cache+0x38/0x84 [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22bd78] [f19ef7a4] btrfs_free_block_groups+0x164/0x24c [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22bdb8] [f1a0193c] close_ctree+0x230/0x2c4 [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22bdf8] [c01ab508] generic_shutdown_super+0x80/0x110
Aug 14 19:32:56 T600 kernel: [ed22be18] [c01ab718] kill_anon_super+0x18/0x30
Aug 14 19:32:56 T600 kernel: [ed22be38] [f19d88b4] btrfs_kill_super+0x18/0x30 [btrfs]
Aug 14 19:32:56 T600 kernel: [ed22be58] [c01abdbc] deactivate_locked_super+0x54/0xa4
Aug 14 19:32:56 T600 kernel: [ed22be78] [c01cbcb4] cleanup_mnt+0x6c/0xe4
Aug 14 19:32:56 T600 kernel: [ed22bea8] [c0054f50] task_work_run+0xa0/0xc0
Aug 14 19:32:56 T600 kernel: [ed22bed8] [c000bc44] do_notify_resume+0x160/0x2c8
Aug 14 19:32:56 T600 kernel: [ed22bf38] [c0014800] do_user_signal+0x2c/0x34
Aug 14 19:32:56 T600 kernel: --- interrupt: c00 at 0x5a93d4
                                 LR = 0x5a93b8
Aug 14 19:32:56 T600 kernel: FIX kmalloc-4k: Restoring 0xb1102a74-0x308f4f85=0xcc
Aug 14 19:32:56 T600 kernel: FIX kmalloc-4k: Object at 0x6ef21f55 not freed
Aug 14 19:32:56 T600 kernel: =============================================================================
Aug 14 19:32:56 T600 kernel: BUG kmalloc-4k (Tainted: P    B   W        ): Redzone overwritten
Aug 14 19:32:56 T600 kernel: -----------------------------------------------------------------------------
Aug 14 19:32:56 T600 kernel: INFO: 0xddb8467c-0xf2d9726e. First byte 0x0 instead of 0xcc
Aug 14 19:32:56 T600 kernel: INFO: Allocated in __load_free_space_cache+0x420/0x61c [btrfs] age=900 cpu=1 pid=252
Aug 14 19:32:56 T600 kernel:         __slab_alloc.constprop.74+0x40/0x6c
Aug 14 19:32:56 T600 kernel:         kmem_cache_alloc_trace+0x7c/0x1a0
Aug 14 19:32:57 T600 kernel:         __load_free_space_cache+0x420/0x61c [btrfs]
Aug 14 19:32:57 T600 kernel:         load_free_space_cache+0xe8/0x1bc [btrfs]
Aug 14 19:32:57 T600 kernel:         cache_block_group+0x1cc/0x3b4 [btrfs]
Aug 14 19:32:57 T600 kernel:         find_free_extent+0x56c/0xe70 [btrfs]
Aug 14 19:32:57 T600 kernel:         btrfs_reserve_extent+0xec/0x220 [btrfs]
Aug 14 19:32:57 T600 kernel:         btrfs_alloc_tree_block+0x144/0x35c [btrfs]
Aug 14 19:32:57 T600 kernel:         alloc_tree_block_no_bg_flush+0x88/0x98 [btrfs]
Aug 14 19:32:57 T600 kernel:         __btrfs_cow_block+0x140/0x4d0 [btrfs]
Aug 14 19:32:57 T600 kernel:         btrfs_cow_block+0x144/0x23c [btrfs]
Aug 14 19:32:57 T600 kernel:         commit_cowonly_roots+0x50/0x294 [btrfs]
Aug 14 19:32:57 T600 kernel:         btrfs_commit_transaction+0x5e4/0x994 [btrfs]
Aug 14 19:32:57 T600 kernel:         close_ctree+0xf4/0x2c4 [btrfs]
Aug 14 19:32:57 T600 kernel:         generic_shutdown_super+0x80/0x110
Aug 14 19:32:57 T600 kernel:         kill_anon_super+0x18/0x30
Aug 14 19:32:57 T600 kernel: INFO: Freed in proc_cgroup_show+0xbc/0x24c age=1018 cpu=0 pid=95
Aug 14 19:32:57 T600 kernel:         kfree+0x264/0x29c
Aug 14 19:32:57 T600 kernel:         proc_cgroup_show+0xbc/0x24c
Aug 14 19:32:57 T600 kernel:         proc_single_show+0x54/0x74
Aug 14 19:32:57 T600 kernel:         seq_read+0x27c/0x460
Aug 14 19:32:57 T600 kernel:         __vfs_read+0x3c/0x10c
Aug 14 19:32:57 T600 kernel:         vfs_read+0xa8/0xf8
Aug 14 19:32:57 T600 kernel:         ksys_read+0x7c/0xd0
Aug 14 19:32:57 T600 kernel:         ret_from_syscall+0x0/0x34
Aug 14 19:32:57 T600 kernel: INFO: Slab 0x5fe33d40 objects=7 used=7 fp=0xcc30daf2 flags=0x10201
Aug 14 19:32:57 T600 kernel: INFO: Object 0xf510daf3 @offset=17128 fp=0xcc30daf2
Aug 14 19:32:57 T600 kernel: Redzone ddb8467c: 00 00 00 00 00 00 00 00                          ........
[...]
Aug 14 19:32:57 T600 kernel: Redzone 2ef2f036: cc cc cc cc                                      ....
Aug 14 19:32:57 T600 kernel: Padding 6004979e: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Aug 14 19:32:57 T600 kernel: CPU: 1 PID: 252 Comm: umount Tainted: P    B   W         5.3.0-rc4+ #1
Aug 14 19:32:57 T600 kernel: Call Trace:
Aug 14 19:32:57 T600 kernel: [ed22bb68] [c0532654] dump_stack+0xa0/0xfc (unreliable)
Aug 14 19:32:57 T600 kernel: [ed22bb98] [c0195540] check_bytes_and_report+0xc8/0xf0
Aug 14 19:32:57 T600 kernel: [ed22bbc8] [c0195f80] check_object+0x50/0x224
Aug 14 19:32:57 T600 kernel: [ed22bbf8] [c0197d6c] free_debug_processing+0x17c/0x27c
Aug 14 19:32:57 T600 kernel: [ed22bc48] [c0197ff4] __slab_free+0x188/0x338
Aug 14 19:32:57 T600 kernel: [ed22bcc8] [c0198408] kfree+0x264/0x29c
Aug 14 19:32:57 T600 kernel: [ed22bd18] [f1a53444] free_bitmap+0x24/0x68 [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22bd38] [f1a534f0] __btrfs_remove_free_space_cache_locked+0x68/0x6c [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22bd58] [f1a566c8] btrfs_remove_free_space_cache+0x38/0x84 [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22bd78] [f19ef7a4] btrfs_free_block_groups+0x164/0x24c [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22bdb8] [f1a0193c] close_ctree+0x230/0x2c4 [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22bdf8] [c01ab508] generic_shutdown_super+0x80/0x110
Aug 14 19:32:57 T600 kernel: [ed22be18] [c01ab718] kill_anon_super+0x18/0x30
Aug 14 19:32:57 T600 kernel: [ed22be38] [f19d88b4] btrfs_kill_super+0x18/0x30 [btrfs]
Aug 14 19:32:57 T600 kernel: [ed22be58] [c01abdbc] deactivate_locked_super+0x54/0xa4
Aug 14 19:32:57 T600 kernel: [ed22be78] [c01cbcb4] cleanup_mnt+0x6c/0xe4
Aug 14 19:32:57 T600 kernel: [ed22bea8] [c0054f50] task_work_run+0xa0/0xc0
Aug 14 19:32:57 T600 kernel: [ed22bed8] [c000bc44] do_notify_resume+0x160/0x2c8
Aug 14 19:32:57 T600 kernel: [ed22bf38] [c0014800] do_user_signal+0x2c/0x34
Aug 14 19:32:57 T600 kernel: --- interrupt: c00 at 0x5a93d4
                                 LR = 0x5a93b8
Aug 14 19:32:57 T600 kernel: FIX kmalloc-4k: Restoring 0xddb8467c-0xf2d9726e=0xcc
Aug 14 19:32:57 T600 kernel: FIX kmalloc-4k: Object at 0xf510daf3 not freed
Comment 24 Christophe Leroy 2019-08-14 20:28:11 UTC
It confirms what I suspected: due to some debug options, kzalloc() doesn't provide aligned areas.

In __load_free_space_cache() can you replace 
e->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
By
e->bitmap = (void *)__get_free_page(GFP_NOFS | __GFP_ZERO);

And same in insert_into_bitmap()

Then replace the three kfree() which free bitmaps by something like free_page((unsigned long)entry->bitmap)
Comment 25 Christophe Leroy 2019-08-14 20:33:51 UTC
You can use get_zeroed_page(GFP_NOFS) instead of __get_free_page(GFP_NOFS | __GFP_ZERO)
Comment 26 Erhard F. 2019-08-15 14:38:50 UTC
On Wed, 14 Aug 2019 20:33:51 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:


> https://bugzilla.kernel.org/show_bug.cgi?id=204371
>
>--- Comment #24 from Christophe Leroy (christophe.leroy@c-s.fr) ---
>It confirms what I suspected: due to some debug options, kzalloc() doesn't
>provide aligned areas.
>
>In __load_free_space_cache() can you replace 
>e->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
>By
>e->bitmap = (void *)__get_free_page(GFP_NOFS | __GFP_ZERO);
>
>And same in insert_into_bitmap()
>
>Then replace the three kfree() which free bitmaps by something like
>free_page((unsigned long)entry->bitmap)
> 
> --- Comment #25 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> You can use get_zeroed_page(GFP_NOFS) instead of __get_free_page(GFP_NOFS |
> __GFP_ZERO)
Think I got everything right after a bit of searching...

Now I get this nice compact output:
[..]
[   46.579181] ------------[ cut here ]------------
[   46.579378] kernel BUG at mm/slub.c:3952!
[   46.579513] Oops: Exception in kernel mode, sig: 5 [#1]
[   46.579699] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
[   46.579898] Modules linked in: b43legacy led_class mac80211 cfg80211 snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa rfkill libarc4 evdev btrfs therm_windtunnel ohci_pci xor zstd_decompress zstd_compress zlib_deflate raid6_pq zlib_inflate radeon sr_mod firewire_ohci sungem hwmon snd_aoa_i2sbus i2c_algo_bit backlight cdrom firewire_core sungem_phy crc_itu_t snd_aoa_soundbus snd_pcm drm_kms_helper ohci_hcd syscopyarea ehci_pci snd_timer sysfillrect snd sysimgblt fb_sys_fops ttm ehci_hcd soundcore drm drm_panel_orientation_quirks usbcore uninorth_agp usb_common agpgart ssb lzo lzo_compress lzo_decompress zram zsmalloc
[   46.582252] CPU: 0 PID: 261 Comm: umount Tainted: G        W         5.3.0-rc4+ #2
[   46.582533] NIP:  c0198228 LR: c0198204 CTR: c01981a4
[   46.582708] REGS: dbbc1c10 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[   46.582990] MSR:  00029032 <EE,ME,IR,DR,RI>  CR: 22008824  XER: 00000000
[   46.583243] 
               GPR00: f110b444 dbbc1cc8 ec2944a0 ef4329f4 c07fe5f8 8950b76e 00001032 00000000 
               GPR08: 2854c000 00000001 00000000 dbbc1d18 c01981a4 008f5ff4 00000000 00000000 
               GPR16: 00000000 00000000 bff5d9dc bff5d9c4 00000001 00000000 00000000 00000000 
               GPR24: 00000000 f110b444 00000100 dba800b8 f11b0000 c07fe5f8 ed3d5000 ef4329f4 
[   46.584505] NIP [c0198228] kfree+0x84/0x29c
[   46.584642] LR [c0198204] kfree+0x60/0x29c
[   46.584774] Call Trace:
[   46.585254] [dbbc1cc8] [f110b4e0] __btrfs_remove_free_space_cache_locked+0x58/0x6c [btrfs] (unreliable)
[   46.585717] [dbbc1d18] [f110b444] free_bitmap+0x24/0x68 [btrfs]
[   46.586008] [dbbc1d38] [f110b4f0] __btrfs_remove_free_space_cache_locked+0x68/0x6c [btrfs]
[   46.586388] [dbbc1d58] [f110e6ac] btrfs_remove_free_space_cache+0x38/0x84 [btrfs]
[   46.586732] [dbbc1d78] [f10a77a4] btrfs_free_block_groups+0x164/0x24c [btrfs]
[   46.587073] [dbbc1db8] [f10b993c] close_ctree+0x230/0x2c4 [btrfs]
[   46.587303] [dbbc1df8] [c01ab508] generic_shutdown_super+0x80/0x110
[   46.587531] [dbbc1e18] [c01ab718] kill_anon_super+0x18/0x30
[   46.587802] [dbbc1e38] [f10908b4] btrfs_kill_super+0x18/0x30 [btrfs]
[   46.588039] [dbbc1e58] [c01abdbc] deactivate_locked_super+0x54/0xa4
[   46.588269] [dbbc1e78] [c01cbcb4] cleanup_mnt+0x6c/0xe4
[   46.588456] [dbbc1ea8] [c0054f50] task_work_run+0xa0/0xc0
[   46.588645] [dbbc1ed8] [c000bc44] do_notify_resume+0x160/0x2c8
[   46.588857] [dbbc1f38] [c0014800] do_user_signal+0x2c/0x34
[   46.589052] --- interrupt: c00 at 0x7593d4
                   LR = 0x7593b8
[   46.589252] Instruction dump:
[   46.589340] 4bffade1 7c7f1b78 4bffadbd 81230000 71290200 40a200f8 813f0000 552987ff 
[   46.589644] 4082000c 813f0004 552907fe 69290001 <0f090000> 7fe3fb78 4bffadcd 7c641b78 
[   46.589961] ---[ end trace 0164244520bfd23a ]---
Comment 27 Christophe Leroy 2019-08-15 16:45:11 UTC
Can you post the changes you did ?

Did you replace the two kzalloc() by get_zeroed_page()  as suggested ?
If so, it looks like you missed one kfree() (in free_bitmap()) to be replaced by free_page().
Comment 28 Erhard F. 2019-08-15 17:07:59 UTC
On Thu, 15 Aug 2019 16:45:11 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #27 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> Can you post the changes you did ?
> 
> Did you replace the two kzalloc() by get_zeroed_page()  as suggested ?
> If so, it looks like you missed one kfree() (in free_bitmap()) to be replaced
> by free_page().

Ah yes, I added the (unsigned long) part but forgot to replace kfree() with free_page(). Now looks like this:

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 062be9dde4c6..c3eed8c3d3fe 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -764,7 +764,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
                } else {
                        ASSERT(num_bitmaps);
                        num_bitmaps--;
-                       e->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+                       e->bitmap = (void *)get_zeroed_page(GFP_NOFS);
                        if (!e->bitmap) {
                                kmem_cache_free(
                                        btrfs_free_space_cachep, e);
@@ -1881,7 +1881,7 @@ static void free_bitmap(struct btrfs_free_space_ctl *ctl,
                        struct btrfs_free_space *bitmap_info)
 {
        unlink_free_space(ctl, bitmap_info);
-       kfree(bitmap_info->bitmap);
+       free_page((unsigned long)bitmap_info->bitmap);
        kmem_cache_free(btrfs_free_space_cachep, bitmap_info);
        ctl->total_bitmaps--;
        ctl->op->recalc_thresholds(ctl);
@@ -2135,7 +2135,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
                }
 
                /* allocate the bitmap */
-               info->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+               info->bitmap = (void *)get_zeroed_page(GFP_NOFS);
                spin_lock(&ctl->tree_lock);
                if (!info->bitmap) {
                        ret = -ENOMEM;
@@ -2146,7 +2146,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
 
 out:
        if (info) {
-               kfree(info->bitmap);
+               free_page((unsigned long)info->bitmap);
                kmem_cache_free(btrfs_free_space_cachep, info);
        }
 
@@ -2802,7 +2802,7 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group_cache *block_group,
        if (entry->bytes == 0) {
                ctl->free_extents--;
                if (entry->bitmap) {
-                       kfree(entry->bitmap);
+                       free_page((unsigned long)entry->bitmap);
                        ctl->total_bitmaps--;
                        ctl->op->recalc_thresholds(ctl);
                }
Comment 29 Christophe Leroy 2019-08-15 17:11:36 UTC
Looks good. Does it work better ?
Comment 30 Erhard F. 2019-08-15 17:56:35 UTC
On Thu, 15 Aug 2019 17:11:36 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #29 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> Looks good. Does it work better ?
Had some trouble getting the dmesg. With these modifications the btrfs module hiccups as soon as it gets loaded, during btrfs selftests:

[...]
[  167.258266] Btrfs loaded, crc32c=crc32c-generic, debug=on
[  167.259388] BTRFS: selftest: sectorsize: 4096  nodesize: 4096
[  167.259602] BTRFS: selftest: running btrfs free space cache tests
[  167.259943] BTRFS: selftest: running extent only tests
[  167.260201] BTRFS: selftest: running bitmap only tests
[  167.260501] BTRFS: selftest: running bitmap and extent tests
[  167.260963] WARNING: CPU: 0 PID: 266 at mm/slub.c:1846 ___slab_alloc.constprop.75+0x2ac/0x380
[  167.261277] Modules linked in: btrfs(+) auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc input_leds joydev b43legacy led_class hid_generic mac80211 usbhid hid snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill libarc4 evdev ohci_pci xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate therm_windtunnel ehci_pci ohci_hcd hwmon i2c_algo_bit ehci_hcd backlight drm_kms_helper sungem firewire_ohci syscopyarea sungem_phy sysfillrect sr_mod firewire_core sysimgblt fb_sys_fops cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus ttm snd_pcm usbcore snd_timer drm usb_common snd soundcore ssb uninorth_agp drm_panel_orientation_quirks agpgart lzo lzo_compress lzo_decompress zram zsmalloc
[  167.263795] CPU: 0 PID: 266 Comm: modprobe Tainted: G        W         5.3.0-rc4+ #3
[  167.264074] NIP:  c0196ddc LR: c0196dd4 CTR: c019711c
[  167.264236] REGS: ecde9a70 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[  167.264488] MSR:  00021032 <ME,IR,DR,RI>  CR: 28224222  XER: 00000000
[  167.264709] 
               GPR00: c0196dd4 ecde9b28 eb353380 00000000 ef3d3600 0003000e 00000000 0007000f 
               GPR08: 00000001 00000001 ef42488c ecde9b28 48244222 00a9eff4 00a64e74 00000004 
               GPR16: 00000000 ee800810 00000000 c0800000 00000000 ee800824 f203de58 c07fea34 
               GPR24: 00210d00 ef424888 00000d40 ee800800 ef3d3600 ee8032e0 00000000 eedb1af8 
[  167.265857] NIP [c0196ddc] ___slab_alloc.constprop.75+0x2ac/0x380
[  167.266052] LR [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380
[  167.266248] Call Trace:
[  167.266308] [ecde9b28] [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380 (unreliable)
[  167.266567] [ecde9bb8] [c0196ef0] __slab_alloc.constprop.74+0x40/0x6c
[  167.266776] [ecde9be8] [c0197198] kmem_cache_alloc_trace+0x7c/0x1a0
[  167.267321] [ecde9c28] [f203de58] test_add_free_space_entry+0xf0/0x214 [btrfs]
[  167.267672] [ecde9c78] [f207e238] btrfs_test_free_space_cache+0x918/0x1308 [btrfs]
[  167.268012] [ecde9cd8] [f207ad3c] btrfs_run_sanity_tests+0x8c/0x144 [btrfs]
[  167.268327] [ecde9d08] [f1678cd0] init_btrfs_fs+0xd4/0x12c [btrfs]
[  167.268536] [ecde9d28] [c00052cc] do_one_initcall+0x54/0x288
[  167.282204] [ecde9d98] [c00bebf8] do_init_module+0x60/0x1dc
[  167.295710] [ecde9dc8] [c00c0ad4] load_module+0x1ca4/0x1e18
[  167.308943] [ecde9ea8] [c00c0df4] sys_finit_module+0x98/0xb8
[  167.322086] [ecde9f38] [c0014274] ret_from_syscall+0x0/0x34
[  167.335151] --- interrupt: c01 at 0x8ed2c4
                   LR = 0xa757c4
[  167.361356] Instruction dump:
[  167.374279] 7e048378 7f83e378 60e70001 90e10030 4bffda0d 2f830000 41beff20 7f84e378 
[  167.387497] 7f63db78 4bffdd35 7e090034 5529d97e <0f090000> 2f900000 41beff00 7e527a14 
[  167.400820] irq event stamp: 0
[  167.414008] hardirqs last  enabled at (0): [<00000000>] 0x0
[  167.427196] hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  167.440311] softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  167.453198] softirqs last disabled at (0): [<00000000>] 0x0
[  167.465906] ---[ end trace 74450a6aa18e595d ]---
[  167.481091] BTRFS: selftest: running space stealing from bitmap to extent tests
[  167.496220] BTRFS: selftest: running extent buffer operation tests
[  167.510905] BTRFS: selftest: running btrfs_split_item tests
[  167.525961] BTRFS: selftest: running extent I/O tests
[  167.540569] BTRFS: selftest: running find delalloc tests
[  168.104731] BTRFS: selftest: running find_first_clear_extent_bit test
[  168.119119] BTRFS: selftest: running extent buffer bitmap tests
[  168.463591] BTRFS: selftest: running inode tests
[  168.477316] BTRFS: selftest: running btrfs_get_extent tests
[  168.491784] BTRFS: selftest: running hole first btrfs_get_extent test
[  168.506234] BTRFS: selftest: running outstanding_extents tests
[  168.520925] BTRFS: selftest: running qgroup tests
[  168.535039] BTRFS: selftest: running qgroup add/remove tests
[  168.549344] BTRFS: selftest: running qgroup multiple refs test
[  168.563713] BTRFS: selftest: running free space tree tests
[  168.706099] BTRFS: selftest: sectorsize: 4096  nodesize: 8192
[  168.718674] BTRFS: selftest: running btrfs free space cache tests
[  168.731349] BTRFS: selftest: running extent only tests
[  168.744380] BTRFS: selftest: running bitmap only tests
[  168.757505] BTRFS: selftest: running bitmap and extent tests
[  168.770615] WARNING: CPU: 0 PID: 266 at mm/slub.c:1846 ___slab_alloc.constprop.75+0x2ac/0x380
[  168.783738] Modules linked in: btrfs(+) auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc input_leds joydev b43legacy led_class hid_generic mac80211 usbhid hid snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill libarc4 evdev ohci_pci xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate therm_windtunnel ehci_pci ohci_hcd hwmon i2c_algo_bit ehci_hcd backlight drm_kms_helper sungem firewire_ohci syscopyarea sungem_phy sysfillrect sr_mod firewire_core sysimgblt fb_sys_fops cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus ttm snd_pcm usbcore snd_timer drm usb_common snd soundcore ssb uninorth_agp drm_panel_orientation_quirks agpgart lzo lzo_compress lzo_decompress zram zsmalloc
[  168.828086] CPU: 0 PID: 266 Comm: modprobe Tainted: G        W         5.3.0-rc4+ #3
[  168.843577] NIP:  c0196ddc LR: c0196dd4 CTR: c019711c
[  168.859054] REGS: ecde9a70 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[  168.874772] MSR:  00021032 <ME,IR,DR,RI>  CR: 28224242  XER: 00000000
[  168.890402] 
               GPR00: c0196dd4 ecde9b28 eb353380 00000000 ef3d3960 0001000e 00000000 0007000f 
               GPR08: 00000001 00000001 ef432578 ecde9b28 48244242 00a9eff4 00a64e74 00000006 
               GPR16: 00000000 ee800810 00000000 c0800000 00000000 ee800824 f203de58 c07fea34 
               GPR24: 00210d00 ef432574 00000d40 ee800800 ef3d3960 ee8032e0 00000000 eedb1af8 
[  168.960711] NIP [c0196ddc] ___slab_alloc.constprop.75+0x2ac/0x380
[  168.974148] LR [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380
[  168.987536] Call Trace:
[  169.000772] [ecde9b28] [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380 (unreliable)
[  169.014236] [ecde9bb8] [c0196ef0] __slab_alloc.constprop.74+0x40/0x6c
[  169.027634] [ecde9be8] [c0197198] kmem_cache_alloc_trace+0x7c/0x1a0
[  169.041326] [ecde9c28] [f203de58] test_add_free_space_entry+0xf0/0x214 [btrfs]
[  169.054891] [ecde9c78] [f207dee4] btrfs_test_free_space_cache+0x5c4/0x1308 [btrfs]
[  169.068328] [ecde9cd8] [f207ad3c] btrfs_run_sanity_tests+0x8c/0x144 [btrfs]
[  169.081776] [ecde9d08] [f1678cd0] init_btrfs_fs+0xd4/0x12c [btrfs]
[  169.095039] [ecde9d28] [c00052cc] do_one_initcall+0x54/0x288
[  169.108161] [ecde9d98] [c00bebf8] do_init_module+0x60/0x1dc
[  169.121179] [ecde9dc8] [c00c0ad4] load_module+0x1ca4/0x1e18
[  169.134179] [ecde9ea8] [c00c0df4] sys_finit_module+0x98/0xb8
[  169.147133] [ecde9f38] [c0014274] ret_from_syscall+0x0/0x34
[  169.159933] --- interrupt: c01 at 0x8ed2c4
                   LR = 0xa757c4
[  169.185350] Instruction dump:
[  169.197936] 7e048378 7f83e378 60e70001 90e10030 4bffda0d 2f830000 41beff20 7f84e378 
[  169.210814] 7f63db78 4bffdd35 7e090034 5529d97e <0f090000> 2f900000 41beff00 7e527a14 
[  169.223824] irq event stamp: 0
[  169.236770] hardirqs last  enabled at (0): [<00000000>] 0x0
[  169.249906] hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  169.263077] softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  169.276147] softirqs last disabled at (0): [<00000000>] 0x0
[  169.289173] ---[ end trace 74450a6aa18e595e ]---
[  169.305352] WARNING: CPU: 0 PID: 266 at mm/slub.c:1846 ___slab_alloc.constprop.75+0x2ac/0x380
[  169.318278] Modules linked in: btrfs(+) auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc input_leds joydev b43legacy led_class hid_generic mac80211 usbhid hid snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill libarc4 evdev ohci_pci xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate therm_windtunnel ehci_pci ohci_hcd hwmon i2c_algo_bit ehci_hcd backlight drm_kms_helper sungem firewire_ohci syscopyarea sungem_phy sysfillrect sr_mod firewire_core sysimgblt fb_sys_fops cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus ttm snd_pcm usbcore snd_timer drm usb_common snd soundcore ssb uninorth_agp drm_panel_orientation_quirks agpgart lzo lzo_compress lzo_decompress zram zsmalloc
[  169.363085] CPU: 0 PID: 266 Comm: modprobe Tainted: G        W         5.3.0-rc4+ #3
[  169.378753] NIP:  c0196ddc LR: c0196dd4 CTR: c019711c
[  169.394382] REGS: ecde9a70 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[  169.410156] MSR:  00021032 <ME,IR,DR,RI>  CR: 28224222  XER: 00000000
[  169.425900] 
               GPR00: c0196dd4 ecde9b28 eb353380 00000000 ef3d3ba0 0001000e 00000000 0007000f 
               GPR08: 00000001 00000001 ef188f3c ecde9b28 48244222 00a9eff4 00a64e74 00000006 
               GPR16: 00000000 ee800810 00000000 c0800000 00000000 ee800824 f203de58 c07fea34 
               GPR24: 00210d00 ef188f38 00000d40 ee800800 ef3d3ba0 ee8032e0 00000000 eedb1af8 
[  169.506070] NIP [c0196ddc] ___slab_alloc.constprop.75+0x2ac/0x380
[  169.522457] LR [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380
[  169.538919] Call Trace:
[  169.555301] [ecde9b28] [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380 (unreliable)
[  169.572079] [ecde9bb8] [c0196ef0] __slab_alloc.constprop.74+0x40/0x6c
[  169.588847] [ecde9be8] [c0197198] kmem_cache_alloc_trace+0x7c/0x1a0
[  169.605861] [ecde9c28] [f203de58] test_add_free_space_entry+0xf0/0x214 [btrfs]
[  169.623013] [ecde9c78] [f207e08c] btrfs_test_free_space_cache+0x76c/0x1308 [btrfs]
[  169.640288] [ecde9cd8] [f207ad3c] btrfs_run_sanity_tests+0x8c/0x144 [btrfs]
[  169.657589] [ecde9d08] [f1678cd0] init_btrfs_fs+0xd4/0x12c [btrfs]
[  169.674812] [ecde9d28] [c00052cc] do_one_initcall+0x54/0x288
[  169.692019] [ecde9d98] [c00bebf8] do_init_module+0x60/0x1dc
[  169.709242] [ecde9dc8] [c00c0ad4] load_module+0x1ca4/0x1e18
[  169.726533] [ecde9ea8] [c00c0df4] sys_finit_module+0x98/0xb8
[  169.743904] [ecde9f38] [c0014274] ret_from_syscall+0x0/0x34
[  169.761110] --- interrupt: c01 at 0x8ed2c4
                   LR = 0xa757c4
[  169.793422] Instruction dump:
[  169.808483] 7e048378 7f83e378 60e70001 90e10030 4bffda0d 2f830000 41beff20 7f84e378 
[  169.823241] 7f63db78 4bffdd35 7e090034 5529d97e <0f090000> 2f900000 41beff00 7e527a14 
[  169.838004] irq event stamp: 0
[  169.852643] hardirqs last  enabled at (0): [<00000000>] 0x0
[  169.867316] hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  169.882061] softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  169.896525] softirqs last disabled at (0): [<00000000>] 0x0
[  169.910840] ---[ end trace 74450a6aa18e595f ]---
[  169.928722] BTRFS: selftest: running space stealing from bitmap to extent tests
[  169.943052] WARNING: CPU: 1 PID: 266 at mm/slub.c:1846 ___slab_alloc.constprop.75+0x2ac/0x380
[  169.957190] Modules linked in: btrfs(+) auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc input_leds joydev b43legacy led_class hid_generic mac80211 usbhid hid snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill libarc4 evdev ohci_pci xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate therm_windtunnel ehci_pci ohci_hcd hwmon i2c_algo_bit ehci_hcd backlight drm_kms_helper sungem firewire_ohci syscopyarea sungem_phy sysfillrect sr_mod firewire_core sysimgblt fb_sys_fops cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus ttm snd_pcm usbcore snd_timer drm usb_common snd soundcore ssb uninorth_agp drm_panel_orientation_quirks agpgart lzo lzo_compress lzo_decompress zram zsmalloc
[  170.003934] CPU: 1 PID: 266 Comm: modprobe Tainted: G        W         5.3.0-rc4+ #3
[  170.019968] NIP:  c0196ddc LR: c0196dd4 CTR: c019711c
[  170.035984] REGS: ecde9a70 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[  170.052020] MSR:  00021032 <ME,IR,DR,RI>  CR: 28224242  XER: 00000000
[  170.068087] 
               GPR00: c0196dd4 ecde9b28 eb353380 00000000 ef3d3cc0 0004000e 00000000 0007000f 
               GPR08: 00000001 00000001 ef188eac ecde9b28 48244242 00a9eff4 00a64e74 00000003 
               GPR16: 00000000 ee800810 00000000 c0800000 00000000 ee800824 f203de58 c07fea34 
               GPR24: 00210d00 ef188ea8 00000d40 ee800800 ef3d3cc0 ee8032e0 00000000 eedceaf8 
[  170.149103] NIP [c0196ddc] ___slab_alloc.constprop.75+0x2ac/0x380
[  170.165571] LR [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380
[  170.182082] Call Trace:
[  170.198482] [ecde9b28] [c0196dd4] ___slab_alloc.constprop.75+0x2a4/0x380 (unreliable)
[  170.215245] [ecde9bb8] [c0196ef0] __slab_alloc.constprop.74+0x40/0x6c
[  170.232091] [ecde9be8] [c0197198] kmem_cache_alloc_trace+0x7c/0x1a0
[  170.249175] [ecde9c28] [f203de58] test_add_free_space_entry+0xf0/0x214 [btrfs]
[  170.266059] [ecde9c78] [f207e44c] btrfs_test_free_space_cache+0xb2c/0x1308 [btrfs]
[  170.282918] [ecde9cd8] [f207ad3c] btrfs_run_sanity_tests+0x8c/0x144 [btrfs]
[  170.299700] [ecde9d08] [f1678cd0] init_btrfs_fs+0xd4/0x12c [btrfs]
[  170.316388] [ecde9d28] [c00052cc] do_one_initcall+0x54/0x288
[  170.333159] [ecde9d98] [c00bebf8] do_init_module+0x60/0x1dc
[  170.349916] [ecde9dc8] [c00c0ad4] load_module+0x1ca4/0x1e18
[  170.366610] [ecde9ea8] [c00c0df4] sys_finit_module+0x98/0xb8
[  170.383179] [ecde9f38] [c0014274] ret_from_syscall+0x0/0x34
[  170.399101] --- interrupt: c01 at 0x8ed2c4
                   LR = 0xa757c4
[  170.428916] Instruction dump:
[  170.442826] 7e048378 7f83e378 60e70001 90e10030 4bffda0d 2f830000 41beff20 7f84e378 
[  170.456986] 7f63db78 4bffdd35 7e090034 5529d97e <0f090000> 2f900000 41beff00 7e527a14 
[  170.471190] irq event stamp: 0
[  170.485188] hardirqs last  enabled at (0): [<00000000>] 0x0
[  170.499283] hardirqs last disabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  170.513322] softirqs last  enabled at (0): [<c0032f0c>] copy_process+0x474/0x1368
[  170.527207] softirqs last disabled at (0): [<00000000>] 0x0
[  170.540920] ---[ end trace 74450a6aa18e5960 ]---
[  170.558438] ------------[ cut here ]------------
[  170.572459] kernel BUG at mm/slub.c:3952!
[  170.586496] Oops: Exception in kernel mode, sig: 5 [#1]
[  170.600564] BE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2 PowerMac
[  170.614556] Modules linked in: btrfs(+) auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc input_leds joydev b43legacy led_class hid_generic mac80211 usbhid hid snd_aoa_codec_tas snd_aoa_fabric_layout snd_aoa cfg80211 rfkill libarc4 evdev ohci_pci xor zstd_decompress zstd_compress zlib_deflate radeon raid6_pq zlib_inflate therm_windtunnel ehci_pci ohci_hcd hwmon i2c_algo_bit ehci_hcd backlight drm_kms_helper sungem firewire_ohci syscopyarea sungem_phy sysfillrect sr_mod firewire_core sysimgblt fb_sys_fops cdrom crc_itu_t snd_aoa_i2sbus snd_aoa_soundbus ttm snd_pcm usbcore snd_timer drm usb_common snd soundcore ssb uninorth_agp drm_panel_orientation_quirks agpgart lzo lzo_compress lzo_decompress zram zsmalloc
[  170.662386] CPU: 0 PID: 266 Comm: modprobe Tainted: G        W         5.3.0-rc4+ #3
[  170.678893] NIP:  c0198228 LR: c0198204 CTR: c01981a4
[  170.695309] REGS: ecde9b10 TRAP: 0700   Tainted: G        W          (5.3.0-rc4+)
[  170.711791] MSR:  00029032 <EE,ME,IR,DR,RI>  CR: 28242424  XER: 00000000
[  170.728343] 
               GPR00: f207aa60 ecde9bc8 eb353380 ef3d3a80 c07fe5f8 c01600dc 00003c40 00000000 
               GPR08: 00001032 00000001 00000000 ecde9c18 c01981a4 00a9eff4 00a64e74 c0710cd4 
               GPR16: f16a1fac 000011ad 00000001 00000124 c00bbeac c0e7e55c 00000001 ea8ed15c 
               GPR24: ea8ed16c f207aa60 f209a370 f209a343 00002000 c07fe5f8 ea9a6458 ef3d3a80 
[  170.812601] NIP [c0198228] kfree+0x84/0x29c
[  170.829631] LR [c0198204] kfree+0x60/0x29c
[  170.846753] Call Trace:
[  170.863878] [ecde9bc8] [f20d0000] test_error+0xd44/0xfffba69c [btrfs] (unreliable)
[  170.881178] [ecde9c18] [f207aa60] btrfs_free_dummy_fs_info+0x168/0x1e0 [btrfs]
[  170.898408] [ecde9c78] [f207da3c] btrfs_test_free_space_cache+0x11c/0x1308 [btrfs]
[  170.915026] [ecde9cd8] [f207ad3c] btrfs_run_sanity_tests+0x8c/0x144 [btrfs]
[  170.931323] [ecde9d08] [f1678cd0] init_btrfs_fs+0xd4/0x12c [btrfs]
[  170.948575] [ecde9d28] [c00052cc] do_one_initcall+0x54/0x288
[  170.965784] [ecde9d98] [c00bebf8] do_init_module+0x60/0x1dc
[  170.982897] [ecde9dc8] [c00c0ad4] load_module+0x1ca4/0x1e18
[  170.999911] [ecde9ea8] [c00c0df4] sys_finit_module+0x98/0xb8
[  171.016767] [ecde9f38] [c0014274] ret_from_syscall+0x0/0x34
[  171.032967] --- interrupt: c01 at 0x8ed2c4
                   LR = 0xa757c4
[  171.063206] Instruction dump:
[  171.077290] 4bffade1 7c7f1b78 4bffadbd 81230000 71290200 40a200f8 813f0000 552987ff 
[  171.091622] 4082000c 813f0004 552907fe 69290001 <0f090000> 7fe3fb78 4bffadcd 7c641b78 
[  171.106020] ---[ end trace 74450a6aa18e5961 ]---
Comment 31 Christophe Leroy 2019-08-15 18:33:21 UTC
Problem 1: test_add_free_space_entry() contains a kzalloc() to allocate a bitmap. That's the problem.


Problem 2: btrfs_free_dummy_fs_info() has 3 kfree(). Need to know which one is creating your last warning (kernel BUG at mm/slub.c:3952!)
Comment 32 Christophe Leroy 2019-08-16 08:22:31 UTC
I think first thing is to fix test_add_free_space_entry() :
- replace the map = kzalloc(...) by map = (void *)get_zeroed_page(...) like in other places.
- replace the kfree(map); by free_page((unsigned long)map);

Then see if the WARNING on kfree() in  btrfs_free_dummy_fs_info() is still there.
Comment 33 Erhard F. 2019-08-16 14:35:55 UTC
On Fri, 16 Aug 2019 08:22:31 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #32 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> I think first thing is to fix test_add_free_space_entry() :
> - replace the map = kzalloc(...) by map = (void *)get_zeroed_page(...) like
> in
> other places.
> - replace the kfree(map); by free_page((unsigned long)map);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 062be9dde4c6..ed15645b4321 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -764,7 +764,7 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
 		} else {
 			ASSERT(num_bitmaps);
 			num_bitmaps--;
-			e->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+			e->bitmap = (void *)get_zeroed_page(GFP_NOFS);
 			if (!e->bitmap) {
 				kmem_cache_free(
 					btrfs_free_space_cachep, e);
@@ -1881,7 +1881,7 @@ static void free_bitmap(struct btrfs_free_space_ctl *ctl,
 			struct btrfs_free_space *bitmap_info)
 {
 	unlink_free_space(ctl, bitmap_info);
-	kfree(bitmap_info->bitmap);
+	free_page((unsigned long)bitmap_info->bitmap);
 	kmem_cache_free(btrfs_free_space_cachep, bitmap_info);
 	ctl->total_bitmaps--;
 	ctl->op->recalc_thresholds(ctl);
@@ -2135,7 +2135,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
 		}
 
 		/* allocate the bitmap */
-		info->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+		info->bitmap = (void *)get_zeroed_page(GFP_NOFS);
 		spin_lock(&ctl->tree_lock);
 		if (!info->bitmap) {
 			ret = -ENOMEM;
@@ -2146,7 +2146,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
 
 out:
 	if (info) {
-		kfree(info->bitmap);
+		free_page((unsigned long)info->bitmap);
 		kmem_cache_free(btrfs_free_space_cachep, info);
 	}
 
@@ -2802,7 +2802,7 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group_cache *block_group,
 	if (entry->bytes == 0) {
 		ctl->free_extents--;
 		if (entry->bitmap) {
-			kfree(entry->bitmap);
+			free_page((unsigned long)entry->bitmap);
 			ctl->total_bitmaps--;
 			ctl->op->recalc_thresholds(ctl);
 		}
@@ -3606,7 +3606,7 @@ int test_add_free_space_entry(struct btrfs_block_group_cache *cache,
 	}
 
 	if (!map) {
-		map = kzalloc(PAGE_SIZE, GFP_NOFS);
+		map = (void *)get_zeroed_page(GFP_NOFS);
 		if (!map) {
 			kmem_cache_free(btrfs_free_space_cachep, info);
 			return -ENOMEM;
@@ -3635,7 +3635,7 @@ int test_add_free_space_entry(struct btrfs_block_group_cache *cache,
 
 	if (info)
 		kmem_cache_free(btrfs_free_space_cachep, info);
-	kfree(map);
+	free_page((unsigned long)map);
 	return 0;
 }
Comment 34 Erhard F. 2019-08-16 14:38:54 UTC
On Fri, 16 Aug 2019 08:22:31 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #32 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> Then see if the WARNING on kfree() in  btrfs_free_dummy_fs_info() is still
> there.
With latest changes there are no complaints of the kernel any longer. btrfs selftests pass, mounting and unmounting a btrfs partition works without any suspicious dmesg output.
Comment 35 Christophe Leroy 2019-08-16 15:20:47 UTC
Le 16/08/2019 à 16:38, bugzilla-daemon@bugzilla.kernel.org a écrit :
> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #34 from Erhard F. (erhard_f@mailbox.org) ---
> On Fri, 16 Aug 2019 08:22:31 +0000
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
>> https://bugzilla.kernel.org/show_bug.cgi?id=204371
>>
>> --- Comment #32 from Christophe Leroy (christophe.leroy@c-s.fr) ---
>> Then see if the WARNING on kfree() in  btrfs_free_dummy_fs_info() is still
>> there.
> With latest changes there are no complaints of the kernel any longer. btrfs
> selftests pass, mounting and unmounting a btrfs partition works without any
> suspicious dmesg output.
> 

That's good news. Will you handle submitting the patch to BTRFS file 
system ?
Comment 36 Erhard F. 2019-08-16 18:02:07 UTC
On Fri, 16 Aug 2019 15:20:47 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #35 from Christophe Leroy (christophe.leroy@c-s.fr) ---
> That's good news. Will you handle submitting the patch to BTRFS file 
> system ?
Thats nice of you. But as my part in this process was only searching & replacing some code without deeper knowledge of what it's doing, I guess the patch is yours. ;) Also if any questions or follow-up patches arise I am not the right person to ask.

And probably I should test it on the G5 first, the 'BUG kmalloc-4k (Tainted: G        W        ): Object padding overwritten' happened here too.
Comment 37 Erhard F. 2019-08-16 20:51:35 UTC
On Fri, 16 Aug 2019 15:20:47 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

Ok, tested the G5 + patch now. It boots from a btrfs partition with SLUB debugging + btrfs debug & selftests enabled. So at least on the PowerPC side everything is back to working condition again.
Comment 38 Christophe Leroy 2019-08-17 08:09:51 UTC
Le 30/07/2019 à 20:52, bugzilla-daemon@bugzilla.kernel.org a écrit :
> https://bugzilla.kernel.org/show_bug.cgi?id=204371
> 
> --- Comment #2 from Andrew Morton (akpm@linux-foundation.org) ---
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).

Reply all replies to bugzilla-daemon@bugzilla.kernel.org only.


[...]


> 
> cc'ing various people here.

Hum ... only got that email through the bugzilla interface, and CC'ed 
people don't show up.


> 
> I suspect proc_cgroup_show() is innocent and that perhaps
> bpf_prepare_filter() had a memory scribble.  iirc there has been at
> least one recent pretty serious bpf fix applied recently.  Can others
> please take a look?
> 
> (Seriously - please don't modify this report via the bugzilla web interface!)
> 

Haven't got the original CC'ed list, so please reply with missing Cc's 
if any.

We have well progressed on this case.

Erhard made a relation being this "Object padding overwritten" issue 
arising on any driver, and the presence of the BTRFS driver.

Then he was able to bisect the issue to:

commit 69d2480456d1baf027a86e530989d7bedd698d5f
Author: David Sterba <dsterba@suse.com>
Date:   Fri Jun 29 10:56:44 2018 +0200

     btrfs: use copy_page for copying pages instead of memcpy

     Use the helper that's possibly optimized for full page copies.

     Signed-off-by: David Sterba <dsterba@suse.com>



After looking in the code, it has appeared that some of the said "pages" 
were allocated with "kzalloc()".

Using the patch https://patchwork.ozlabs.org/patch/1148033/ Erhard 
confirmed that some btrfs functions were calling copy_page() with 
misaligned destinations.

copy_page(), at least on powerpc, expects cache aligned destination.

The patch https://patchwork.ozlabs.org/patch/1148606/ fixes the issue.

Christophe

---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Comment 39 David Sterba 2019-08-22 13:39:43 UTC
Though I don't like neither of the patches, I'll apply one of them so it works and we can think of a better fix later.

Note You need to log in before you can comment on or make changes to this bug.