Recent commits to 5.15 btrfs caused null pointer dereference on 32-bit ARM platforms. I've bisected this, but keep in mind real culprit likely few commits before. Zstd only crashes because filesystem using it. Would happily crash in e.g. LZO if filesystem uses it as well, so commits in zstd and lzo are probably not issue on their own, but rather preceding page/mem related refactor itself. [ 20.206835][ T210] Unable to handle kernel NULL pointer dereference at virtual address 00000000 [ 20.216301][ T210] pgd = c4159ed3 [ 20.220709][ T210] [00000000] *pgd=00000000 [ 20.225178][ T210] Internal error: Oops: 5 [#1] PREEMPT SMP ARM [ 20.231231][ T210] Modules linked in: [ 20.235028][ T210] CPU: 0 PID: 210 Comm: kworker/u2:3 Not tainted 5.14.0-rc79+ #12 [ 20.242725][ T210] Hardware name: Allwinner sun4i/sun5i Families [ 20.248853][ T210] Workqueue: btrfs-delalloc btrfs_work_helper [ 20.254853][ T210] PC is at mmiocpy+0x48/0x330 [ 20.259439][ T210] LR is at ZSTD_compressStream_generic+0x15c/0x28c [ 20.265832][ T210] pc : [<c0649588>] lr : [<c0629648>] psr: 20010013 [ 20.272821][ T210] sp : c20e9cb4 ip : 00000000 fp : 00001000 [ 20.278768][ T210] r10: c20e9d28 r9 : 00001000 r8 : 00001000 [ 20.284714][ T210] r7 : 00000000 r6 : c2e27000 r5 : c2e27000 r4 : c2400008 [ 20.291966][ T210] r3 : 00002000 r2 : 00000f80 r1 : 00000000 r0 : c24027c0 [ 20.299218][ T210] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 20.307085][ T210] Control: 10c5387d Table: 429cc019 DAC: 00000051 [ 20.313549][ T210] Register r0 information: non-slab/vmalloc memory [ 20.319953][ T210] Register r1 information: NULL pointer [ 20.325394][ T210] Register r2 information: non-paged memory [ 20.331179][ T210] Register r3 information: non-paged memory [ 20.336964][ T210] Register r4 information: non-slab/vmalloc memory [ 20.343358][ T210] Register r5 information: non-slab/vmalloc memory [ 20.349751][ T210] Register r6 information: non-slab/vmalloc memory [ 20.356143][ T210] Register r7 information: NULL pointer [ 20.361582][ T210] Register r8 information: non-paged memory [ 20.367367][ T210] Register r9 information: non-paged memory [ 20.373151][ T210] Register r10 information: non-slab/vmalloc memory [ 20.379630][ T210] Register r11 information: non-paged memory [ 20.385503][ T210] Register r12 information: NULL pointer [ 20.391028][ T210] Process kworker/u2:3 (pid: 210, stack limit = 0xca2ca311) [ 20.398202][ T210] Stack: (0xc20e9cb4 to 0xc20ea000) [ 20.403294][ T210] 9ca0: c2e27000 c2e27000 00000000 [ 20.412207][ T210] 9cc0: 00001000 c24027c0 c2400008 c0629648 00000000 00000000 ef356bc4 00000000 [ 20.421121][ T210] 9ce0: c1140e80 c2e28000 00001000 00002000 00000000 c2091d74 c2091d68 c100b008 [ 20.430032][ T210] 9d00: 00000000 00001180 ef5e7d7c 00000000 00000000 c06297dc c20e9d24 00000000 [ 20.438943][ T210] 9d20: c20e9ea8 00001000 00001000 27fc220f c2091d58 c2091d58 00000000 00002000 [ 20.447853][ T210] 9d40: 00000001 c049444c 00000004 00000006 00000004 00000000 00000000 00000000 [ 20.456765][ T210] 9d60: 00001180 00000000 c2400000 0027cd1c c11a0820 c2400008 c2091d74 c2091d68 [ 20.465676][ T210] 9d80: ef356bc4 00000002 00023000 00000000 c2d6fc80 c1a902c0 c20e9ea0 c100b008 [ 20.474586][ T210] 9da0: c20e9ea8 c20e9ea4 c2091d58 0000000d 0000000d 0000000d 00000004 00000004 [ 20.483497][ T210] 9dc0: 00000006 00000004 00000000 00000000 00000000 27fc220f ef3618bc 00000003 [ 20.492409][ T210] 9de0: c1a902c0 00023000 00000000 c20e9ea0 c2091d58 00023000 00000000 c0496798 [ 20.501320][ T210] 9e00: c2d6fc80 c20e9ea0 c20e9ea8 c20e9ea4 00000003 c1a901b0 00001000 00000000 [ 20.510231][ T210] 9e20: 00000000 c043e850 00000003 c043c068 c2d6fc80 c20e9ea0 c20e9ea8 c20e9ea4 [ 20.519143][ T210] 9e40: 00000000 00009000 c2dfe328 c042b954 c1f85ed0 1fed424e 00024180 00000000 [ 20.528054][ T210] 9e60: 00000000 c2d6fc80 00024fff 00000000 00025000 00000000 00000003 c2dfea08 [ 20.536966][ T210] 9e80: 00000025 00001000 00000000 c1a90000 c1f7b000 c100b008 c11492c8 00000000 [ 20.545877][ T210] 9ea0: 00000000 00000000 00000000 27fc220f 20010093 c2dfea30 c2dfea3c c1406e00 [ 20.554789][ T210] 9ec0: c2dfea30 c043e850 00000040 c2dfea40 c28358bc c043c4ec c23f5f00 c2dfea3c [ 20.563702][ T210] 9ee0: c1406e00 c0475c3c c23f5c88 27fc220f c23fd873 c2dfea3c c2835880 c1406e00 [ 20.572614][ T210] 9f00: c1408100 00000000 00000040 c2dfea40 c28358bc c014ef68 c1003d00 c1406e18 [ 20.581525][ T210] 9f20: c1406e00 00000000 c1408c00 c2835880 c2835894 c1406e00 00000088 c1003d00 [ 20.590438][ T210] 9f40: c1406e18 c1406e00 ffffe000 c014f210 60000113 c1141070 c0d4783c c0afc420 [ 20.599349][ T210] 9f60: c2861ea4 c2861e80 c2861dc0 c014f1cc c2835880 00000000 c20e8000 c20f5eac [ 20.608260][ T210] 9f80: c2861ea4 c0156aa4 00000000 c2861dc0 c0156924 00000000 00000000 00000000 [ 20.617170][ T210] 9fa0: 00000000 00000000 00000000 c0100150 00000000 00000000 00000000 00000000 [ 20.626079][ T210] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 20.634988][ T210] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 20.643897][ T210] [<c0649588>] (mmiocpy) from [<c0629648>] (ZSTD_compressStream_generic+0x15c/0x28c) [ 20.653271][ T210] [<c0629648>] (ZSTD_compressStream_generic) from [<c06297dc>] (ZSTD_compressStream+0x64/0xa0) [ 20.663496][ T210] [<c06297dc>] (ZSTD_compressStream) from [<c049444c>] (zstd_compress_pages+0x170/0x488) [ 20.673217][ T210] [<c049444c>] (zstd_compress_pages) from [<c0496798>] (btrfs_compress_pages+0x124/0x12c) [ 20.683019][ T210] [<c0496798>] (btrfs_compress_pages) from [<c043c068>] (compress_file_range+0x3c0/0x834) [ 20.692836][ T210] [<c043c068>] (compress_file_range) from [<c043c4ec>] (async_cow_start+0x10/0x28) [ 20.702034][ T210] [<c043c4ec>] (async_cow_start) from [<c0475c3c>] (btrfs_work_helper+0x100/0x230) [ 20.711232][ T210] [<c0475c3c>] (btrfs_work_helper) from [<c014ef68>] (process_one_work+0x1b4/0x418) [ 20.720517][ T210] [<c014ef68>] (process_one_work) from [<c014f210>] (worker_thread+0x44/0x524) [ 20.729357][ T210] [<c014f210>] (worker_thread) from [<c0156aa4>] (kthread+0x180/0x1b0) [ 20.737504][ T210] [<c0156aa4>] (kthread) from [<c0100150>] (ret_from_fork+0x14/0x24) [ 20.745471][ T210] Exception stack(0xc20e9fb0 to 0xc20e9ff8) [ 20.751254][ T210] 9fa0: 00000000 00000000 00000000 00000000 [ 20.760164][ T210] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 20.769070][ T210] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 git bisect start # bad: [519d81956ee277b4419c723adfb154603c2565ba] Linux 5.15-rc6 git bisect bad 519d81956ee277b4419c723adfb154603c2565ba # good: [6efb943b8616ec53a5e444193dccf1af9ad627b5] Linux 5.13-rc1 git bisect good 182c7355be0383c202572eb06d8d2110d6cda003 # good: [6efb943b8616ec53a5e444193dccf1af9ad627b5] Linux 5.13-rc1 git bisect good 182c7355be0383c202572eb06d8d2110d6cda003 # good: [6efb943b8616ec53a5e444193dccf1af9ad627b5] Linux 5.13-rc1 git bisect good 182c7355be0383c202572eb06d8d2110d6cda003 # good: [e083bbd6040f4efa5c13633fb4e460b919d69dae] Merge tag 'arm-dt-5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc git bisect good e083bbd6040f4efa5c13633fb4e460b919d69dae # bad: [0d290223a6c77107b1c3988959e49279a8dafaba] Merge tag 'sound-5.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound git bisect bad 0d290223a6c77107b1c3988959e49279a8dafaba # good: [9c849ce86e0fa93a218614eac562ace44053d7ce] Merge tag '5.15-rc-smb3-fixes-part1' of git://git.samba.org/sfrench/cifs-2.6 git bisect good 9c849ce86e0fa93a218614eac562ace44053d7ce # good: [29ce8f9701072fc221d9c38ad952de1a9578f95c] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net git bisect good 29ce8f9701072fc221d9c38ad952de1a9578f95c # bad: [c6c3c5704ba70820f6b632982abde06661b7222a] Merge tag 'driver-core-5.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core git bisect bad c6c3c5704ba70820f6b632982abde06661b7222a # bad: [359f3d743f3a762cc2cc7ddb7c6fb4c57b9a06cc] Merge tag 'mmc-v5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc git bisect bad 359f3d743f3a762cc2cc7ddb7c6fb4c57b9a06cc # bad: [916d636e0a2df48be48b573d8ec9070408d7681f] Merge tag 'vfs-5.15-merge-1' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux git bisect bad 916d636e0a2df48be48b573d8ec9070408d7681f # bad: [63fb5879db7ca94fefac12cf7a5a051cee889c12] btrfs: zoned: add asserts on splitting extent_map git bisect bad 63fb5879db7ca94fefac12cf7a5a051cee889c12 # bad: [c2832898126fc320a0e2915b07bf8924cf54770e] btrfs: make relocate_one_page() handle subpage case git bisect bad c2832898126fc320a0e2915b07bf8924cf54770e # bad: [2ac691d8b3b1dd300a48b1763fa3a1434863070b] btrfs: avoid unnecessary lock and leaf splits when updating inode in the log git bisect bad 2ac691d8b3b1dd300a48b1763fa3a1434863070b # bad: [bbaf9715f3f5b5ff0de71da91fcc34ee9c198ed8] btrfs: compression: drop kmap/kunmap from zstd git bisect bad bbaf9715f3f5b5ff0de71da91fcc34ee9c198ed8 # good: [67d5e289a193c643a70ceda437c625e2bc876dbc] btrfs: remove max argument from generic_bin_search git bisect good 67d5e289a193c643a70ceda437c625e2bc876dbc # good: [b0ee5e1ec44afda53aaa37f8c41ad00d170506cb] btrfs: drop from __GFP_HIGHMEM all allocations git bisect good b0ee5e1ec44afda53aaa37f8c41ad00d170506cb # good: [696ab562e6df9fbafd6052d8ce4aafcb2ed16069] btrfs: compression: drop kmap/kunmap from zlib git bisect good 696ab562e6df9fbafd6052d8ce4aafcb2ed16069 # first bad commit: [bbaf9715f3f5b5ff0de71da91fcc34ee9c198ed8] btrfs: compression: drop kmap/kunmap from zstd
NB: disregard kernel version in log - byproduct of bisecting and going some pre-5.15-rc1 commits. Bug definitely tested to exist up to 5.15-rc6
Thanks for the report. This is most likely caused by my patches removing kmap calls. This affects 32bit architectures with enabled HIGHMEM. There was another report https://lore.kernel.org/all/CAJCQCtT+OuemovPO7GZk8Y8=qtOObr0XTDp8jh4OHD6y84AFxw@mail.gmail.com/ I'll send a revert as it's still in unreleased 5.15-rc.
I've got similar impression as bisect revolved all around these and after of couple of experiments I've got same very thing in LZO so bisect found where it finally became in effect in my config but real issue apparently happened before. Thanks, fix would be really appreciated as it kinda breaks btrfs on 5.15 for me and otherwise I'd had to skip 5.15 on at least ARM32 targets (maybe others, arms were first where it surfaced upon giving -rc a try). And in the end btrfs is really nice thing, I'll be sad if some users would get other impression.
Yeah, though intel 32bit is almost extinct nowadays, the 32bit ARM machines are not and used for low power storage boxes. A fix would have to be merged either way, pre 5.15 or after release via stable trees but potentially causing more damage.
I see revert arrived and I've tested it (as of a379fbbcb88bcf43d886977c6fb91fb43f330b84 commit). It has corrected problem, thanks! Yay! On side note, btrfs got strong points even beyond "storage boxes". For example, DUP storage scheme improves chances system would complete boot even if conditions were imperfect (and fatal for most of other FS) and boot sequence not really assumed there can be more than 1 boot device involved (e.g. booting from SDcard or eMMC). Its very sad state of things when single sporadic bad sector, etc on underlying storage ruins whole boot sequence. Especially in automatic/unsupervised systems. That's where btrfs comes to help: it would usually say "bad csum at XXX, corrected", repairing from another copy and if it repeats often I'd to schedule maintenance, while it still keeps going so its normal course of actions instead of emergency. Way better vs suddenly getting unbootable system, especially if it's been control or automation running unsupervised. Self healing properties of btrfs and lack of fsck proven to be really neat in these cases. And best of all: it easy to get going and flexible (e.g. unlike RAID it not cares of device size, so can be easily grown/shrinked as needed, minimizing OS image after composing it, or expanding FS to full available device size as needed, etc). So I eventually test how upcoming kernels perform. I actually caught it about -rc2 but wasn't sure what it is me got something wrong in kernel configuration or something. Then I've got dragged into few other things, so only got to fullfledged bisect around -rc6 and quickly got idea it revolves around suspicious refactor and likely isn't my fault. I'm sorry for rather late timings. Either way, thanks for fixing. Its nice this bullet has been dodged. I have pleasure to mark this RESOLVED as I can't reproduce in my configs anymore :)