Created attachment 276381 [details] The (compressed) crafted image which causes crash - Overview ext4: use-after-free() detected by KASAN when mounting and operating a crafted ext4 image - Reproduce (on a KASAN build of 4.17-rc4) # mkdir mnt # mount -t ext4 242.img mnt # gcc -o poc poc.c # ./poc ./mnt - POC (poc.c) #define _GNU_SOURCE #include <sys/types.h> #include <sys/mount.h> #include <sys/mman.h> #include <sys/stat.h> #include <sys/xattr.h> #include <dirent.h> #include <errno.h> #include <error.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> static void activity(char *mpoint) { char *xattr; int err; err = asprintf(&xattr, "%s/foo/bar/xattr", mpoint); char buf2[113]; memset(buf2, 0, sizeof(buf2)); listxattr(xattr, buf2, sizeof(buf2)); } int main(int argc, char *argv[]) { activity(argv[1]); return 0; } - Kernel message [ 382.459225] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 389.596693] ================================================================== [ 389.598395] BUG: KASAN: use-after-free in ext4_xattr_list_entries+0x120/0x190 [ 389.599925] Read of size 4 at addr ffff8801f0d00080 by task a.out/1376 [ 389.601669] CPU: 0 PID: 1376 Comm: a.out Not tainted 4.17.0-rc4+ #5 [ 389.601672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 389.601679] Call Trace: [ 389.601719] dump_stack+0x7b/0xb5 [ 389.601754] print_address_description+0x70/0x290 [ 389.601760] kasan_report+0x291/0x390 [ 389.601765] ? ext4_xattr_list_entries+0x120/0x190 [ 389.601771] __asan_load4+0x78/0x80 [ 389.601776] ext4_xattr_list_entries+0x120/0x190 [ 389.601781] ext4_listxattr+0x32e/0x5f0 [ 389.601787] ? ext4_xattr_get+0x4d0/0x4d0 [ 389.601791] ? __kmalloc_node+0x11e/0x2e0 [ 389.601796] ? ext4_xattr_get+0x4d0/0x4d0 [ 389.601813] vfs_listxattr+0x9d/0xc0 [ 389.601818] listxattr+0x58/0xd0 [ 389.601823] path_listxattr+0xb9/0x120 [ 389.601828] ? listxattr+0xd0/0xd0 [ 389.601839] ? vm_brk+0x20/0x20 [ 389.601845] __x64_sys_listxattr+0x48/0x50 [ 389.601869] do_syscall_64+0x78/0x170 [ 389.601885] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 389.601903] RIP: 0033:0x7fcf8ac8a0d7 [ 389.601906] RSP: 002b:00007ffc8733aa58 EFLAGS: 00000202 ORIG_RAX: 00000000000000c2 [ 389.601916] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8ac8a0d7 [ 389.601918] RDX: 0000000000000071 RSI: 00007ffc8733aa80 RDI: 00000000015a5080 [ 389.601921] RBP: 00007ffc8733ab00 R08: 00000000015a5010 R09: 0000000000000000 [ 389.601923] R10: 00000000000001ab R11: 0000000000000202 R12: 0000000000400550 [ 389.601926] R13: 00007ffc8733ac00 R14: 0000000000000000 R15: 0000000000000000 [ 389.602286] Allocated by task 464: [ 389.603041] save_stack+0x46/0xd0 [ 389.603046] kasan_kmalloc+0xad/0xe0 [ 389.603050] kasan_slab_alloc+0x11/0x20 [ 389.603054] kmem_cache_alloc+0xd1/0x1e0 [ 389.603080] skb_clone+0x91/0x160 [ 389.603102] netlink_broadcast_filtered+0x4f7/0x600 [ 389.603110] netlink_sendmsg+0x676/0x6e0 [ 389.603121] sock_sendmsg+0x76/0x80 [ 389.603126] ___sys_sendmsg+0x4cd/0x4e0 [ 389.603130] __sys_sendmsg+0xea/0x170 [ 389.603134] __x64_sys_sendmsg+0x48/0x50 [ 389.603138] do_syscall_64+0x78/0x170 [ 389.603143] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 389.603486] Freed by task 464: [ 389.604150] save_stack+0x46/0xd0 [ 389.604154] __kasan_slab_free+0x13c/0x1a0 [ 389.604158] kasan_slab_free+0xe/0x10 [ 389.604161] kmem_cache_free+0x89/0x1e0 [ 389.604165] kfree_skbmem+0x92/0xa0 [ 389.604169] kfree_skb+0x5e/0xf0 [ 389.604173] netlink_broadcast_filtered+0x470/0x600 [ 389.604176] netlink_sendmsg+0x676/0x6e0 [ 389.604180] sock_sendmsg+0x76/0x80 [ 389.604184] ___sys_sendmsg+0x4cd/0x4e0 [ 389.604187] __sys_sendmsg+0xea/0x170 [ 389.604191] __x64_sys_sendmsg+0x48/0x50 [ 389.604195] do_syscall_64+0x78/0x170 [ 389.604200] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 389.604547] The buggy address belongs to the object at ffff8801f0d00000 which belongs to the cache skbuff_head_cache of size 208 [ 389.607302] The buggy address is located 128 bytes inside of 208-byte region [ffff8801f0d00000, ffff8801f0d000d0) [ 389.609737] The buggy address belongs to the page: [ 389.610763] page:ffffea0007c34000 count:1 mapcount:0 mapping:0000000000000000 index:0x0 [ 389.612457] flags: 0x2ffff0000000100(slab) [ 389.613340] raw: 02ffff0000000100 0000000000000000 0000000000000000 00000001000c000c [ 389.614972] raw: ffffea00077d8940 0000000b0000000b ffff8801f5808540 0000000000000000 [ 389.616588] page dumped because: kasan: bad access detected [ 389.618099] Memory state around the buggy address: [ 389.625195] ffff8801f0cfff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 389.626736] ffff8801f0d00000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 389.628279] >ffff8801f0d00080: fb fb fb fb fb fb fb fb fb fb fc fc fc fc fc fc [ 389.629799] ^ [ 389.630497] ffff8801f0d00100: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb [ 389.632027] ffff8801f0d00180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 389.633542] ================================================================== [ 389.635079] Disabling lock debugging due to kernel taint - Analysis https://elixir.bootlin.com/linux/v4.17-rc7/source/fs/ext4/xattr.c#L663 for (; !IS_LAST_ENTRY(entry); entry = EXT4_XATTR_NEXT(entry)) { const struct xattr_handler *handler = ext4_xattr_handler(entry->e_name_index); Based on KASAN report, a dangling ext4_xattr_entry pointer is accessed during the execution of POC. Reported by Wen Xu (wen.xu@gatech.edu) from SSLab at Gatech.
This replicates on v4.17, but it doesn't replicate with the tip of ext4.git tree (which was pulled by Linus during the post 4.17 merge window). Still TBD which of the patches fixes this particular POC replication.
(In reply to Theodore Tso from comment #1) > This replicates on v4.17, but it doesn't replicate with the tip of ext4.git > tree (which was pulled by Linus during the post 4.17 merge window). > > Still TBD which of the patches fixes this particular POC replication. Hi Ted, In fact I reproduce with latest ext4 dev branch while fuzzing with Linus' branch. The last commit of the ext4.git branch I use to reproduce is commit 4f2f76f751433908364ccff82f437a57d0e6e9b7 Author: Jan Kara <jack@suse.cz> Date: Fri May 25 12:51:25 2018 -0400 ext4: fix fencepost error in check for inode count overflow during resize ext4_resize_fs() has an off-by-one bug when checking whether growing of a filesystem will not overflow inode count. As a result it allows a filesystem with 8192 inodes per group to grow to 64TB which overflows inode count to 0 and makes filesystem unusable. Fix it. Cc: stable@vger.kernel.org Fixes: 3f8a6411fbada1fa482276591e037f3b1adcf55b Reported-by: Jaco Kroon <jaco@uls.co.za> Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Andreas Dilger <adilger@dilger.ca> And I can still get the following information with POC: [ 159.253263] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 160.963191] ================================================================== [ 160.964784] BUG: KASAN: slab-out-of-bounds in ext4_xattr_list_entries+0x120/0x190 [ 160.966274] Read of size 4 at addr ffff8801e33d3080 by task a.out/1348 [ 160.967882] CPU: 0 PID: 1348 Comm: a.out Not tainted 4.17.0-rc4+ #5 [ 160.967886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 160.967889] Call Trace: [ 160.967901] dump_stack+0x7b/0xb5 [ 160.967913] print_address_description+0x70/0x290 [ 160.967919] kasan_report+0x291/0x390 [ 160.967926] ? ext4_xattr_list_entries+0x120/0x190 [ 160.967933] __asan_load4+0x78/0x80 [ 160.967940] ext4_xattr_list_entries+0x120/0x190 [ 160.967947] ext4_listxattr+0x32e/0x5f0 [ 160.967955] ? ext4_xattr_get+0x4d0/0x4d0 [ 160.967961] ? __kmalloc_node+0x11e/0x2e0 [ 160.967967] ? ext4_xattr_get+0x4d0/0x4d0 [ 160.967975] vfs_listxattr+0x9d/0xc0 [ 160.967981] listxattr+0x58/0xd0 [ 160.967988] path_listxattr+0xb9/0x120 [ 160.967994] ? listxattr+0xd0/0xd0 [ 160.968002] ? vm_brk+0x20/0x20 [ 160.968009] __x64_sys_listxattr+0x48/0x50 [ 160.968017] do_syscall_64+0x78/0x170 [ 160.968027] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 160.968031] RIP: 0033:0x7ff483fe20d7 [ 160.968035] RSP: 002b:00007ffd3a2bc418 EFLAGS: 00000206 ORIG_RAX: 00000000000000c2 [ 160.968041] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff483fe20d7 [ 160.968044] RDX: 0000000000000071 RSI: 00007ffd3a2bc440 RDI: 0000000000af7080 [ 160.968047] RBP: 00007ffd3a2bc4c0 R08: 0000000000af7010 R09: 0000000000000000 [ 160.968050] R10: 00000000000001ab R11: 0000000000000206 R12: 0000000000400550 [ 160.968053] R13: 00007ffd3a2bc5c0 R14: 0000000000000000 R15: 0000000000000000 [ 160.968377] Allocated by task 0: [ 160.969027] (stack is not available) [ 160.970057] Freed by task 0: [ 160.970650] (stack is not available) [ 160.971681] The buggy address belongs to the object at ffff8801e33d3080 which belongs to the cache pid(250:lvm2-lvmetad.service) of size 64 [ 160.974417] The buggy address is located 0 bytes inside of 64-byte region [ffff8801e33d3080, ffff8801e33d30c0) [ 160.976653] The buggy address belongs to the page: [ 160.977605] page:ffffea00078cf4c0 count:1 mapcount:0 mapping:0000000000000000 index:0xffff8801e33d3000 [ 160.979440] flags: 0x2ffff0000000100(slab) [ 160.980260] raw: 02ffff0000000100 0000000000000000 ffff8801e33d3000 000000018020001f [ 160.981780] raw: dead000000000100 dead000000000200 ffff8801e0c2da40 ffff8801f3c7bb80 [ 160.983305] page dumped because: kasan: bad access detected [ 160.984406] page->mem_cgroup:ffff8801f3c7bb80 [ 160.985584] Memory state around the buggy address: [ 160.989718] ffff8801e33d2f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 160.991161] ffff8801e33d3000: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc [ 160.992579] >ffff8801e33d3080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 160.993997] ^ [ 160.994660] ffff8801e33d3100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 160.996073] ffff8801e33d3180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 160.997486] ================================================================== [ 160.998913] Disabling lock debugging due to kernel taint The POC is simplified. I do not know whether use my original POC can help to reproduce but I attached.
Created attachment 276473 [details] poc.c
The original POC and your simplified POC bug are catching different problems. Part of the reason for this is that your fuzzed image has many different file system corruptions. So for example, inode #7 is completely corrupted in some strange way that causes e2fsck to not be able to fix file system at all. The kernel problems are triggered by it still trigger if you use debugfs to "clri <7>". This makes it harder for me to figure out what was the specific file system corruption that was causing the problem being reported in a specific bug. So with the simplified poc, I'm no longer seeing a problem with the tip of the ext4 branch because the simplified poc is only testing ext4's handling a specific corruption, and that's now fixed: [ 32.516640] EXT4-fs error (device loop0): ext4_xattr_ibody_find:2191: inode #14: comm poc-199997: corrupted in-inode xattr With your full POC, we're now tripping on the block 35 problem. I have a fix for it that I fixed for Bugzilla #200001, and I'm finding that one of the patches that I used to fix that problem also now fixes Bugzilla #19997, and it also fixes Bugzilla #199977 with your expanded poc.c: [ 26.736417] EXT4-fs error (device loop0): ext4_xattr_block_list:708: inode #15: comm poc-199977: corrupted xattr block 35 The specific problem here is that block 35 is being used as a block allocation bitmap. On a number of your fuzzed images, one or more of the inodes use that same block as an external xattr block. So the problem is that block 35 is verified as an allocation bitmap; and then the buffer_verified flag is set. Then when we call ext4_xattr_check_block(), the code looks at the buffer_verified flag, and says, "Hurr durr.... this block has already been verified (as an allocation bitmap) so it must be OK; no need to verify it again (as an xattr block)". So now we get to an interesting philosophical question --- is the kernel bug fundamentally about the fuzzed image, or the poc.c code that was submitted to trigger the problem? And if you have a fuzzed image with many many corruptions, and a different poc.c is used to trigger kernel bug related to a different aspect of the image corruption, and with a completely different kernel signature, is it the same bug or a different bug?
(In reply to Theodore Tso from comment #4) > The original POC and your simplified POC bug are catching different > problems. Part of the reason for this is that your fuzzed image has many > different file system corruptions. So for example, inode #7 is completely > corrupted in some strange way that causes e2fsck to not be able to fix file > system at all. The kernel problems are triggered by it still trigger if you > use debugfs to "clri <7>". This makes it harder for me to figure out what > was the specific file system corruption that was causing the problem being > reported in a specific bug. > > So with the simplified poc, I'm no longer seeing a problem with the tip of > the ext4 branch because the simplified poc is only testing ext4's handling a > specific corruption, and that's now fixed: > > [ 32.516640] EXT4-fs error (device loop0): ext4_xattr_ibody_find:2191: > inode #14: comm poc-199997: corrupted in-inode xattr > > With your full POC, we're now tripping on the block 35 problem. I have a > fix for it that I fixed for Bugzilla #200001, and I'm finding that one of > the patches that I used to fix that problem also now fixes Bugzilla #19997, > and it also fixes Bugzilla #199977 with your expanded poc.c: > > [ 26.736417] EXT4-fs error (device loop0): ext4_xattr_block_list:708: > inode #15: comm poc-199977: corrupted xattr block 35 > > The specific problem here is that block 35 is being used as a block > allocation bitmap. On a number of your fuzzed images, one or more of the > inodes use that same block as an external xattr block. So the problem is > that block 35 is verified as an allocation bitmap; and then the > buffer_verified flag is set. Then when we call ext4_xattr_check_block(), > the code looks at the buffer_verified flag, and says, "Hurr durr.... this > block has already been verified (as an allocation bitmap) so it must be OK; > no need to verify it again (as an xattr block)". > I see, it seems to be a block type confusion :) > So now we get to an interesting philosophical question --- is the kernel bug > fundamentally about the fuzzed image, or the poc.c code that was submitted > to trigger the problem? And if you have a fuzzed image with many many > corruptions, and a different poc.c is used to trigger kernel bug related to > a different aspect of the image corruption, and with a completely different > kernel signature, is it the same bug or a different bug? Hmm, to me, a fs bug requires both fuzzed image and POC to trigger. They are all important. And I think it is better to use the root cause (the improper method to handle a particular field of a disk image appearing in the whole implementation) to differentiate bugs. The bugs of the same root cause but are triggered at different places because of different POCs should be treated as one bug. Otherwise, it is better to consider them as different ones.
OK, so based on that, I'll treat #199977 as the original simplified poc.c and image. In which case it should be a dup of one of your other older, closed bugs. I'll also then mark #199997 as a dup of #200001, which is fixed via the two patches I sent out for review. (There is also a discussion on that thread about how we should fix the more general problem of block type confusion and verification.) With these two patches, as far as I can tell we've cleared all of the bugs you've opened relating to extended attributes, with the possible exception of #20005, which is related to inline data, but which may also have some relationship to ext4's xattr codepaths.
(In reply to Theodore Tso from comment #6) > OK, so based on that, I'll treat #199977 as the original simplified poc.c > and image. In which case it should be a dup of one of your other older, > closed bugs. > > I'll also then mark #199997 as a dup of #200001, which is fixed via the two > patches I sent out for review. (There is also a discussion on that thread > about how we should fix the more general problem of block type confusion and > verification.) > > With these two patches, as far as I can tell we've cleared all of the bugs > you've opened relating to extended attributes, with the possible exception > of #20005, which is related to inline data, but which may also have some > relationship to ext4's xattr codepaths. Got it. I will also do some fuzzing tests on xattr again once the patches are committed. Thanks.