Bug 219504
| Summary: | iomap/buffered-io/XFS crashes with kernel Version > 6.1.91. Perhaps Changes in kernel 6.1.92 (and up) for XFS/iomap causing the problems? | ||
|---|---|---|---|
| Product: | File System | Reporter: | Mike-SPC (speedcracker) |
| Component: | Other | Assignee: | FileSystem/XFS Default Virtual Assignee (filesystem_xfs) |
| Status: | RESOLVED CODE_FIX | ||
| Severity: | normal | CC: | leo.lilong |
| Priority: | P3 | ||
| Hardware: | AMD | ||
| OS: | Linux | ||
| Kernel Version: | since 6.1.92 and up | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
|
Description
Mike-SPC
2024-11-17 13:42:44 UTC
Hi, Mike:
Look at the code of 6.1.106:
970 /*
971 * If there is no more data to scan, all that is left is to
972 * punch out the remaining range.
973 */
974 if (start_byte == -ENXIO || start_byte == scan_end_byte)
975 break;
976 if (start_byte < 0) {
977 error = start_byte;
978 goto out_unlock;
979 }
980 WARN_ON_ONCE(start_byte < punch_start_byte);
981 WARN_ON_ONCE(start_byte > scan_end_byte);
982
983 /*
984 * We find the end of this contiguous cached data range by
985 * seeking from start_byte to the beginning of the next hole.
986 */
987 data_end = mapping_seek_hole_data(inode->i_mapping, start_byte,
988 scan_end_byte, SEEK_HOLE);
989 if (data_end < 0) {
990 error = data_end;
991 goto out_unlock;
992 }
993 WARN_ON_ONCE(data_end <= start_byte);
994 WARN_ON_ONCE(data_end > scan_end_byte);
995
996 error = iomap_write_delalloc_scan(inode, &punch_start_byte,
997 start_byte, data_end, punch);
Looking at your warning stack, it reminds me of a problem[1] I tried to solve before, but it seems different. In my case, there was only a warning on line 993. Perhaps it's not the same issue. Below is a link to my attempted fix patch, which wasn't accepted, but hopefully it can be helpful to you.
[1] https://patchwork.kernel.org/project/xfs/patch/20231216115559.3823359-1-leo.lilong@huawei.com/
Long Li
Hello Long Li, thanks for investigation. It seems, that the patch went into the kernel 6.1.113: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=e3aa99b13a99405d935910306d1bbf419edfd679 It looks like this: --- snip --- diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 98617f00101d68..1833608f39318e 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -990,7 +990,15 @@ static int iomap_write_delalloc_release(struct inode *inode, error = data_end; goto out_unlock; } - WARN_ON_ONCE(data_end <= start_byte); + + /* + * If we race with post-direct I/O invalidation of the page cache, + * there might be no data left at start_byte. + */ + if (data_end == start_byte) + continue; + + WARN_ON_ONCE(data_end < start_byte); WARN_ON_ONCE(data_end > scan_end_byte); error = iomap_write_delalloc_scan(inode, &punch_start_byte, --- snap --- Looks like your's: https://patchwork.kernel.org/project/xfs/patch/20231216115559.3823359-1-leo.lilong@huawei.com/ Due to lack of time, I haven't gotten around to testing the latest kernel with this patch in it. Regards, Mike Hi there, rechecked the newest kernel version 6.1.120. Get the same problem. Here is the kernel-backtrace: [Thu Dec 19 14:03:21 2024] ------------[ cut here ]------------ [Thu Dec 19 14:03:21 2024] WARNING: CPU: 0 PID: 24318 at fs/iomap/buffered-io.c:980 iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O) dlm quota_v2 quota_tree autofs4 tcp_bbr sch_fq udf crc_itu_t ses enclosure hid_generic wmi_bmof usbhid edac_mce_amd uas crc32_pclmul aesni_intel hid crypto_simd usb_storage rapl bnx2 i2c_piix4 r8169 pcspkr k10temp ccp i2c_core sha1_generic video mpt3sas backlight wmi [Thu Dec 19 14:03:21 2024] CPU: 0 PID: 24318 Comm: disk014_0 Tainted: G S O 6.1.120_LFS_FILE01 #1 [Thu Dec 19 14:03:21 2024] Hardware name: To Be Filled By O.E.M. X370 Pro4/X370 Pro4, BIOS P10.08 01/22/2024 [Thu Dec 19 14:03:21 2024] EIP: iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] Code: 84 8d 00 00 00 8b 45 e0 8b 40 20 83 c0 10 e8 2f 31 e0 ff 83 c4 44 89 f0 5b 5e 5f 5d e9 f5 31 8e 00 90 0f 0b e9 13 fe ff ff 90 <0f> 0b e9 fd fd ff ff 90 0f 0b e9 6b fe ff ff 90 0f 0b 8d b6 00 00 [Thu Dec 19 14:03:21 2024] EAX: bb68e000 EBX: ffffff85 ECX: bb68f000 EDX: 0000007b [Thu Dec 19 14:03:21 2024] ESI: bb68f000 EDI: 00000000 EBP: 5137fbd0 ESP: 5137fb80 [Thu Dec 19 14:03:21 2024] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010293 [Thu Dec 19 14:03:21 2024] CR0: 80050033 CR2: 0175a2d0 CR3: 03d1b720 CR4: 00350ef0 [Thu Dec 19 14:03:21 2024] Call Trace: [Thu Dec 19 14:03:21 2024] ? show_regs.cold+0x16/0x1b [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] ? __warn+0x87/0xe0 [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] ? report_bug+0xe5/0x170 [Thu Dec 19 14:03:21 2024] ? exc_overflow+0x60/0x60 [Thu Dec 19 14:03:21 2024] ? handle_bug+0x2a/0x50 [Thu Dec 19 14:03:21 2024] ? exc_invalid_op+0x1e/0x70 [Thu Dec 19 14:03:21 2024] ? handle_exception+0x101/0x101 [Thu Dec 19 14:03:21 2024] ? exc_overflow+0x60/0x60 [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] ? exc_overflow+0x60/0x60 [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write_punch_delalloc+0x398/0x440 [Thu Dec 19 14:03:21 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Thu Dec 19 14:03:21 2024] ? xfs_buffered_write_iomap_end+0x52/0xc0 [Thu Dec 19 14:03:21 2024] ? xfs_buffered_write_iomap_end+0xc0/0xc0 [Thu Dec 19 14:03:21 2024] ? iomap_iter+0xce/0x4b0 [Thu Dec 19 14:03:21 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Thu Dec 19 14:03:21 2024] ? iomap_file_buffered_write+0xa9/0x420 [Thu Dec 19 14:03:21 2024] ? xfs_file_buffered_write+0x9d/0x2e0 [Thu Dec 19 14:03:21 2024] ? xfs_file_write_iter+0xc9/0x100 [Thu Dec 19 14:03:21 2024] ? fileio_exec_async+0x25e/0x3a0 [scst_vdisk] [Thu Dec 19 14:03:21 2024] ? fileio_exec_write+0x2ce/0x400 [scst_vdisk] [Thu Dec 19 14:03:21 2024] ? vdev_do_job+0x36/0xe0 [scst_vdisk] [Thu Dec 19 14:03:21 2024] ? fileio_exec+0x1f/0x30 [scst_vdisk] [Thu Dec 19 14:03:21 2024] ? scst_do_real_exec+0x51/0x130 [scst] [Thu Dec 19 14:03:21 2024] ? scst_exec_check_blocking+0xa8/0x220 [scst] [Thu Dec 19 14:03:21 2024] ? scst_process_active_cmd+0x200/0x18f0 [scst] [Thu Dec 19 14:03:21 2024] ? scst_cmd_thread+0x15c/0x500 [scst] [Thu Dec 19 14:03:21 2024] ? prepare_to_wait_event+0x160/0x160 [Thu Dec 19 14:03:21 2024] ? kthread+0xd2/0x100 [Thu Dec 19 14:03:21 2024] ? scst_cmd_done_local+0x90/0x90 [scst] [Thu Dec 19 14:03:21 2024] ? kthread_complete_and_exit+0x20/0x20 [Thu Dec 19 14:03:21 2024] ? ret_from_fork+0x1c/0x28 [Thu Dec 19 14:03:21 2024] ---[ end trace 0000000000000000 ]--- Thanks in advance and for investigation, Mike I think this might be the same problem I'm running in to, which only seems to happen on 32-bit kernels, starting with commit f43dc4dc3eff0. Problem is still present in 6.13. Easy repro steps are as follows: hash mkfs.xfs || apt install -y xfsprogs rm -f xfsimg.bin truncate -s 6G xfsimg.bin # I can reproduce this with an xfs image, but not with ext4 mkfs.xfs xfsimg.bin mkdir -p xfs mount xfsimg.bin xfs truncate -s 5G xfs/diskimg.bin mkfs.ext4 xfs/diskimg.bin # this can probably be any fs type, it happens with fat too mkdir -p mnt mount xfs/diskimg.bin mnt dd if=/dev/zero of=mnt/file.bin bs=1M of=mnt/file.bin bs=1M The above almost immediately prints a warning to the kernel log, after which there is a kworker thread hogging the CPU With the big caveat that I'm completely unfamiliar with this code, it seems to me the problem is that here: https://github.com/torvalds/linux/blame/ccb98ccef0e543c2bd4ef1a72270461957f3d8d0/mm/filemap.c#L2989 "bsz" is a 32-bit type on 32-bit kernels, and so when it gets used later in that same function to mask the 64-bit "start" value with "~(bsz - 1)", it's effectively truncating "start" to 32 bits. This is more or less confirmed by the actual values of "start_byte" and "punch_start_byte" when that WARN_ON_ONCE in buffer-io.c triggers, with one being (close to) a 32-bit truncated version of the other. Changing bsz to a 64-bit type fixes the problem for me. I am experiencing the same issue on a 32-bit system when using Kernel version 6.1.92 and above. As a non-programmer, I find this problem challenging to address independently. It would be greatly appreciated if a fix could be provided in the form of a patch. Could the maintainers consider releasing one? Thank you! On Sat, Jan 18, 2025 at 6:00 AM <bugzilla-daemon@kernel.org> wrote: > > https://bugzilla.kernel.org/show_bug.cgi?id=219504 > > --- Comment #6 from Mike-SPC (speedcracker@hotmail.com) --- > I am experiencing the same issue on a 32-bit system when using Kernel version > 6.1.92 and above. > > As a non-programmer, I find this problem challenging to address > independently. > > It would be greatly appreciated if a fix could be provided in the form of a > patch. Could the maintainers consider releasing one? There are patches for 2 separate 32-bit issues, both of which are in linux-next, though only one of them appears to have been selected for 6.1, 6.6 and 6.12. These patches are: https://lore.kernel.org/linux-xfs/20250102190540.1356838-1-marco.nelissen@gmail.com/ https://lore.kernel.org/linux-xfs/20250109041253.2494374-1-marco.nelissen@gmail.com/ (In reply to marco.nelissen from comment #7) > There are patches for 2 separate 32-bit issues, both of which are in > linux-next, > though only one of them appears to have been selected for 6.1, 6.6 and 6.12. > These patches are: > https://lore.kernel.org/linux-xfs/20250102190540.1356838-1-marco. > nelissen@gmail.com/ > https://lore.kernel.org/linux-xfs/20250109041253.2494374-1-marco. > nelissen@gmail.com/ It looks like both patches are in kernel 6.1.127. When I have time again, I will test kernel 6.1.127 (or higher) and report back. Thanks so far. >(In reply to marco.nelissen from comment #7) > On Sat, Jan 18, 2025 at 6:00 AM <bugzilla-daemon@kernel.org> wrote: > There are patches for 2 separate 32-bit issues, both of which are in > linux-next, > though only one of them appears to have been selected for 6.1, 6.6 and 6.12. > These patches are: > https://lore.kernel.org/linux-xfs/20250102190540.1356838-1-marco. > nelissen@gmail.com/ > https://lore.kernel.org/linux-xfs/20250109041253.2494374-1-marco. > nelissen@gmail.com/ Hi there, I've tested the latest kernel (6.1.128) and it works like it should be. :) Thanks again for investigation and fixing the bug. Regards, Mike |