I've been unable to boot two of my systems after upgrading past kernel version 5.18.1. Both of them are using F2FS with compression enabled. This was the command used to create them back on 5.13.6: mkfs.f2fs -l root -O extra_attr,inode_checksum,sb_checksum,compression /dev/sda3 The command used to mount: mount -o compress_algorithm=lz4,compress_chksum,atgc,gc_merge,noatime /dev/sda3 /mnt A simple ls /mnt/usr/bin will report "Structure needs cleaning" on almost all of the critical system files on kernels past 5.18.1. Downgrading to 5.18.1 fixes everything.
Created attachment 301323 [details] broken f2fs image
I haven't had the time to bisect, but I've made a small image to demonstrate. It's mountable on 5.18.1 but not 5.18.3 and after. Due to limited time I was not able to test 5.18.2. Maybe someone more knowledgeable than me could take a look?
I suspect one issue recent reported with respect to compression vs. inline_data. Could you please run fsck from this? https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev Especially this patch should fix the corruption. "fsck.f2fs: drop compression bit if inline_data is set" And, in order to avoid the runtime issue, we need this from Linus's tree. https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/commit/?h=dev&id=29eeafc661f283a7412ecf8d9899a5ffd0c80f59 I think v5.18 stable branch hasn't merge the fixes yet. By any chance, could you please apply the below from the Linus tree on top of 5.18? "f2fs: attach inline_data after setting compression"
I did compile and run f2fs-tools-git just now, it doesn't do anything unfortunately. The mentioned patch is in the stable tree as of 5.18.8 and doesn't fix this problem either.
According to git bisect that I just ran, 198fd9faa271dd54dca6fc8eb6873f42dfd3b4d8 is the first bad commit. "f2fs: fix to do sanity check for inline inode" I reverted that on linux-stable v5.18.9 and the problem is indeed no longer present. Wile doing that I've found that my initial description as well es the attached image were missing a command: chattr -R +c /mnt/etc Without it the compression wouldn't be enabled and the errors wouldn't be visible.
Another week, another point release and the issue still persists on 5.18.10. Can I ask for the offending commit to please be reverted?
I confirm for 5.18.11 bug persists
(In reply to ghtm2 from comment #5) > According to git bisect that I just ran, > 198fd9faa271dd54dca6fc8eb6873f42dfd3b4d8 is the first bad commit. > "f2fs: fix to do sanity check for inline inode" > > I reverted that on linux-stable v5.18.9 and the problem is indeed no longer > present. > > Wile doing that I've found that my initial description as well es the > attached image were missing a command: > chattr -R +c /mnt/etc > Without it the compression wouldn't be enabled and the errors wouldn't be > visible. I can reproduce your issue and repair this w/ last f2fs-tools: [reproduce] mount -o loop ./broken.img /mnt/f2fs/ touch /mnt/f2fs/file lsattr /mnt/f2fs/file -----------------N-- /mnt/f2fs/file chattr +c /mnt/f2fs/file lsattr /mnt/f2fs/file --------c--------N-- /mnt/f2fs/file umount /mnt/f2fs mount -o loop ./broken.img /mnt/f2fs/ lsattr /mnt/f2fs/file lsattr: Structure needs cleaning while trying to stat /mnt/f2fs/file [repair] fsck.f2fs ./broken.img output: [FIX] (fsck_chk_inode_blk: 728) --> [0x505] i_flags=0x4 -> 0x0 [ASSERT] (fsck_chk_inode_blk:1182) --> ino: 0x505 chksum:0xe7b707a8, but calculated one is: 0x3f6daf58 [FIX] (fsck_chk_inode_blk:1188) --> ino: 0x505 recover, i_inode_checksum= 0xe7b707a8 -> 0x3f6daf58 [verify] mount -o loop ./broken.img /mnt/f2fs/ stat /mnt/f2fs/file File: /mnt/f2fs/file Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 70fh/1807d Inode: 1285 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-07-17 11:51:41.705230802 +0800 Modify: 2022-07-17 11:51:41.705230802 +0800 Change: 2022-07-17 11:51:57.641110482 +0800 Birth: - lsattr /mnt/f2fs/file -----------------N-- /mnt/f2fs/file ---- compressed flag was removed by fsck.
Created attachment 301448 [details] broken f2fs image
Which f2fs-tools are those? On my end neither f2fs-tools-1.15 nor f2fs-tools-git commit c1144bf472f69fdfac8f3fd7634aab19b771c32e fix anything. This is the output: Info: MKFS version "Linux version 5.13.6-arch1-1 (linux@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Thu, 29 Jul 2021 00:21:06 +0000" Info: FSCK version from "Linux version 5.13.6-arch1-1 (linux@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Thu, 29 Jul 2021 00:21:06 +0000" to "Linux version 5.18.7-arch1-1 (linux@archlinux) (gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT_DYNAMIC Sat, 25 Jun 2022 20:22:01 +0000" Info: superblock features = 2828 : extra_attr inode_checksum sb_checksum compression Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000 Info: Segments per section = 1 Info: Sections per zone = 1 Info: total FS sectors = 262144 (128 MB) Info: CKPT version = 1a027a03 Info: Checked valid nat_bits in checkpoint Info: checkpoint state = d5 : nat_bits crc fsck compacted_summary unmount [FSCK] Check node 1 / 1282 (0.08%) [FSCK] Check node 129 / 1282 (10.08%) [FSCK] Check node 257 / 1282 (20.08%) [FSCK] Check node 385 / 1282 (30.08%) [FSCK] Check node 513 / 1282 (40.08%) [FSCK] Check node 641 / 1282 (50.08%) [FSCK] Check node 769 / 1282 (60.08%) [FSCK] Check node 897 / 1282 (70.08%) [FSCK] Check node 1025 / 1282 (80.08%) [FSCK] Check node 1153 / 1282 (90.08%) [FSCK] Check node 1281 / 1282 (100.08%) [FSCK] Max image size: 36 MB, Free space: 52 MB [FSCK] Unreachable nat entries [Ok..] [0x0] [FSCK] SIT valid block bitmap checking [Ok..] [FSCK] Hard link checking for regular file [Ok..] [0x0] [FSCK] valid_block_count matching with CP [Ok..] [0x9be] [FSCK] valid_node_count matching with CP (de lookup) [Ok..] [0x502] [FSCK] valid_node_count matching with CP (nat lookup) [Ok..] [0x502] [FSCK] valid_inode_count matched with CP [Ok..] [0x502] [FSCK] free segment_count matched with CP [Ok..] [0x2f] [FSCK] next block offset is free [Ok..] [FSCK] fixing SIT types [FSCK] other corrupted bugs [Ok..] Done: 0.000000 secs I've updated the broken image attached to this bug report.
Please switch to dev-test/dev branch, due to fixing patch has not been committed to master branch now... https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/?h=dev-test
Ah, I somehow completely missed that the dev branched was linked before in comment 3, sorry. That branch does indeed fix the issue, thanks for the hint. Maybe the fix warrants a push to master and a new release? Most distros aren't exactly fast in packaging up new releases, the majority are still on f2fs-tools-1. 14 or older...