Bug 208325 - f2fs inconsistent node block
Summary: f2fs inconsistent node block
Status: NEEDINFO
Alias: None
Product: File System
Classification: Unclassified
Component: f2fs (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Default virtual assignee for f2fs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-26 10:22 UTC by zKqri0
Modified: 2020-07-05 07:40 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.7.2-arch1-1
Tree: Mainline
Regression: No


Attachments

Description zKqri0 2020-06-26 10:22:46 UTC
i was working with an android kernel tree and once when i was trying to build i started getting errors like "make[1]: stat: drivers/input/touchscreen/Kconfig: Invalid argument". checked my laptop's dmesg and there were a bunch of errors like "F2FS-fs (sda2): inconsistent node block, nid:1761978, node_footer[nid:540703074,ino:1937074503,ofs:212733452,cpver:7020105085017473824,blkaddr:1008758642]".


rebooted and the problem is still there whenever i try to build that kernel tree. probably could just delete and clone it again but i kept it in case having it will help with finding what caused this bug. its probably not caused by ssd failure as i didnt get any sata errors and ive never got files corrupted before and after this happened.
Comment 1 Chao Yu 2020-06-28 12:44:25 UTC
Hi, thanks for the report.

What's you mkfs/mount option?

I've no idea whether this is a f2fs bug or not, as you said device can be trusted, so almostly it should be a software bug.

One case I can image could be that apps bypassing filesystem to write data via LBA directly, then data can be corrupted.

If possible, could you please help to add below three patches to recompile the kernel

https://lore.kernel.org/linux-f2fs-devel/20200628122940.29665-1-yuchao0@huawei.com/T/#t

[f2fs-dev] [PATCH 1/3] f2fs: fix wrong return value of f2fs_bmap_compress()
[f2fs-dev] [PATCH 2/3] f2fs: support to trace f2fs_bmap()
[f2fs-dev] [PATCH 3/3] f2fs: support to trace f2fs_fiemap()

Then, use below commands to see whether there is apps are lookuping LBA:

echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_bmap/enable
echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_fiemap/enable
cat /sys/kernel/debug/tracing/trace_pipe |grep f2fs
Comment 2 zKqri0 2020-06-29 06:17:37 UTC
(In reply to Chao Yu from comment #1)
> Hi, thanks for the report.
> 
> What's you mkfs/mount option?
> 
> I've no idea whether this is a f2fs bug or not, as you said device can be
> trusted, so almostly it should be a software bug.
> 
> One case I can image could be that apps bypassing filesystem to write data
> via LBA directly, then data can be corrupted.
> 
> If possible, could you please help to add below three patches to recompile
> the kernel
> 
> https://lore.kernel.org/linux-f2fs-devel/20200628122940.29665-1-
> yuchao0@huawei.com/T/#t
> 
> [f2fs-dev] [PATCH 1/3] f2fs: fix wrong return value of f2fs_bmap_compress()
> [f2fs-dev] [PATCH 2/3] f2fs: support to trace f2fs_bmap()
> [f2fs-dev] [PATCH 3/3] f2fs: support to trace f2fs_fiemap()
> 
> Then, use below commands to see whether there is apps are lookuping LBA:
> 
> echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_bmap/enable
> echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_fiemap/enable
> cat /sys/kernel/debug/tracing/trace_pipe |grep f2fs

Mount options are "/dev/sda2 on / type f2fs (rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,fsync_mode=posix)".

I patched my laptop's kernel with those patches but I don't see anything in "trace_pipe" while I'm getting invalid argument errors. Also I noticed that the "nid" and "node_footer" are the same always in the error so its only one node block that's messed up. 

Maybe a raw copy of that node block will help find what caused it ?
Comment 3 Chao Yu 2020-06-30 14:08:37 UTC
(In reply to zKqri0 from comment #2)
> Mount options are "/dev/sda2 on / type f2fs
> (rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,
> inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,
> mode=adaptive,active_logs=6,alloc_mode=default,fsync_mode=posix)".

It looks it's default mount options.

Did you use any special mkfs options? like -O [feature_name]?

> 
> I patched my laptop's kernel with those patches but I don't see anything in
> "trace_pipe" while I'm getting invalid argument errors. Also I noticed that
> the "nid" and "node_footer" are the same always in the error so its only one
> node block that's messed up. 
> 
> Maybe a raw copy of that node block will help find what caused it ?

Yes, please, I can parse it with dentry_block, inode or dnode structure to see what it looks like, and what kind of fields are corrupted.
Comment 4 zKqri0 2020-07-01 05:01:03 UTC
(In reply to Chao Yu from comment #3)
> (In reply to zKqri0 from comment #2)
> > Mount options are "/dev/sda2 on / type f2fs
> > (rw,relatime,lazytime,background_gc=on,discard,no_heap,user_xattr,
> > inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,
> > mode=adaptive,active_logs=6,alloc_mode=default,fsync_mode=posix)".
> 
> It looks it's default mount options.
> 
> Did you use any special mkfs options? like -O [feature_name]?
> 
> > 
> > I patched my laptop's kernel with those patches but I don't see anything in
> > "trace_pipe" while I'm getting invalid argument errors. Also I noticed that
> > the "nid" and "node_footer" are the same always in the error so its only
> one
> > node block that's messed up. 
> > 
> > Maybe a raw copy of that node block will help find what caused it ?
> 
> Yes, please, I can parse it with dentry_block, inode or dnode structure to
> see what it looks like, and what kind of fields are corrupted.


I used default mkfs options. Here is output of using dump.f2fs on that inode


>sudo dump.f2fs -i 1761978 /dev/sda2


>Info: [/dev/sda2] Disk Model: Samsung SSD 850 
>Info: Segments per section = 1
>Info: Sections per zone = 1
>Info: sector size = 512
>Info: total sectors = 102539264 (50068 MB)
>Info: MKFS version
>  "Linux version 4.20.0-arch1-1-ARCH (builduser@heftig-29859) (gcc version
>  8.2.1 20181127 (GCC)) #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018"
>Info: FSCK version
>  from "Linux version 5.7.2-arch1-1 (linux@archlinux) (gcc version 10.1.0
>  (GCC), GNU ld (GNU Binutils) 2.34.0) #1 SMP PREEMPT Wed, 10 Jun 2020
>  20:36:24 +0000"
>    to "Linux version 5.7.2-arch1-1 (linux@archlinux) (gcc version 10.1.0
>    (GCC), GNU ld (GNU Binutils) 2.34.0) #1 SMP PREEMPT Wed, 10 Jun 2020
>    20:36:24 +0000"
>Info: superblock features = 0 : 
>Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
>Info: total FS sectors = 102539264 (50068 MB)
>Info: CKPT version = 64d05005
>[print_node_info: 271] Node ID [0x203a7962:540703074] is direct node or
>>indirect node.
>[0]                    [0x5452202c : 1414668332]
>[1]                    [0x45445f4d : 1162108749]
>[2]                    [0x4444414c : 1145323852]
>[3]                    [0x202c2952 : 539765074]
>[4]                    [0x6e657665 : 1852143205]
>[5]                    [0x6f687420 : 1869116448]
>[6]                    [0x20686775 : 543713141]
>[7]                    [0x69207469 : 1763734633]
>[8]                    [0x6f6e2073 : 1869488243]
>[9]                    [0x46202e74 : 1176514164]
>[10]                   [0x69207869 : 1763735657]
>Invalid (i)node block

>Info: checkpoint state = 51 :  crc fsck unmount

>Done: 0.063481 secs
Comment 5 Chao Yu 2020-07-01 06:55:42 UTC
(In reply to zKqri0 from comment #4)
> >[print_node_info: 271] Node ID [0x203a7962:540703074] is direct node or
> >>indirect node.
> >[0]                    [0x5452202c : 1414668332]
> >[1]                    [0x45445f4d : 1162108749]
> >[2]                    [0x4444414c : 1145323852]
> >[3]                    [0x202c2952 : 539765074]
> >[4]                    [0x6e657665 : 1852143205]
> >[5]                    [0x6f687420 : 1869116448]
> >[6]                    [0x20686775 : 543713141]
> >[7]                    [0x69207469 : 1763734633]
> >[8]                    [0x6f6e2073 : 1869488243]
> >[9]                    [0x46202e74 : 1176514164]
> >[10]                   [0x69207869 : 1763735657]
> >Invalid (i)node block

I don't see any valid information from this, could you please upload the raw block if possible?
Comment 6 zKqri0 2020-07-05 07:40:37 UTC
(In reply to Chao Yu from comment #5)
> (In reply to zKqri0 from comment #4)
> > >[print_node_info: 271] Node ID [0x203a7962:540703074] is direct node or
> > >>indirect node.
> > >[0]                    [0x5452202c : 1414668332]
> > >[1]                    [0x45445f4d : 1162108749]
> > >[2]                    [0x4444414c : 1145323852]
> > >[3]                    [0x202c2952 : 539765074]
> > >[4]                    [0x6e657665 : 1852143205]
> > >[5]                    [0x6f687420 : 1869116448]
> > >[6]                    [0x20686775 : 543713141]
> > >[7]                    [0x69207469 : 1763734633]
> > >[8]                    [0x6f6e2073 : 1869488243]
> > >[9]                    [0x46202e74 : 1176514164]
> > >[10]                   [0x69207869 : 1763735657]
> > >Invalid (i)node block
> 
> I don't see any valid information from this, could you please upload the raw
> block if possible?

yeah there probably isnt because it seems like blk_addr is pointing to an invalid address. i took a dump of the node with "dd if=/dev/sda2 of=./node.bin bs=4096 skip=2900324352 count=4096 iflag=skip_bytes,count_bytes" with 2900324352 being "blk_addr >> 12" and it was part of a random git commit message and not a node block. anything else that would be useful to dump ?

Note You need to log in before you can comment on or make changes to this bug.