Bug 207165

Summary: Persistent ext4_search_dir: bad entry in directory: directory entry too close to block end
Product: File System Reporter: Elvis Pranskevichus (el)
Component: ext4Assignee: fs_ext4 (fs_ext4)
Severity: normal CC: jack, tytso
Priority: P1    
Hardware: Intel   
OS: Linux   
Kernel Version: 5.5.16 Tree: Mainline
Regression: No
Attachments: [PATCH] ext4: Fix checking of entry validity

Description Elvis Pranskevichus 2020-04-08 19:05:55 UTC
Since 5.5 I started getting persistent hits at the check added in 109ba779d6cca (ext4: check for directory entries too close to block end).  It is 100% reproducible when running docker containers on overlayfs2.  Here's an example log entry:

kernel: EXT4-fs error (device dm-0): ext4_search_dir:1395: inode #28320400: block 113246792: comm dockerd: bad entry in directory: directory entry too close to block end - offset=0, inode=28320403, rec_len=32, name_len=8, size=4096
dockerd[5315]: time="2020-04-08T11:03:35.148433258-07:00" level=error msg="Error removing mounted layer c520f6ce1d0b493e51aa9cdaea2240c6f65f104c3da8fb9767999dc526086f85: unlinkat /var/lib/docker/overlay2/01c0c02ee4841227fefe595eeef8912fee32bc2b63a2264cb513f924e6366950/diff/tmp/apt-key-gpghome.TauCtRwzyD: directory not empty"

To clarify, this error happened elsewhere as well, so this doesn't seem to be overlayfs2-specific.

At first I thought that my filesystem was borked somehow, so I went so far as to reformat the partition, but that didn't help.
Comment 1 Theodore Tso 2020-04-09 03:19:54 UTC
Could you run dumpe2fs -h on the file system and attach it to the bug?

Many thanks!!
Comment 2 Elvis Pranskevichus 2020-07-09 18:31:30 UTC
dumpe2fs 1.45.6 (20-Mar-2020)
Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          1ca09d01-202a-4a0c-a150-8d078c57d751
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg inline_data sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              31031296
Block count:              124111616
Reserved block count:     6205580
Free blocks:              37127036
Free inodes:              26630302
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Apr  7 23:31:35 2020
Last mount time:          Thu Jul  9 09:47:02 2020
Last write time:          Thu Jul  9 11:30:13 2020
Mount count:              6
Maximum mount count:      38
Last checked:             Fri May 22 14:10:16 2020
Check interval:           15552000 (6 months)
Next check after:         Wed Nov 18 13:10:16 2020
Lifetime writes:          2756 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       7122511
Default directory hash:   half_md4
Directory Hash Seed:      7fd796d0-3b05-456b-8550-2734924aa361
Journal backup:           inode blocks
FS Error count:           2
First error time:         Thu Jul  9 11:30:13 2020
First error function:     ext4_search_dir
First error line #:       1399
First error inode #:      28320400
First error block #:      113246792
Last error time:          Thu Jul  9 11:30:13 2020
Last error function:      ext4_search_dir
Last error line #:        1399
Last error inode #:       28328032
Last error block #:       113247269
Checksum type:            crc32c
Checksum:                 0x69cb04e1
Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3
Journal size:             1024M
Journal length:           262144
Journal sequence:         0x0052bb7e
Journal start:            66275
Journal checksum type:    crc32c
Journal checksum:         0x17d02966
Comment 3 Elvis Pranskevichus 2020-07-19 23:37:44 UTC
I can still trigger this pretty reliably with docker on overlayfs.  Anything I can do to help narrow this down?
Comment 4 Theodore Tso 2020-07-23 14:47:49 UTC
Can you give a reliable repro that will work everywhere?

Also, can you try reformatting the file system without inline_data and see if the problem goes away?   Inline_data is not something which I consider as mature as other ext4 features.
Comment 5 Jan Kara 2020-07-29 15:40:58 UTC
FWIW I've checked fs/ext4/inline.c and the way it calls ext4_search_dir() which ends up calling ext4_check_dir_entry() indeed looks broken. I'll have a look into fixing that.
Comment 6 Jan Kara 2020-07-31 16:23:05 UTC
Created attachment 290705 [details]
[PATCH] ext4: Fix checking of entry validity

This patch fixes the failures for me. I've submitted it to Ted for inclusion.
Comment 7 Elvis Pranskevichus 2020-08-07 23:50:49 UTC
The patch fixes the bug for me as well. Thanks Jan!
Comment 8 Jan Kara 2020-08-17 15:33:38 UTC
Thanks for info. Ted has picked up the patch so I'm closing the bug.