Bug 10882

Summary: Different kernel crashes on corrupted filesystems
Product: File System Reporter: Sami Liedes (sami.liedes)
Component: ext3Assignee: Andrew Morton (akpm)
Status: CLOSED CODE_FIX    
Severity: normal CC: bunk, duaneg
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.25.4 Subsystem:
Regression: --- Bisected commit-id:
Attachments: Test case 1, hdb.25.softlockup, gzipped
Test case 2, hdb.20000009, gzipped
Test case 3, hdb.20000057, gzipped

Description Sami Liedes 2008-06-07 09:19:33 UTC
Latest working kernel version:
Earliest failing kernel version:
Distribution: Debian sid (unstable)
Hardware Environment: qemu x86
Software Environment: Minimal Debian sid
Problem Description:

The attached (intentionally) broken ext3 images break ext3 in different ways.

Steps to reproduce:

For hdb.25.softlockup.gz:

1. gunzip the image
2. mount hdb.25.softlockup /mnt -o loop
3. cd /mnt
4. find >&/dev/null
   -> lockup

For hdb.20000009.softlockup.gz:

1. gunzip the image
2. mount hdb.20000009.softlockup /mnt -o loop
   -> orphan list check failed, lockup

For hdb.20000057.nullderef.gz:

1. gunzip the image
2. mount hdb.20000057.nullderef /mnt -o loop
3. rm -rf /mnt
   -> oops
Comment 1 Sami Liedes 2008-06-07 09:21:14 UTC
Created attachment 16427 [details]
Test case 1, hdb.25.softlockup, gzipped

This is what happens:

fstest:~# mount /dev/hdb /mnt
[   21.013647] kjournald starting.  Commit interval 5 seconds
[   21.031354] EXT3 FS on hdb, internal journal
[   21.031354] EXT3-fs: mounted filesystem with ordered data mode.
fstest:~# cd /mnt
fstest:/mnt# find >&/dev/null
[   26.993073] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1586: rec_len is smaller than minimal - offset=524, inode=0, rec_len=0, name_len=0
[   27.005762] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1538: directory entry across blocks - offset=0, inode=1538, rec_len=4108, name_len=1
[   27.011934] attempt to access beyond end of device
[   27.012696] hdb: rw=32, want=68262, limit=20480
[   27.019444] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1555: inode out of bounds - offset=0, inode=134219283, rec_len=12, name_len=1
[   27.024280] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #760: inode out of bounds - offset=64, inode=134218491, rec_len=28, name_len=19
[   27.041863] EXT3-fs warning (device hdb): dx_probe: dx entry: no count or count > limit
[   27.041863] EXT3-fs warning (device hdb): dx_probe: Corrupt dir inode 1281, running e2fsck is recommended.
[   27.041863] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: directory entry across blocks - offset=532, inode=1796, rec_len=16400, name_len=7
[   27.045094] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: inode out of bounds - offset=12, inode=268437174, rec_len=16, name_len=5
[   27.048078] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: inode out of bounds - offset=48, inode=132784, rec_len=16, name_len=5
[   27.062202] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=1688, inode=268436964, rec_len=24, name_len=14
[   27.081892] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: rec_len is too small for name_len - offset=10240, inode=1414, rec_len=16, name_len=37
[   27.091913] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=2500, inode=536871373, rec_len=64, name_len=55
[   27.101939] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: rec_len is smaller than minimal - offset=6144, inode=0, rec_len=0, name_len=0
[   27.111965] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=12372, inode=2098508, rec_len=12, name_len=3
[   27.121992] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=4552, inode=262487, rec_len=16, name_len=5
[   27.132019] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #313: inode out of bounds - offset=24, inode=1073742138, rec_len=1000, name_len=39
[   27.150063] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #281: inode out of bounds - offset=24, inode=1048858, rec_len=1000, name_len=92
[   27.155989] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #243: rec_len % 4 != 0 - offset=44, inode=1885745784, rec_len=26979, name_len=48
[   27.172050] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #215: inode out of bounds - offset=12, inode=16777356, rec_len=12, name_len=2
[   27.194256] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #337: inode out of bounds - offset=0, inode=4194641, rec_len=12, name_len=1
[   27.212109] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #141: rec_len is too small for name_len - offset=0, inode=141, rec_len=12, name_len=65
[   27.222138] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #411: rec_len is smaller than minimal - offset=768, inode=0, rec_len=0, name_len=0
[   92.342166] BUG: soft lockup - CPU#0 stuck for 61s! [find:639]
[   92.342166] 
[   92.342166] Pid: 639, comm: find Not tainted (2.6.25.4 #3)
[   92.342166] EIP: 0060:[<c02de129>] EFLAGS: 00000206 CPU: 0
[   92.342166] EIP is at ext3_find_entry+0x3cb/0x6cb
[   92.342166] EAX: 00000000 EBX: c71c8c00 ECX: 00000000 EDX: 0000057d
[   92.342166] ESI: 00010000 EDI: c7a12000 EBP: c7ac1dd4 ESP: c7ac1ce4
[   92.342166]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   92.342166] CR0: 80050033 CR2: 09463094 CR3: 07acb000 CR4: 00000690
[   92.342166] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[   92.342166] DR6: 00000000 DR7: 00000000
[   92.342166]  [<c027c6af>] ? __getblk+0x27/0x29a
[   92.342166]  [<c02d8921>] ? __ext3_get_inode_loc+0xe8/0x334
[   92.342166]  [<c053d089>] ? _spin_lock+0x32/0x38
[   92.342166]  [<c026e1fc>] ? __d_lookup+0x4a/0x174
[   92.342166]  [<c02deb05>] ext3_lookup+0x2d/0xb4
[   92.342166]  [<c026e914>] ? d_alloc+0x136/0x183
[   92.342166]  [<c0264555>] do_lookup+0x15c/0x188
[   92.342166]  [<c0265ba7>] __link_path_walk+0x5f9/0xd88
[   92.342166]  [<c026420e>] ? path_put+0x20/0x23
[   92.342166]  [<c053d089>] ? _spin_lock+0x32/0x38
[   92.342166]  [<c053cfe1>] ? _spin_unlock+0x1d/0x20
[   92.342166]  [<c044f361>] ? _atomic_dec_and_lock+0x25/0x34
[   92.342166]  [<c026420e>] ? path_put+0x20/0x23
[   92.342166]  [<c0266380>] path_walk+0x4a/0x99
[   92.342166]  [<c02665a5>] do_path_lookup+0x82/0x1c6
[   92.342166]  [<c0267041>] __user_walk_fd+0x32/0x4a
[   92.342166]  [<c0260732>] vfs_lstat_fd+0x18/0x3e
[   92.342166]  [<c053d089>] ? _spin_lock+0x32/0x38
[   92.342167]  [<c053cfe1>] ? _spin_unlock+0x1d/0x20
[   92.342167]  [<c044f361>] ? _atomic_dec_and_lock+0x25/0x34
[   92.342167]  [<c0260847>] sys_fstatat64+0x46/0x5a
[   92.342167]  [<c025e8ce>] ? fput+0x18/0x1b
[   92.342167]  [<c025bdc2>] ? filp_close+0x41/0x5f
[   92.342167]  [<c025be47>] ? sys_close+0x67/0xab
[   92.342167]  [<c0202cde>] ? syscall_exit+0x8/0x1a
[   92.342167]  [<c0202cd2>] syscall_call+0x7/0xb
[   92.342167]  =======================
Comment 2 Sami Liedes 2008-06-07 09:22:27 UTC
Created attachment 16428 [details]
Test case 2, hdb.20000009, gzipped

What happens:

fstest:~# mount /dev/hdb /mnt
[   27.743805] kjournald starting.  Commit interval 5 seconds
[   27.756299] EXT3 FS on hdb, internal journal
[   27.758994] EXT3 Inode c7479930: orphan list check failed!
[   27.759575] c7479930: 2e2f2e2e 64732f2e 00003165 00000000 
[   27.760161] c7479940: 00000000 00000000 00000000 00000000 
[   27.760661] c7479950: 00000000 00000000 00000000 00000000 
[   27.761175] c7479960: 00000000 00000000 00000000 00000000 
[   27.761663] c7479970: 00000000 00000000 00000000 00000000 
[   27.762197] c7479980: 00000000 00000000 00000000 00000000 
[   27.763142] c7479990: 00000001 dead4ead ffffffff ffffffff 
[   27.763619] c74799a0: c0948278 00000000 c05eff5b c74799ac 
[   27.764092] c74799b0: c74799ac c093c080 00000000 c05c68c8 
[   27.764574] c74799c0: ffffffff ffffffff c7ab430c c7ab430c 
[   27.765087] c74799d0: 0000000a 00000000 00000000 00000001 
[   27.765578] c74799e0: 00000001 dead4ead ffffffff ffffffff 
[   27.766095] c74799f0: c086c3b0 00000000 c05bd0e4 c74799fc 
[   27.766639] c7479a00: c74799fc 00000000 00000000 c74799dc 
[   27.767148] c7479a10: c093c078 00000000 c05c68d7 00000000 
[   27.767662] c7479a20: 00000000 c7479a24 c7479a24 c7479a2c 
[   27.768183] c7479a30: c7479a2c c7479a34 c7479a34 00000020 
[   27.768688] c7479a40: 00000000 00000001 00000000 00000000 
[   27.769240] c7479a50: 00000000 00000001 00000000 0000000a 
[   27.769713] c7479a60: 00000000 484aa7ca 00000000 484aa7c2 
[   27.770232] c7479a70: 00000000 484aa7ca 00000000 0000000a 
[   27.770698] c7479a80: 00000000 a1ff0000 00000001 dead4ead 
[   27.771223] c7479a90: ffffffff ffffffff c06127b0 00000000 
[   27.771707] c7479aa0: c05c4453 00000001 00000001 dead4ead 
[   27.772219] c7479ab0: ffffffff ffffffff c086c3b0 00000000 
[   27.772746] c7479ac0: c05bd0e4 c7479ac4 c7479ac4 00000000 
[   27.773120] c7479ad0: 00000000 c7479aa4 c06127b8 00000000 
[   27.773444] c7479ae0: c05c447b 00000000 00000001 dead4ead 
[   27.775340] c7479af0: ffffffff ffffffff c0948278 00000000 
[   27.777652] c7479b00: c05eff5b c7479b04 c7479b04 c06127c8 
[   27.779780] c7479b10: 00000000 c05c44a8 c0551540 c093a6c0 
[   27.782634] c7479b20: c79f9000 00000000 c7479b2c c7479a1c 
[   27.783133] c7479b30: 00000000 00000020 00000000 deaf1eed 
[   27.785528] c7479b40: ffffffff ffffffff c093a80c 00000000 
[   27.787929] c7479b50: c05c44dc 00000000 00000000 00010001 
[   27.790307] c7479b60: c7479b60 c7479b60 00000001 dead4ead 
[   27.793127] c7479b70: ffffffff ffffffff c093a804 00000000 
[   27.795482] c7479b80: c05c44f5 00000000 00000000 00000000 
[   27.797787] c7479b90: c093a7a0 001200d2 c7a169fc 00000001 
[   27.800154] c7479ba0: dead4ead ffffffff ffffffff c093a7fc 
[   27.803119] c7479bb0: 00000000 c05c4510 c7479bb8 c7479bb8 
[   27.805507] c7479bc0: 00000000 c7479bc4 c7479bc4 00000000 
[   27.807872] c7479bd0: 00000000 35a13a0c 00000000 00000000 
[   27.810202] c7479be0: c7479be0 c7479be0 00000001 00000001 
[   27.813118] c7479bf0: dead4ead ffffffff ffffffff c086c3b0 
[   27.815467] c7479c00: 00000000 c05bd0e4 c7479c08 c7479c08 
[   27.817876] c7479c10: 00000000 00000000 c7479be8 c093a7f4 
[   27.820244] c7479c20: 00000000 c05c452c 00000040 00000000 
[   27.823120] c7479c30: 00000000 00000000 00000000 
[   27.825193] Pid: 637, comm: mount Not tainted 2.6.25.4 #3
[   27.827522]  [<c02e047b>] ext3_destroy_inode+0x76/0x78
[   27.829819]  [<c026f83d>] destroy_inode+0x23/0x39
[   27.831874]  [<c027046d>] generic_drop_inode+0x122/0x157
[   27.834331]  [<c026f4ae>] iput+0x64/0x6b
[   27.836060]  [<c02e2b08>] ext3_fill_super+0x185a/0x19b4
[   27.838294]  [<c029697c>] ? disk_name+0xa7/0xb2
[   27.840297]  [<c025ea62>] ? test_bdev_super+0x0/0x11
[   27.842438]  [<c025fa7f>] get_sb_bdev+0x108/0x139
[   27.844079]  [<c02727dd>] ? mntput_no_expire+0x16/0x67
[   27.846377]  [<c053d089>] ? _spin_lock+0x32/0x38
[   27.848398]  [<c02dfa11>] ext3_get_sb+0x21/0x27
[   27.850332]  [<c02e12ae>] ? ext3_fill_super+0x0/0x19b4
[   27.853373]  [<c025eab7>] vfs_kern_mount+0x3a/0x8b
[   27.853749]  [<c025eb52>] do_kern_mount+0x33/0xbd
[   27.854088]  [<c027296e>] do_new_mount+0x59/0x77
[   27.854469]  [<c02737c7>] do_mount+0x185/0x1b0
[   27.854812]  [<c0244727>] ? __get_free_pages+0x29/0x62
[   27.855242]  [<c0271f95>] ? copy_mount_options+0x2e/0x11e
[   27.855663]  [<c027386d>] sys_mount+0x7b/0xae
[   27.856023]  [<c0202cd2>] syscall_call+0x7/0xb
[   27.863142]  =======================
[   92.343090] BUG: soft lockup - CPU#0 stuck for 61s! [mount:637]
[   92.343090] 
[   92.343090] Pid: 637, comm: mount Not tainted (2.6.25.4 #3)
[   92.343090] EIP: 0060:[<c0217bf2>] EFLAGS: 00000282 CPU: 0
[   92.343090] EIP is at vprintk+0x1f0/0x359
[   92.343090] EAX: c7aac000 EBX: c066a7b8 ECX: 00000000 EDX: 00000001
[   92.343090] ESI: 00000000 EDI: c066a7b7 EBP: c7ac3d64 ESP: c7ac3cf8
[   92.343090]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   92.343090] CR0: 8005003b CR2: b7e55210 CR3: 07ac5000 CR4: 00000690
[   92.343090] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[   92.343090] DR6: 00000000 DR7: 00000000
[   92.343090]  [<c02d771c>] ? ext3_orphan_get+0x133/0x1ca
[   92.343090]  [<c026f83d>] ? destroy_inode+0x23/0x39
[   92.343090]  [<c02e2af6>] ? ext3_fill_super+0x1848/0x19b4
[   92.343090]  [<c029697c>] ? disk_name+0xa7/0xb2
[   92.343090]  [<c025ea62>] ? test_bdev_super+0x0/0x11
[   92.343090]  [<c025fa7f>] ? get_sb_bdev+0x108/0x139
[   92.343090]  [<c02727dd>] ? mntput_no_expire+0x16/0x67
[   92.343090]  [<c053d089>] ? _spin_lock+0x32/0x38
[   92.343090]  [<c02dfa11>] ? ext3_get_sb+0x21/0x27
[   92.343090]  [<c02e12ae>] ? ext3_fill_super+0x0/0x19b4
[   92.343090]  [<c025eab7>] ? vfs_kern_mount+0x3a/0x8b
[   92.343090]  [<c025eb52>] ? do_kern_mount+0x33/0xbd
[   92.343090]  [<c027296e>] ? do_new_mount+0x59/0x77
[   92.343090]  [<c02737c7>] ? do_mount+0x185/0x1b0
[   92.343090]  [<c0244727>] ? __get_free_pages+0x29/0x62
[   92.343090]  [<c0271f95>] ? copy_mount_options+0x2e/0x11e
[   92.343090]  [<c027386d>] ? sys_mount+0x7b/0xae
[   92.343090]  [<c0202cd2>] ? syscall_call+0x7/0xb
[   92.343090]  =======================
[  157.843080] BUG: soft lockup - CPU#0 stuck for 61s! [mount:637]
[  157.843091] 
[  157.843091] Pid: 637, comm: mount Not tainted (2.6.25.4 #3)
[  157.843091] EIP: 0060:[<c0458807>] EFLAGS: 00000246 CPU: 0
[  157.843091] EIP is at _raw_spin_lock+0x1e/0x100
[  157.843091] EAX: c0610fd0 EBX: c0610fd0 ECX: 00000000 EDX: 00000000
[  157.843091] ESI: c1155d2c EDI: c79f9000 EBP: c7ac3cc4 ESP: c7ac3c9c
[  157.843091]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[  157.843091] CR0: 8005003b CR2: b7e55210 CR3: 07ac5000 CR4: 00000690
[  157.843091] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  157.843091] DR6: 00000000 DR7: 00000000
[  157.843091]  [<c027c5ba>] ? __find_get_block+0x70/0x13e
[  157.843091]  [<c0244832>] ? mapping_tagged+0x53/0x5c
[  157.843091]  [<c053d089>] _spin_lock+0x32/0x38
[  157.843091]  [<c026f672>] ? ifind_fast+0x19/0x8d
[  157.843091]  [<c026f672>] ifind_fast+0x19/0x8d
[  157.843091]  [<c026fa40>] iget_locked+0x2a/0x12d
[  157.843091]  [<c02d8b7b>] ext3_iget+0xe/0x384
[  157.843091]  [<c02d7671>] ext3_orphan_get+0x88/0x1ca
[  157.843091]  [<c026f83d>] ? destroy_inode+0x23/0x39
[  157.843091]  [<c02e2a7f>] ext3_fill_super+0x17d1/0x19b4
[  157.843091]  [<c029697c>] ? disk_name+0xa7/0xb2
[  157.843091]  [<c025ea62>] ? test_bdev_super+0x0/0x11
[  157.843091]  [<c025fa7f>] get_sb_bdev+0x108/0x139
[  157.843091]  [<c02727dd>] ? mntput_no_expire+0x16/0x67
[  157.843091]  [<c053d089>] ? _spin_lock+0x32/0x38
[  157.843091]  [<c02dfa11>] ext3_get_sb+0x21/0x27
[  157.843091]  [<c02e12ae>] ? ext3_fill_super+0x0/0x19b4
[  157.843091]  [<c025eab7>] vfs_kern_mount+0x3a/0x8b
[  157.843091]  [<c025eb52>] do_kern_mount+0x33/0xbd
[  157.843091]  [<c027296e>] do_new_mount+0x59/0x77
[  157.843091]  [<c02737c7>] do_mount+0x185/0x1b0
[  157.843091]  [<c0244727>] ? __get_free_pages+0x29/0x62
[  157.843091]  [<c0271f95>] ? copy_mount_options+0x2e/0x11e
[  157.843091]  [<c027386d>] sys_mount+0x7b/0xae
[  157.843091]  [<c0202cd2>] syscall_call+0x7/0xb
[  157.843091]  =======================
Comment 3 Sami Liedes 2008-06-07 09:23:25 UTC
Created attachment 16429 [details]
Test case 3, hdb.20000057, gzipped

What happens:

fstest:~# mount /dev/hdb /mnt
[   18.051022] kjournald starting.  Commit interval 5 seconds
[   18.054932] EXT3 FS on hdb, internal journal
[   18.055588] EXT3-fs: mounted filesystem with ordered data mode.
fstest:~# rm -rf /mnt
[   20.148396] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #760: rec_len % 4 != 0 - offset=52, inode=2037411683, rec_len=26994, name_len=103
[   20.156383] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 38956, count = 1
[   20.160759] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1073741824, count = 1
[   20.163825] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 32768, count = 1
[   20.167152] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 524288, count = 1
[   20.170810] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 128, count = 1
[   20.174700] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 131072, count = 1
[   20.178294] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 67108864, count = 1
[   20.182943] EXT3-fs error (device hdb): empty_dir: bad entry in directory #760: rec_len % 4 != 0 - offset=52, inode=2037411683, rec_len=26994, name_len=103
[   20.188861] attempt to access beyond end of device
[   20.189960] hdb: rw=0, want=262146, limit=20480
[   20.190926] EXT3-fs error (device hdb): ext3_free_branches: Read failure, inode=1519, block=131072
[   20.195111] attempt to access beyond end of device
[   20.196147] hdb: rw=0, want=2147483650, limit=20480
[   20.197166] EXT3-fs error (device hdb): ext3_free_branches: Read failure, inode=1519, block=1073741824
[   20.202385] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.205915] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.209697] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.213103] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.216394] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.219921] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.223665] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.227183] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 12898, count = 1
[   20.230485] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757455, count = 1
[   20.233619] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.237051] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.240546] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.243585] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.247325] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.250643] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 8
[   20.254101] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.257682] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.261055] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 101, count = 1
[   20.264780] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757456, count = 1
[   20.268227] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.271577] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 9
[   20.273986] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.277418] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.280784] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.284721] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.288058] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.291316] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.294621] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 100, count = 1
[   20.297885] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757457, count = 1
[   20.301279] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.304864] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 10
[   20.307899] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.311339] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.314657] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.318137] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.321387] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.324787] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.328315] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 12643, count = 1
[   20.332457] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757458, count = 1
[   20.335778] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.339257] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 10
[   20.342070] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.344595] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.348334] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.352829] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.356223] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.359625] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.363083] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 12641, count = 1
[   20.366346] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757459, count = 1
[   20.369836] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.373487] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 10
[   20.376422] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.379988] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.383398] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.386735] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.389991] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.393665] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.397357] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 12642, count = 1
[   20.400772] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1048576, count = 1
[   20.403601] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757460, count = 1
[   20.406932] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.410219] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 9
[   20.413811] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.417294] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.420705] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.424656] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.428002] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.431276] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.434633] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 99, count = 1
[   20.438067] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 128, count = 1
[   20.441403] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757461, count = 1
[   20.444616] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 65577471, count = 1
[   20.447919] EXT3-fs error (device hdb): ext3_free_blocks_sb: bit already cleared for block 9
[   20.451075] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.454573] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.457979] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1212852168, count = 1
[   20.461295] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 66536, count = 1
[   20.464655] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 774843950, count = 1
[   20.468225] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 1685270318, count = 1
[   20.471731] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks in system zones - Block = 97, count = 1
[   20.474579] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 899757462, count = 1
[   20.478164] EXT3-fs error (device hdb): ext3_free_blocks: Freeing blocks not in datazone - block = 134217728, count = 1
[   20.482722] BUG: unable to handle kernel NULL pointer dereference at 0000000c
[   20.484311] IP: [<c03089b7>] journal_dirty_metadata+0x37/0xea
[   20.485547] *pde = 00000000 
[   20.486269] Oops: 0000 [#1] 
[   20.486905] 
[   20.487268] Pid: 639, comm: rm Not tainted (2.6.25.4 #3)
[   20.488393] EIP: 0060:[<c03089b7>] EFLAGS: 00000246 CPU: 0
[   20.489408] EIP is at journal_dirty_metadata+0x37/0xea
[   20.490496] EAX: 00000000 EBX: c746cf18 ECX: c7ab5400 EDX: c7a90080
[   20.491867] ESI: c746cf18 EDI: 00000000 EBP: c7ac7d58 ESP: c7ac7d48
[   20.492303]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[   20.492303] Process rm (pid: 639, ti=c7ac6000 task=c7a22ea0 task.ti=c7ac6000)
[   20.492303] Stack: c7435000 c746cf18 c7435000 c746cf18 c7ac7d78 c02e4e05 00000001 08000000 
[   20.492303]        c0550eb3 c746cf18 08000000 c71783fc c7ac7da8 c02db214 08000000 00000001 
[   20.492303]        c71783fc c7178400 c746cf18 c74793fc c7435000 c746cf18 c717b334 00000100 
[   20.492303] Call Trace:
[   20.492303]  [<c02e4e05>] ? __ext3_journal_dirty_metadata+0x19/0x3f
[   20.492303]  [<c02db214>] ? ext3_free_data+0x96/0xe8
[   20.492303]  [<c02db48b>] ? ext3_free_branches+0x225/0x22d
[   20.492303]  [<c02db30e>] ? ext3_free_branches+0xa8/0x22d
[   20.492303]  [<c02db30e>] ? ext3_free_branches+0xa8/0x22d
[   20.492303]  [<c02dba58>] ? ext3_truncate+0x5c5/0x935
[   20.492303]  [<c030a261>] ? journal_start+0xd1/0x106
[   20.492303]  [<c030a23e>] ? journal_start+0xae/0x106
[   20.492303]  [<c02e2ccb>] ? ext3_journal_start_sb+0x29/0x4a
[   20.492303]  [<c02dbe94>] ? ext3_delete_inode+0xcc/0xd8
[   20.492303]  [<c02dbdc8>] ? ext3_delete_inode+0x0/0xd8
[   20.492303]  [<c02702d8>] ? generic_delete_inode+0x62/0xd5
[   20.492303]  [<c0270486>] ? generic_drop_inode+0x13b/0x157
[   20.492303]  [<c026f4ae>] ? iput+0x64/0x6b
[   20.492303]  [<c0266c06>] ? do_unlinkat+0xe3/0x13f
[   20.492303]  [<c0268bc8>] ? vfs_readdir+0x60/0x85
[   20.492303]  [<c026890c>] ? filldir64+0x0/0xd7
[   20.492303]  [<c0268c84>] ? sys_getdents64+0x97/0xa1
[   20.492303]  [<c0266d61>] ? sys_unlinkat+0x23/0x3b
[   20.492303]  [<c0202cd2>] ? syscall_call+0x7/0xb
[   20.492303]  =======================
[   20.492303] Code: 89 d6 8b 10 8b 0a 8b 7e 20 f6 40 10 04 75 6b f6 01 02 75 66 0f ba 2e 15 19 c0 85 c0 74 0d 8b 06 a9 00 00 20 00 74 ed f3 90 eb f3 <8b> 47 0c 85 c0 75 1e c7 47 0c 01 00 00 00 8b 5d f0 8b 43 04 85 
[   20.492303] EIP: [<c03089b7>] journal_dirty_metadata+0x37/0xea SS:ESP 0068:c7ac7d48
[   20.492352] ---[ end trace aacc63c886859a2a ]---
Segmentation fault
fstest:~#
Comment 4 Solofo Ramangalahy 2008-06-09 02:36:34 UTC
Hi Sami,

Thanks for the report.

(In reply to comment #1)
> The attached (intentionally) broken ext3 images

Could you describe how you proceeded (fsfuzzer, killing qemu?)

> break ext3 in different ways.

Did you try e2fsck before trying to mount the images?

I briefly tested the first case and it seems ok after "e2fsck -y" (on Debian Sid 
e2fsprogs 1.40.8-2).

I am not clear about what is the expected behavior here.

On the one hand, every program should check its inputs, on the other hand, it may be the job of e2fsck to check filesystems (before passing them to the kernel). But checks are expensive. Maybe mount should trigger the fsck...

This seems to be a borderline case, with expected behavior depending on how the corruption happened, or if the mount triggers something not strictly related to the corruption.

Will try to figure out...

I think it would be better to fill separate issues for each corruption (this way subject can be refined as analysis progresses).
Comment 5 Sami Liedes 2008-06-09 02:54:49 UTC
Hi,

I use the zzuf program (google for it, or Debian package zzuf), which just flips each bit in the image with a preset probability. Here's a script I use inside qemu (varying some of the parameters):

-----
#!/bin/sh

if [ "`hostname`" != "fstest" ]; then
   echo "This is a dangerous script."
   echo "Set your hostname to \`fstest\' if you want to use it."
   exit 1
fi

umount /dev/hdb
umount /dev/hdc
/etc/init.d/sysklogd stop
/etc/init.d/klogd stop
/etc/init.d/cron stop
mount /dev/hda / -t ext3 -o remount,ro || exit 1

ulimit -t 20

for ((s=$1; s<1000000000; s++)); do
  umount /mnt
  echo '***** zzuffing *****' seed $s
  zzuf -r 0.0001:0.05 -s $s </dev/hdc >/dev/hdb || exit
  mount /dev/hdb /mnt || continue
  cd /mnt || continue
  cp -r doc doc2 >&/dev/null
  find -xdev >&/dev/null
  find -xdev -print0 2>/dev/null |xargs -0 touch -- 2>/dev/null
  mkdir tmp >&/dev/null
  echo whoah >tmp/filu 2>/dev/null
  rm -rf /mnt/* >&/dev/null
  cd /
done
-----

The kernel dying (or worse, if it can be exploited to execute arbitrary code) on mounting corrupted filesystems can be a problem at least on the currently quite common setup on e.g. universities, since many modern distributions allow the user to plug in a USB memory stick and mount it. It's quite easy for a malicious user to put an arbitrary file system on an USB stick.

So at least if the crashes are easily avoidable by changing the code, I think it definitely should be done. It also means more robustness against unintentionally corrupt code.

No, I didn't try e2fsck, since my intention was to get the kernel to crash.
Comment 6 Solofo Ramangalahy 2008-06-09 05:56:41 UTC
(In reply to comment #5)
> I use the zzuf program (google for it, or Debian package zzuf), which just
> flips each bit in the image with a preset probability. Here's a script I use
> inside qemu (varying some of the parameters):

Thanks for the nice script!

> The kernel dying (or worse, if it can be exploited to execute arbitrary code)
> on mounting corrupted filesystems can be a problem at least on the currently
> quite common setup on e.g. universities, since many modern distributions
> allow
> the user to plug in a USB memory stick and mount it. It's quite easy for a
> malicious user to put an arbitrary file system on an USB stick.

Or worse: a malicious DPL :-)

I see the misuse case: privilege escalation with carefully crafted corrupted fs.

I would say kernel dying is less worrisome since physical access to plug USB stick probably means also access to pull the power plug or more creative ways to crash or damage the machine via the USB connector.

Back in the days, we used mtools to access data on removable media at the University. It still works with USB sticks.

> So at least if the crashes are easily avoidable by changing the code, I think
> it definitely should be done. It also means more robustness against
> unintentionally corrupt code.

Agreed.

This leaves the question "which code?" (kernel, mount, mounting scripts, security levels,...).

> No, I didn't try e2fsck, since my intention was to get the kernel to crash.

The 3 images do not seem to issue errors after e2fsck.
Comment 7 Solofo Ramangalahy 2008-06-10 00:07:43 UTC
Testing on 2.6.26-rc5:

. hdb.25.softlockup reproduced

. hdb.20000009.softlockup not reproduced (mount running taking 100% cpu, unkillable)

. hdb.20000057.null.deref oops not reproduced, no error on console, rm gives error:
"
rm: WARNING: Circular directory structure.
This almost certainly means that you have a corrupted file system.
NOTIFY YOUR SYSTEM MANAGER.
The following directory is part of the cycle:
  `/mnt/test/doc/adduser/examples/adduser.local.conf.examples/skel.o4her/index.html/index.html/index.html'
"
Comment 8 Duane Griffin 2008-06-20 08:30:39 UTC
The first case, hdb.25.softlockup, is happening inside ext3_dx_find_entry. It is caused by ext3_next_entry being used to iterate through directory entries without checking for the rec_len == 0 case.

There are a number of places in fs/ext3/namei.c where similar usage seems problematic. I'll look into fixing them.
Comment 9 Sami Liedes 2008-06-20 08:45:45 UTC
Hi,

If you have any experimental patches, I can try them and rerun the tests to validate them and find more problems if you wish. Of course you can do that yourself too with the script in #5, but it does require some setting up.
Comment 10 Duane Griffin 2008-06-20 09:33:04 UTC
Thanks, I'll let you know. I've reproduced it here, so hopefully we won't need too much back & forth.
Comment 11 Duane Griffin 2008-06-21 10:19:27 UTC
Hi Sami, you should have received my patch for case 1 via email (the second version should be fine to test with, despite the typo already pointed out in it). Let me know if it works for you.

For case two it looks like the orphan list is pointing at a valid non-orphaned inode. Since it is valid it doesn't get processed and removed from the orphan list and we keep looping around. I'm looking into the best way to fix that now.
Comment 12 Sami Liedes 2008-06-22 10:53:33 UTC
The patch for case 1 seems to resolve at least that crash. Running tests now to see if I can get some new kind of crashes.

By the way, do you think it would be more productive for me to run the tests with latest stable rc (or even 2.6 head) rather than the latest stable? It's basically no extra work for me since I have to compile a minimal non-modular version of the kernel anyway.
Comment 13 Duane Griffin 2008-06-23 17:46:43 UTC
> By the way, do you think it would be more productive for me to run the tests
> with latest stable rc (or even 2.6 head) rather than the latest stable?

I'm not sure. On one hand, you might find new bugs before they went into a release and you wouldn't find already fixed bugs. On the other, it would be more work for you and you may find transient issues that are already being worked on, especially if you track the head.

I'd say what you're currently doing is very valuable and working well at finding real existing bugs. Perhaps you should keep on as you are for now and switch to testing -git if and when you stop finding unfixed bugs.
Comment 14 Duane Griffin 2008-06-23 19:35:59 UTC
I still see the OOPS with case 3 with recent git. Looks like the following line from journal_dirty_metadata in fs/jbd/transaction.c:

if (jh->b_modified == 0) {

Which implies bh->b_private is corrupted.
Comment 15 Sami Liedes 2008-06-24 12:50:32 UTC
These cases seem to be now fixed by the various patches you sent to lkml and me. I found one more soft lockup even with your patches applied and filed a separate bug, #10976.
Comment 16 Duane Griffin 2008-06-24 17:23:35 UTC
Thanks, Sami. Excellent work with the testing. I'll take a look at the new bug and comment there. I'm not sure what the usual procedure for closing these tickets is but I suppose once the patches hit mainline would be a sensible time to do so.
Comment 17 Adrian Bunk 2008-07-15 13:16:19 UTC
the fixes are now in Linus' tree
Comment 18 Sami Liedes 2008-08-04 15:42:15 UTC
I can reproduce the crash with attachment #1 [details] (hdb.25.softlockup) with 2.6.26.1, so reopening this bug. The stack trace is slightly different, but the steps to reproduce are same. Here's the backtrace:

----------
fstest:~# mount /dev/hdb /mnt
[   29.064533] kjournald starting.  Commit interval 5 seconds
[   29.073527] EXT3 FS on hdb, internal journal
[   29.074477] EXT3-fs: mounted filesystem with ordered data mode.
fstest:~# cd /mnt
fstest:/mnt# find -xdev >&/dev/null
[   36.866225] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1586: rec_len is smaller than minimal - offset=524, inode=0, rec_len=0, name_len=0
[   36.873003] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1538: directory entry across blocks - offset=0, inode=1538, rec_len=4108, name_len=1
[   36.876934] attempt to access beyond end of device
[   36.877227] hdb: rw=32, want=68262, limit=20480
[   36.884383] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #1555: inode out of bounds - offset=0, inode=134219283, rec_len=12, name_len=1
[   36.887207] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #760: inode out of bounds - offset=64, inode=134218491, rec_len=28, name_len=19
[   36.890514] EXT3-fs warning (device hdb): dx_probe: dx entry: no count or count > limit
[   36.890904] EXT3-fs warning (device hdb): dx_probe: Corrupt dir inode 1281, running e2fsck is recommended.
[   36.895856] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: directory entry across blocks - offset=532, inode=1796, rec_len=16400, name_len=7
[   36.898112] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: inode out of bounds - offset=12, inode=268437174, rec_len=16, name_len=5
[   36.900128] EXT3-fs error (device hdb): ext3_readdir: bad entry in directory #1281: inode out of bounds - offset=48, inode=132784, rec_len=16, name_len=5
[   36.912984] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=1688, inode=268436964, rec_len=24, name_len=14
[   36.915434] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: rec_len is too small for name_len - offset=10240, inode=1414, rec_len=16, name_len=37
[   36.918578] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=2500, inode=536871373, rec_len=64, name_len=55
[   36.922538] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: rec_len is smaller than minimal - offset=6144, inode=0, rec_len=0, name_len=0
[   36.924868] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=12372, inode=2098508, rec_len=12, name_len=3
[   36.926991] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #140: inode out of bounds - offset=4552, inode=262487, rec_len=16, name_len=5
[   36.933880] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #313: inode out of bounds - offset=24, inode=1073742138, rec_len=1000, name_len=39
[   36.949007] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #281: inode out of bounds - offset=24, inode=1048858, rec_len=1000, name_len=92
[   36.953536] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #243: rec_len % 4 != 0 - offset=44, inode=1885745784, rec_len=26979, name_len=48
[   36.962146] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #215: inode out of bounds - offset=12, inode=16777356, rec_len=12, name_len=2
[   36.987669] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #337: inode out of bounds - offset=0, inode=4194641, rec_len=12, name_len=1
[   36.991557] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #141: rec_len is too small for name_len - offset=0, inode=141, rec_len=12, name_len=65
[   36.996444] EXT3-fs error (device hdb): htree_dirblock_to_tree: bad entry in directory #411: rec_len is smaller than minimal - offset=768, inode=0, rec_len=0, name_len=0
[  102.010046] BUG: soft lockup - CPU#0 stuck for 61s! [find:663]
[  102.010046] 
[  102.010046] Pid: 663, comm: find Not tainted (2.6.26.1 #2)
[  102.010046] EIP: 0060:[<c02e2280>] EFLAGS: 00000202 CPU: 0
[  102.010046] EIP is at ext3_find_entry+0x3ea/0x6cb
[  102.010046] EAX: 00000000 EBX: c716ac00 ECX: 00000000 EDX: 00000580
[  102.010046] ESI: 00010000 EDI: c7a29000 EBP: c7ae5dd4 ESP: c7ae5ce4
[  102.010046]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[  102.010046] CR0: 80050033 CR2: 0860206c CR3: 07ada000 CR4: 00000690
[  102.010046] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  102.010046] DR6: 00000000 DR7: 00000000
[  102.010046]  [<c027f592>] ? __find_get_block+0x70/0x13e
[  102.010046]  [<c027f687>] ? __getblk+0x27/0x28a
[  102.010046]  [<c02e2c3d>] ext3_lookup+0x2d/0xb4
[  102.010046]  [<c0270ccc>] ? d_alloc+0x136/0x183
[  102.010046]  [<c02663ad>] do_lookup+0x15c/0x188
[  102.010046]  [<c02676e5>] __link_path_walk+0x2ef/0xd9c
[  102.010046]  [<c053e231>] ? _spin_lock+0x32/0x38
[  102.010046]  [<c053e189>] ? _spin_unlock+0x1d/0x20
[  102.010046]  [<c0453b49>] ? _atomic_dec_and_lock+0x25/0x3c
[  102.010046]  [<c0266066>] ? path_put+0x20/0x23
[  102.010046]  [<c02681dc>] path_walk+0x4a/0x99
[  102.010046]  [<c0268404>] do_path_lookup+0x82/0x1c7
[  102.010046]  [<c0268fd6>] __user_walk_fd+0x32/0x4a
[  102.010046]  [<c02625c1>] vfs_lstat_fd+0x18/0x3e
[  102.010046]  [<c053e231>] ? _spin_lock+0x32/0x38
[  102.010046]  [<c053e189>] ? _spin_unlock+0x1d/0x20
[  102.010046]  [<c0453b49>] ? _atomic_dec_and_lock+0x25/0x3c
[  102.010046]  [<c02626d6>] sys_fstatat64+0x46/0x5a
[  102.010046]  [<c0260691>] ? fput+0x18/0x20
[  102.010046]  [<c025dac2>] ? filp_close+0x41/0x5f
[  102.010046]  [<c025db47>] ? sys_close+0x67/0xab
[  102.010046]  [<c0202e92>] ? syscall_exit+0x8/0x1a
[  102.010046]  [<c0202e86>] syscall_call+0x7/0xb
[  102.010046]  =======================
[  167.510046] BUG: soft lockup - CPU#0 stuck for 61s! [find:663]
[  167.510046] 
[  167.510046] Pid: 663, comm: find Not tainted (2.6.26.1 #2)
[  167.510046] EIP: 0060:[<c02e2280>] EFLAGS: 00000202 CPU: 0
[  167.510046] EIP is at ext3_find_entry+0x3ea/0x6cb
[  167.510046] EAX: 00000000 EBX: c716ac00 ECX: 00000000 EDX: 00000580
[  167.510046] ESI: 00010000 EDI: c7a29000 EBP: c7ae5dd4 ESP: c7ae5ce4
[  167.510046]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[  167.510046] CR0: 80050033 CR2: 0860206c CR3: 07ada000 CR4: 00000690
[  167.510046] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  167.510046] DR6: 00000000 DR7: 00000000
[  167.510046]  [<c027f592>] ? __find_get_block+0x70/0x13e
[  167.510046]  [<c027f687>] ? __getblk+0x27/0x28a
[  167.510046]  [<c02e2c3d>] ext3_lookup+0x2d/0xb4
[  167.510046]  [<c0270ccc>] ? d_alloc+0x136/0x183
[  167.510046]  [<c02663ad>] do_lookup+0x15c/0x188
[  167.510046]  [<c02676e5>] __link_path_walk+0x2ef/0xd9c
[  167.510046]  [<c053e231>] ? _spin_lock+0x32/0x38
[  167.510046]  [<c053e189>] ? _spin_unlock+0x1d/0x20
[  167.510046]  [<c0453b49>] ? _atomic_dec_and_lock+0x25/0x3c
[  167.510046]  [<c0266066>] ? path_put+0x20/0x23
[  167.510046]  [<c02681dc>] path_walk+0x4a/0x99
[  167.510046]  [<c0268404>] do_path_lookup+0x82/0x1c7
[  167.510046]  [<c0268fd6>] __user_walk_fd+0x32/0x4a
[  167.510046]  [<c02625c1>] vfs_lstat_fd+0x18/0x3e
[  167.510046]  [<c053e231>] ? _spin_lock+0x32/0x38
[  167.510046]  [<c053e189>] ? _spin_unlock+0x1d/0x20
[  167.510046]  [<c0453b49>] ? _atomic_dec_and_lock+0x25/0x3c
[  167.510046]  [<c02626d6>] sys_fstatat64+0x46/0x5a
[  167.510046]  [<c0260691>] ? fput+0x18/0x20
[  167.510046]  [<c025dac2>] ? filp_close+0x41/0x5f
[  167.510046]  [<c025db47>] ? sys_close+0x67/0xab
[  167.510046]  [<c0202e92>] ? syscall_exit+0x8/0x1a
[  167.510046]  [<c0202e86>] syscall_call+0x7/0xb
[  167.510046]  =======================
----------
Comment 19 Sami Liedes 2008-08-04 15:44:45 UTC
Sorry for the noise, Duane says this only went in for 2.6.27, so it's not supposed to work on 2.6.26.1. Closing again.