Most recent kernel where this bug did *NOT* occur: Distribution: Debian Hardware Environment: x86_64 SMP Software Environment: Problem Description: Steps to reproduce: $ file fs.img fs.img: Unix Fast File system [v2] (big-endian) last mounted on /cell_mw_cfs, last written at Sun Mar 25 05:49:34 2007, clean flag 1, readonly flag 0, number of blocks 5242878, number of data blocks 5077077, number of cylinder groups 56, block size 16384, fragment size 2048, average file size 16384, average number of files in dir 64, pending blocks to free 0, pending inodes to free 0, system-wide uuid 0, minimum percentage of free blocks 8, TIME optimization $ mount -o loop,ufstype=ufs2,ro fs.img tmp/ $ find tmp/ Hits a few errors, dmesg gets some stuff: [827280.036853] UFS-fs error (device loop1): ufs_check_page: bad entry in directory #141312: unaligned directory entry - offset=0, rec_len=61846, name_len=144 [827280.036917] UFS-fs error (device loop1): ufs_readdir: bad page in #141312 [827280.054615] UFS-fs error (device loop1): ufs_read_inode: inode 6 has zero nlink [827280.054616] [827280.054662] init_special_inode: bogus i_mode (0) [827280.054691] UFS-fs error (device loop1): ufs_read_inode: inode 8 has zero nlink [827280.054692] [827280.054736] init_special_inode: bogus i_mode (0) [827280.054755] UFS-fs error (device loop1): ufs_read_inode: inode 10 has zero nlink [827280.054756] [827280.054800] init_special_inode: bogus i_mode (0) [827280.054817] UFS-fs error (device loop1): ufs_read_inode: inode 11 has zero nlink [827280.054819] [827280.054863] init_special_inode: bogus i_mode (0) [827280.054882] UFS-fs error (device loop1): ufs_read_inode: inode 12 has zero nlink [827280.054883] [827280.054927] init_special_inode: bogus i_mode (0) [827280.054946] UFS-fs error (device loop1): ufs_read_inode: inode 15 has zero nlink [827280.054947] [827280.054991] init_special_inode: bogus i_mode (0) [827280.055019] UFS-fs error (device loop1): ufs_read_inode: inode 20 has zero nlink [827280.055021] [827280.055065] init_special_inode: bogus i_mode (0) [827280.055083] UFS-fs error (device loop1): ufs_read_inode: inode 22 has zero nlink [827280.055084] [827280.055128] init_special_inode: bogus i_mode (0) [827280.084792] UFS-fs error (device loop1): ufs_read_inode: inode 23561 has zero nlink [827280.084793] [827280.084842] init_special_inode: bogus i_mode (0) [827280.084863] UFS-fs error (device loop1): ufs_read_inode: inode 23562 has zero nlink [827280.084865] [827280.084910] init_special_inode: bogus i_mode (0) [827280.084930] UFS-fs error (device loop1): ufs_read_inode: inode 23563 has zero nlink [827280.084931] [827280.084975] init_special_inode: bogus i_mode (0) [827280.084993] UFS-fs error (device loop1): ufs_read_inode: inode 23564 has zero nlink [827280.084995] [827280.085039] init_special_inode: bogus i_mode (0) [827280.085056] UFS-fs error (device loop1): ufs_read_inode: inode 23565 has zero nlink [827280.085058] [827280.085102] init_special_inode: bogus i_mode (0) [827280.086067] UFS-fs error (device loop1): ufs_check_page: bad entry in directory #1248485: unaligned directory entry - offset=0, rec_len=56645, name_len=252 [827280.086124] UFS-fs error (device loop1): ufs_readdir: bad page in #1248485 [827280.086471] UFS-fs error (device loop1): ufs_check_page: bad entry in directory #1271989: directory entry across blocks - offset=0, rec_len=52012, name_len=245 [827280.086543] UFS-fs error (device loop1): ufs_readdir: bad page in #1271989 [827280.086725] UFS-fs error (device loop1): ufs_read_inode: inode 23603 has zero nlink [827280.086727] [827280.086771] init_special_inode: bogus i_mode (0) [827280.117270] UFS-fs error (device loop1): ufs_read_inode: inode 94208 has zero nlink [827280.117272] [827280.117317] init_special_inode: bogus i_mode (0) [827280.241836] UFS-fs error (device loop1): ufs_check_page: bad entry in directory #423936: directory entry across blocks - offset=0, rec_len=65280, name_len=0 [827280.241894] UFS-fs error (device loop1): ufs_readdir: bad page in #423936 [827280.355618] UFS-fs error (device loop1): ufs_read_inode: inode 541696 has zero nlink [827280.355620] [827280.355666] init_special_inode: bogus i_mode (0) That part is not completely unexpected, it might be corrupted or have some non-standard structure. But, on unmounting the filesystem, [827297.979496] Unable to handle kernel NULL pointer dereference at 00000000000000b8 RIP: [827297.979514] [<ffffffff88393735>] :ufs:ufs_read_cylinder+0x41/0x2c5 [827297.979585] PGD b40cf067 PUD b34e9067 PMD 0 [827297.979615] Oops: 0000 [1] SMP [827297.979641] CPU 0 [827297.979663] Modules linked in: ufs nls_iso8859_1 isofs i2c_isa w83793 hwmon_vid nfs nfsd exportfs lockd nfs_acl sunrpc button ac battery ipv6 iptable_raw xt_policy xt_multiport ipt_ULOG ipt_TTL ipt_ttl ipt_TOS ipt_tos ipt_TCPMSS ipt_SAME ipt_REJECT ipt_REDIRECT ipt_recent ipt_owner ipt_NETMAP ipt_MASQUERADE ipt_LOG ipt_iprange ipt_ECN ipt_ecn ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_physdev xt_NFQUEUE xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_dccp xt_conntrack xt_CLASSIFY xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables raid456 xor kvm_intel kvm fuse loop evdev parport_pc parport shpchp pci_hotplug floppy i2c_i801 i2c_core pcspkr ext3 jbd mbcache dm_mirror dm_snapshot dm_mod raid1 md_mod ide_generic ata_generic ata_piix sd_mod piix generic ide_core ahci sata_sil24 e1000 ehci_hcd uhci_hcd thermal processor fan [827297.980157] Pid: 11019, comm: umount Not tainted 2.6.20.1 #1 [827297.980185] RIP: 0010:[<ffffffff88393735>] [<ffffffff88393735>] :ufs:ufs_read_cylinder+0x41/0x2c5 [827297.980238] RSP: 0018:ffff810024ad9d08 EFLAGS: 00010246 [827297.980265] RAX: 0000000000000000 RBX: ffff81013939ac00 RCX: 0000000000000000 [827297.980309] RDX: 0000000000000000 RSI: ffff81013b4e5c00 RDI: 00000000000000b8 [827297.980353] RBP: 0000000000000000 R08: 0000000000000024 R09: 0000000000000000 [827297.980397] R10: ffff81013cf5e528 R11: 00000000fffffffa R12: ffff810136a4b800 [827297.980442] R13: ffff810136a4b800 R14: ffff81013939ac00 R15: 0000000000000017 [827297.980487] FS: 00002ace8cbed1d0(0000) GS:ffffffff804fb000(0000) knlGS:0000000000000000 [827297.980532] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [827297.980560] CR2: 00000000000000b8 CR3: 00000000b34f6000 CR4: 00000000000026e0 [827297.980605] Process umount (pid: 11019, threadinfo ffff810024ad8000, task ffff81013b42b040) [827297.980650] Stack: ffff810100000000 ffff81013939ac00 ffff810025753a18 0000000000000017 [827297.980705] ffff810136a4b800 0000000000513c20 0000000000005c00 ffffffff88393bc2 [827297.980756] ffff810025753a18 0000000000084400 ffff810025753a18 ffff810136a4b800 [827297.980790] Call Trace: [827297.980836] [<ffffffff88393bc2>] :ufs:ufs_load_cylinder+0x12c/0x13a [827297.980872] [<ffffffff8839511b>] :ufs:ufs_free_inode+0x8f/0x308 [827297.980908] [<ffffffff88395dee>] :ufs:ufs_delete_inode+0x0/0x93 [827297.980942] [<ffffffff88395e7b>] :ufs:ufs_delete_inode+0x8d/0x93 [827297.980974] [<ffffffff8022c05e>] generic_delete_inode+0xc0/0x136 [827297.981007] [<ffffffff802c0999>] shrink_dcache_for_umount_subtree+0x203/0x250 [827297.981055] [<ffffffff802c0e95>] shrink_dcache_for_umount+0x2f/0x3d [827297.981087] [<ffffffff802bac62>] generic_shutdown_super+0x19/0xf1 [827297.981119] [<ffffffff802bad60>] kill_block_super+0x26/0x3b [827297.981149] [<ffffffff802bae2f>] deactivate_super+0x6a/0x83 [827297.981180] [<ffffffff802c2e0f>] sys_umount+0x247/0x27d [827297.981212] [<ffffffff802215d3>] sys_newstat+0x19/0x31 [827297.981250] [<ffffffff8025611e>] system_call+0x7e/0x83 [827297.981284] [827297.981303] [827297.981304] Code: 48 8b 04 38 4c 8b 68 28 75 0c 44 89 f8 0f af 86 14 01 00 00 [827297.981404] RIP [<ffffffff88393735>] :ufs:ufs_read_cylinder+0x41/0x2c5 [827297.981438] RSP <ffff810024ad9d08> [827297.981461] CR2: 00000000000000b8 [827297.981821] BUG: at kernel/exit.c:860 do_exit() [827297.981939] [827297.981940] Call Trace: [827297.982067] [<ffffffff802135ca>] do_exit+0x51/0x7ff [827297.982138] [<ffffffff8025c84b>] _spin_unlock_irqrestore+0x8/0x9 [827297.982213] [<ffffffff8020aa0b>] do_page_fault+0x72d/0x7ad [827297.982291] [<ffffffff802371dd>] d_instantiate+0x52/0x8a [827297.982362] [<ffffffff8023c61c>] d_rehash+0x21/0x34 [827297.982432] [<ffffffff8022d703>] d_splice_alias+0x114/0x11c [827297.982511] [<ffffffff8025c9fd>] error_exit+0x0/0x84 [827297.982592] [<ffffffff88393735>] :ufs:ufs_read_cylinder+0x41/0x2c5 [827297.982673] [<ffffffff88393bc2>] :ufs:ufs_load_cylinder+0x12c/0x13a [827297.982753] [<ffffffff8839511b>] :ufs:ufs_free_inode+0x8f/0x308 [827297.982832] [<ffffffff88395dee>] :ufs:ufs_delete_inode+0x0/0x93 [827297.982908] [<ffffffff88395e7b>] :ufs:ufs_delete_inode+0x8d/0x93 [827297.982981] [<ffffffff8022c05e>] generic_delete_inode+0xc0/0x136 [827297.983054] [<ffffffff802c0999>] shrink_dcache_for_umount_subtree+0x203/0x250 [827297.983145] [<ffffffff802c0e95>] shrink_dcache_for_umount+0x2f/0x3d [827297.983218] [<ffffffff802bac62>] generic_shutdown_super+0x19/0xf1 [827297.983292] [<ffffffff802bad60>] kill_block_super+0x26/0x3b [827297.983365] [<ffffffff802bae2f>] deactivate_super+0x6a/0x83 [827297.983437] [<ffffffff802c2e0f>] sys_umount+0x247/0x27d [827297.983512] [<ffffffff802215d3>] sys_newstat+0x19/0x31 [827297.983591] [<ffffffff8025611e>] system_call+0x7e/0x83 A friend tells me that he got an oops doing the same thing on a 32-bit UP system, but I'm not sure the exact kernel version (probably 2.6.18).
>$ file fs.img Thanks for reporting this. May be I'm missing something, is it possible to get this image somewhere?
Unfortunately I can't share it, and I haven't reproduced it with other images. I can probably pull out pieces if there is some particular table or entry you'd like to see, or if you'd like any extra debugging added to ufs I can do that too.
>Unfortunately I can't share it, and I haven't reproduced it with other images. >I can probably pull out pieces if there is some particular table or entry you'd >like to see, or if you'd like any extra debugging added to ufs I can do that >too. There is UFS's option avaible on kernel build stage: >UFS debugging (UFS_DEBUG) > >If you are experiencing any problems with the UFS filesystem, say >Y here. This will result in _many_ additional debugging messages to be >written to the system log. can you turn on it and post result here, messages should appear somewhere like /var/log/messages. And before any rebuild of kernel can you post here result of "objdump -S path/to/ufs/module"? This should help "parse" oops message.
Created attachment 10984 [details] objdump -S ufs.ko
Created attachment 10985 [details] objdump -S ufs.ko
Disassembly added. I'll want to move to a different machine before I start triggering the oops on purpose, so it might take me a few days to get the debugging output.
>Disassembly added. Thanks. After looking on trace: 827297.980942] [<ffffffff88395e7b>] :ufs:ufs_delete_inode+0x8d/0x93 [827297.980974] [<ffffffff8022c05e>] generic_delete_inode+0xc0/0x136 the problem happened, because of a)filesystem mounted read-only b)filesystem was damaged, and inode has zero link count c)ufs_delete_inode was called Can you try this patch?
Created attachment 11104 [details] ufs_unmount_zero_link fix This patch should fix oops on unmount, if cache contains inode with zero links, and filesystem mounted readonly.
I tested it: ufs-proper-handling-of-zero-link-case.patch fixes the oops on unmount. Thank you!
It looks like the patch is in the tree, the problem can be closed. Thanks.