It crashed again shortly after I brought it back up:


Unable to handle kernel paging request at virtual address ffffff83
 printing eip:
c013df80
*pde = 00003067
*pte = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: ipv6 lp ipt_REJECT ipt_state ip_conntrack iptable_filter
ip_tables ohci_hcd
CPU:    0
EIP:    0060:[<c013df80>]    Not tainted
EFLAGS: 00010046   (2.6.7-ql)
EIP is at cache_alloc_refill+0x190/0x220
eax: ffffff7f   ebx: f7c65078   ecx: dc432000   edx: f7cb2118
esi: 00000009   edi: dc432018   ebp: f7cb2100   esp: f625d9a8
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 2804, threadinfo=f625d000 task=c20ba680)
Stack: dc432018 f7cb2118 f7c65088 f7cb2120 000000d0 00000246 000000d0 f7cb2100
       f65ba400 c013e248 000007b6 0b3bd42c f7cb2100 c02273e3 c67043bc 0c608b60
       00000000 c313b9e0 00000000 00000000 0b3bd42c 00000000 00000000 c020e6b5
Call Trace:
 [<c013e248>] kmem_cache_alloc+0x58/0x70
 [<c02273e3>] xfs_iread+0x53/0x240
 [<c020e6b5>] xfs_da_brelse+0xc5/0x110
 [<c0224781>] xfs_iget_core+0x111/0x5e0
 [<c0224d90>] xfs_iget+0x140/0x180
 [<c02409f4>] xfs_dir_lookup_int+0x94/0x100
 [<c0245da6>] xfs_lookup+0x66/0x90
 [<c02517b4>] linvfs_lookup+0x64/0xa0
 [<c0160c69>] __lookup_hash+0x89/0xb0
 [<c0160cf0>] lookup_one_len+0x50/0x60
 [<c01d5afe>] compose_entry_fh+0x5e/0x130
 [<c01d6024>] encode_entry+0x454/0x580
 [<c024f4c6>] linvfs_readdir+0x196/0x240
 [<c015fb26>] permission+0x36/0x40
 [<c01d61a0>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c0153c4c>] open_private_file+0x1c/0x90
 [<c01648b9>] vfs_readdir+0x99/0xb0
 [<c01d61a0>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c01cd1d9>] nfsd_readdir+0x79/0xc0
 [<c03cfe86>] svcauth_unix_accept+0x286/0x2a0
 [<c01d2f20>] nfsd3_proc_readdirplus+0xe0/0x1c0
 [<c01d61a0>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c01d50f0>] nfs3svc_decode_readdirplusargs+0x0/0x180
 [<c01c777e>] nfsd_dispatch+0xbe/0x19c
 [<c03ceb92>] svc_authenticate+0xb2/0xd0
 [<c03cc0f2>] svc_process+0x4d2/0x5f9
 [<c0119db0>] default_wake_function+0x0/0x10
 [<c01c7530>] nfsd+0x1b0/0x340
 [<c01c7380>] nfsd+0x0/0x340
 [<c0103fad>] kernel_thread_helper+0x5/0x18
 
Code: 89 50 04 89 02 66 83 79 14 ff c7 01 00 01 10 00 c7 41 04 00


So I upgraded to the 2.6.8-rc2 hoping it has fixed this problem... When taking
down the 2.6.7-kernel, I got another oops:

Unmounting file systems:  Debug: sleeping function called from invalid context
at include/linux/rwsem.h:43
in_atomic():0, irqs_disabled():1
 [<c01163d0>] do_page_fault+0x0/0x521
 [<c011b3cf>] __might_sleep+0x9f/0xb0
 [<c029da32>] __make_request+0x252/0x500
 [<c0116439>] do_page_fault+0x69/0x521
 [<c01163d0>] do_page_fault+0x0/0x521
 [<c01067c1>] error_code+0x2d/0x38
 [<c013e072>] free_block+0x62/0x100
 [<c013e1e5>] cache_flusharray+0xd5/0xe0
 [<c013e619>] kmem_cache_free+0x49/0x50
 [<c0248eb5>] xfs_finish_reclaim+0xe5/0x110
 [<c0254946>] vn_reclaim+0x56/0x60
 [<c0254e28>] vn_purge+0x138/0x160
 [<c0245984>] xfs_inactive+0xf4/0x4b0
 [<c013e0e3>] free_block+0xd3/0x100
 [<c0254fb8>] vn_remove+0x58/0x5a
 [<c0253adf>] linvfs_clear_inode+0xf/0x20
 [<c016ae44>] clear_inode+0xb4/0xd0
 [<c016ae9c>] dispose_list+0x3c/0x80
 [<c016b02d>] invalidate_inodes+0x8d/0xb0
 [<c0158e7a>] generic_shutdown_super+0x8a/0x190
 [<c01599f7>] kill_block_super+0x17/0x40
 [<c0158d2b>] deactivate_super+0x6b/0x90
 [<c016e18b>] sys_umount+0x3b/0x90
 [<c0105cdf>] syscall_call+0x7/0xb
 
Unable to handle kernel paging request at virtual address 00020004
 printing eip:
c013e072
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: ipv6 lp ipt_REJECT ipt_state ip_conntrack iptable_filter
ip_tables ohci_hcd
CPU:    0
EIP:    0060:[<c013e072>]    Not tainted
EFLAGS: 00010016   (2.6.7-ql)
EIP is at free_block+0x62/0x100
eax: 00020000   ebx: f3ff5000   ecx: f3ff58d0   edx: f4013000
esi: f7cb2100   edi: 00000002   ebp: f7cb2118   esp: c22b4e34
ds: 007b   es: 007b   ss: 0068
Process umount (pid: 27574, threadinfo=c22b4000 task=c2283830)
Stack: f7cb2128 0000001b f7c65088 f7c65088 00000296 f4011480 f7c63c00 c013e1e5
       0000001b f7c65078 f7cb2100 f7c65078 00000296 f4011480 f4c4fc40 c013e619
       f4011480 00000001 00000001 c0248eb5 f4c4fc40 00000000 01afcf60 c0254946
Call Trace:
 [<c013e1e5>] cache_flusharray+0xd5/0xe0
 [<c013e619>] kmem_cache_free+0x49/0x50
 [<c0248eb5>] xfs_finish_reclaim+0xe5/0x110
 [<c0254946>] vn_reclaim+0x56/0x60
 [<c0254e28>] vn_purge+0x138/0x160
 [<c0245984>] xfs_inactive+0xf4/0x4b0
 [<c013e0e3>] free_block+0xd3/0x100
 [<c0254fb8>] vn_remove+0x58/0x5a
 [<c0253adf>] linvfs_clear_inode+0xf/0x20
 [<c016ae44>] clear_inode+0xb4/0xd0
 [<c016ae9c>] dispose_list+0x3c/0x80
 [<c016b02d>] invalidate_inodes+0x8d/0xb0
 [<c0158e7a>] generic_shutdown_super+0x8a/0x190
 [<c01599f7>] kill_block_super+0x17/0x40
 [<c0158d2b>] deactivate_super+0x6b/0x90
 [<c016e18b>] sys_umount+0x3b/0x90
 [<c0105cdf>] syscall_call+0x7/0xb
 
Code: 89 50 04 89 02 8b 43 0c c7 03 00 01 10 00 31 d2 c7 43 04 00

BTW, in case it might matter, the kernel is also complaining about

     nfs warning: mount version older than kernel

and my mount version is:

     # rpm -q mount
     mount-2.11y-31.1

The crash is always occuring in the XFS code. Unless it can be reproduced on a
non-XFS filesystem, I suggest reassigning it to the XFS team.

Another crash.. Now while running the 2.6.8-rc2 kernel :(

Unable to handle kernel paging request at virtual address 01000004
 printing eip:
c013e562
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: ipv6 lp ipt_REJECT ipt_state ip_conntrack iptable_filter
ip_tables ohci_hcd dm_mod
CPU:    1
EIP:    0060:[<c013e562>]    Not tainted
EFLAGS: 00010006   (2.6.8-rc2)
EIP is at free_block+0x52/0xe0
eax: 01000000   ebx: c57b5000   ecx: c57b5e90   edx: ce45f000
esi: f7ca0100   edi: 0000000e   ebp: f7ca0118   esp: f6473754
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 2789, threadinfo=f6473000 task=c2289830)
Stack: f7ca0128 0000001b f7cee0b0 f7cee0b0 00000292 df3b5310 f7cefc00 c013e697
       0000001b f7cee0a0 f7ca0100 f7cee0a0 00000292 df3b5310 d606a9c0 c013ea09
       df3b5310 00000001 00000001 c02465a5 d606a9c0 00000000 01af7f60 c0252056
Call Trace:
 [<c013e697>] cache_flusharray+0xa7/0xb0
 [<c013ea09>] kmem_cache_free+0x49/0x50
 [<c02465a5>] xfs_finish_reclaim+0xe5/0x110
 [<c0252056>] vn_reclaim+0x56/0x60
 [<c0252538>] vn_purge+0x138/0x160
 [<c0243074>] xfs_inactive+0xf4/0x4b0
 [<c013e5d3>] free_block+0xc3/0xe0
 [<c02526c8>] vn_remove+0x58/0x5a
 [<c025128f>] linvfs_clear_inode+0xf/0x20
 [<c016bbe4>] clear_inode+0xd4/0x110
 [<c016bc5c>] dispose_list+0x3c/0x80
 [<c016bf7a>] prune_icache+0xca/0x200
 [<c016c0e5>] shrink_icache_memory+0x35/0x40
 [<c01409a6>] shrink_slab+0x126/0x190
 [<c0141c7a>] try_to_free_pages+0xba/0x180
 [<c0139fd5>] __alloc_pages+0x1e5/0x350
 [<c020d7a5>] xfs_da_do_buf+0x405/0x850
 [<c013a15f>] __get_free_pages+0x1f/0x40
 [<c013d63e>] kmem_getpages+0x1e/0xd0
 [<c013e27c>] cache_grow+0x9c/0x130
 [<c013e3d5>] cache_alloc_refill+0xc5/0x200
 [<c013e6f8>] kmem_cache_alloc+0x58/0x70
 [<c0248439>] kmem_zone_alloc+0x49/0xa0
 [<c02149de>] xfs_dir2_leaf_lookup_int+0x16e/0x290
 [<c02484a5>] kmem_zone_zalloc+0x15/0x50
 [<c0225de4>] xfs_iread+0x34/0x1f0
 [<c02232d1>] xfs_iget_core+0x111/0x5d0
 [<c02238d0>] xfs_iget+0x140/0x180
 [<c023e224>] xfs_dir_lookup_int+0x94/0x100
 [<c0243496>] xfs_lookup+0x66/0x90
 [<c024ef74>] linvfs_lookup+0x64/0xa0
 [<c0161589>] __lookup_hash+0x89/0xb0
 [<c0161610>] lookup_one_len+0x50/0x60
 [<c01d6a0e>] compose_entry_fh+0x5e/0x130
 [<c01d6e9b>] encode_entry+0x3bb/0x4f0
 [<c024cbf6>] linvfs_readdir+0x196/0x240
 [<c015ff26>] permission+0x36/0x40
 [<c01d7020>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c0153fdc>] open_private_file+0x1c/0x90
 [<c0165549>] vfs_readdir+0x99/0xb0
 [<c01d7020>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c01ce0a9>] nfsd_readdir+0x79/0xe0
 [<c03d0be6>] svcauth_unix_accept+0x286/0x2a0
 [<c01d3e30>] nfsd3_proc_readdirplus+0xe0/0x1c0
 [<c01d7020>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c01d6000>] nfs3svc_decode_readdirplusargs+0x0/0x180
 [<c01c857e>] nfsd_dispatch+0xbe/0x19c
 [<c03cf8f2>] svc_authenticate+0xb2/0xd0
 [<c03cce62>] svc_process+0x4d2/0x5f9
 [<c0119990>] default_wake_function+0x0/0x10
 [<c01c8330>] nfsd+0x1b0/0x340
 [<c01c8180>] nfsd+0x0/0x340
 [<c0103fdd>] kernel_thread_helper+0x5/0x18
Code: 89 50 04 89 02 8b 43 0c c7 03 00 01 10 00 31 d2 c7 43 04 00

BTW: I'm running with vm.vfs_cache_pressure=10000 because of oom-killer problems.

(I'm leaving for vacation tomorrow, so I probably woun't be able to give more
information for the next two weeks..)

BTW: I'm running with 4KSTACKS. Will remove this and tell the other engineers to
boot this kernel if it crashes again before I'm back from vacation..

Darn.. down again (before the 8Kstack kernel was ready):

Unable to handle kernel paging request at virtual address 00020004
 printing eip:
c013e562
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: ipv6 lp ipt_REJECT ipt_state ip_conntrack iptable_filter
ip_tables ohci_hcd dm_mod
CPU:    1
EIP:    0060:[<c013e562>]    Not tainted
EFLAGS: 00010016   (2.6.8-rc2)
EIP is at free_block+0x52/0xe0
eax: 00020000   ebx: da46d000   ecx: da46d040   edx: d88540a0
esi: f7ca0380   edi: 00000000   ebp: f7ca0398   esp: f7c52e8c
ds: 007b   es: 007b   ss: 0068
Process kswapd0 (pid: 43, threadinfo=f7c52000 task=f7c7ebd0)
Stack: f7ca03a8 0000001b f7cdf070 f7cdf070 00000296 f2428520 f7cd1000 c013e697
       0000001b f7cdf060 f7ca0380 f7cdf060 00000296 f2428520 00000080 c013ea09
       f2428540 f7c52efc 0000000f c016b968 f2428540 c016bc63 f144b988 f144b980
Call Trace:
 [<c013e697>] cache_flusharray+0xa7/0xb0
 [<c013ea09>] kmem_cache_free+0x49/0x50
 [<c016b968>] destroy_inode+0x28/0x40
 [<c016bc63>] dispose_list+0x43/0x80
 [<c016bf7a>] prune_icache+0xca/0x200
 [<c016c0e5>] shrink_icache_memory+0x35/0x40
 [<c01409a6>] shrink_slab+0x126/0x190
 [<c0141f2a>] balance_pgdat+0x1ea/0x270
 [<c0142085>] kswapd+0xd5/0xe0
 [<c011b3d0>] autoremove_wake_function+0x0/0x50
 [<c011b3d0>] autoremove_wake_function+0x0/0x50
 [<c0141fb0>] kswapd+0x0/0xe0
 [<c0103fdd>] kernel_thread_helper+0x5/0x18
Code: 89 50 04 89 02 8b 43 0c c7 03 00 01 10 00 31 d2 c7 43 04 00

I'm going to go with "it's a stack problem."

Trond - cc'ing you, please see below....

If I trust the script kaos wrote a while ago, and if we trust the stack
from the oops, here's the stack usage:

0x20 cache_flusharray
0x20 kmem_cache_free
0x10 xfs_finish_reclaim
0x8 vn_reclaim
0x44 vn_purge
0x4c xfs_inactive
0x8 free_block
0x20 free_block
0x28 vn_remove
0x8 linvfs_clear_inode
0x4 clear_inode
0xc dispose_list
0x14 prune_icache
0x4 shrink_icache_memory
0x34 shrink_slab
0x2c try_to_free_pages
0x38 __alloc_pages
0xac xfs_da_do_buf
0x8 __get_free_pages
0x4 kmem_getpages
0x18 cache_grow
0x20 cache_alloc_refill
0x10 kmem_cache_alloc
0xc kmem_zone_alloc
0x50 xfs_dir2_leaf_lookup_int
0x14 kmem_zone_zalloc
0x2c xfs_iread
0x4c xfs_iget_core
0x38 xfs_iget
0x2c xfs_dir_lookup_int
0x2c xfs_lookup
0x24 linvfs_lookup
0x18 __lookup_hash
0x18 lookup_one_len
0x18 compose_entry_fh
0x214 encode_entry
0x70 linvfs_readdir
0x10 permission
0x8 open_private_file
0x18 vfs_readdir
0xb4 nfsd_readdir
0x54 svcauth_unix_accept
0x24 nfsd3_proc_readdirplus
0xc nfs3svc_decode_readdirplusargs
0xc nfsd_dispatch
0x14 svc_authenticate
0x3c svc_process
0x2c nfsd
0x2c nfsd

If I added this up right, it's around 7k of stack...  Even allowing for
incorrect entries or sizes in there, it's a very long stack.

Note that encode_entry is using 0x213 (532 decimal), or > 25% of your 4k stack
in a single function.  Due in large part to the struct svc_fh plopped in the
middle of the (nfs) function.

nfsd_readdir is also taking 180 bytes off the stack...

On the xfs side, xfs_da_do_buf is using 172 bytes, linvfs_readdir is using 112...

Those seem to be the big hitters.

If this is a stack issue, not sure where to file the "bug" - or if it even
is one.  I'm not a big fan of 4KSTACKS :)

-Eric Sandeen

Er, 532 is only 13% of 4K, not 25%... but still.  Pretty big :)

Is this issue still present in recent 2.6 kernels?

No, I think it got stable somewher around 2.6.11, and has been happily running
2.6.12.2 for the last 151 days. So please close this ticket.


# uname -a
Linux nfsserver 2.6.12.2 #1 SMP Mon Jul 4 22:28:25 CEST 2005 i686 i686 i386
GNU/Linux
# uptime
 14:43:11  up 151 days, 22:57,  1 user,  load average: 0.01, 0.07, 0.04