Bug 7364
Summary: | nbd dead-lock/panic with 4k stack | ||
---|---|---|---|
Product: | IO/Storage | Reporter: | Bruno Prémont (bonbons) |
Component: | Block Layer | Assignee: | Jens Axboe (axboe) |
Status: | RESOLVED CODE_FIX | ||
Severity: | normal | CC: | akpm, alan, protasnb, xfs-masters |
Priority: | P2 | ||
Hardware: | i386 | ||
OS: | Linux | ||
Kernel Version: | 2.6.22-rc5 | Subsystem: | |
Regression: | No | Bisected commit-id: | |
Attachments: |
alt + sysreq + t output
alt + sysreq + t output on vanila 2.6.18.1 alt + sysreq + t output on vanila 2.6.19-rc3 Kernel config used for 2.6.19-rc3 in comment #11 |
Description
Bruno Prémont
2006-10-14 15:27:21 UTC
Oh wonderful. The call trace is garbage because the kernel oopsed while trying to generate the call trace. Can you please try 2.6.19-rc2 (or at least 2.6.18, but 2.6.19-rc2 has more fixes) and also disable CONFIG_UNWIND_INFO? Hopefully that'll get us a decent backtrace. Also the backtraces you've included here got wordwrapped, which makes them hard to read. Thanks. With 2.6.18.1 I have troubles getting stacktrace, there netconsole stops working "just before" kernel panics and outputs dumps to screen. From netconsole I get output of normal printk(), but no more from the stack-traces which are at least two 0x31A screens high. Serial console is not possible as I have no serial port on that host and no nullmodem cable either (for use with usb2serial). With 2.6.18.1 in most cases I get no output at all, only with pushing all output to console (dmesg -n 8) I get some stacktrace. Otherwise it's like a deadlock, sysrq is not answering either in that state. Regarding 2.6.19-rc2 I will try to reproduce there as well and report back! Here is one trace that it accepted to send via netconsole: [ 275.396000] BUG: unable to handle kernel paging request at virtual address 8bec5593 [ 275.400000] BUG: unable to handle kernel paging request at virtual address 8bec5593 [ 275.400000] printing eip: [ 275.400000] c0117544 [ 275.400000] *pde = 00000000 [ 275.400000] Oops: 0002 [#1] [ 275.400000] PREEMPT [ 275.400000] Modules linked in: xfs squashfs zlib_inflate loop nbd netconsole tg3 rtc usbcore ext2 mbcache evdev cpufreq_ondemand [ 275.400000] CPU: 0 [ 275.400000] EIP: 0060:[<c0117544>] Not tainted VLI [ 275.400000] EFLAGS: 00210093 (2.6.18.1-restena #1) [ 275.400000] EIP is at scheduler_tick+0x84/0x320 [ 275.400000] eax: 8bec558b ebx: c02c67db ecx: bfc026f0 edx: 27763ce2 [ 275.400000] esi: 1f1e8200 edi: 00000040 ebp: c047ff98 esp: c047ff84 [ 275.400000] ds: 007b es: 007b ss: 0068 [ 275.400000] Process �딍v (pid: -1996488704, ti=c047f000 task=c02c67db task.ti=8bec558b) [ 275.400000] Stack: c01213bf c02c67db c02c67db 00000000 00000000 c047ffac c0125caf c047efb8 [ 275.400000] 00000000 c047efb8 c047ffb8 c0106cc1 c03bb520 c047ffd4 c0141993 00000000 [ 275.400000] 00000000 c0441a40 00000000 c0441a68 c047fff8 c0141a4f f729c2e0 00000001 [ 275.400000] Call Trace: [ 275.400000] [<c010416a>] show_stack_log_lvl+0x9a/0xc0 [ 275.400000] [<c0104396>] show_registers+0x1a6/0x220 [ 275.400000] [<c01045b4>] die+0x104/0x210 [ 275.400000] [<c0115998>] do_page_fault+0x2d8/0x570 [ 275.400000] [<c0103dd9>] error_code+0x39/0x40 [ 275.400000] [<c0125caf>] update_process_times+0x8f/0xb0 [ 275.400000] [<c0106cc1>] timer_interrupt+0x41/0xb0 [ 275.400000] [<c0141993>] handle_IRQ_event+0x33/0x70 [ 275.400000] [<c0141a4f>] __do_IRQ+0x7f/0x100 [ 275.400000] [<c010595d>] do_IRQ+0x6d/0xc0 [ 275.400000] ======================= [ 275.400000] [<c0103cd1>] common_interrupt+0x25/0x2c [ 275.400000] [<c0105afe>] do_softirq+0x9e/0x100 [ 275.400000] ======================= [ 275.400000] [<c012125c>] irq_exit+0x4c/0x50 [ 275.400000] [<c0105964>] do_IRQ+0x74/0xc0 [ 275.400000] [<c0103cd1>] common_interrupt+0x25/0x2c [ 275.400000] [<c011c328>] printk+0x18/0x20 [ 275.400000] [<c01159fc>] do_page_fault+0x33c/0x570 [ 275.400000] [<c0103dd9>] error_code+0x39/0x40 [ 275.400000] [<f8ddb575>] nbd_send_req+0x145/0x260 [nbd] [ 275.400000] [<f8ddbb53>] do_nbd_request+0xa3/0x230 [nbd] [ 275.400000] [<c01f3fb8>] __generic_unplug_device+0x28/0x30 [ 275.400000] [<c01f553e>] __make_request+0x10e/0x310 [ 275.400000] [<c01f586d>] generic_make_request+0x9d/0x110 [ 275.400000] [<c01f5951>] submit_bio+0x71/0x120 [ 275.400000] [<f99a8d2d>] _xfs_buf_ioapply+0x1ad/0x2d0 [xfs] [ 275.400000] [<f99a8e7a>] xfs_buf_iorequest+0x2a/0x80 [xfs] [ 275.400000] [<f998b80c>] xlog_bdstrat_cb+0x1c/0x50 [xfs] [ 275.400000] [<f998c1d6>] xlog_sync+0x246/0x520 [xfs] [ 275.400000] [<f998da3a>] xlog_state_release_iclog+0xfa/0x120 [xfs] [ 275.400000] [<f998df19>] xlog_state_sync+0x179/0x270 [xfs] [ 275.400000] [<f998ac64>] _xfs_log_force+0x44/0x70 [xfs] [ 275.400000] [<f9951b5d>] xfs_alloc_search_busy+0xdd/0xe0 [xfs] [ 275.400000] [<f994f3b5>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs] [ 275.400000] [<f9951755>] xfs_alloc_vextent+0x335/0x440 [xfs] [ 275.400000] [<f996045d>] xfs_bmap_btalloc+0x2dd/0x780 [xfs] [ 275.400000] [<f996091e>] xfs_bmap_alloc+0x1e/0x30 [xfs] [ 275.400000] [<f9963cdc>] xfs_bmapi+0xcbc/0x1360 [xfs] [ 275.400000] [<f9970614>] xfs_dir2_grow_inode+0xe4/0x3a0 [xfs] [ 275.400000] [<f9972124>] xfs_dir2_sf_to_block+0xb4/0x5a0 [xfs] [ 275.400000] [<f9978078>] xfs_dir2_sf_addname+0x108/0x130 [xfs] [ 275.400000] [<f996ffe0>] xfs_dir_createname+0xd0/0x110 [xfs] [ 275.400000] [<f99a0f9e>] xfs_create+0x48e/0x690 [xfs] [ 275.400000] [<f99acd60>] xfs_vn_mknod+0x210/0x450 [xfs] [ 275.400000] [<f99acfb2>] xfs_vn_create+0x12/0x20 [xfs] [ 275.400000] [<c0172fbb>] vfs_create+0xcb/0x110 [ 275.400000] [<c017329f>] open_namei+0xbf/0x650 [ 275.400000] [<c0161d0c>] do_filp_open+0x2c/0x60 [ 275.400000] [<c01620a2>] do_sys_open+0x52/0xe0 [ 275.400000] [<c016214c>] sys_open+0x1c/0x20 [ 275.400000] [<c01031b1>] sysenter_past_esp+0x56/0x8d [ 275.400000] ======================= [ 275.400000] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034 [ 275.400000] printing eip: [ 275.400000] c010406b [ 275.400000] *pde = 00000000 [ 275.400000] Oops: 0000 [#2] [ 275.400000] PREEMPT [ 275.400000] Modules linked in: xfs squashfs zlib_inflate loop nbd netconsole tg3 rtc usbcore ext2 mbcache evdev cpufreq_ondemand [ 275.400000] CPU: 0 [ 275.400000] EIP: 0060:[<c010406b>] Not tainted VLI [ 275.400000] EFLAGS: 00210003 (2.6.18.1-restena #1) [ 275.400000] EIP is at show_trace_log_lvl+0x6b/0xb0 [ 275.400000] eax: 00000ffd ebx: f729c000 ecx: 00000001 edx: 00000001 [ 275.400000] esi: f729c000 edi: 00000000 ebp: c047fe60 esp: c047fe3c [ 275.400000] ds: 007b es: 007b ss: 0068 [ 275.400000] Process �딍v (pid: -1996488704, ti=c047f000 task=c02c67db task.ti=8bec558b) [ 275.400000] Stack: c036d99f c036d9fa c047ffe4 00000018 00000000 c047fe60 c047ffe4 00000018 [ 275.400000] 00000000 c047fe84 c010416a c036d9fa c036d9fa c047ff50 c047ff84 c02c6967 [ 275.400000] 00000000 c047ff50 c047fec8 c0104396 c036d9fa 00000010 c02c6967 89000000 [ 275.400000] Call Trace: [ 275.400000] [<c010416a>] show_stack_log_lvl+0x9a/0xc0 [ 275.400000] [<c0104396>] show_registers+0x1a6/0x220 [ 275.400000] [<c01045b4>] die+0x104/0x210 [ 275.400000] [<c0115998>] do_page_fault+0x2d8/0x570 [ 275.400000] [<c0103dd9>] error_code+0x39/0x40 [ 275.400000] [<c010416a>] show_stack_log_lvl+0x9a/0xc0 [ 275.400000] [<c0104396>] show_registers+0x1a6/0x220 [ 275.400000] [<c01045b4>] die+0x104/0x210 [ 275.400000] [<c0115998>] do_page_fault+0x2d8/0x570 [ 275.400000] [<c0103dd9>] error_code+0x39/0x40 [ 275.400000] [<c0125caf>] update_process_times+0x8f/0xb0 [ 275.400000] [<c0106cc1>] timer_interrupt+0x41/0xb0 [ 275.400000] [<c0141993>] handle_IRQ_event+0x33/0x70 [ 275.400000] [<c0141a4f>] __do_IRQ+0x7f/0x100 [ 275.400000] [<c010595d>] do_IRQ+0x6d/0xc0 [ 275.400000] ======================= [ 275.400000] [<c0103cd1>] common_interrupt+0x25/0x2c [ 275.400000] [<c0105afe>] do_softirq+0x9e/0x100 [ 275.400000] ======================= [ 275.400000] [<c012125c>] irq_exit+0x4c/0x50 [ 275.400000] [<c0105964>] do_IRQ+0x74/0xc0 [ 275.400000] [<c0103cd1>] common_interrupt+0x25/0x2c [ 275.400000] [<c011c328>] printk+0x18/0x20 [ 275.400000] [<c01159fc>] do_page_fault+0x33c/0x570 [ 275.400000] [<c0103dd9>] error_code+0x39/0x40 [ 275.400000] [<f8ddb575>] nbd_send_req+0x145/0x260 [nbd] [ 275.400000] [<f8ddbb53>] do_nbd_request+0xa3/0x230 [nbd] [ 275.400000] [<c01f3fb8>] __generic_unplug_device+0x28/0x30 [ 275.400000] [<c01f553e>] __make_request+0x10e/0x310 [ 275.400000] [<c01f586d>] generic_make_request+0x9d/0x110 [ 275.400000] [<c01f5951>] submit_bio+0x71/0x120 [ 275.400000] [<f99a8d2d>] _xfs_buf_ioapply+0x1ad/0x2d0 [xfs] [ 275.400000] [<f99a8e7a>] xfs_buf_iorequest+0x2a/0x80 [xfs] [ 275.400000] [<f998b80c>] xlog_bdstrat_cb+0x1c/0x50 [xfs] [ 275.400000] [<f998c1d6>] xlog_sync+0x246/0x520 [xfs] [ 275.400000] [<f998da3a>] xlog_state_release_iclog+0xfa/0x120 [xfs] [ 275.400000] [<f998df19>] xlog_state_sync+0x179/0x270 [xfs] [ 275.400000] [<f998ac64>] _xfs_log_force+0x44/0x70 [xfs] [ 275.400000] [<f9951b5d>] xfs_alloc_search_busy+0xdd/0xe0 [xfs] [ 275.400000] [<f994f3b5>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs] [ 275.400000] [<f9951755>] xfs_alloc_vextent+0x335/0x440 [xfs] [ 275.400000] [<f996045d>] xfs_bmap_btalloc+0x2dd/0x780 [xfs] [ 275.400000] [<f996091e>] xfs_bmap_alloc+0x1e/0x30 [xfs] [ 275.400000] [<f9963cdc>] xfs_bmapi+0xcbc/0x1360 [xfs] [ 275.400000] [<f9970614>] xfs_dir2_grow_inode+0xe4/0x3a0 [xfs] [ 275.400000] [<f9972124>] xfs_dir2_sf_to_block+0xb4/0x5a0 [xfs] [ 275.400000] [<f9978078>] xfs_dir2_sf_addname+0x108/0x130 [xfs] [ 275.400000] [<f996ffe0>] xfs_dir_createname+0xd0/0x110 [xfs] [ 275.400000] [<f99a0f9e>] xfs_create+0x48e/0x690 [xfs] [ 275.400000] [<f99acd60>] xfs_vn_mknod+0x210/0x450 [xfs] [ 275.400000] [<f99acfb2>] xfs_vn_create+0x12/0x20 [xfs] [ 275.400000] [<c0172fbb>] vfs_create+0xcb/0x110 [ 275.400000] [<c017329f>] open_namei+0xbf/0x650 [ 275.400000] [<c0161d0c>] do_filp_open+0x2c/0x60 [ 275.400000] [<c01620a2>] do_sys_open+0x52/0xe0 [ 275.400000] [<c016214c>] sys_open+0x1c/0x20 [ 275.400000] [<c01031b1>] sysenter_past_esp+0x56/0x8d [ 275.400000] ======================= [ 275.400000] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034 [ 275.400000] printing eip: [ 275.400000] c010406b [ 275.400000] *pde = 00000000 [ 275.400000] Recursive die() failure, output suppressed [ 275.400000] <0>Kernel panic - not syncing: Fatal exception in interrupt Another trace (much smaller) happening at another try (did not get through netconsole), this one looks like it's only a crash in crash handling (as in my previous post): http://homepage.internet.lu/brunop/IMG_1835.JPG At each try the output differs (a lot), an incomplete, uncaptured one mentionned spinlock issue. I tried 2.6.19-rc2 as well, same kind of crash with page-fault. As for 2.6.18.1 I don't get the trace over netconsole but have to take a photo of it: http://homepage.internet.lu/brunop/IMG_1841.JPG On this one, top is page_fault dump, below spurious ACKs and spinlock appeared half a minute later. Hope this helps figuring out what's happening. From memory point of view, system has 2GB and has plenty of unused memory (fresh start in console mode only) dd if=/dev/ndb0 of=/dev/null works without an issue, copying lots of files to empty partition on nbd0 worked fine, just (intense) mixed read/write access seems to cause trouble. Have you tried setting CONFIG_4KSTACKS=n? On Sun, 15 Oct 2006 04:42:37 -0700 bugme-daemon@bugzilla.kernel.org wrote: > http://homepage.internet.lu/brunop/IMG_1841.JPG yup "esp:f72c305c". You've run out of stack. CONFIG_4KSTACKS=n should get it going. You're right, without 4k stacks it works. Looks like nbd description should at least have a warning regarding 4K stacks, or nbd would need an own stack in case of 4k stacks (possibly the way loop does) On Sun, 15 Oct 2006 12:20:50 -0700 bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=7364 We have a repeatable stack overrun with 4k stacks, XFS over NBD. Maybe something like the following will help: gcc (GCC) 4.1.1 (Gentoo 4.1.1-r1) with CONFIG_FORCED_INLINING=y (same as Bruno) on i386 make checkstack before: 0x0005b7b6 xfs_vn_mknod: 400 after: 0x0005b83a xfs_vn_mknod: 128 ====================== begin patch ====================== --- linux/fs/xfs/linux-2.6/xfs_iops.c.orig 2006-10-16 02:22:19.000000000 +0200 +++ linux/fs/xfs/linux-2.6/xfs_iops.c 2006-10-16 03:45:51.000000000 +0200 @@ -262,7 +262,11 @@ return (task->fs != init_task.fs); } -STATIC inline void +/* + * Do not inline. We only hit this in the error path and struct dentry needs + * huge amounts of stack space + */ +STATIC void xfs_cleanup_inode( vnode_t *dvp, vnode_t *vp, ====================== end patch ====================== @XFS-People: you should realy do something about your stack usage. You nest deep and some funcs use some stack, which sums up. Eg xfs_bmapi -> 398 bytes, lots of funcs with 100 bytes (i think this is vattr_t, and btw, all these typedefs...) Greetings Jan Created attachment 9358 [details]
alt + sysreq + t output
Unfrtuately the stack-overflows with 4k stacks are not the only troubles I
experience with nbd...
For now I've disabled 4k Stack but kept preemption (with preemptible big kernel
lock) and regularly get heavy-io processes accessing my nbd device ending up in
D state.
Attached is the output of sysreq+t in such a locked state (with minimum of
processes around)
Most intresting are probably the following tasks:
pdflush (PID 160)
ebuild.sh (PID 20785)
Both seem to be blocked by some XFS spinlock/mutex.
ebuild.sh is a bash process doing some parsing for the Gentoo emerge
(calculating dependencies for system update with empty cache)
That call of emerge (like explicit metadata updates) produces a lot of
disc activity on the cache (ndb in my case) and the portage tree (squashfs in
my case). The probably important point is that there are lots of reads AND
writes happening. Just reading or just writing never caused such deadlocks.
This problem can be reproduced very easily.
My setup:
2.6.18.1 kernel, based on config posted above (only limited changes)
The kernel running at that time is patched with linux-vserver 2.1.1-rc42 and
squashfs 3.1
CPU: PentiumM second generation (Family 6, model 13, steppting 8)
RAM: 2GB
GCC 3.4.6 + glibc 2.3.6 used to compile kernel + userspace
GCC 4.1.1 + glibc 2.4-r3 used to compile userspace in chroot/context
running emerge (PIDs: 9501, 9530, 9547, 20784, 20785)
I will try to reproduce with vanilla 2.6.18.1 and 2.6.19-rc3, to be sure that
linux-vserver does not interfere, and report back.
If more specific information is useful, please indicate what information I
should provide!
Created attachment 9364 [details]
alt + sysreq + t output on vanila 2.6.18.1
Did similar working with vanilla 2.6.18.1 kernel and still get processes on NBD
(XFS partition) to deadlock.
This time stuck while doing a rm -rf.
Using 2.6.19-rc3 I have not yet been able to reproduce these locks... maybe
fixed by one of the XFS changes?...
Created attachment 9370 [details]
alt + sysreq + t output on vanila 2.6.19-rc3
Also got it with vanilla 2.6.19-rc3, but it happens less often / less
immediately...
The output from kernel.log for this dead-lock contains two alt+sysReq+t, the
second issued about 6 minutes after the first (no reboot in between, just
umounting some other filesystems / flushing buffers on other block devices)
The weird thing is that I never get such deadlocks with XFS on other
blockdevices (loop, sd/usb-storage, local drive)...
Created attachment 9371 [details] Kernel config used for 2.6.19-rc3 in comment #11 See http://bugzilla.kernel.org/show_bug.cgi?id=7364 This guy has XFS deadlocks. And stack overruns, but we knew about that. Perhaps the deadlocks are due to NBD? On Wed, Jan 31, 2007 at 12:39:33AM -0800, Andrew Morton wrote:
>
> See http://bugzilla.kernel.org/show_bug.cgi?id=7364
>
> This guy has XFS deadlocks. And stack overruns, but we knew about that.
>
> Perhaps the deadlocks are due to NBD?
The first set of traces show:
- pdflush trying to write the superblock and sleeping because
the superblock buffer is pinned in memory.
- a create waiting for the superblock buffer lock so
it can include the superblock in a transaction
The create is backed up behind superblock write, but hte superblock
write can't make progress until the the pinned buffer is unpinned
during I/O completion of the log buffer that contains the
transaction that the superblock modification was recorded in. So it
would appear we are hung up waiting for I/O to complete.
And in the 2.6.18.1 trace we have:
- rm doing an inode read from disk which is waiting for
I/O completion.
- pdflush trying to write the superblock and sleeping because
the superblock buffer is pinned in memory.
If the inode read wasn't a give-away, the pdflush thread should make
it is obvious we are hung up waiting for I/O completion.
And from 2.6.19-rc3, we have a pdflush thread asleep in
xlog_state_get_iclog_space() waiting for a in-core log buffer to
become available so it can commit an allocation transaction to the
log. We only wait here when all the iclog buffers are under I/O
and we have to wait for them to complete their I/O.....
So, there's no deadlocks in XFS here - XFs is simply waiting for the
I/O it issued to complete. I think this pretty clearly indicates an
NBD problem.
Cheers,
Dave.
Reply-To: paul.clements@steeleye.com David Chinner wrote: > On Wed, Jan 31, 2007 at 12:39:33AM -0800, Andrew Morton wrote: >> See http://bugzilla.kernel.org/show_bug.cgi?id=7364 >> >> This guy has XFS deadlocks. And stack overruns, but we knew about that. >> >> Perhaps the deadlocks are due to NBD? > So, there's no deadlocks in XFS here - XFs is simply waiting for the > I/O it issued to complete. I think this pretty clearly indicates an > NBD problem. Maybe a network problem, but there's nothing in these stack traces to implicate nbd. The only process that's in nbd code is the nbd-client, which is doing what it normally does: sitting and waiting for replies from the server. Any details about what was going on at the time of the deadlock? -- Paul Bruno, Could you please give some more info on your application and setup. Also, have you tried the latest kernel again for the record? Thanks. I just tried to reproduce on 2.6.22-rc5 now and got the following output (see end for description of my setup): [ 7753.510000] BUG: scheduling while atomic: emerge/0xd0117331/4970 [ 7753.510000] [<c0104afa>] show_trace_log_lvl+0x1a/0x30 [ 7753.520000] [<c0104b22>] show_trace+0x12/0x20 [ 7753.520000] [<c0104c45>] dump_stack+0x15/0x20 [ 7753.530000] [<c03066f7>] schedule+0x497/0x5c0 [ 7753.530000] [<c01188c3>] __cond_resched+0x23/0x50 [ 7753.540000] [<c0306d29>] cond_resched+0x39/0x40 [ 7753.540000] [<e3af4435>] xfs_buf_iowait+0x35/0x60 [xfs] [ 7753.550000] [<e3af3f49>] xfs_buf_iostart+0x79/0x90 [xfs] [ 7753.550000] [<e3af3830>] xfs_buf_read_flags+0x50/0x70 [xfs] [ 7753.550000] [<e3ae58b5>] xfs_trans_read_buf+0x195/0x320 [xfs] [ 7753.560000] [<e3ab8345>] xfs_btree_read_bufs+0x55/0x70 [xfs] [ 7753.560000] [<e3a9f6a6>] xfs_alloc_lookup+0xe6/0x3c0 [xfs] [ 7753.570000] [<e3aa0d24>] xfs_alloc_lookup_eq+0x14/0x20 [xfs] [ 7753.570000] [<e3a9bbab>] xfs_alloc_fixup_trees+0x17b/0x330 [xfs] [ 7753.580000] [<e3a9c7a1>] xfs_alloc_ag_vextent_near+0x6e1/0x9b0 [xfs] [ 7753.580000] [<e3a9beee>] xfs_alloc_ag_vextent+0xee/0x100 [xfs] [ 7753.590000] [<e3a9e250>] xfs_alloc_vextent+0x330/0x430 [xfs] [ 7753.590000] [<e3aad057>] xfs_bmap_btalloc+0x2e7/0x790 [xfs] [ 7753.590000] [<e3aad51e>] xfs_bmap_alloc+0x1e/0x30 [xfs] [ 7753.600000] [<e3ab085c>] xfs_bmapi+0xccc/0x13a0 [xfs] [ 7753.600000] [<e3abcf14>] xfs_dir2_grow_inode+0xe4/0x3a0 [xfs] [ 7753.610000] [<e3abea34>] xfs_dir2_sf_to_block+0xb4/0x5a0 [xfs] [ 7753.610000] [<e3ac49e8>] xfs_dir2_sf_addname+0x108/0x130 [xfs] [ 7753.620000] [<e3abc8e0>] xfs_dir_createname+0xd0/0x110 [xfs] [ 7753.620000] [<e3ae2013>] xfs_rename+0x2d3/0x8c0 [xfs] [ 7753.620000] [<e3af80ba>] xfs_vn_rename+0x3a/0x90 [xfs] [ 7753.630000] [<c0166ec6>] vfs_rename_other+0xb6/0x100 [ 7753.630000] [<c016701d>] vfs_rename+0x10d/0x240 [ 7753.640000] [<c01672c1>] do_rename+0x171/0x1a0 [ 7753.640000] [<c016735e>] sys_renameat+0x6e/0x80 [ 7753.650000] [<c0167399>] sys_rename+0x29/0x30 [ 7753.650000] [<c0103dd6>] sysenter_past_esp+0x5f/0x85 [ 7753.660000] ======================= [ 7753.660000] BUG: scheduling while atomic: emerge/0xc0117331/4970 [ 7753.670000] [<c0104afa>] show_trace_log_lvl+0x1a/0x30 [ 7753.670000] [<c0104b22>] show_trace+0x12/0x20 [ 7753.670000] [<c0104c45>] dump_stack+0x15/0x20 [ 7753.680000] [<c03066f7>] schedule+0x497/0x5c0 [ 7753.680000] [<c0307773>] __down+0x63/0xc0 [ 7753.690000] [<c03076a2>] __down_failed+0xa/0x10 [ 7753.690000] [<e3af4442>] xfs_buf_iowait+0x42/0x60 [xfs] [ 7753.690000] [<e3af3f49>] xfs_buf_iostart+0x79/0x90 [xfs] [ 7753.700000] [<e3af3830>] xfs_buf_read_flags+0x50/0x70 [xfs] [ 7753.700000] [<e3ae58b5>] xfs_trans_read_buf+0x195/0x320 [xfs] [ 7753.710000] [<e3ab8345>] xfs_btree_read_bufs+0x55/0x70 [xfs] [ 7753.710000] [<e3a9f6a6>] xfs_alloc_lookup+0xe6/0x3c0 [xfs] [ 7753.710000] [<e3aa0d24>] xfs_alloc_lookup_eq+0x14/0x20 [xfs] [ 7753.720000] [<e3a9bbab>] xfs_alloc_fixup_trees+0x17b/0x330 [xfs] [ 7753.720000] [<e3a9c7a1>] xfs_alloc_ag_vextent_near+0x6e1/0x9b0 [xfs] [ 7753.720000] [<e3a9beee>] xfs_alloc_ag_vextent+0xee/0x100 [xfs] [ 7753.730000] [<e3a9e250>] xfs_alloc_vextent+0x330/0x430 [xfs] [ 7753.730000] [<e3aad057>] xfs_bmap_btalloc+0x2e7/0x790 [xfs] [ 7753.730000] [<e3aad51e>] xfs_bmap_alloc+0x1e/0x30 [xfs] [ 7753.740000] [<e3ab085c>] xfs_bmapi+0xccc/0x13a0 [xfs] [ 7753.740000] [<e3abcf14>] xfs_dir2_grow_inode+0xe4/0x3a0 [xfs] [ 7753.740000] [<e3abea34>] xfs_dir2_sf_to_block+0xb4/0x5a0 [xfs] [ 7753.750000] [<e3ac49e8>] xfs_dir2_sf_addname+0x108/0x130 [xfs] [ 7753.750000] [<e3abc8e0>] xfs_dir_createname+0xd0/0x110 [xfs] [ 7753.750000] [<e3ae2013>] xfs_rename+0x2d3/0x8c0 [xfs] [ 7753.760000] [<e3af80ba>] xfs_vn_rename+0x3a/0x90 [xfs] [ 7753.760000] [<c0166ec6>] vfs_rename_other+0xb6/0x100 [ 7753.760000] [<c016701d>] vfs_rename+0x10d/0x240 [ 7753.770000] [<c01672c1>] do_rename+0x171/0x1a0 [ 7753.770000] [<c016735e>] sys_renameat+0x6e/0x80 [ 7753.770000] [<c0167399>] sys_rename+0x29/0x30 [ 7753.780000] [<c0103dd6>] sysenter_past_esp+0x5f/0x85 [ 7753.780000] ======================= [ 7753.790000] BUG: scheduling while atomic: emerge/0xc0117331/4970 [ 7753.790000] [<c0104afa>] show_trace_log_lvl+0x1a/0x30 [ 7753.790000] [<c0104b22>] show_trace+0x12/0x20 [ 7753.790000] [<c0104c45>] dump_stack+0x15/0x20 [ 7753.790000] [<c03066f7>] schedule+0x497/0x5c0 [ 7753.790000] [<c013d0ba>] refrigerator+0x4a/0x70 [ 7753.790000] [<c012557c>] get_signal_to_deliver+0x23c/0x250 [ 7753.800000] [<c0103bcb>] do_signal+0x5b/0x170 [ 7753.800000] [<c0103d1a>] do_notify_resume+0x3a/0x3c [ 7753.800000] [<c0103eda>] work_notifysig+0x13/0x19 [ 7753.800000] ======================= [ 7753.800000] NOHZ: local_softirq_pending 2a The setup is a follows: * nbd client Gentoo Linux, gcc 3.4.6, glibc 2.5, nbd 2.9.3 On /dev/nbd0 I link an image of an XFS / filesystem (also Gentoo) and chroot into the mountpoint to run emerge --metadata (update of metadata with lots of I/O, reading and writing portage tree metadata) * nbd server Gentoo Linux, gcc 4.1.2, glibc 2.5, nbd 2.9.3 It exports the file-based image of that root filesystem Both machines are on the same subnet, after the BUG() messages in kernel log the emerge process is blocked in D state, no traffic pending on nbd connection on either side according to netstat. Filesystem is still working for other processes (e.g. listing files) and show network traffic. Please verify against a modern kernel as XFS went on a stack diet Seems to survive a run with: Server side: nbd-client-2.9.11 Client side: ndb-client-2.9.11 (using 1k blocks), linux-2.6.29 (CONFIG_4KSTACKS=y), XFS filesystem, [ 2106.706056] nbd: registered device at major 43 [ 2144.206120] Filesystem "nbd0": Disabling barriers, trial barrier write failed [ 2144.207097] XFS mounting filesystem nbd0 [ 2144.323944] Starting XFS recovery on filesystem: nbd0 (logdev: internal) [ 2144.392744] Ending XFS recovery on filesystem: nbd0 (logdev: internal) [ 4788.331401] ldconfig used greatest stack depth: 856 bytes left I will do some more testing during week-end, doing some more I/O load. Until now it survived two emerge --metadata runs and compiling a few packages. |