Bug 63571

Summary:	Frequent hard lock ups with btrfs-transaction and btrfs-flush_del eating CPU
Product:	File System	Reporter:	Sean Clarke (sean.clarke)
Component:	btrfs	Assignee:	Josef Bacik (josef)
Status:	RESOLVED CODE_FIX
Severity:	high	CC:	dsterba, juergen.richtsfeld
Priority:	P1
Hardware:	x86-64
OS:	Linux
Kernel Version:	3.11 3.12rc	Subsystem:
Regression:	Yes	Bisected commit-id:
Attachments:	Error filled syslog Latest kernel errors

Description Sean Clarke 2013-10-23 21:24:13 UTC

I have an Intel Core i7 based fileserver with 18TB BTRFS in a 6x
3TB RAID 1+0 configuration. The system was working fine running Ubuntu
13.04 (kernel 3.11.0-12-generic). The system was upgraded to Ubuntu
13.10 (kernel 3.11) and began to lock up daily, sometimes every couple
of hours. Previously it never crashed and was only taken down for
maintenance so a bug was files on launchpad
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1237794).

I have updated the system with mainline kernels "3.12.0-999-generic
#201310090426" and "3.12.0-999-generic #201310170405" and still the
lockups occur.

Absolutely nothing is logged in the syslog, it is a complete freeze. I
could get the system to hang by attempting to delete a 95GB tar file
from a client (over NFS), it would lock up every time. At other times
it just locks up (overnight for example).

I have had a top session running a couple of times when it locks up
and btrfs-transaction and btrfs-flush_del are using 3 to 5 cores at 50
to 100% (see top extract below).

top - 12:06:37 up 1:51, 1 user, load average: 3.33, 0.87, 0.33
Tasks: 226 total, 5 running, 221 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 48.5 sy, 0.0 ni, 50.9 id, 0.1 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem: 12295372 total, 12138296 used, 157076 free, 312 buffers
KiB Swap: 7831536 total, 56 used, 7831480 free, 24644 cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1005  root   20  0   0       0      0      R   100.1   0.0 0:50.16 btrfs-transacti
 2462  root   20  0   0       0      0      R    75.8   0.0 0:06.66  btrfs-flush_del
 2459  root   20  0   0       0      0      S    72.5   0.0 0:10.18  btrfs-flush_del
 2463  root   20  0   0       0      0      S    70.5   0.0 0:14.41  btrfs-flush_del
 2457  root   20  0   0       0      0      R    38.6   0.0 0:28.35  btrfs-flush_del
 2458  root   20  0   0       0      0      S    17.0   0.0 0:39.31  btrfs-flush_del
 1959  root   20  0   0       0      0      R     9.6   0.0 0:31.68  btrfs-flush_del
  100  root    20  0   0       0      0      S    7.6    0.0 0:00.31 kswap

Comment 1 Sean Clarke 2013-11-02 09:08:44 UTC

Another load of errors in the syslog this morning:

Nov  2 06:52:02 enterprise kernel: [28154.404374] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:1003]
Nov  2 06:52:02 enterprise kernel: [28154.404406] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:02 enterprise kernel: [28154.404455] CPU: 1 PID: 1003 Comm: btrfs-transacti Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:02 enterprise kernel: [28154.404457] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:02 enterprise kernel: [28154.404459] task: ffff8800ba0f4710 ti: ffff8800ba170000 task.ti: ffff8800ba170000
Nov  2 06:52:02 enterprise kernel: [28154.404460] RIP: 0010:[<ffffffff81746772>]  [<ffffffff81746772>] _raw_spin_lock+0x32/0x50
Nov  2 06:52:02 enterprise kernel: [28154.404466] RSP: 0018:ffff8800ba171d38  EFLAGS: 00000293
Nov  2 06:52:02 enterprise kernel: [28154.404467] RAX: 0000000000002445 RBX: ffffffff810135ca RCX: 000000000000000c
Nov  2 06:52:02 enterprise kernel: [28154.404468] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880039420ec8
Nov  2 06:52:02 enterprise kernel: [28154.404470] RBP: ffff8800ba171d38 R08: ffffe8ffffc209d0 R09: 0000000000000000
Nov  2 06:52:02 enterprise kernel: [28154.404471] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88032e7a6600
Nov  2 06:52:02 enterprise kernel: [28154.404472] R13: ffff8800ba171cb8 R14: 0000000000000020 R15: 0000000000000000
Nov  2 06:52:02 enterprise kernel: [28154.404473] FS:  0000000000000000(0000) GS:ffff88033fc20000(0000) knlGS:0000000000000000
Nov  2 06:52:02 enterprise kernel: [28154.404475] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:02 enterprise kernel: [28154.404476] CR2: 00000000016bedb8 CR3: 0000000001c0d000 CR4: 00000000000007e0
Nov  2 06:52:02 enterprise kernel: [28154.404477] Stack:
Nov  2 06:52:02 enterprise kernel: [28154.404478]  ffff8800ba171d58 ffffffff811d63d7 ffff880039420d98 0000000000000000
Nov  2 06:52:02 enterprise kernel: [28154.404481]  ffff8800ba171dd8 ffffffffa017e149 ffff8800ba171dc8 ffff88032cdea7d0
Nov  2 06:52:02 enterprise kernel: [28154.404483]  ffff88032cdea720 ffff8800ba171d88 ffff880039420d98 ffff880039420558
Nov  2 06:52:02 enterprise kernel: [28154.404485] Call Trace:
Nov  2 06:52:02 enterprise kernel: [28154.404491]  [<ffffffff811d63d7>] igrab+0x27/0x70
Nov  2 06:52:02 enterprise kernel: [28154.404508]  [<ffffffffa017e149>] btrfs_run_ordered_operations+0xe9/0x2c0 [btrfs]
Nov  2 06:52:02 enterprise kernel: [28154.404519]  [<ffffffffa0165042>] btrfs_commit_transaction+0x32/0xa20 [btrfs]
Nov  2 06:52:02 enterprise kernel: [28154.404523]  [<ffffffff81074ef0>] ? call_timer_fn+0x160/0x160
Nov  2 06:52:02 enterprise kernel: [28154.404533]  [<ffffffffa015efd5>] transaction_kthread+0x1c5/0x230 [btrfs]
Nov  2 06:52:02 enterprise kernel: [28154.404543]  [<ffffffffa015ee10>] ? btrfs_destroy_delayed_refs+0x350/0x350 [btrfs]
Nov  2 06:52:02 enterprise kernel: [28154.404546]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:02 enterprise kernel: [28154.404549]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:02 enterprise kernel: [28154.404552]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:02 enterprise kernel: [28154.404554]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:02 enterprise kernel: [28154.404555] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Nov  2 06:52:10 enterprise kernel: [28162.429570] BUG: soft lockup - CPU#3 stuck for 23s! [btrfs-delalloc-:2161]
Nov  2 06:52:10 enterprise kernel: [28162.429616] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:10 enterprise kernel: [28162.429674] CPU: 3 PID: 2161 Comm: btrfs-delalloc- Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:10 enterprise kernel: [28162.429676] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:10 enterprise kernel: [28162.429678] task: ffff88032bca2f60 ti: ffff88026ecb0000 task.ti: ffff88026ecb0000
Nov  2 06:52:10 enterprise kernel: [28162.429680] RIP: 0010:[<ffffffff81746772>]  [<ffffffff81746772>] _raw_spin_lock+0x32/0x50
Nov  2 06:52:10 enterprise kernel: [28162.429687] RSP: 0018:ffff88026ecb1bb8  EFLAGS: 00000202
Nov  2 06:52:10 enterprise kernel: [28162.429688] RAX: 000000000000449a RBX: ffff88026ecb1ca7 RCX: 0000000000001780
Nov  2 06:52:10 enterprise kernel: [28162.429690] RDX: 0000000000001782 RSI: 0000000000001782 RDI: ffff88032a2b28e4
Nov  2 06:52:10 enterprise kernel: [28162.429692] RBP: ffff88026ecb1bb8 R08: 0000000000001000 R09: ffff8802de229d60
Nov  2 06:52:10 enterprise kernel: [28162.429693] R10: ffffffffa01a6ab3 R11: 0000000000000000 R12: 0000000200004000
Nov  2 06:52:10 enterprise kernel: [28162.429695] R13: ffff88032c9d2500 R14: 000000c9f08da000 R15: ffff88026ecb1b48
Nov  2 06:52:10 enterprise kernel: [28162.429697] FS:  0000000000000000(0000) GS:ffff88033fc60000(0000) knlGS:0000000000000000
Nov  2 06:52:10 enterprise kernel: [28162.429699] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:10 enterprise kernel: [28162.429700] CR2: 00000000015d2918 CR3: 0000000001c0d000 CR4: 00000000000007e0
Nov  2 06:52:10 enterprise kernel: [28162.429702] Stack:
Nov  2 06:52:10 enterprise kernel: [28162.429703]  ffff88026ecb1c18 ffffffffa017d0c3 0000000000001000 000000c9f08da000
Nov  2 06:52:10 enterprise kernel: [28162.429707]  ffff88026ecb1be8 ffff88032d26b290 ffff88026ecb1c38 0000000000000000
Nov  2 06:52:10 enterprise kernel: [28162.429710]  0000000000004000 0000000000001000 ffff88032d26b1b8 ffff88032d26b360
Nov  2 06:52:10 enterprise kernel: [28162.429713] Call Trace:
Nov  2 06:52:10 enterprise kernel: [28162.429735]  [<ffffffffa017d0c3>] __btrfs_add_ordered_extent+0x333/0x3c0 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429749]  [<ffffffffa017d171>] btrfs_add_ordered_extent+0x21/0x30 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429763]  [<ffffffffa016d9c6>] cow_file_range+0x226/0x3d0 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429776]  [<ffffffffa016e87a>] submit_compressed_extents+0x1ea/0x4c0 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429789]  [<ffffffffa016efcb>] ? compress_file_range+0x3eb/0x5d0 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429802]  [<ffffffffa016ebd7>] async_cow_submit+0x87/0x90 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429817]  [<ffffffffa0192a43>] run_ordered_completions+0x83/0x110 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429831]  [<ffffffffa01937e3>] worker_loop+0xb3/0x330 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429845]  [<ffffffffa0193730>] ? check_pending_worker_creates.isra.1+0xe0/0xe0 [btrfs]
Nov  2 06:52:10 enterprise kernel: [28162.429849]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:10 enterprise kernel: [28162.429853]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:10 enterprise kernel: [28162.429856]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:10 enterprise kernel: [28162.429859]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:10 enterprise kernel: [28162.429860] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Nov  2 06:52:30 enterprise kernel: [28182.394590] BUG: soft lockup - CPU#0 stuck for 22s! [nfsd:1547]
Nov  2 06:52:30 enterprise kernel: [28182.394592] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs-transacti:1003]
Nov  2 06:52:30 enterprise kernel: [28182.394625] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:30 enterprise kernel: [28182.394630] CPU: 1 PID: 1003 Comm: btrfs-transacti Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:30 enterprise kernel: [28182.394631] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:30 enterprise kernel: [28182.394632] task: ffff8800ba0f4710 ti: ffff8800ba170000 task.ti: ffff8800ba170000
Nov  2 06:52:30 enterprise kernel: [28182.394637] RIP: 0010:[<ffffffff81746772>]  [<ffffffff81746772>] _raw_spin_lock+0x32/0x50
Nov  2 06:52:30 enterprise kernel: [28182.394638] RSP: 0018:ffff8800ba171d38  EFLAGS: 00000293
Nov  2 06:52:30 enterprise kernel: [28182.394639] RAX: 0000000000003910 RBX: ffffffff810135ca RCX: 000000000000000c
Nov  2 06:52:30 enterprise kernel: [28182.394639] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880039420ec8
Nov  2 06:52:30 enterprise kernel: [28182.394640] RBP: ffff8800ba171d38 R08: ffffe8ffffc209d0 R09: 0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394642] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88032e7a6600
Nov  2 06:52:30 enterprise kernel: [28182.394643] R13: ffff8800ba171cb8 R14: 0000000000000020 R15: 0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394645] FS:  0000000000000000(0000) GS:ffff88033fc20000(0000) knlGS:0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394646] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:30 enterprise kernel: [28182.394648] CR2: 00000000016bedb8 CR3: 0000000001c0d000 CR4: 00000000000007e0
Nov  2 06:52:30 enterprise kernel: [28182.394649] Stack:
Nov  2 06:52:30 enterprise kernel: [28182.394653]  ffff8800ba171d58 ffffffff811d63d7 ffff880039420d98 0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394657]  ffff8800ba171dd8 ffffffffa017e149 ffff8800ba171dc8 ffff88032cdea7d0
Nov  2 06:52:30 enterprise kernel: [28182.394658]  ffff88032cdea720 ffff8800ba171d88 ffff880039420d98
Nov  2 06:52:30 enterprise kernel: [28182.394660] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:30 enterprise kernel: [28182.394733] CPU: 0 PID: 1547 Comm: nfsd Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:30 enterprise kernel: [28182.394735]  ffff880039420558
Nov  2 06:52:30 enterprise kernel: [28182.394736] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:30 enterprise kernel: [28182.394738] task: ffff88032b61af60 ti: ffff8800ba068000 task.ti: ffff8800ba068000
Nov  2 06:52:30 enterprise kernel: [28182.394742] RIP: 0010:[<ffffffff81746772>]  [<ffffffff81746772>] _raw_spin_lock+0x32/0x50
Nov  2 06:52:30 enterprise kernel: [28182.394743] Call Trace:
Nov  2 06:52:30 enterprise kernel: [28182.394745] RSP: 0018:ffff8800ba069b28  EFLAGS: 00000206
Nov  2 06:52:30 enterprise kernel: [28182.394749]  [<ffffffff811d63d7>] igrab+0x27/0x70
Nov  2 06:52:30 enterprise kernel: [28182.394751] RAX: 0000000000004846 RBX: ffffffffa013dbaa RCX: 0000000000001780
Nov  2 06:52:30 enterprise kernel: [28182.394767]  [<ffffffffa017e149>] btrfs_run_ordered_operations+0xe9/0x2c0 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394770] RDX: 0000000000001786 RSI: 0000000000001786 RDI: ffff88032a2b28e4
Nov  2 06:52:30 enterprise kernel: [28182.394780]  [<ffffffffa0165042>] btrfs_commit_transaction+0x32/0xa20 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394784]  [<ffffffff81074ef0>] ? call_timer_fn+0x160/0x160
Nov  2 06:52:30 enterprise kernel: [28182.394785] RBP: ffff8800ba069b28 R08: 000060fcc0000c20 R09: ffffea0000d9cd80
Nov  2 06:52:30 enterprise kernel: [28182.394787] R10: ffffffffa013dbaa R11: 0000000000000000 R12: 0000000000004072
Nov  2 06:52:30 enterprise kernel: [28182.394788] R13: ffff880329b73000 R14: 000000000000000b R15: 0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394790] FS:  0000000000000000(0000) GS:ffff88033fc00000(0000) knlGS:0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.394791] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:30 enterprise kernel: [28182.394793] CR2: 00000000016c6d38 CR3: 0000000001c0d000 CR4: 00000000000007f0
Nov  2 06:52:30 enterprise kernel: [28182.394794] Stack:
Nov  2 06:52:30 enterprise kernel: [28182.394795]  ffff8800ba069b58 ffffffffa017ee0c
Nov  2 06:52:30 enterprise kernel: [28182.394796]  ffff8800ba069b58 ffff88032d26d880
Nov  2 06:52:30 enterprise kernel: [28182.394809]  [<ffffffffa015efd5>] transaction_kthread+0x1c5/0x230 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394799]  ffff8801766dc480 0000000000000000 ffff8800ba069c18
Nov  2 06:52:30 enterprise kernel: [28182.394811]  ffffffffa01739e5
Nov  2 06:52:30 enterprise kernel: [28182.394823]  [<ffffffffa015ee10>] ? btrfs_destroy_delayed_refs+0x350/0x350 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394813]  000000000000da8a ffffffff81330499 ffff8800ba069ba8 010100ff81336b2b
Nov  2 06:52:30 enterprise kernel: [28182.394824]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:30 enterprise kernel: [28182.394829]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.394832]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.394835]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.394838] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Nov  2 06:52:30 enterprise kernel: [28182.394885] 
Nov  2 06:52:30 enterprise kernel: [28182.394886] Call Trace:
Nov  2 06:52:30 enterprise kernel: [28182.394902]  [<ffffffffa017ee0c>] btrfs_add_ordered_operation+0x5c/0xc0 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394923]  [<ffffffffa01739e5>] btrfs_rename+0x4d5/0x750 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.394932]  [<ffffffff81330499>] ? aa_alloc_task_context+0x39/0x40
Nov  2 06:52:30 enterprise kernel: [28182.394941]  [<ffffffff811c77fd>] vfs_rename_other+0xcd/0x120
Nov  2 06:52:30 enterprise kernel: [28182.394946]  [<ffffffff811caf9b>] vfs_rename+0xbb/0x250
Nov  2 06:52:30 enterprise kernel: [28182.394958]  [<ffffffffa0612264>] nfsd_rename+0x3c4/0x410 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.394967]  [<ffffffffa0619de1>] nfsd3_proc_rename+0xd1/0x1a0 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.394976]  [<ffffffffa061735f>] ? nfsd_cache_lookup+0x20f/0x4c0 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.394983]  [<ffffffffa060bd65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.394999]  [<ffffffffa0595f55>] svc_process_common+0x345/0x680 [sunrpc]
Nov  2 06:52:30 enterprise kernel: [28182.395012]  [<ffffffffa05965e3>] svc_process+0x103/0x160 [sunrpc]
Nov  2 06:52:30 enterprise kernel: [28182.395019]  [<ffffffffa060b8cf>] nfsd+0xbf/0x130 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.395026]  [<ffffffffa060b810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.395031]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:30 enterprise kernel: [28182.395039]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.395046]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.395049]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.395050] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Nov  2 06:52:30 enterprise kernel: [28182.410583] BUG: soft lockup - CPU#2 stuck for 22s! [nfsd:1552]
Nov  2 06:52:30 enterprise kernel: [28182.410605] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:30 enterprise kernel: [28182.410643] CPU: 2 PID: 1552 Comm: nfsd Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:30 enterprise kernel: [28182.410644] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:30 enterprise kernel: [28182.410646] task: ffff8800369caf60 ti: ffff88032c1c6000 task.ti: ffff88032c1c6000
Nov  2 06:52:30 enterprise kernel: [28182.410647] RIP: 0010:[<ffffffff81746770>]  [<ffffffff81746770>] _raw_spin_lock+0x30/0x50
Nov  2 06:52:30 enterprise kernel: [28182.410650] RSP: 0018:ffff88032c1c7b28  EFLAGS: 00000202
Nov  2 06:52:30 enterprise kernel: [28182.410652] RAX: 00000000000031fd RBX: ffffffffa013dbaa RCX: 0000000000001780
Nov  2 06:52:30 enterprise kernel: [28182.410653] RDX: 0000000000001784 RSI: 0000000000001784 RDI: ffff88032a2b28e4
Nov  2 06:52:30 enterprise kernel: [28182.410654] RBP: ffff88032c1c7b28 R08: 000060fcc0000c20 R09: ffffea0000d971c0
Nov  2 06:52:30 enterprise kernel: [28182.410655] R10: ffffffffa013dbaa R11: 0000000000000000 R12: 0000000000004072
Nov  2 06:52:30 enterprise kernel: [28182.410656] R13: ffff880329b70800 R14: 000000000000000b R15: 0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.410658] FS:  0000000000000000(0000) GS:ffff88033fc40000(0000) knlGS:0000000000000000
Nov  2 06:52:30 enterprise kernel: [28182.410659] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:30 enterprise kernel: [28182.410661] CR2: 00007f991210e7e2 CR3: 0000000001c0d000 CR4: 00000000000007e0
Nov  2 06:52:30 enterprise kernel: [28182.410662] Stack:
Nov  2 06:52:30 enterprise kernel: [28182.410662]  ffff88032c1c7b58 ffffffffa017ee0c ffff88032c1c7b58 ffff88032d276900
Nov  2 06:52:30 enterprise kernel: [28182.410665]  ffff8801766dc240 0000000000000000 ffff88032c1c7c18 ffffffffa01739e5
Nov  2 06:52:30 enterprise kernel: [28182.410667]  000000000000da89 ffffffff81330499 ffff88032c1c7ba8 010100ff81336b2b
Nov  2 06:52:30 enterprise kernel: [28182.410670] Call Trace:
Nov  2 06:52:30 enterprise kernel: [28182.410683]  [<ffffffffa017ee0c>] btrfs_add_ordered_operation+0x5c/0xc0 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.410694]  [<ffffffffa01739e5>] btrfs_rename+0x4d5/0x750 [btrfs]
Nov  2 06:52:30 enterprise kernel: [28182.410697]  [<ffffffff81330499>] ? aa_alloc_task_context+0x39/0x40
Nov  2 06:52:30 enterprise kernel: [28182.410699]  [<ffffffff811c77fd>] vfs_rename_other+0xcd/0x120
Nov  2 06:52:30 enterprise kernel: [28182.410702]  [<ffffffff811caf9b>] vfs_rename+0xbb/0x250
Nov  2 06:52:30 enterprise kernel: [28182.410708]  [<ffffffffa0612264>] nfsd_rename+0x3c4/0x410 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410713]  [<ffffffffa0619de1>] nfsd3_proc_rename+0xd1/0x1a0 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410719]  [<ffffffffa061735f>] ? nfsd_cache_lookup+0x20f/0x4c0 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410723]  [<ffffffffa060bd65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410732]  [<ffffffffa0595f55>] svc_process_common+0x345/0x680 [sunrpc]
Nov  2 06:52:30 enterprise kernel: [28182.410740]  [<ffffffffa05965e3>] svc_process+0x103/0x160 [sunrpc]
Nov  2 06:52:30 enterprise kernel: [28182.410745]  [<ffffffffa060b8cf>] nfsd+0xbf/0x130 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410749]  [<ffffffffa060b810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Nov  2 06:52:30 enterprise kernel: [28182.410751]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:30 enterprise kernel: [28182.410754]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.410756]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.410758]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:30 enterprise kernel: [28182.410759] Code: 55 48 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 <f3> 90 83 e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 
Nov  2 06:52:31 enterprise kernel: [28182.474560] BUG: soft lockup - CPU#7 stuck for 22s! [nfsd:1549]
Nov  2 06:52:31 enterprise kernel: [28182.474596] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek mxm_wmi wmi video ttm snd_hda_intel drm_kms_helper snd_hda_codec drm snd_hwdep psmouse snd_pcm microcode i2c_algo_bit snd_page_alloc serio_raw snd_timer lpc_ich i7core_edac snd soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid firewire_ohci pata_jmicron raid6_pq firewire_core raid1 r8169 usb_storage crc_itu_t ahci raid0 mii pata_marvell libahci multipath linear
Nov  2 06:52:31 enterprise kernel: [28182.474644] CPU: 7 PID: 1549 Comm: nfsd Not tainted 3.12.0-999-generic #201310260405
Nov  2 06:52:31 enterprise kernel: [28182.474646] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Nov  2 06:52:31 enterprise kernel: [28182.474648] task: ffff88032a2faf60 ti: ffff88032d7dc000 task.ti: ffff88032d7dc000
Nov  2 06:52:31 enterprise kernel: [28182.474649] RIP: 0010:[<ffffffff81746772>]  [<ffffffff81746772>] _raw_spin_lock+0x32/0x50
Nov  2 06:52:31 enterprise kernel: [28182.474654] RSP: 0018:ffff88032d7ddb28  EFLAGS: 00000202
Nov  2 06:52:31 enterprise kernel: [28182.474655] RAX: 00000000000020d0 RBX: ffffffffa013dbaa RCX: 0000000000001780
Nov  2 06:52:31 enterprise kernel: [28182.474657] RDX: 0000000000001788 RSI: 0000000000001788 RDI: ffff88032a2b28e4
Nov  2 06:52:31 enterprise kernel: [28182.474658] RBP: ffff88032d7ddb28 R08: 000060fcc0000c20 R09: ffffea000cb67880
Nov  2 06:52:31 enterprise kernel: [28182.474660] R10: ffffffffa013dbaa R11: 0000000000000000 R12: 00000000000064dd
Nov  2 06:52:31 enterprise kernel: [28182.474661] R13: ffff88032ae9f800 R14: 000000000000000b R15: 0000000000000000
Nov  2 06:52:31 enterprise kernel: [28182.474664] FS:  0000000000000000(0000) GS:ffff88033fce0000(0000) knlGS:0000000000000000
Nov  2 06:52:31 enterprise kernel: [28182.474665] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov  2 06:52:31 enterprise kernel: [28182.474667] CR2: 00000000015c0ffc CR3: 0000000001c0d000 CR4: 00000000000007e0
Nov  2 06:52:31 enterprise kernel: [28182.474668] Stack:
Nov  2 06:52:31 enterprise kernel: [28182.474669]  ffff88032d7ddb58 ffffffffa017ee0c ffff88032d7ddb58 ffff88032d276d20
Nov  2 06:52:31 enterprise kernel: [28182.474672]  ffff8801766dc3c0 0000000000000000 ffff88032d7ddc18 ffffffffa01739e5
Nov  2 06:52:31 enterprise kernel: [28182.474675]  000000000004c67b ffffffff81330499 ffff88032d7ddba8 010100ff81336b2b
Nov  2 06:52:31 enterprise kernel: [28182.474678] Call Trace:
Nov  2 06:52:31 enterprise kernel: [28182.474694]  [<ffffffffa017ee0c>] btrfs_add_ordered_operation+0x5c/0xc0 [btrfs]
Nov  2 06:52:31 enterprise kernel: [28182.474707]  [<ffffffffa01739e5>] btrfs_rename+0x4d5/0x750 [btrfs]
Nov  2 06:52:31 enterprise kernel: [28182.474710]  [<ffffffff81330499>] ? aa_alloc_task_context+0x39/0x40
Nov  2 06:52:31 enterprise kernel: [28182.474713]  [<ffffffff811c77fd>] vfs_rename_other+0xcd/0x120
Nov  2 06:52:31 enterprise kernel: [28182.474716]  [<ffffffff811caf9b>] vfs_rename+0xbb/0x250
Nov  2 06:52:31 enterprise kernel: [28182.474723]  [<ffffffffa0612264>] nfsd_rename+0x3c4/0x410 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474730]  [<ffffffffa0619de1>] nfsd3_proc_rename+0xd1/0x1a0 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474737]  [<ffffffffa061735f>] ? nfsd_cache_lookup+0x20f/0x4c0 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474742]  [<ffffffffa060bd65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474752]  [<ffffffffa0595f55>] svc_process_common+0x345/0x680 [sunrpc]
Nov  2 06:52:31 enterprise kernel: [28182.474763]  [<ffffffffa05965e3>] svc_process+0x103/0x160 [sunrpc]
Nov  2 06:52:31 enterprise kernel: [28182.474768]  [<ffffffffa060b8cf>] nfsd+0xbf/0x130 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474773]  [<ffffffffa060b810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Nov  2 06:52:31 enterprise kernel: [28182.474776]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Nov  2 06:52:31 enterprise kernel: [28182.474779]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:31 enterprise kernel: [28182.474782]  [<ffffffff8174f7bc>] ret_from_fork+0x7c/0xb0
Nov  2 06:52:31 enterprise kernel: [28182.474785]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Nov  2 06:52:31 enterprise kernel: [28182.474786] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Nov  2 06:52:37 enterprise kernel: [28188.816343] INFO: rcu_sched self-detected stall on CPU { 1}  (t=15000 jiffies g=215146 c=215145 q=0)
Nov  2 06:52:37 enterprise kernel: [28188.816403] sending NMI to all CPUs:
Nov  2 06:52:37 enterprise kernel: [28188.816408] NMI backtrace for cpu 0

Comment 2 Sean Clarke 2013-11-02 09:11:18 UTC

Created attachment 113071 [details]
Error filled syslog

Comment 3 juergen.richtsfeld 2013-11-15 08:33:02 UTC

I think this affects me too. Using Ubuntu 13.10 with the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-trusty/ , sometimes file IO is really slow, and I can see the process btrfs-flush_del using all cpu (one core). So I watched writing files taking minutes, sometimes making the whole system unusable until it finishes (happened just minutes ago when writing a ~ 750 MB file which took ~3 minutes on a SSD)

mount options for my /home are
defaults,compress=no,subvol=@home

Comment 4 Sean Clarke 2013-12-05 23:23:59 UTC

Another log dump - this time hopefully it will contain some helpful info:

Dec  1 12:22:05 enterprise kernel: [47431.860169] BUG: soft lockup - CPU#3 stuck for 22s! [btrfs-transacti:982]
Dec  1 12:22:05 enterprise kernel: [47431.860200] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek snd_hda_intel mxm_wmi wmi video ttm microcode drm_kms_helper snd_hda_codec psmouse serio_raw drm snd_hwdep snd_pcm i2c_algo_bit snd_page_alloc lpc_ich snd_timer snd i7core_edac soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq firewire_ohci raid1 r8169 firewire_core usbhid pata_jmicron raid0 crc_itu_t mii ahci hid pata_marvell libahci multipath linear
Dec  1 12:22:05 enterprise kernel: [47431.860247] CPU: 3 PID: 982 Comm: btrfs-transacti Not tainted 3.12.0-031200-generic #201311031935
Dec  1 12:22:05 enterprise kernel: [47431.860249] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Dec  1 12:22:05 enterprise kernel: [47431.860250] task: ffff88032bf32f60 ti: ffff88032a566000 task.ti: ffff88032a566000
Dec  1 12:22:05 enterprise kernel: [47431.860252] RIP: 0010:[<ffffffff811d6220>]  [<ffffffff811d6220>] igrab+0x50/0x70
Dec  1 12:22:05 enterprise kernel: [47431.860258] RSP: 0018:ffff88032a567d48  EFLAGS: 00000202
Dec  1 12:22:05 enterprise kernel: [47431.860260] RAX: 00000000487c487c RBX: 0000000000000000 RCX: ffff88032d2edb50
Dec  1 12:22:05 enterprise kernel: [47431.860261] RDX: 000000000000487c RSI: 00000000487c487a RDI: ffff88000fb6c048
Dec  1 12:22:05 enterprise kernel: [47431.860262] RBP: ffff88032a567d58 R08: ffffe8ffffc609c0 R09: 0000000000000000
Dec  1 12:22:05 enterprise kernel: [47431.860263] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810135ca
Dec  1 12:22:05 enterprise kernel: [47431.860264] R13: ffff88032a567d38 R14: ffff88032e7a6a00 R15: ffff88032a567cb8
Dec  1 12:22:05 enterprise kernel: [47431.860266] FS:  0000000000000000(0000) GS:ffff88033fc60000(0000) knlGS:0000000000000000
Dec  1 12:22:05 enterprise kernel: [47431.860267] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  1 12:22:05 enterprise kernel: [47431.860268] CR2: 00007f4c22cb3000 CR3: 0000000001c0d000 CR4: 00000000000007e0
Dec  1 12:22:05 enterprise kernel: [47431.860269] Stack:
Dec  1 12:22:05 enterprise kernel: [47431.860270]  ffff88000fb6bf18 0000000000000000 ffff88032a567dd8 ffffffffa0177149
Dec  1 12:22:05 enterprise kernel: [47431.860273]  ffff88032a567dc8 ffff88032d2edc00 ffff88032d2edb50 ffff88032a567d88
Dec  1 12:22:05 enterprise kernel: [47431.860275]  ffff88000fb6bf18 ffff88028ab399f8 ffff88032a567d98 ffff88032a567d98
Dec  1 12:22:05 enterprise kernel: [47431.860277] Call Trace:
Dec  1 12:22:05 enterprise kernel: [47431.860295]  [<ffffffffa0177149>] btrfs_run_ordered_operations+0xe9/0x2c0 [btrfs]
Dec  1 12:22:05 enterprise kernel: [47431.860306]  [<ffffffffa015e042>] btrfs_commit_transaction+0x32/0xa20 [btrfs]
Dec  1 12:22:05 enterprise kernel: [47431.860310]  [<ffffffff81074ef0>] ? call_timer_fn+0x160/0x160
Dec  1 12:22:05 enterprise kernel: [47431.860320]  [<ffffffffa0157fd5>] transaction_kthread+0x1c5/0x230 [btrfs]
Dec  1 12:22:05 enterprise kernel: [47431.860330]  [<ffffffffa0157e10>] ? btrfs_destroy_delayed_refs+0x350/0x350 [btrfs]
Dec  1 12:22:05 enterprise kernel: [47431.860334]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Dec  1 12:22:05 enterprise kernel: [47431.860336]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:05 enterprise kernel: [47431.860340]  [<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Dec  1 12:22:05 enterprise kernel: [47431.860342]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:05 enterprise kernel: [47431.860343] Code: 57 00 f6 83 a0 00 00 00 30 75 20 f0 ff 83 30 01 00 00 4c 89 e7 e8 61 04 57 00 48 89 d8 4c 8b 65 f8 48 8b 5d f0 c9 c3 0f 1f 40 00 <4c> 89 e7 31 db e8 46 04 57 00 48 89 d8 4c 8b 65 f8 48 8b 5d f0 
Dec  1 12:22:09 enterprise kernel: [47435.874771] BUG: soft lockup - CPU#4 stuck for 23s! [btrfs-delalloc-:3072]
Dec  1 12:22:09 enterprise kernel: [47435.874813] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek snd_hda_intel mxm_wmi wmi video ttm microcode drm_kms_helper snd_hda_codec psmouse serio_raw drm snd_hwdep snd_pcm i2c_algo_bit snd_page_alloc lpc_ich snd_timer snd i7core_edac soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq firewire_ohci raid1 r8169 firewire_core usbhid pata_jmicron raid0 crc_itu_t mii ahci hid pata_marvell libahci multipath linear
Dec  1 12:22:09 enterprise kernel: [47435.874863] CPU: 4 PID: 3072 Comm: btrfs-delalloc- Not tainted 3.12.0-031200-generic #201311031935
Dec  1 12:22:09 enterprise kernel: [47435.874865] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Dec  1 12:22:09 enterprise kernel: [47435.874867] task: ffff880329c6af60 ti: ffff8802f4442000 task.ti: ffff8802f4442000
Dec  1 12:22:09 enterprise kernel: [47435.874869] RIP: 0010:[<ffffffff817466d2>]  [<ffffffff817466d2>] _raw_spin_lock+0x32/0x50
Dec  1 12:22:09 enterprise kernel: [47435.874874] RSP: 0018:ffff8802f4443bb8  EFLAGS: 00000202
Dec  1 12:22:09 enterprise kernel: [47435.874876] RAX: 0000000000003fe0 RBX: ffff8802f4443ca7 RCX: 00000000000059e2
Dec  1 12:22:09 enterprise kernel: [47435.874877] RDX: 00000000000059e4 RSI: 00000000000059e4 RDI: ffff88003604a8e4
Dec  1 12:22:09 enterprise kernel: [47435.874879] RBP: ffff8802f4443bb8 R08: 0000000000001000 R09: ffff8800367b7658
Dec  1 12:22:09 enterprise kernel: [47435.874880] R10: ffffffffa017438e R11: 0000000000000000 R12: 0000000200000000
Dec  1 12:22:09 enterprise kernel: [47435.874882] R13: ffff880329e68700 R14: 000000e3728a8000 R15: ffff8802f4443b48
Dec  1 12:22:09 enterprise kernel: [47435.874884] FS:  0000000000000000(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  1 12:22:09 enterprise kernel: [47435.874886] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  1 12:22:09 enterprise kernel: [47435.874887] CR2: 00007fff991f6608 CR3: 0000000001c0d000 CR4: 00000000000007e0
Dec  1 12:22:09 enterprise kernel: [47435.874888] Stack:
Dec  1 12:22:09 enterprise kernel: [47435.874890]  ffff8802f4443c18 ffffffffa01760c3 0000000000001000 000000e3728a8000
Dec  1 12:22:09 enterprise kernel: [47435.874893]  ffff8802f4443be8 ffff88032cc54310 ffff8802f4443c38 0000000000000000
Dec  1 12:22:09 enterprise kernel: [47435.874896]  000000000039d000 0000000000001000 ffff88032cc54238 ffff88032cc543e0
Dec  1 12:22:09 enterprise kernel: [47435.874899] Call Trace:
Dec  1 12:22:09 enterprise kernel: [47435.874915]  [<ffffffffa01760c3>] __btrfs_add_ordered_extent+0x333/0x3c0 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874929]  [<ffffffffa0176171>] btrfs_add_ordered_extent+0x21/0x30 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874942]  [<ffffffffa01669c6>] cow_file_range+0x226/0x3d0 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874955]  [<ffffffffa016787a>] submit_compressed_extents+0x1ea/0x4c0 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874968]  [<ffffffffa0167fcb>] ? compress_file_range+0x3eb/0x5d0 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874982]  [<ffffffffa0167bd7>] async_cow_submit+0x87/0x90 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.874996]  [<ffffffffa018ba43>] run_ordered_completions+0x83/0x110 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.875010]  [<ffffffffa018c7e3>] worker_loop+0xb3/0x330 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.875024]  [<ffffffffa018c730>] ? check_pending_worker_creates.isra.1+0xe0/0xe0 [btrfs]
Dec  1 12:22:09 enterprise kernel: [47435.875027]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Dec  1 12:22:09 enterprise kernel: [47435.875031]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:09 enterprise kernel: [47435.875033]  [<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Dec  1 12:22:09 enterprise kernel: [47435.875036]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:09 enterprise kernel: [47435.875038] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Dec  1 12:22:17 enterprise kernel: [47443.832001] BUG: soft lockup - CPU#0 stuck for 22s! [nfsd:1519]
Dec  1 12:22:17 enterprise kernel: [47443.832037] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek snd_hda_intel mxm_wmi wmi video ttm microcode drm_kms_helper snd_hda_codec psmouse serio_raw drm snd_hwdep snd_pcm i2c_algo_bit snd_page_alloc lpc_ich snd_timer snd i7core_edac soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq firewire_ohci raid1 r8169 firewire_core usbhid pata_jmicron raid0 crc_itu_t mii ahci hid pata_marvell libahci multipath linear
Dec  1 12:22:17 enterprise kernel: [47443.832084] CPU: 0 PID: 1519 Comm: nfsd Not tainted 3.12.0-031200-generic #201311031935
Dec  1 12:22:17 enterprise kernel: [47443.832086] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Dec  1 12:22:17 enterprise kernel: [47443.832088] task: ffff88032bf0dec0 ti: ffff88032cb24000 task.ti: ffff88032cb24000
Dec  1 12:22:17 enterprise kernel: [47443.832089] RIP: 0010:[<ffffffff817466d2>]  [<ffffffff817466d2>] _raw_spin_lock+0x32/0x50
Dec  1 12:22:17 enterprise kernel: [47443.832093] RSP: 0018:ffff88032cb25b28  EFLAGS: 00000202
Dec  1 12:22:17 enterprise kernel: [47443.832095] RAX: 000000000000271d RBX: ffffffffa0136baa RCX: 00000000000059e2
Dec  1 12:22:17 enterprise kernel: [47443.832096] RDX: 00000000000059e6 RSI: 00000000000059e6 RDI: ffff88003604a8e4
Dec  1 12:22:17 enterprise kernel: [47443.832098] RBP: ffff88032cb25b28 R08: 000060fcc0000c00 R09: ffffea000cb44340
Dec  1 12:22:17 enterprise kernel: [47443.832100] R10: ffffffffa0136baa R11: 0000000000000000 R12: 0000000000593f69
Dec  1 12:22:17 enterprise kernel: [47443.832101] R13: ffff880329da9000 R14: 000000000000000b R15: 0000000000000000
Dec  1 12:22:17 enterprise kernel: [47443.832103] FS:  0000000000000000(0000) GS:ffff88033fc00000(0000) knlGS:0000000000000000
Dec  1 12:22:17 enterprise kernel: [47443.832105] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  1 12:22:17 enterprise kernel: [47443.832106] CR2: 00007fcee1962000 CR3: 0000000001c0d000 CR4: 00000000000007f0
Dec  1 12:22:17 enterprise kernel: [47443.832108] Stack:
Dec  1 12:22:17 enterprise kernel: [47443.832109]  ffff88032cb25b58 ffffffffa0177e0c ffff88032cb25b58 ffff88032d387da0
Dec  1 12:22:17 enterprise kernel: [47443.832112]  ffff880284dd6540 0000000000000000 ffff88032cb25c18 ffffffffa016c9e5
Dec  1 12:22:17 enterprise kernel: [47443.832115]  00000000000070fe ffffffff81330289 ffff88032cb25ba8 010100ff8133691b
Dec  1 12:22:17 enterprise kernel: [47443.832118] Call Trace:
Dec  1 12:22:17 enterprise kernel: [47443.832133]  [<ffffffffa0177e0c>] btrfs_add_ordered_operation+0x5c/0xc0 [btrfs]
Dec  1 12:22:17 enterprise kernel: [47443.832146]  [<ffffffffa016c9e5>] btrfs_rename+0x4d5/0x750 [btrfs]
Dec  1 12:22:17 enterprise kernel: [47443.832150]  [<ffffffff81330289>] ? aa_alloc_task_context+0x39/0x40
Dec  1 12:22:17 enterprise kernel: [47443.832154]  [<ffffffff811c784d>] vfs_rename_other+0xcd/0x120
Dec  1 12:22:17 enterprise kernel: [47443.832157]  [<ffffffff811cafeb>] vfs_rename+0xbb/0x250
Dec  1 12:22:17 enterprise kernel: [47443.832167]  [<ffffffffa0608264>] nfsd_rename+0x3c4/0x410 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832174]  [<ffffffffa060fde1>] nfsd3_proc_rename+0xd1/0x1a0 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832180]  [<ffffffffa060d35f>] ? nfsd_cache_lookup+0x20f/0x4c0 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832186]  [<ffffffffa0601d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832199]  [<ffffffffa058bf55>] svc_process_common+0x345/0x680 [sunrpc]
Dec  1 12:22:17 enterprise kernel: [47443.832209]  [<ffffffffa058c5e3>] svc_process+0x103/0x160 [sunrpc]
Dec  1 12:22:17 enterprise kernel: [47443.832215]  [<ffffffffa06018cf>] nfsd+0xbf/0x130 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832220]  [<ffffffffa0601810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Dec  1 12:22:17 enterprise kernel: [47443.832223]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Dec  1 12:22:17 enterprise kernel: [47443.832226]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:17 enterprise kernel: [47443.832229]  [<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Dec  1 12:22:17 enterprise kernel: [47443.832231]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:17 enterprise kernel: [47443.832233] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66 
Dec  1 12:22:33 enterprise kernel: [47459.850424] BUG: soft lockup - CPU#3 stuck for 22s! [btrfs-transacti:982]
Dec  1 12:22:33 enterprise kernel: [47459.850449] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek snd_hda_intel mxm_wmi wmi video ttm microcode drm_kms_helper snd_hda_codec psmouse serio_raw drm snd_hwdep snd_pcm i2c_algo_bit snd_page_alloc lpc_ich snd_timer snd i7core_edac soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq firewire_ohci raid1 r8169 firewire_core usbhid pata_jmicron raid0 crc_itu_t mii ahci hid pata_marvell libahci multipath linear
Dec  1 12:22:33 enterprise kernel: [47459.850486] CPU: 3 PID: 982 Comm: btrfs-transacti Not tainted 3.12.0-031200-generic #201311031935
Dec  1 12:22:33 enterprise kernel: [47459.850487] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Dec  1 12:22:33 enterprise kernel: [47459.850488] task: ffff88032bf32f60 ti: ffff88032a566000 task.ti: ffff88032a566000
Dec  1 12:22:33 enterprise kernel: [47459.850489] RIP: 0010:[<ffffffff8174668f>]  [<ffffffff8174668f>] _raw_spin_unlock+0x1f/0x30
Dec  1 12:22:33 enterprise kernel: [47459.850493] RSP: 0018:ffff88032a567d38  EFLAGS: 00000282
Dec  1 12:22:33 enterprise kernel: [47459.850494] RAX: 0000000099069906 RBX: ffffffff810135ca RCX: ffff88032d2edb50
Dec  1 12:22:33 enterprise kernel: [47459.850495] RDX: 0000000000009906 RSI: 0000000099089906 RDI: ffff88000fb6c048
Dec  1 12:22:33 enterprise kernel: [47459.850496] RBP: ffff88032a567d38 R08: ffffe8ffffc609c0 R09: 0000000000000000
Dec  1 12:22:33 enterprise kernel: [47459.850498] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88032e7a6a00
Dec  1 12:22:33 enterprise kernel: [47459.850499] R13: ffff88032a567cb8 R14: 0000000000000020 R15: 0000000000000000
Dec  1 12:22:33 enterprise kernel: [47459.850500] FS:  0000000000000000(0000) GS:ffff88033fc60000(0000) knlGS:0000000000000000
Dec  1 12:22:33 enterprise kernel: [47459.850501] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  1 12:22:33 enterprise kernel: [47459.850503] CR2: 00007f4c22cb3000 CR3: 0000000001c0d000 CR4: 00000000000007e0
Dec  1 12:22:33 enterprise kernel: [47459.850504] Stack:
Dec  1 12:22:33 enterprise kernel: [47459.850505]  ffff88032a567d58 ffffffff811d622a ffff88000fb6bf18 0000000000000000
Dec  1 12:22:33 enterprise kernel: [47459.850507]  ffff88032a567dd8 ffffffffa0177149 ffff88032a567dc8 ffff88032d2edc00
Dec  1 12:22:33 enterprise kernel: [47459.850509]  ffff88032d2edb50 ffff88032a567d88 ffff88000fb6bf18 ffff88028ab399f8
Dec  1 12:22:33 enterprise kernel: [47459.850511] Call Trace:
Dec  1 12:22:33 enterprise kernel: [47459.850514]  [<ffffffff811d622a>] igrab+0x5a/0x70
Dec  1 12:22:33 enterprise kernel: [47459.850527]  [<ffffffffa0177149>] btrfs_run_ordered_operations+0xe9/0x2c0 [btrfs]
Dec  1 12:22:33 enterprise kernel: [47459.850538]  [<ffffffffa015e042>] btrfs_commit_transaction+0x32/0xa20 [btrfs]
Dec  1 12:22:33 enterprise kernel: [47459.850540]  [<ffffffff81074ef0>] ? call_timer_fn+0x160/0x160
Dec  1 12:22:33 enterprise kernel: [47459.850550]  [<ffffffffa0157fd5>] transaction_kthread+0x1c5/0x230 [btrfs]
Dec  1 12:22:33 enterprise kernel: [47459.850560]  [<ffffffffa0157e10>] ? btrfs_destroy_delayed_refs+0x350/0x350 [btrfs]
Dec  1 12:22:33 enterprise kernel: [47459.850563]  [<ffffffff8108c140>] kthread+0xc0/0xd0
Dec  1 12:22:33 enterprise kernel: [47459.850565]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:33 enterprise kernel: [47459.850567]  [<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Dec  1 12:22:33 enterprise kernel: [47459.850569]  [<ffffffff8108c080>] ? flush_kthread_worker+0xb0/0xb0
Dec  1 12:22:33 enterprise kernel: [47459.850570] Code: 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 e9 0a 00 00 00 66 83 07 02 5d c3 0f 1f 40 00 8b 37 f0 66 83 07 02 <f6> 47 02 01 74 ed e8 2b 75 fe ff 5d c3 0f 1f 40 00 66 66 66 66 
Dec  1 12:22:37 enterprise kernel: [47463.865082] BUG: soft lockup - CPU#4 stuck for 23s! [btrfs-delalloc-:3072]
Dec  1 12:22:37 enterprise kernel: [47463.865124] Modules linked in: xt_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache coretemp kvm_intel kvm nouveau snd_hda_codec_realtek snd_hda_intel mxm_wmi wmi video ttm microcode drm_kms_helper snd_hda_codec psmouse serio_raw drm snd_hwdep snd_pcm i2c_algo_bit snd_page_alloc lpc_ich snd_timer snd i7core_edac soundcore edac_core asus_atk0110 mac_hid lp parport btrfs libcrc32c pata_acpi raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq firewire_ohci raid1 r8169 firewire_core usbhid pata_jmicron raid0 crc_itu_t mii ahci hid pata_marvell libahci multipath linear
Dec  1 12:22:37 enterprise kernel: [47463.865171] CPU: 4 PID: 3072 Comm: btrfs-delalloc- Not tainted 3.12.0-031200-generic #201311031935
Dec  1 12:22:37 enterprise kernel: [47463.865172] Hardware name: System manufacturer System Product Name/P6T SE, BIOS 0805    02/24/2010
Dec  1 12:22:37 enterprise kernel: [47463.865174] task: ffff880329c6af60 ti: ffff8802f4442000 task.ti: ffff8802f4442000
Dec  1 12:22:37 enterprise kernel: [47463.865176] RIP: 0010:[<ffffffff817466d2>]  [<ffffffff817466d2>] _raw_spin_lock+0x32/0x50
Dec  1 12:22:37 enterprise kernel: [47463.865179] RSP: 0018:ffff8802f4443bb8  EFLAGS: 00000202
Dec  1 12:22:37 enterprise kernel: [47463.865181] RAX: 00000000000047a3 RBX: ffff8802f4443ca7 RCX: 00000000000059e2
Dec  1 12:22:37 enterprise kernel: [47463.865182] RDX: 00000000000059e4 RSI: 00000000000059e4 RDI: ffff88003604a8e4
Dec  1 12:22:37 enterprise kernel: [47463.865184] RBP: ffff8802f4443bb8 R08: 0000000000001000 R09: ffff8800367b7658
Dec  1 12:22:37 enterprise kernel: [47463.865185] R10: ffffffffa017438e R11: 0000000000000000 R12: 0000000200000000
Dec  1 12:22:37 enterprise kernel: [47463.865187] R13: ffff880329e68700 R14: 000000e3728a8000 R15: ffff8802f4443b48
Dec  1 12:22:37 enterprise kernel: [47463.865189] FS:  0000000000000000(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  1 12:22:37 enterprise kernel: [47463.865191] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b

Comment 5 Sean Clarke 2013-12-05 23:25:13 UTC

Created attachment 117711 [details]
Latest kernel errors

Comment 6 Sean Clarke 2013-12-22 23:25:04 UTC

Now running 3.13.0-031300rc3-generic and have had the longest continual uptime since moving to 3.11 (i.e. upgraded to Ubuntu 13.10) - 8 days.

Will keep and eye on the situation, but I also have had backups running and so far it looks stable.

Will continue to monitor.

Comment 7 Sean Clarke 2013-12-30 11:42:29 UTC

Now over 2 weeks without a failure, and up until this kernel it never lasted anywhere near this.

Comment 8 Sean Clarke 2014-01-04 12:53:00 UTC

No further issues - happy to close as fixed in 3.13

Comment 9 David Sterba 2014-01-06 14:13:32 UTC

Thanks, closing.