===== Server Ubuntu 12.04 with mainline kernel installed Architecture: amd64 dmi.bios.date: 04/15/2011 dmi.bios.vendor: Phoenix Technologies LTD dmi.bios.version: 6.00 dmi.board.name: 440BX Desktop Reference Platform dmi.board.vendor: Intel Corporation dmi.board.version: None dmi.chassis.asset.tag: No Asset Tag dmi.chassis.type: 1 dmi.chassis.vendor: No Enclosure dmi.chassis.version: N/A dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd04/15/2011:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A: dmi.product.name: VMware Virtual Platform dmi.product.version: None dmi.sys.vendor: VMware, Inc. ==== In /var/log/kern.log Aug 14 01:09:17 h3oserver2 kernel: [19063.160843] BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:1:64] Aug 14 01:09:17 h3oserver2 kernel: [19063.491358] BUG: soft lockup - CPU#4 stuck for 22s! [flush-252:0:384] Aug 14 01:09:17 h3oserver2 kernel: [19064.151216] Modules linked in: xt_multiport dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state vmw_balloon ppdev ip6table_filter ip6_tables Aug 14 01:09:17 h3oserver2 kernel: [19066.533308] Modules linked in: nf_conntrack_netbios_ns xt_multiport dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state vmw_balloon ppdev ip6table_filter ip6_tables nf_conntrack_netbios_ns Aug 14 01:09:17 h3oserver2 kernel: [19066.689922] nf_conntrack_broadcast nf_conntrack_broadcast nf_nat_ftp Aug 14 01:09:17 h3oserver2 kernel: [19066.845802] nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 Aug 14 01:09:17 h3oserver2 kernel: [19067.408740] nf_nat coretemp nf_conntrack_ftp nf_conntrack Aug 14 01:09:17 h3oserver2 kernel: [19067.799885] nf_conntrack_ipv4 psmouse nf_defrag_ipv4 Aug 14 01:09:17 h3oserver2 kernel: [19067.967818] iptable_filter coretemp nf_conntrack_ftp nf_conntrack psmouse iptable_filter Aug 14 01:09:17 h3oserver2 kernel: [19068.139213] microcode ip_tables Aug 14 01:09:17 h3oserver2 kernel: [19068.550568] microcode x_tables ip_tables x_tables serio_raw Aug 14 01:09:17 h3oserver2 kernel: [19068.913761] serio_raw acpi_memhotplug Aug 14 01:09:17 h3oserver2 kernel: [19068.966588] acpi_memhotplug parport_pc mac_hid Aug 14 01:09:17 h3oserver2 kernel: [19069.077272] parport_pc mac_hid Aug 14 01:09:17 h3oserver2 kernel: [19069.164442] i2c_piix4 i2c_piix4 Aug 14 01:09:17 h3oserver2 kernel: [19069.164871] shpchp shpchp Aug 14 01:09:17 h3oserver2 kernel: [19069.165144] lp parport Aug 14 01:09:17 h3oserver2 kernel: [19069.165657] lp parport Aug 14 01:09:17 h3oserver2 kernel: [19069.165805] vmxnet3 vmxnet3 floppy vmw_pvscsi Aug 14 01:09:17 h3oserver2 kernel: [19069.305445] floppy vmw_pvscsi Aug 14 01:09:17 h3oserver2 kernel: [19069.305576] Aug 14 01:09:17 h3oserver2 kernel: [19069.324876] CPU 4 Aug 14 01:09:17 h3oserver2 kernel: [19069.347067] CPU 1 Modules linked in: xt_multiport Aug 14 01:09:17 h3oserver2 kernel: [19069.411029] Modules linked in: xt_multiport Aug 14 01:09:17 h3oserver2 kernel: [19069.411159] dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit Aug 14 01:09:17 h3oserver2 kernel: [19069.411279] dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit Aug 14 01:09:17 h3oserver2 kernel: [19069.411471] xt_tcpudp xt_tcpudp xt_addrtype xt_state Aug 14 01:09:17 h3oserver2 kernel: [19069.411480] xt_addrtype vmw_balloon ppdev ip6table_filter Aug 14 01:09:17 h3oserver2 kernel: [19069.411495] xt_state vmw_balloon ppdev ip6table_filter Aug 14 01:09:18 h3oserver2 kernel: [19069.421235] ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 coretemp nf_conntrack_ftp nf_conntrack Aug 14 01:09:18 h3oserver2 kernel: [19069.421475] ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 coretemp nf_conntrack_ftp nf_conntrack psmouse Aug 14 01:09:18 h3oserver2 kernel: [19069.462067] psmouse iptable_filter microcode ip_tables x_tables Aug 14 01:09:18 h3oserver2 kernel: [19069.462097] iptable_filter microcode ip_tables Aug 14 01:09:18 h3oserver2 kernel: [19069.462109] serio_raw acpi_memhotplug parport_pc mac_hid i2c_piix4 shpchp x_tables serio_raw Aug 14 01:09:18 h3oserver2 kernel: [19069.462126] lp parport vmxnet3 floppy vmw_pvscsi Aug 14 01:09:18 h3oserver2 kernel: [19069.462137] acpi_memhotplug parport_pc mac_hid i2c_piix4 shpchp lp parport vmxnet3 floppy vmw_pvscsi Aug 14 01:09:18 h3oserver2 kernel: [19069.517932] Aug 14 01:09:18 h3oserver2 kernel: [19069.518044] Aug 14 01:09:18 h3oserver2 kernel: [19069.600105] Pid: 384, comm: flush-252:0 Not tainted 3.5.1-030501-generic #201208091310 Aug 14 01:09:18 h3oserver2 kernel: [19069.600261] Pid: 64, comm: kworker/1:1 Not tainted 3.5.1-030501-generic #201208091310 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Aug 14 01:09:18 h3oserver2 kernel: [19069.690429] VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Aug 14 01:09:18 h3oserver2 kernel: [19069.690450] RIP: 0010:[<ffffffff81085e50>] Aug 14 01:09:18 h3oserver2 kernel: [19070.478212] [<ffffffff81085e50>] finish_task_switch+0x50/0xf0 Aug 14 01:09:18 h3oserver2 kernel: [19069.690456] RIP: 0010:[<ffffffff81133a70>] [<ffffffff81133a70>] sys_readahead+0xa0/0xa0 Aug 14 01:09:18 h3oserver2 kernel: [19070.797865] RSP: 0000:ffff880171175938 EFLAGS: 00010246 Aug 14 01:09:18 h3oserver2 kernel: [19070.797872] RAX: 0000000000000001 RBX: ffff8801711758c0 RCX: 0000000000000000 Aug 14 01:09:18 h3oserver2 kernel: [19070.797883] RDX: ffff8801711759e8 RSI: ffff8801774d55f8 RDI: ffff880171175968 Aug 14 01:09:18 h3oserver2 kernel: [19070.797887] RBP: ffff880171175a20 R08: 000000000000000e R09: 7fffffffffffffff Aug 14 01:09:18 h3oserver2 kernel: [19070.797905] R10: ffff88017a010c00 R11: ffff880171175c38 R12: ffffffff81349260 Aug 14 01:09:18 h3oserver2 kernel: [19070.797908] R13: ffff8801712c6d80 R14: 00000001779e9000 R15: ffffffffa00018be Aug 14 01:09:20 h3oserver2 kernel: [19070.797921] RSP: 0018:ffff880173723dc0 EFLAGS: 00000286 Aug 14 01:09:20 h3oserver2 kernel: [19070.797924] RAX: ffff880177dedc00 RBX: ffff8801735e8048 RCX: 0000000000000001 Aug 14 01:09:20 h3oserver2 kernel: [19070.797926] RDX: ffff880173723fd8 RSI: ffff8801735e8000 RDI: ffff88017fc33940 Aug 14 01:09:20 h3oserver2 kernel: [19070.797927] RBP: ffff880173723de0 R08: ffff880173722000 R09: 0000000000000001 Aug 14 01:09:20 h3oserver2 kernel: [19070.797928] R10: 00000000000e7ef0 R11: 0000000000000001 R12: ffff880173723d60 Aug 14 01:09:20 h3oserver2 kernel: [19070.797929] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001 Aug 14 01:09:20 h3oserver2 kernel: [19070.797974] FS: 0000000000000000(0000) GS:ffff88017fc80000(0000) knlGS:0000000000000000 Aug 14 01:09:20 h3oserver2 kernel: [19070.797980] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 14 01:09:20 h3oserver2 kernel: [19070.797982] CR2: 00007ff4201ae000 CR3: 0000000162943000 CR4: 00000000000006e0 Aug 14 01:09:20 h3oserver2 kernel: [19070.798004] FS: 0000000000000000(0000) GS:ffff88017fc20000(0000) knlGS:0000000000000000 Aug 14 01:09:20 h3oserver2 kernel: [19070.798006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 14 01:09:20 h3oserver2 kernel: [19070.798008] CR2: 00007f1ff1a35000 CR3: 0000000162943000 CR4: 00000000000006e0 Aug 14 01:09:20 h3oserver2 kernel: [19070.848830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 14 01:09:20 h3oserver2 kernel: [19070.848842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 14 01:09:20 h3oserver2 kernel: [19070.848876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 14 01:09:20 h3oserver2 kernel: [19070.848883] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 14 01:09:20 h3oserver2 kernel: [19070.848889] Process kworker/1:1 (pid: 64, threadinfo ffff880173722000, task ffff8801735e8000) Aug 14 01:09:20 h3oserver2 kernel: [19070.848895] Process flush-252:0 (pid: 384, threadinfo ffff880171174000, task ffff880170295c00) Aug 14 01:09:20 h3oserver2 kernel: [19070.848898] Stack: Aug 14 01:09:20 h3oserver2 kernel: [19070.848906] Stack: Aug 14 01:09:20 h3oserver2 kernel: [19070.848901] 0000000000013940 Aug 14 01:09:20 h3oserver2 kernel: [19070.848910] ffffffff81220f12 0000000000000000 ffff880173155e80 0000000000000001 Aug 14 01:09:20 h3oserver2 kernel: [19070.848936] ffff880173723e60 ffffffff81696094 ffff88017fc307a0 ffff880177eca080 Aug 14 01:09:20 h3oserver2 kernel: [19070.848939] ffff880173723fd8 ffff880173723fd8 ffff880173723fd8 0000000000013940 Aug 14 01:09:20 h3oserver2 kernel: [19070.848942] Call Trace: Aug 14 01:09:20 h3oserver2 kernel: [19070.848946] ffff880171175950 ffff88017300bd40 ffff88017300bd40 Aug 14 01:09:20 h3oserver2 kernel: [19070.848952] ffff88017300bd40 Aug 14 01:09:20 h3oserver2 kernel: [19070.848954] ffff8801711759a0 0000000000000000 0000000000000000 Aug 14 01:09:20 h3oserver2 kernel: [19070.848964] ffff880173049800 [<ffffffff81696094>] __schedule+0x3c4/0x700 Aug 14 01:09:20 h3oserver2 kernel: [19071.500857] [<ffffffff816966e9>] schedule+0x29/0x70 Aug 14 01:09:20 h3oserver2 kernel: [19071.553927] [<ffffffff81072bae>] worker_thread+0x24e/0x370 Aug 14 01:09:20 h3oserver2 kernel: [19071.563020] ffff8801736a9000 ffff880173049800 ffff8801711759a0 Aug 14 01:09:20 h3oserver2 kernel: [19071.563040] Call Trace: Aug 14 01:09:20 h3oserver2 kernel: [19071.644174] [<ffffffff81072960>] ? manage_workers.isra.29+0x130/0x130 Aug 14 01:09:20 h3oserver2 kernel: [19071.644191] [<ffffffff81077873>] kthread+0x93/0xa0 Aug 14 01:09:20 h3oserver2 kernel: [19071.716927] [<ffffffff816a1164>] kernel_thread_helper+0x4/0x10 Aug 14 01:09:20 h3oserver2 kernel: [19071.716957] [<ffffffff810777e0>] ? kthread_freezable_should_stop+0x70/0x70 Aug 14 01:09:20 h3oserver2 kernel: [19071.716961] [<ffffffff816a1160>] ? gs_change+0x13/0x13 Aug 14 01:09:20 h3oserver2 kernel: [19071.716963] Code: Aug 14 01:09:20 h3oserver2 kernel: [19071.716973] [<ffffffff81220f12>] ? ext4_num_dirty_pages.isra.43+0xa2/0x200 Aug 14 01:09:20 h3oserver2 kernel: [19071.716976] 48 89 Aug 14 01:09:20 h3oserver2 kernel: [19071.935852] [<ffffffff81226f40>] ext4_da_writepages+0x5c0/0x620 Aug 14 01:09:20 h3oserver2 kernel: [19071.935877] fb [<ffffffff81226980>] ? write_cache_pages_da+0x450/0x450 Aug 14 01:09:20 h3oserver2 kernel: [19071.936018] [<ffffffff81132ed0>] do_writepages+0x20/0x40 Aug 14 01:09:20 h3oserver2 kernel: [19072.128836] [<ffffffff811af5ff>] __writeback_single_inode.isra.32+0x3f/0x190 Aug 14 01:09:20 h3oserver2 kernel: [19072.232022] 4c 8b 36 65 48 8b 34 25 00 c7 00 00 66 Aug 14 01:09:20 h3oserver2 kernel: [19072.232037] 66 [<ffffffff811afd20>] writeback_sb_inodes+0x1a0/0x350 Aug 14 01:09:20 h3oserver2 kernel: [19072.232042] 66 66 90 41 Aug 14 01:09:20 h3oserver2 kernel: [19072.232063] [<ffffffff811aff6e>] __writeback_inodes_wb+0x9e/0xd0 Aug 14 01:09:20 h3oserver2 kernel: [19072.232068] [<ffffffff811b022b>] wb_writeback+0x28b/0x340 Aug 14 01:09:20 h3oserver2 kernel: [19072.232074] [<ffffffff811b037f>] wb_check_old_data_flush+0x9f/0xb0 Aug 14 01:09:20 h3oserver2 kernel: [19072.232079] [<ffffffff811b1979>] wb_do_writeback+0x149/0x1d0 Aug 14 01:09:20 h3oserver2 kernel: [19072.232085] [<ffffffff810636d0>] ? usleep_range+0x50/0x50 Aug 14 01:09:20 h3oserver2 kernel: [19072.255121] [<ffffffff811b1a8b>] bdi_writeback_thread+0x8b/0x290 Aug 14 01:09:20 h3oserver2 kernel: [19072.255137] [<ffffffff811b1a00>] ? wb_do_writeback+0x1d0/0x1d0 Aug 14 01:09:20 h3oserver2 kernel: [19072.255144] [<ffffffff81077873>] kthread+0x93/0xa0 Aug 14 01:09:20 h3oserver2 kernel: [19072.255158] [<ffffffff816a1164>] kernel_thread_helper+0x4/0x10 Aug 14 01:09:20 h3oserver2 kernel: [19072.255181] [<ffffffff810777e0>] ? kthread_freezable_should_stop+0x70/0x70 Aug 14 01:09:20 h3oserver2 kernel: [19072.255188] [<ffffffff816a1160>] ? gs_change+0x13/0x13 Aug 14 01:09:20 h3oserver2 kernel: [19072.255196] Code: c7 44 24 28 00 00 00 00 48 89 df Aug 14 01:09:20 h3oserver2 kernel: [19072.255215] d1 48 c1 e8 Aug 14 01:09:20 h3oserver2 kernel: [19072.255223] e8 56 ac fb ff 66 90 fb 66 66 90 <66> 66 90 65 48 8b 04 25 00 c7 00 00 48 8b 98 e0 01 00 00 48 85 Aug 14 01:09:20 h3oserver2 kernel: [19072.255302] 0c 48 01 c1 e8 7e fb ff ff 48 89 df e8 26 4f 05 00 4c 89 f0 48 8b 5d e0 4c 8b 65 e8 4c 8b 6d f0 4c 8b 75 f8 c9 c3 90 <55> 48 89 e5 53 48 83 ec 08 66 66 66 66 90 48 89 fb 48 89 f7 48 Reference https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1035855
This looks like an I/O never completed but it may be a pointer to something else (eg sys_readahead/ext4 interaction). Assigning to the ext4 folk see if they have any thoughts but thats half guesswork 8)
can Another oop dump be useful? Thanks Aug 16 07:55:28 h3oserver2 kernel: [147758.832781] BUG: soft lockup - CPU#2 stuck for 23s! [netstat:18634] Aug 16 07:55:28 h3oserver2 kernel: [147759.046380] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi Aug 16 07:55:28 h3oserver2 kernel: [147759.630769] CPU 2 Aug 16 07:55:28 h3oserver2 kernel: [147759.630772] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi Aug 16 07:55:28 h3oserver2 kernel: [147759.652655] Aug 16 07:55:28 h3oserver2 kernel: [147759.678170] Pid: 18634, comm: netstat Not tainted 3.5.1-030501-generic #201208091310 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Aug 16 07:55:28 h3oserver2 kernel: [147759.688391] RIP: 0010:[<ffffffff815d0b5e>] [<ffffffff815d0b5e>] established_get_next+0xfe/0x190 Aug 16 07:55:28 h3oserver2 kernel: [147759.928918] RSP: 0018:ffff880161f8fde8 EFLAGS: 00010286 Aug 16 07:55:28 h3oserver2 kernel: [147759.928923] RAX: 0000000000060500 RBX: ffff880018139400 RCX: 000000000007ffff Aug 16 07:55:28 h3oserver2 kernel: [147759.928925] RDX: ffffc90010e82000 RSI: ffffc90011487000 RDI: ffffffff815d0b28 Aug 16 07:55:28 h3oserver2 kernel: [147759.928926] RBP: ffff880161f8fdf8 R08: 0000000000000014 R09: 000000000000ffff Aug 16 07:55:28 h3oserver2 kernel: [147759.928928] R10: 0000000000000000 R11: 000000000000000f R12: ffff0014ff0a0000 Aug 16 07:55:28 h3oserver2 kernel: [147759.928930] R13: ffff8801ffffffff R14: ffff8801630898b5 R15: 000000000000074b Aug 16 07:55:28 h3oserver2 kernel: [147759.928984] FS: 00007f5d094bf700(0000) GS:ffff88017fc40000(0000) knlGS:0000000000000000 Aug 16 07:55:28 h3oserver2 kernel: [147759.928987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 16 07:55:28 h3oserver2 kernel: [147759.928988] CR2: 0000000000415f95 CR3: 000000011ccfb000 CR4: 00000000000006e0 Aug 16 07:55:28 h3oserver2 kernel: [147759.974228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 16 07:55:28 h3oserver2 kernel: [147759.974266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 16 07:55:28 h3oserver2 kernel: [147760.003496] Process netstat (pid: 18634, threadinfo ffff880161f8e000, task ffff880173125c00) Aug 16 07:55:28 h3oserver2 kernel: [147760.003505] Stack: Aug 16 07:55:28 h3oserver2 kernel: [147760.003507] ffff880018139400 ffff8801745c7440 ffff880161f8fe28 ffffffff815d0cef Aug 16 07:55:28 h3oserver2 kernel: [147760.003514] ffff880161f8fe28 ffff8801622b0d00 ffff880018139400 ffff8801684c0000 Aug 16 07:55:28 h3oserver2 kernel: [147760.003518] ffff880161f8fea8 ffffffff811a8d08 ffff88013ffa4530 0002000000000001 Aug 16 07:55:28 h3oserver2 kernel: [147760.003522] Call Trace: Aug 16 07:55:28 h3oserver2 kernel: [147760.003535] [<ffffffff815d0cef>] tcp_seq_next+0x3f/0xa0 Aug 16 07:55:28 h3oserver2 kernel: [147760.087408] [<ffffffff811a8d08>] seq_read+0x238/0x400 Aug 16 07:55:28 h3oserver2 kernel: [147760.087434] [<ffffffff811a8ad0>] ? seq_put_decimal_ll+0x60/0x60 Aug 16 07:55:28 h3oserver2 kernel: [147760.158567] [<ffffffff811e7402>] proc_reg_read+0x82/0xc0 Aug 16 07:55:28 h3oserver2 kernel: [147760.178084] [<ffffffff811873c0>] vfs_read+0xb0/0x180 Aug 16 07:55:28 h3oserver2 kernel: [147760.178108] [<ffffffff811874da>] sys_read+0x4a/0x90 Aug 16 07:55:28 h3oserver2 kernel: [147760.291955] [<ffffffff8169fe69>] system_call_fastpath+0x16/0x1b Aug 16 07:55:28 h3oserver2 kernel: [147760.291966] Code: 0d 58 bb 95 00 48 8b 15 41 bb 95 00 c7 43 1c 00 00 00 00 83 c0 01 39 c8 89 43 18 0f 87 8e 00 00 00 48 63 f0 48 c1 e6 04 48 01 d6 <f6> 06 01 75 73 23 05 2b bb 95 00 48 8d 3c 85 00 00 00 00 48 03
I made 2 modifications on the server running linux and currently no more soft lockup has happened 1) removed irqbalance daemon from ubuntu server 2) unchecked "Synchronize guest time with host" from VM setting (VMWare esx) Modification 1) most probably solved the problem.
Hi, I had the same issue with Ubuntu 14.04 + Xen-4.4 and the same solution worked: remove irqbalance. Though I suspect it could be a question of lock timing and this problem could come back. 3.13.0-32-generic Aug 16 17:11:42 mytv kernel: [ 59.284157] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631] Aug 16 17:11:42 mytv kernel: [ 59.284160] Modules linked in: xen_gntdev xen_evtchn xenfs xen_privcmd nfsv 3 nfsv4 bridge stp llc dm_crypt nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 eeepc_wmi a sus_wmi sparse_keymap video mxm_wmi snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hwdep snd_pcm aesni_intel aes_x86_64 lrw gf128mul snd_page_alloc glue_help er ablk_helper cryptd snd_seq_midi snd_seq_midi_event serio_raw snd_rawmidi fam15h_power k10temp edac_core edac_mce_amd snd_seq joydev snd_seq_device snd_timer snd sp5100_tco i2c_piix4 soundcore nvidia(POF) parport _pc ppdev lp parport wmi mac_hid dm_mirror dm_region_hash dm_log hid_generic usbhid hid psmouse e1000e r816 9 mii ptp ahci pps_core libahci Aug 16 17:11:42 mytv kernel: [ 59.284210] CPU: 2 PID: 3631 Comm: Xorg Tainted: PF O 3.13.0-32-ge neric #57-Ubuntu Aug 16 17:11:42 mytv kernel: [ 59.284212] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SA BERTOOTH 990FX R2.0, BIOS 2501 04/08/2014 Aug 16 17:11:42 mytv kernel: [ 59.284216] task: ffff880425155fc0 ti: ffff8804265f6000 task.ti: ffff880426 5f6000 Aug 16 17:11:42 mytv kernel: [ 59.284218] RIP: e030:[<ffffffffa077d0da>] [<ffffffffa077d0da>] rm_shutdow n_gvi_device+0x187/0x295 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284315] RSP: e02b:ffff8804265f78d0 EFLAGS: 00000282 Aug 16 17:11:42 mytv kernel: [ 59.284317] RAX: ffff8800000dc97c RBX: ffff880415192e24 RCX: ffffffffa0b616 70 Aug 16 17:11:42 mytv kernel: [ 59.284318] RDX: 00000000000000ff RSI: 00000000000dc97c RDI: 00000000000dc9 7c Aug 16 17:11:42 mytv kernel: [ 59.284319] RBP: ffff880415192e20 R08: 0000000000000001 R09: ffffffffa0b6d1 40 Aug 16 17:11:42 mytv kernel: [ 59.284320] R10: ffff880425ace808 R11: ffffffffa079c8db R12: ffff880415192e 28 Aug 16 17:11:42 mytv kernel: [ 59.284321] R13: ffff880415192e2c R14: 0000000000009bed R15: ffff880415192e 90 Aug 16 17:11:42 mytv kernel: [ 59.284325] FS: 00007fc4235f79c0(0000) GS:ffff880447280000(0000) knlGS:000 0000000000000 Aug 16 17:11:42 mytv kernel: [ 59.284327] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 16 17:11:42 mytv kernel: [ 59.284328] CR2: 00007fd1e4edcff0 CR3: 0000000425a8a000 CR4: 00000000000406 60 Aug 16 17:11:42 mytv kernel: [ 59.284329] Stack: Aug 16 17:11:42 mytv kernel: [ 59.284330] ffffffffa079c8db ffffffffa077f9da ffff8804151b8008 00000000000 0cfde Aug 16 17:11:42 mytv kernel: [ 59.284333] ffff880415192e8c ffffffffa0789d58 ffff8804265f7910 ffff8804151 b8008 Aug 16 17:11:42 mytv kernel: [ 59.284336] 000000000000cfde ffffffffa077f976 ffff8804151b8008 ffffffffa07 7d639 Aug 16 17:11:42 mytv kernel: [ 59.284338] Call Trace: Aug 16 17:11:42 mytv kernel: [ 59.284384] [<ffffffffa079c8db>] ? os_free_mem+0x1b/0x30 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284428] [<ffffffffa077f9da>] ? _nv007991rm+0x33/0x56 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284472] [<ffffffffa0789d58>] ? _nv019222rm+0x970b/0xcee3 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284517] [<ffffffffa077f976>] ? _nv001242rm+0x83/0xa4 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284560] [<ffffffffa077d639>] ? _nv014418rm+0x1f0/0xc97 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284605] [<ffffffffa076ad9e>] ? _nv014859rm+0xce/0x3da [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284653] [<ffffffffa076b176>] ? _nv014923rm+0x4c/0x59 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284716] [<ffffffffa02e083c>] ? _nv018571rm+0x35/0x79 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284765] [<ffffffffa072e0ab>] ? _nv018447rm+0x46/0xbf [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284827] [<ffffffffa02a3ce1>] ? _nv012729rm+0x551/0x13cb [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284890] [<ffffffffa02c05a5>] ? _nv017460rm+0x405/0x42f [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284954] [<ffffffffa03111f5>] ? _nv005690rm+0x1e5/0x1f3 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285033] [<ffffffffa03df9ee>] ? _nv005044rm+0x9a/0xc4 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285129] [<ffffffffa0539d22>] ? _nv004050rm+0x8858/0xaef1 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285225] [<ffffffffa0538573>] ? _nv004050rm+0x70a9/0xaef1 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285264] [<ffffffffa0166e80>] ? _nv010039rm+0x25/0x40 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285310] [<ffffffffa0779558>] ? _nv015014rm+0x808/0x982 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285354] [<ffffffffa077a538>] ? _nv001097rm+0x483/0x6b8 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285398] [<ffffffffa0772aa4>] ? rm_init_adapter+0xac/0x146 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285443] [<ffffffffa0792b71>] ? nv_kern_open+0x191/0x810 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285448] [<ffffffff811c14ef>] ? chrdev_open+0x9f/0x1d0 Aug 16 17:11:42 mytv kernel: [ 59.285451] [<ffffffff811ba033>] ? do_dentry_open+0x233/0x2e0 Aug 16 17:11:42 mytv kernel: [ 59.285453] [<ffffffff811c1450>] ? cdev_put+0x30/0x30 Aug 16 17:11:42 mytv kernel: [ 59.285455] [<ffffffff811ba369>] ? vfs_open+0x49/0x50 Aug 16 17:11:42 mytv kernel: [ 59.285464] [<ffffffff811c8f04>] ? do_last+0x554/0x1200 Aug 16 17:11:42 mytv kernel: [ 59.285467] [<ffffffff81311bdb>] ? apparmor_file_alloc_security+0x5b/0x180 Aug 16 17:11:42 mytv kernel: [ 59.285470] [<ffffffff811cc38b>] ? path_openat+0xbb/0x640 Aug 16 17:11:42 mytv kernel: [ 59.285473] [<ffffffff812d3d5e>] ? security_inode_alloc+0x1e/0x20 Aug 16 17:11:42 mytv kernel: [ 59.285476] [<ffffffff811e2988>] ? simple_xattr_get+0x68/0xb0 Aug 16 17:11:42 mytv kernel: [ 59.285478] [<ffffffff811cd76a>] ? do_filp_open+0x3a/0x90 Aug 16 17:11:42 mytv kernel: [ 59.285481] [<ffffffff811da527>] ? __alloc_fd+0xa7/0x130 Aug 16 17:11:42 mytv kernel: [ 59.285484] [<ffffffff811bbe89>] ? do_sys_open+0x129/0x280 Aug 16 17:11:42 mytv kernel: [ 59.285487] [<ffffffff81020d45>] ? syscall_trace_enter+0x145/0x250 Aug 16 17:11:42 mytv kernel: [ 59.285489] [<ffffffff811bbffe>] ? SyS_open+0x1e/0x20 Aug 16 17:11:42 mytv kernel: [ 59.285492] [<ffffffff8172c87f>] ? tracesys+0xe1/0xe6 Aug 16 17:11:42 mytv kernel: [ 59.285493] Code: c3 e8 e2 28 00 00 b8 00 00 00 00 48 83 c4 08 c3 48 83 ec 08 be 01 00 00 00 e8 96 ff ff ff ba 00 00 00 00 48 85 c0 74 03 0f b6 10 <89> d0 48 83 c4 08 c3 48 83 ec 08 be 02 00 00 00 e8 74 ff ff ff Aug 16 17:12:10 mytv kernel: [ 87.283580] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631] and loop... Xorg never starts properly though ssh through works. All attempts to shutdown cleanly hang as does tcpdump if it matters.