Bug 45941

Summary: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:1:64] BUG: soft lockup - CPU#4 stuck for 22s! [flush-252:0:384]
Product: File System Reporter: Loris Luise (loris.luise)
Component: ext4Assignee: fs_ext4 (fs_ext4)
Status: RESOLVED OBSOLETE    
Severity: normal CC: alan, vmware
Priority: P1    
Hardware: x86-64   
OS: Linux   
URL: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.1-quantal/
Kernel Version: 3.5.1 Subsystem:
Regression: No Bisected commit-id:

Description Loris Luise 2012-08-14 08:02:49 UTC
=====
Server Ubuntu 12.04 with mainline kernel installed
Architecture: amd64
dmi.bios.date: 04/15/2011
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd04/15/2011:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.
====

In /var/log/kern.log

Aug 14 01:09:17 h3oserver2 kernel: [19063.160843] BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:1:64]
Aug 14 01:09:17 h3oserver2 kernel: [19063.491358] BUG: soft lockup - CPU#4 stuck for 22s! [flush-252:0:384]
Aug 14 01:09:17 h3oserver2 kernel: [19064.151216] Modules linked in: xt_multiport dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state vmw_balloon ppdev ip6table_filter ip6_tables
Aug 14 01:09:17 h3oserver2 kernel: [19066.533308] Modules linked in: nf_conntrack_netbios_ns xt_multiport dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state vmw_balloon ppdev ip6table_filter ip6_tables nf_conntrack_netbios_ns
Aug 14 01:09:17 h3oserver2 kernel: [19066.689922]  nf_conntrack_broadcast nf_conntrack_broadcast nf_nat_ftp
Aug 14 01:09:17 h3oserver2 kernel: [19066.845802]  nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
Aug 14 01:09:17 h3oserver2 kernel: [19067.408740]  nf_nat coretemp nf_conntrack_ftp nf_conntrack
Aug 14 01:09:17 h3oserver2 kernel: [19067.799885]  nf_conntrack_ipv4 psmouse nf_defrag_ipv4
Aug 14 01:09:17 h3oserver2 kernel: [19067.967818]  iptable_filter coretemp nf_conntrack_ftp nf_conntrack psmouse iptable_filter
Aug 14 01:09:17 h3oserver2 kernel: [19068.139213]  microcode ip_tables
Aug 14 01:09:17 h3oserver2 kernel: [19068.550568]  microcode x_tables ip_tables x_tables serio_raw
Aug 14 01:09:17 h3oserver2 kernel: [19068.913761]  serio_raw acpi_memhotplug
Aug 14 01:09:17 h3oserver2 kernel: [19068.966588]  acpi_memhotplug parport_pc mac_hid
Aug 14 01:09:17 h3oserver2 kernel: [19069.077272]  parport_pc mac_hid
Aug 14 01:09:17 h3oserver2 kernel: [19069.164442]  i2c_piix4 i2c_piix4
Aug 14 01:09:17 h3oserver2 kernel: [19069.164871]  shpchp shpchp
Aug 14 01:09:17 h3oserver2 kernel: [19069.165144]  lp parport
Aug 14 01:09:17 h3oserver2 kernel: [19069.165657]  lp parport
Aug 14 01:09:17 h3oserver2 kernel: [19069.165805]  vmxnet3 vmxnet3 floppy vmw_pvscsi
Aug 14 01:09:17 h3oserver2 kernel: [19069.305445]  floppy vmw_pvscsi
Aug 14 01:09:17 h3oserver2 kernel: [19069.305576] 
Aug 14 01:09:17 h3oserver2 kernel: [19069.324876] CPU 4 
Aug 14 01:09:17 h3oserver2 kernel: [19069.347067] CPU 1 Modules linked in: xt_multiport
Aug 14 01:09:17 h3oserver2 kernel: [19069.411029] Modules linked in: xt_multiport
Aug 14 01:09:17 h3oserver2 kernel: [19069.411159]  dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit
Aug 14 01:09:17 h3oserver2 kernel: [19069.411279]  dm_crypt pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit
Aug 14 01:09:17 h3oserver2 kernel: [19069.411471]  xt_tcpudp xt_tcpudp xt_addrtype xt_state
Aug 14 01:09:17 h3oserver2 kernel: [19069.411480]  xt_addrtype vmw_balloon ppdev ip6table_filter
Aug 14 01:09:17 h3oserver2 kernel: [19069.411495]  xt_state vmw_balloon ppdev ip6table_filter
Aug 14 01:09:18 h3oserver2 kernel: [19069.421235]  ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 coretemp nf_conntrack_ftp nf_conntrack
Aug 14 01:09:18 h3oserver2 kernel: [19069.421475]  ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 coretemp nf_conntrack_ftp nf_conntrack psmouse
Aug 14 01:09:18 h3oserver2 kernel: [19069.462067]  psmouse iptable_filter microcode ip_tables x_tables
Aug 14 01:09:18 h3oserver2 kernel: [19069.462097]  iptable_filter microcode ip_tables
Aug 14 01:09:18 h3oserver2 kernel: [19069.462109]  serio_raw acpi_memhotplug parport_pc mac_hid i2c_piix4 shpchp x_tables serio_raw
Aug 14 01:09:18 h3oserver2 kernel: [19069.462126]  lp parport vmxnet3 floppy vmw_pvscsi
Aug 14 01:09:18 h3oserver2 kernel: [19069.462137]  acpi_memhotplug parport_pc mac_hid i2c_piix4 shpchp lp parport vmxnet3 floppy vmw_pvscsi
Aug 14 01:09:18 h3oserver2 kernel: [19069.517932] 
Aug 14 01:09:18 h3oserver2 kernel: [19069.518044] 
Aug 14 01:09:18 h3oserver2 kernel: [19069.600105] Pid: 384, comm: flush-252:0 Not tainted 3.5.1-030501-generic #201208091310
Aug 14 01:09:18 h3oserver2 kernel: [19069.600261] Pid: 64, comm: kworker/1:1 Not tainted 3.5.1-030501-generic #201208091310 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Aug 14 01:09:18 h3oserver2 kernel: [19069.690429]  VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Aug 14 01:09:18 h3oserver2 kernel: [19069.690450] RIP: 0010:[<ffffffff81085e50>] 
Aug 14 01:09:18 h3oserver2 kernel: [19070.478212]  [<ffffffff81085e50>] finish_task_switch+0x50/0xf0
Aug 14 01:09:18 h3oserver2 kernel: [19069.690456] RIP: 0010:[<ffffffff81133a70>]  [<ffffffff81133a70>] sys_readahead+0xa0/0xa0
Aug 14 01:09:18 h3oserver2 kernel: [19070.797865] RSP: 0000:ffff880171175938  EFLAGS: 00010246
Aug 14 01:09:18 h3oserver2 kernel: [19070.797872] RAX: 0000000000000001 RBX: ffff8801711758c0 RCX: 0000000000000000
Aug 14 01:09:18 h3oserver2 kernel: [19070.797883] RDX: ffff8801711759e8 RSI: ffff8801774d55f8 RDI: ffff880171175968
Aug 14 01:09:18 h3oserver2 kernel: [19070.797887] RBP: ffff880171175a20 R08: 000000000000000e R09: 7fffffffffffffff
Aug 14 01:09:18 h3oserver2 kernel: [19070.797905] R10: ffff88017a010c00 R11: ffff880171175c38 R12: ffffffff81349260
Aug 14 01:09:18 h3oserver2 kernel: [19070.797908] R13: ffff8801712c6d80 R14: 00000001779e9000 R15: ffffffffa00018be
Aug 14 01:09:20 h3oserver2 kernel: [19070.797921] RSP: 0018:ffff880173723dc0  EFLAGS: 00000286
Aug 14 01:09:20 h3oserver2 kernel: [19070.797924] RAX: ffff880177dedc00 RBX: ffff8801735e8048 RCX: 0000000000000001
Aug 14 01:09:20 h3oserver2 kernel: [19070.797926] RDX: ffff880173723fd8 RSI: ffff8801735e8000 RDI: ffff88017fc33940
Aug 14 01:09:20 h3oserver2 kernel: [19070.797927] RBP: ffff880173723de0 R08: ffff880173722000 R09: 0000000000000001
Aug 14 01:09:20 h3oserver2 kernel: [19070.797928] R10: 00000000000e7ef0 R11: 0000000000000001 R12: ffff880173723d60
Aug 14 01:09:20 h3oserver2 kernel: [19070.797929] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
Aug 14 01:09:20 h3oserver2 kernel: [19070.797974] FS:  0000000000000000(0000) GS:ffff88017fc80000(0000) knlGS:0000000000000000
Aug 14 01:09:20 h3oserver2 kernel: [19070.797980] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 14 01:09:20 h3oserver2 kernel: [19070.797982] CR2: 00007ff4201ae000 CR3: 0000000162943000 CR4: 00000000000006e0
Aug 14 01:09:20 h3oserver2 kernel: [19070.798004] FS:  0000000000000000(0000) GS:ffff88017fc20000(0000) knlGS:0000000000000000
Aug 14 01:09:20 h3oserver2 kernel: [19070.798006] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 14 01:09:20 h3oserver2 kernel: [19070.798008] CR2: 00007f1ff1a35000 CR3: 0000000162943000 CR4: 00000000000006e0
Aug 14 01:09:20 h3oserver2 kernel: [19070.848830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 01:09:20 h3oserver2 kernel: [19070.848842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 01:09:20 h3oserver2 kernel: [19070.848876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 14 01:09:20 h3oserver2 kernel: [19070.848883] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 14 01:09:20 h3oserver2 kernel: [19070.848889] Process kworker/1:1 (pid: 64, threadinfo ffff880173722000, task ffff8801735e8000)
Aug 14 01:09:20 h3oserver2 kernel: [19070.848895] Process flush-252:0 (pid: 384, threadinfo ffff880171174000, task ffff880170295c00)
Aug 14 01:09:20 h3oserver2 kernel: [19070.848898] Stack:
Aug 14 01:09:20 h3oserver2 kernel: [19070.848906] Stack:
Aug 14 01:09:20 h3oserver2 kernel: [19070.848901]  0000000000013940
Aug 14 01:09:20 h3oserver2 kernel: [19070.848910]  ffffffff81220f12 0000000000000000 ffff880173155e80 0000000000000001
Aug 14 01:09:20 h3oserver2 kernel: [19070.848936]  ffff880173723e60 ffffffff81696094 ffff88017fc307a0 ffff880177eca080
Aug 14 01:09:20 h3oserver2 kernel: [19070.848939]  ffff880173723fd8 ffff880173723fd8 ffff880173723fd8 0000000000013940
Aug 14 01:09:20 h3oserver2 kernel: [19070.848942] Call Trace:
Aug 14 01:09:20 h3oserver2 kernel: [19070.848946]  ffff880171175950 ffff88017300bd40 ffff88017300bd40
Aug 14 01:09:20 h3oserver2 kernel: [19070.848952]  ffff88017300bd40
Aug 14 01:09:20 h3oserver2 kernel: [19070.848954]  ffff8801711759a0 0000000000000000 0000000000000000
Aug 14 01:09:20 h3oserver2 kernel: [19070.848964]  ffff880173049800 [<ffffffff81696094>] __schedule+0x3c4/0x700
Aug 14 01:09:20 h3oserver2 kernel: [19071.500857]  [<ffffffff816966e9>] schedule+0x29/0x70
Aug 14 01:09:20 h3oserver2 kernel: [19071.553927]  [<ffffffff81072bae>] worker_thread+0x24e/0x370
Aug 14 01:09:20 h3oserver2 kernel: [19071.563020]  ffff8801736a9000 ffff880173049800 ffff8801711759a0
Aug 14 01:09:20 h3oserver2 kernel: [19071.563040] Call Trace:
Aug 14 01:09:20 h3oserver2 kernel: [19071.644174]  [<ffffffff81072960>] ? manage_workers.isra.29+0x130/0x130
Aug 14 01:09:20 h3oserver2 kernel: [19071.644191]  [<ffffffff81077873>] kthread+0x93/0xa0
Aug 14 01:09:20 h3oserver2 kernel: [19071.716927]  [<ffffffff816a1164>] kernel_thread_helper+0x4/0x10
Aug 14 01:09:20 h3oserver2 kernel: [19071.716957]  [<ffffffff810777e0>] ? kthread_freezable_should_stop+0x70/0x70
Aug 14 01:09:20 h3oserver2 kernel: [19071.716961]  [<ffffffff816a1160>] ? gs_change+0x13/0x13
Aug 14 01:09:20 h3oserver2 kernel: [19071.716963] Code: 
Aug 14 01:09:20 h3oserver2 kernel: [19071.716973]  [<ffffffff81220f12>] ? ext4_num_dirty_pages.isra.43+0xa2/0x200
Aug 14 01:09:20 h3oserver2 kernel: [19071.716976] 48 89 
Aug 14 01:09:20 h3oserver2 kernel: [19071.935852]  [<ffffffff81226f40>] ext4_da_writepages+0x5c0/0x620
Aug 14 01:09:20 h3oserver2 kernel: [19071.935877] fb  [<ffffffff81226980>] ? write_cache_pages_da+0x450/0x450
Aug 14 01:09:20 h3oserver2 kernel: [19071.936018]  [<ffffffff81132ed0>] do_writepages+0x20/0x40
Aug 14 01:09:20 h3oserver2 kernel: [19072.128836]  [<ffffffff811af5ff>] __writeback_single_inode.isra.32+0x3f/0x190
Aug 14 01:09:20 h3oserver2 kernel: [19072.232022] 4c 8b 36 65 48 8b 34 25 00 c7 00 00 66 
Aug 14 01:09:20 h3oserver2 kernel: [19072.232037] 66  [<ffffffff811afd20>] writeback_sb_inodes+0x1a0/0x350
Aug 14 01:09:20 h3oserver2 kernel: [19072.232042] 66 66 90 41 
Aug 14 01:09:20 h3oserver2 kernel: [19072.232063]  [<ffffffff811aff6e>] __writeback_inodes_wb+0x9e/0xd0
Aug 14 01:09:20 h3oserver2 kernel: [19072.232068]  [<ffffffff811b022b>] wb_writeback+0x28b/0x340
Aug 14 01:09:20 h3oserver2 kernel: [19072.232074]  [<ffffffff811b037f>] wb_check_old_data_flush+0x9f/0xb0
Aug 14 01:09:20 h3oserver2 kernel: [19072.232079]  [<ffffffff811b1979>] wb_do_writeback+0x149/0x1d0
Aug 14 01:09:20 h3oserver2 kernel: [19072.232085]  [<ffffffff810636d0>] ? usleep_range+0x50/0x50
Aug 14 01:09:20 h3oserver2 kernel: [19072.255121]  [<ffffffff811b1a8b>] bdi_writeback_thread+0x8b/0x290
Aug 14 01:09:20 h3oserver2 kernel: [19072.255137]  [<ffffffff811b1a00>] ? wb_do_writeback+0x1d0/0x1d0
Aug 14 01:09:20 h3oserver2 kernel: [19072.255144]  [<ffffffff81077873>] kthread+0x93/0xa0
Aug 14 01:09:20 h3oserver2 kernel: [19072.255158]  [<ffffffff816a1164>] kernel_thread_helper+0x4/0x10
Aug 14 01:09:20 h3oserver2 kernel: [19072.255181]  [<ffffffff810777e0>] ? kthread_freezable_should_stop+0x70/0x70
Aug 14 01:09:20 h3oserver2 kernel: [19072.255188]  [<ffffffff816a1160>] ? gs_change+0x13/0x13
Aug 14 01:09:20 h3oserver2 kernel: [19072.255196] Code: c7 44 24 28 00 00 00 00 48 89 df 
Aug 14 01:09:20 h3oserver2 kernel: [19072.255215] d1 48 c1 e8 
Aug 14 01:09:20 h3oserver2 kernel: [19072.255223] e8 56 ac fb ff 66 90 fb 66 66 90 <66> 66 90 65 48 8b 04 25 00 c7 00 00 48 8b 98 e0 01 00 00 48 85 
Aug 14 01:09:20 h3oserver2 kernel: [19072.255302] 0c 48 01 c1 e8 7e fb ff ff 48 89 df e8 26 4f 05 00 4c 89 f0 48 8b 5d e0 4c 8b 65 e8 4c 8b 6d f0 4c 8b 75 f8 c9 c3 90 <55> 48 89 e5 53 48 83 ec 08 66 66 66 66 90 48 89 fb 48 89 f7 48 

Reference
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1035855
Comment 1 Alan 2012-08-14 11:43:28 UTC
This looks like an I/O never completed but it may be a pointer to something else (eg sys_readahead/ext4 interaction). Assigning to the ext4 folk see if they have any thoughts but thats half guesswork 8)
Comment 2 Loris Luise 2012-08-16 13:56:24 UTC
can Another oop dump be useful? Thanks

Aug 16 07:55:28 h3oserver2 kernel: [147758.832781] BUG: soft lockup - CPU#2 stuck for 23s! [netstat:18634]
Aug 16 07:55:28 h3oserver2 kernel: [147759.046380] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi
Aug 16 07:55:28 h3oserver2 kernel: [147759.630769] CPU 2
Aug 16 07:55:28 h3oserver2 kernel: [147759.630772] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi
Aug 16 07:55:28 h3oserver2 kernel: [147759.652655]
Aug 16 07:55:28 h3oserver2 kernel: [147759.678170] Pid: 18634, comm: netstat Not tainted 3.5.1-030501-generic #201208091310 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Aug 16 07:55:28 h3oserver2 kernel: [147759.688391] RIP: 0010:[<ffffffff815d0b5e>]  [<ffffffff815d0b5e>] established_get_next+0xfe/0x190
Aug 16 07:55:28 h3oserver2 kernel: [147759.928918] RSP: 0018:ffff880161f8fde8  EFLAGS: 00010286
Aug 16 07:55:28 h3oserver2 kernel: [147759.928923] RAX: 0000000000060500 RBX: ffff880018139400 RCX: 000000000007ffff
Aug 16 07:55:28 h3oserver2 kernel: [147759.928925] RDX: ffffc90010e82000 RSI: ffffc90011487000 RDI: ffffffff815d0b28
Aug 16 07:55:28 h3oserver2 kernel: [147759.928926] RBP: ffff880161f8fdf8 R08: 0000000000000014 R09: 000000000000ffff
Aug 16 07:55:28 h3oserver2 kernel: [147759.928928] R10: 0000000000000000 R11: 000000000000000f R12: ffff0014ff0a0000
Aug 16 07:55:28 h3oserver2 kernel: [147759.928930] R13: ffff8801ffffffff R14: ffff8801630898b5 R15: 000000000000074b
Aug 16 07:55:28 h3oserver2 kernel: [147759.928984] FS:  00007f5d094bf700(0000) GS:ffff88017fc40000(0000) knlGS:0000000000000000
Aug 16 07:55:28 h3oserver2 kernel: [147759.928987] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 16 07:55:28 h3oserver2 kernel: [147759.928988] CR2: 0000000000415f95 CR3: 000000011ccfb000 CR4: 00000000000006e0
Aug 16 07:55:28 h3oserver2 kernel: [147759.974228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 16 07:55:28 h3oserver2 kernel: [147759.974266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 16 07:55:28 h3oserver2 kernel: [147760.003496] Process netstat (pid: 18634, threadinfo ffff880161f8e000, task ffff880173125c00)
Aug 16 07:55:28 h3oserver2 kernel: [147760.003505] Stack:
Aug 16 07:55:28 h3oserver2 kernel: [147760.003507]  ffff880018139400 ffff8801745c7440 ffff880161f8fe28 ffffffff815d0cef
Aug 16 07:55:28 h3oserver2 kernel: [147760.003514]  ffff880161f8fe28 ffff8801622b0d00 ffff880018139400 ffff8801684c0000
Aug 16 07:55:28 h3oserver2 kernel: [147760.003518]  ffff880161f8fea8 ffffffff811a8d08 ffff88013ffa4530 0002000000000001
Aug 16 07:55:28 h3oserver2 kernel: [147760.003522] Call Trace:
Aug 16 07:55:28 h3oserver2 kernel: [147760.003535]  [<ffffffff815d0cef>] tcp_seq_next+0x3f/0xa0
Aug 16 07:55:28 h3oserver2 kernel: [147760.087408]  [<ffffffff811a8d08>] seq_read+0x238/0x400
Aug 16 07:55:28 h3oserver2 kernel: [147760.087434]  [<ffffffff811a8ad0>] ? seq_put_decimal_ll+0x60/0x60
Aug 16 07:55:28 h3oserver2 kernel: [147760.158567]  [<ffffffff811e7402>] proc_reg_read+0x82/0xc0
Aug 16 07:55:28 h3oserver2 kernel: [147760.178084]  [<ffffffff811873c0>] vfs_read+0xb0/0x180
Aug 16 07:55:28 h3oserver2 kernel: [147760.178108]  [<ffffffff811874da>] sys_read+0x4a/0x90
Aug 16 07:55:28 h3oserver2 kernel: [147760.291955]  [<ffffffff8169fe69>] system_call_fastpath+0x16/0x1b
Aug 16 07:55:28 h3oserver2 kernel: [147760.291966] Code: 0d 58 bb 95 00 48 8b 15 41 bb 95 00 c7 43 1c 00 00 00 00 83 c0 01 39 c8 89 43 18 0f 87 8e 00 00 00 48 63 f0 48 c1 e6 04 48 01 d6 <f6> 06 01 75 73 23 05 2b bb 95 00 48 8d 3c 85 00 00 00 00 48 03
Comment 3 Loris Luise 2012-08-24 10:05:12 UTC
I made 2 modifications on the server running linux and currently no more soft lockup has happened

1) removed irqbalance daemon from ubuntu server
2) unchecked "Synchronize guest time with host" from VM setting (VMWare esx)

Modification 1) most probably solved the problem.
Comment 4 Alain Brossard 2014-08-22 18:30:42 UTC
Hi,
  I had the same issue with Ubuntu 14.04 + Xen-4.4 and the same solution worked: remove irqbalance. Though I suspect it could be a question of lock timing and this problem could come back.

3.13.0-32-generic
Aug 16 17:11:42 mytv kernel: [   59.284157] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631]
Aug 16 17:11:42 mytv kernel: [   59.284160] Modules linked in: xen_gntdev xen_evtchn xenfs xen_privcmd nfsv
3 nfsv4 bridge stp llc dm_crypt nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 eeepc_wmi a
sus_wmi sparse_keymap video mxm_wmi snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
 snd_hda_intel snd_hda_codec snd_hwdep snd_pcm aesni_intel aes_x86_64 lrw gf128mul snd_page_alloc glue_help
er ablk_helper cryptd snd_seq_midi snd_seq_midi_event serio_raw snd_rawmidi fam15h_power k10temp edac_core 
edac_mce_amd snd_seq joydev snd_seq_device snd_timer snd sp5100_tco i2c_piix4 soundcore nvidia(POF) parport
_pc ppdev lp parport wmi mac_hid dm_mirror dm_region_hash dm_log hid_generic usbhid hid psmouse e1000e r816
9 mii ptp ahci pps_core libahci
Aug 16 17:11:42 mytv kernel: [   59.284210] CPU: 2 PID: 3631 Comm: Xorg Tainted: PF          O 3.13.0-32-ge
neric #57-Ubuntu
Aug 16 17:11:42 mytv kernel: [   59.284212] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SA
BERTOOTH 990FX R2.0, BIOS 2501 04/08/2014
Aug 16 17:11:42 mytv kernel: [   59.284216] task: ffff880425155fc0 ti: ffff8804265f6000 task.ti: ffff880426
5f6000
Aug 16 17:11:42 mytv kernel: [   59.284218] RIP: e030:[<ffffffffa077d0da>]  [<ffffffffa077d0da>] rm_shutdow
n_gvi_device+0x187/0x295 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284315] RSP: e02b:ffff8804265f78d0  EFLAGS: 00000282
Aug 16 17:11:42 mytv kernel: [   59.284317] RAX: ffff8800000dc97c RBX: ffff880415192e24 RCX: ffffffffa0b616
70
Aug 16 17:11:42 mytv kernel: [   59.284318] RDX: 00000000000000ff RSI: 00000000000dc97c RDI: 00000000000dc9
7c
Aug 16 17:11:42 mytv kernel: [   59.284319] RBP: ffff880415192e20 R08: 0000000000000001 R09: ffffffffa0b6d1
40
Aug 16 17:11:42 mytv kernel: [   59.284320] R10: ffff880425ace808 R11: ffffffffa079c8db R12: ffff880415192e
28
Aug 16 17:11:42 mytv kernel: [   59.284321] R13: ffff880415192e2c R14: 0000000000009bed R15: ffff880415192e
90
Aug 16 17:11:42 mytv kernel: [   59.284325] FS:  00007fc4235f79c0(0000) GS:ffff880447280000(0000) knlGS:000
0000000000000
Aug 16 17:11:42 mytv kernel: [   59.284327] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 16 17:11:42 mytv kernel: [   59.284328] CR2: 00007fd1e4edcff0 CR3: 0000000425a8a000 CR4: 00000000000406
60
Aug 16 17:11:42 mytv kernel: [   59.284329] Stack:
Aug 16 17:11:42 mytv kernel: [   59.284330]  ffffffffa079c8db ffffffffa077f9da ffff8804151b8008 00000000000
0cfde
Aug 16 17:11:42 mytv kernel: [   59.284333]  ffff880415192e8c ffffffffa0789d58 ffff8804265f7910 ffff8804151
b8008
Aug 16 17:11:42 mytv kernel: [   59.284336]  000000000000cfde ffffffffa077f976 ffff8804151b8008 ffffffffa07
7d639
Aug 16 17:11:42 mytv kernel: [   59.284338] Call Trace:
Aug 16 17:11:42 mytv kernel: [   59.284384]  [<ffffffffa079c8db>] ? os_free_mem+0x1b/0x30 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284428]  [<ffffffffa077f9da>] ? _nv007991rm+0x33/0x56 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284472]  [<ffffffffa0789d58>] ? _nv019222rm+0x970b/0xcee3 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284517]  [<ffffffffa077f976>] ? _nv001242rm+0x83/0xa4 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284560]  [<ffffffffa077d639>] ? _nv014418rm+0x1f0/0xc97 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284605]  [<ffffffffa076ad9e>] ? _nv014859rm+0xce/0x3da [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284653]  [<ffffffffa076b176>] ? _nv014923rm+0x4c/0x59 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284716]  [<ffffffffa02e083c>] ? _nv018571rm+0x35/0x79 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284765]  [<ffffffffa072e0ab>] ? _nv018447rm+0x46/0xbf [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284827]  [<ffffffffa02a3ce1>] ? _nv012729rm+0x551/0x13cb [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284890]  [<ffffffffa02c05a5>] ? _nv017460rm+0x405/0x42f [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.284954]  [<ffffffffa03111f5>] ? _nv005690rm+0x1e5/0x1f3 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285033]  [<ffffffffa03df9ee>] ? _nv005044rm+0x9a/0xc4 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285129]  [<ffffffffa0539d22>] ? _nv004050rm+0x8858/0xaef1 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285225]  [<ffffffffa0538573>] ? _nv004050rm+0x70a9/0xaef1 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285264]  [<ffffffffa0166e80>] ? _nv010039rm+0x25/0x40 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285310]  [<ffffffffa0779558>] ? _nv015014rm+0x808/0x982 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285354]  [<ffffffffa077a538>] ? _nv001097rm+0x483/0x6b8 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285398]  [<ffffffffa0772aa4>] ? rm_init_adapter+0xac/0x146 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285443]  [<ffffffffa0792b71>] ? nv_kern_open+0x191/0x810 [nvidia]
Aug 16 17:11:42 mytv kernel: [   59.285448]  [<ffffffff811c14ef>] ? chrdev_open+0x9f/0x1d0
Aug 16 17:11:42 mytv kernel: [   59.285451]  [<ffffffff811ba033>] ? do_dentry_open+0x233/0x2e0
Aug 16 17:11:42 mytv kernel: [   59.285453]  [<ffffffff811c1450>] ? cdev_put+0x30/0x30
Aug 16 17:11:42 mytv kernel: [   59.285455]  [<ffffffff811ba369>] ? vfs_open+0x49/0x50
Aug 16 17:11:42 mytv kernel: [   59.285464]  [<ffffffff811c8f04>] ? do_last+0x554/0x1200
Aug 16 17:11:42 mytv kernel: [   59.285467]  [<ffffffff81311bdb>] ? apparmor_file_alloc_security+0x5b/0x180
Aug 16 17:11:42 mytv kernel: [   59.285470]  [<ffffffff811cc38b>] ? path_openat+0xbb/0x640
Aug 16 17:11:42 mytv kernel: [   59.285473]  [<ffffffff812d3d5e>] ? security_inode_alloc+0x1e/0x20
Aug 16 17:11:42 mytv kernel: [   59.285476]  [<ffffffff811e2988>] ? simple_xattr_get+0x68/0xb0
Aug 16 17:11:42 mytv kernel: [   59.285478]  [<ffffffff811cd76a>] ? do_filp_open+0x3a/0x90
Aug 16 17:11:42 mytv kernel: [   59.285481]  [<ffffffff811da527>] ? __alloc_fd+0xa7/0x130
Aug 16 17:11:42 mytv kernel: [   59.285484]  [<ffffffff811bbe89>] ? do_sys_open+0x129/0x280
Aug 16 17:11:42 mytv kernel: [   59.285487]  [<ffffffff81020d45>] ? syscall_trace_enter+0x145/0x250
Aug 16 17:11:42 mytv kernel: [   59.285489]  [<ffffffff811bbffe>] ? SyS_open+0x1e/0x20
Aug 16 17:11:42 mytv kernel: [   59.285492]  [<ffffffff8172c87f>] ? tracesys+0xe1/0xe6
Aug 16 17:11:42 mytv kernel: [   59.285493] Code: c3 e8 e2 28 00 00 b8 00 00 00 00 48 83 c4 08 c3 48 83 ec 
08 be 01 00 00 00 e8 96 ff ff ff ba 00 00 00 00 48 85 c0 74 03 0f b6 10 <89> d0 48 83 c4 08 c3 48 83 ec 08 
be 02 00 00 00 e8 74 ff ff ff 
Aug 16 17:12:10 mytv kernel: [   87.283580] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631]
 and loop...  Xorg never starts properly though ssh through works. All attempts to shutdown cleanly hang as does tcpdump if it matters.