Bug 45941
Summary: | BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:1:64] BUG: soft lockup - CPU#4 stuck for 22s! [flush-252:0:384] | ||
---|---|---|---|
Product: | File System | Reporter: | Loris Luise (loris.luise) |
Component: | ext4 | Assignee: | fs_ext4 (fs_ext4) |
Status: | RESOLVED OBSOLETE | ||
Severity: | normal | CC: | alan, vmware |
Priority: | P1 | ||
Hardware: | x86-64 | ||
OS: | Linux | ||
URL: | http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.1-quantal/ | ||
Kernel Version: | 3.5.1 | Subsystem: | |
Regression: | No | Bisected commit-id: |
Description
Loris Luise
2012-08-14 08:02:49 UTC
This looks like an I/O never completed but it may be a pointer to something else (eg sys_readahead/ext4 interaction). Assigning to the ext4 folk see if they have any thoughts but thats half guesswork 8) can Another oop dump be useful? Thanks Aug 16 07:55:28 h3oserver2 kernel: [147758.832781] BUG: soft lockup - CPU#2 stuck for 23s! [netstat:18634] Aug 16 07:55:28 h3oserver2 kernel: [147759.046380] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi Aug 16 07:55:28 h3oserver2 kernel: [147759.630769] CPU 2 Aug 16 07:55:28 h3oserver2 kernel: [147759.630772] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs xt_multiport pcnet32 ext2 xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 dm_crypt ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables ppdev nf_conntrack_netbios_ns nf_conntrack_broadcast vmw_balloon nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack_ftp coretemp nf_conntrack microcode psmouse iptable_filter ip_tables x_tables serio_raw parport_pc lp i2c_piix4 mac_hid shpchp parport vmxnet3 floppy vmw_pvscsi Aug 16 07:55:28 h3oserver2 kernel: [147759.652655] Aug 16 07:55:28 h3oserver2 kernel: [147759.678170] Pid: 18634, comm: netstat Not tainted 3.5.1-030501-generic #201208091310 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Aug 16 07:55:28 h3oserver2 kernel: [147759.688391] RIP: 0010:[<ffffffff815d0b5e>] [<ffffffff815d0b5e>] established_get_next+0xfe/0x190 Aug 16 07:55:28 h3oserver2 kernel: [147759.928918] RSP: 0018:ffff880161f8fde8 EFLAGS: 00010286 Aug 16 07:55:28 h3oserver2 kernel: [147759.928923] RAX: 0000000000060500 RBX: ffff880018139400 RCX: 000000000007ffff Aug 16 07:55:28 h3oserver2 kernel: [147759.928925] RDX: ffffc90010e82000 RSI: ffffc90011487000 RDI: ffffffff815d0b28 Aug 16 07:55:28 h3oserver2 kernel: [147759.928926] RBP: ffff880161f8fdf8 R08: 0000000000000014 R09: 000000000000ffff Aug 16 07:55:28 h3oserver2 kernel: [147759.928928] R10: 0000000000000000 R11: 000000000000000f R12: ffff0014ff0a0000 Aug 16 07:55:28 h3oserver2 kernel: [147759.928930] R13: ffff8801ffffffff R14: ffff8801630898b5 R15: 000000000000074b Aug 16 07:55:28 h3oserver2 kernel: [147759.928984] FS: 00007f5d094bf700(0000) GS:ffff88017fc40000(0000) knlGS:0000000000000000 Aug 16 07:55:28 h3oserver2 kernel: [147759.928987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 16 07:55:28 h3oserver2 kernel: [147759.928988] CR2: 0000000000415f95 CR3: 000000011ccfb000 CR4: 00000000000006e0 Aug 16 07:55:28 h3oserver2 kernel: [147759.974228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 16 07:55:28 h3oserver2 kernel: [147759.974266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 16 07:55:28 h3oserver2 kernel: [147760.003496] Process netstat (pid: 18634, threadinfo ffff880161f8e000, task ffff880173125c00) Aug 16 07:55:28 h3oserver2 kernel: [147760.003505] Stack: Aug 16 07:55:28 h3oserver2 kernel: [147760.003507] ffff880018139400 ffff8801745c7440 ffff880161f8fe28 ffffffff815d0cef Aug 16 07:55:28 h3oserver2 kernel: [147760.003514] ffff880161f8fe28 ffff8801622b0d00 ffff880018139400 ffff8801684c0000 Aug 16 07:55:28 h3oserver2 kernel: [147760.003518] ffff880161f8fea8 ffffffff811a8d08 ffff88013ffa4530 0002000000000001 Aug 16 07:55:28 h3oserver2 kernel: [147760.003522] Call Trace: Aug 16 07:55:28 h3oserver2 kernel: [147760.003535] [<ffffffff815d0cef>] tcp_seq_next+0x3f/0xa0 Aug 16 07:55:28 h3oserver2 kernel: [147760.087408] [<ffffffff811a8d08>] seq_read+0x238/0x400 Aug 16 07:55:28 h3oserver2 kernel: [147760.087434] [<ffffffff811a8ad0>] ? seq_put_decimal_ll+0x60/0x60 Aug 16 07:55:28 h3oserver2 kernel: [147760.158567] [<ffffffff811e7402>] proc_reg_read+0x82/0xc0 Aug 16 07:55:28 h3oserver2 kernel: [147760.178084] [<ffffffff811873c0>] vfs_read+0xb0/0x180 Aug 16 07:55:28 h3oserver2 kernel: [147760.178108] [<ffffffff811874da>] sys_read+0x4a/0x90 Aug 16 07:55:28 h3oserver2 kernel: [147760.291955] [<ffffffff8169fe69>] system_call_fastpath+0x16/0x1b Aug 16 07:55:28 h3oserver2 kernel: [147760.291966] Code: 0d 58 bb 95 00 48 8b 15 41 bb 95 00 c7 43 1c 00 00 00 00 83 c0 01 39 c8 89 43 18 0f 87 8e 00 00 00 48 63 f0 48 c1 e6 04 48 01 d6 <f6> 06 01 75 73 23 05 2b bb 95 00 48 8d 3c 85 00 00 00 00 48 03 I made 2 modifications on the server running linux and currently no more soft lockup has happened 1) removed irqbalance daemon from ubuntu server 2) unchecked "Synchronize guest time with host" from VM setting (VMWare esx) Modification 1) most probably solved the problem. Hi, I had the same issue with Ubuntu 14.04 + Xen-4.4 and the same solution worked: remove irqbalance. Though I suspect it could be a question of lock timing and this problem could come back. 3.13.0-32-generic Aug 16 17:11:42 mytv kernel: [ 59.284157] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631] Aug 16 17:11:42 mytv kernel: [ 59.284160] Modules linked in: xen_gntdev xen_evtchn xenfs xen_privcmd nfsv 3 nfsv4 bridge stp llc dm_crypt nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache nls_iso8859_1 eeepc_wmi a sus_wmi sparse_keymap video mxm_wmi snd_hda_codec_realtek crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hwdep snd_pcm aesni_intel aes_x86_64 lrw gf128mul snd_page_alloc glue_help er ablk_helper cryptd snd_seq_midi snd_seq_midi_event serio_raw snd_rawmidi fam15h_power k10temp edac_core edac_mce_amd snd_seq joydev snd_seq_device snd_timer snd sp5100_tco i2c_piix4 soundcore nvidia(POF) parport _pc ppdev lp parport wmi mac_hid dm_mirror dm_region_hash dm_log hid_generic usbhid hid psmouse e1000e r816 9 mii ptp ahci pps_core libahci Aug 16 17:11:42 mytv kernel: [ 59.284210] CPU: 2 PID: 3631 Comm: Xorg Tainted: PF O 3.13.0-32-ge neric #57-Ubuntu Aug 16 17:11:42 mytv kernel: [ 59.284212] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SA BERTOOTH 990FX R2.0, BIOS 2501 04/08/2014 Aug 16 17:11:42 mytv kernel: [ 59.284216] task: ffff880425155fc0 ti: ffff8804265f6000 task.ti: ffff880426 5f6000 Aug 16 17:11:42 mytv kernel: [ 59.284218] RIP: e030:[<ffffffffa077d0da>] [<ffffffffa077d0da>] rm_shutdow n_gvi_device+0x187/0x295 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284315] RSP: e02b:ffff8804265f78d0 EFLAGS: 00000282 Aug 16 17:11:42 mytv kernel: [ 59.284317] RAX: ffff8800000dc97c RBX: ffff880415192e24 RCX: ffffffffa0b616 70 Aug 16 17:11:42 mytv kernel: [ 59.284318] RDX: 00000000000000ff RSI: 00000000000dc97c RDI: 00000000000dc9 7c Aug 16 17:11:42 mytv kernel: [ 59.284319] RBP: ffff880415192e20 R08: 0000000000000001 R09: ffffffffa0b6d1 40 Aug 16 17:11:42 mytv kernel: [ 59.284320] R10: ffff880425ace808 R11: ffffffffa079c8db R12: ffff880415192e 28 Aug 16 17:11:42 mytv kernel: [ 59.284321] R13: ffff880415192e2c R14: 0000000000009bed R15: ffff880415192e 90 Aug 16 17:11:42 mytv kernel: [ 59.284325] FS: 00007fc4235f79c0(0000) GS:ffff880447280000(0000) knlGS:000 0000000000000 Aug 16 17:11:42 mytv kernel: [ 59.284327] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 16 17:11:42 mytv kernel: [ 59.284328] CR2: 00007fd1e4edcff0 CR3: 0000000425a8a000 CR4: 00000000000406 60 Aug 16 17:11:42 mytv kernel: [ 59.284329] Stack: Aug 16 17:11:42 mytv kernel: [ 59.284330] ffffffffa079c8db ffffffffa077f9da ffff8804151b8008 00000000000 0cfde Aug 16 17:11:42 mytv kernel: [ 59.284333] ffff880415192e8c ffffffffa0789d58 ffff8804265f7910 ffff8804151 b8008 Aug 16 17:11:42 mytv kernel: [ 59.284336] 000000000000cfde ffffffffa077f976 ffff8804151b8008 ffffffffa07 7d639 Aug 16 17:11:42 mytv kernel: [ 59.284338] Call Trace: Aug 16 17:11:42 mytv kernel: [ 59.284384] [<ffffffffa079c8db>] ? os_free_mem+0x1b/0x30 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284428] [<ffffffffa077f9da>] ? _nv007991rm+0x33/0x56 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284472] [<ffffffffa0789d58>] ? _nv019222rm+0x970b/0xcee3 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284517] [<ffffffffa077f976>] ? _nv001242rm+0x83/0xa4 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284560] [<ffffffffa077d639>] ? _nv014418rm+0x1f0/0xc97 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284605] [<ffffffffa076ad9e>] ? _nv014859rm+0xce/0x3da [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284653] [<ffffffffa076b176>] ? _nv014923rm+0x4c/0x59 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284716] [<ffffffffa02e083c>] ? _nv018571rm+0x35/0x79 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284765] [<ffffffffa072e0ab>] ? _nv018447rm+0x46/0xbf [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284827] [<ffffffffa02a3ce1>] ? _nv012729rm+0x551/0x13cb [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284890] [<ffffffffa02c05a5>] ? _nv017460rm+0x405/0x42f [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.284954] [<ffffffffa03111f5>] ? _nv005690rm+0x1e5/0x1f3 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285033] [<ffffffffa03df9ee>] ? _nv005044rm+0x9a/0xc4 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285129] [<ffffffffa0539d22>] ? _nv004050rm+0x8858/0xaef1 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285225] [<ffffffffa0538573>] ? _nv004050rm+0x70a9/0xaef1 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285264] [<ffffffffa0166e80>] ? _nv010039rm+0x25/0x40 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285310] [<ffffffffa0779558>] ? _nv015014rm+0x808/0x982 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285354] [<ffffffffa077a538>] ? _nv001097rm+0x483/0x6b8 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285398] [<ffffffffa0772aa4>] ? rm_init_adapter+0xac/0x146 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285443] [<ffffffffa0792b71>] ? nv_kern_open+0x191/0x810 [nvidia] Aug 16 17:11:42 mytv kernel: [ 59.285448] [<ffffffff811c14ef>] ? chrdev_open+0x9f/0x1d0 Aug 16 17:11:42 mytv kernel: [ 59.285451] [<ffffffff811ba033>] ? do_dentry_open+0x233/0x2e0 Aug 16 17:11:42 mytv kernel: [ 59.285453] [<ffffffff811c1450>] ? cdev_put+0x30/0x30 Aug 16 17:11:42 mytv kernel: [ 59.285455] [<ffffffff811ba369>] ? vfs_open+0x49/0x50 Aug 16 17:11:42 mytv kernel: [ 59.285464] [<ffffffff811c8f04>] ? do_last+0x554/0x1200 Aug 16 17:11:42 mytv kernel: [ 59.285467] [<ffffffff81311bdb>] ? apparmor_file_alloc_security+0x5b/0x180 Aug 16 17:11:42 mytv kernel: [ 59.285470] [<ffffffff811cc38b>] ? path_openat+0xbb/0x640 Aug 16 17:11:42 mytv kernel: [ 59.285473] [<ffffffff812d3d5e>] ? security_inode_alloc+0x1e/0x20 Aug 16 17:11:42 mytv kernel: [ 59.285476] [<ffffffff811e2988>] ? simple_xattr_get+0x68/0xb0 Aug 16 17:11:42 mytv kernel: [ 59.285478] [<ffffffff811cd76a>] ? do_filp_open+0x3a/0x90 Aug 16 17:11:42 mytv kernel: [ 59.285481] [<ffffffff811da527>] ? __alloc_fd+0xa7/0x130 Aug 16 17:11:42 mytv kernel: [ 59.285484] [<ffffffff811bbe89>] ? do_sys_open+0x129/0x280 Aug 16 17:11:42 mytv kernel: [ 59.285487] [<ffffffff81020d45>] ? syscall_trace_enter+0x145/0x250 Aug 16 17:11:42 mytv kernel: [ 59.285489] [<ffffffff811bbffe>] ? SyS_open+0x1e/0x20 Aug 16 17:11:42 mytv kernel: [ 59.285492] [<ffffffff8172c87f>] ? tracesys+0xe1/0xe6 Aug 16 17:11:42 mytv kernel: [ 59.285493] Code: c3 e8 e2 28 00 00 b8 00 00 00 00 48 83 c4 08 c3 48 83 ec 08 be 01 00 00 00 e8 96 ff ff ff ba 00 00 00 00 48 85 c0 74 03 0f b6 10 <89> d0 48 83 c4 08 c3 48 83 ec 08 be 02 00 00 00 e8 74 ff ff ff Aug 16 17:12:10 mytv kernel: [ 87.283580] BUG: soft lockup - CPU#2 stuck for 23s! [Xorg:3631] and loop... Xorg never starts properly though ssh through works. All attempts to shutdown cleanly hang as does tcpdump if it matters. |