Bug 79231

Summary: INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 3, t=15020 jiffies, g=996355, c=996354, q=0)
Product: Other Reporter: Loris Luise (loris.luise)
Component: OtherAssignee: other_other
Status: NEW ---    
Severity: normal    
Priority: P1    
Hardware: All   
OS: Linux   
URL: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.15.1-utopic/
Kernel Version: 3.15.1-031501-generic Subsystem:
Regression: No Bisected commit-id:
Attachments: Additional infos about errors

Description Loris Luise 2014-06-30 09:04:15 UTC 
Created attachment 141461 [details]
Additional infos about errors

Hello,

on a ESXi 4.1 HP ML350G5 host,
I continue getting this sort of errors using linux kernel.

I tried all sorts of versions from UBUNTU mainline...
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.15.1-utopic/


Please, can someone get me if the right direction?

I don't event understand where is locaed the aread that is continuing to generate this kind of errors.

After that I have to rebot the system, because the system wil get nearly unresposive and even the reboot process is getting minutes to perfor.

I'm attaching all infos I collected...

many thanks.

Dif tor eh smusma


[410224.149680] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 3, t=15020 jiffies, g=996355, c=996354, q=0)
[410226.013662] INFO: Stall ended before state dump start
[410233.550339] BUG: soft lockup - CPU#1 stuck for 67s! [atop:25014]
[410233.594140] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs cpuid tcp_htcp sch_codel sch_fq_codel xt_multiport xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables ppdev vmw_balloon irda coretemp serio_raw crc_ccitt parport_pc vmwgfx ttm drm vmw_vmci mac_hid shpchp i2c_piix4 lp parport xfs libcrc32c psmouse floppy vmw_pvscsi vmxnet3
[410236.060588] CPU: 1 PID: 25014 Comm: atop Not tainted 3.15.1-031501-generic #201406161841
[410236.060596] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410236.060599] task: ffff880028dd1930 ti: ffff880024768000 task.ti: ffff880024768000
[410236.060602] RIP: 0010:[<ffffffff81071fa0>]  [<ffffffff81071fa0>] __do_softirq+0x90/0x330
[410236.749024] RSP: 0018:ffff88013fc23f08  EFLAGS: 00000206
[410236.749028] RAX: 0000000000000000 RBX: ffff880028dd1930 RCX: ffff88002476bfd8
[410236.749029] RDX: 0000000000000001 RSI: ffff88013fc23e28 RDI: ffff880028dd1e44
[410236.749031] RBP: ffff88013fc23f68 R08: 000000000000dfbc R09: 0000000000000001
[410236.749032] R10: 0000000000000a7d R11: 0000000000000002 R12: ffff88013fc23e78
[410236.749033] R13: ffffffff817853dd R14: ffff88013fc23f68 R15: 0000000000000000
[410236.749041] FS:  00007f6d6c179740(0000) GS:ffff88013fc20000(0000) knlGS:0000000000000000
[410236.749042] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[410236.749044] CR2: 00007f6b980c8000 CR3: 000000002474c000 CR4: 00000000000006e0
[410236.774877] Stack:
[410236.774883]  ffff88002476bfd8 00000001061a2c57 0000000a00406100 0000000000000001
[410236.774886]  ffff88013fc23f48 ffffffff810a5652 ffff88013fc2e960 0000000000000000
[410236.774889]  ffff880138454b90 ffffffff81828c60 ffffffff81829278 0000000000000006
[410236.774891] Call Trace:
[410236.806600]  <IRQ>
[410236.823547]  [<ffffffff810a5652>] ? __vtime_account_system+0x32/0x40
[410236.888703]  [<ffffffff8107254e>] irq_exit+0x11e/0x140
[410237.203751]  [<ffffffff81786a9a>] smp_apic_timer_interrupt+0x4a/0x60
[410237.203769]  [<ffffffff817853dd>] apic_timer_interrupt+0x6d/0x80
[410237.203770]  <EOI>
[410237.203773]  [<ffffffff81237d10>] ? proc_alloc_inode+0x20/0xb0
[410237.666280]  [<ffffffff8177afa7>] ? _raw_spin_lock+0x37/0x50
[410237.768396]  [<ffffffff811ebc0d>] inode_sb_list_add+0x1d/0x70
[410237.768412]  [<ffffffff811ee22d>] new_inode+0x2d/0x40
[410237.768416]  [<ffffffff811ea6e4>] ? __d_alloc+0x34/0x180
[410237.777696]  [<ffffffff8123c455>] proc_pid_make_inode+0x25/0x110
[410237.777704]  [<ffffffff8123c634>] proc_pident_instantiate+0x24/0xd0
[410237.777706]  [<ffffffff8123c77b>] proc_pident_lookup+0x9b/0xe0
[410237.777708]  [<ffffffff8123c81a>] proc_tgid_base_lookup+0x1a/0x20
[410237.955919]  [<ffffffff811dc03d>] lookup_real+0x1d/0x60
[410237.955986]  [<ffffffff811e1578>] lookup_open+0xc8/0x1d0
[410237.955990]  [<ffffffff811e1954>] do_last+0x2d4/0x7b0
[410238.046444]  [<ffffffff812ef976>] ? security_file_alloc+0x16/0x20
[410238.046459]  [<ffffffff811e1ef4>] path_openat+0xc4/0x4c0
[410238.099704]  [<ffffffff81186e44>] ? tlb_flush_mmu_free+0x34/0x60
[410238.099725]  [<ffffffff81187844>] ? tlb_flush_mmu+0x24/0x30
[410238.099729]  [<ffffffff811e2c63>] do_filp_open+0x43/0xa0
[410238.099733]  [<ffffffff811ef9cd>] ? __alloc_fd+0xcd/0x120
[410238.103425]  [<ffffffff811d1742>] do_sys_open+0x132/0x220
[410238.272278]  [<ffffffff810220dd>] ? syscall_trace_leave+0xdd/0x150
[410238.272296]  [<ffffffff811d184e>] SyS_open+0x1e/0x20
[410238.272300]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410238.272302] Code: 00 00 65 8b 14 25 64 b0 00 00 c7 45 b4 0a 00 00 00 89 55 b8 48 89 4d a0 0f 1f 00 65 c7 04 25 40 19 01 00 00 00 00 00 fb 66 66 90 <66> 66 90 41 bf ff ff ff ff 45 0f bc fe 41 83 c7 01 0f 84 88 00
[410286.360708] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=15020 jiffies, g=996356, c=996355, q=0)
[410286.803644] sending NMI to all CPUs:
[410286.926271] NMI backtrace for cpu 3
[410286.926302] CPU: 3 PID: 2400 Comm: postgres Not tainted 3.15.1-031501-generic #201406161841
[410286.926304] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410286.926306] task: ffff8800b8ae64c0 ti: ffff880138b04000 task.ti: ffff880138b04000
[410286.926308] RIP: 0010:[<ffffffff8104a310>]  [<ffffffff8104a310>] default_send_IPI_mask_sequence_phys+0xa0/0xe0
[410287.139517] RSP: 0018:ffff88013fc63b70  EFLAGS: 00000046
[410287.139521] RAX: 0000000000000400 RBX: 0000000000000002 RCX: 0000000003000000
[410287.139523] RDX: 0000000000000003 RSI: 0000000000000100 RDI: 0000000000000300
[410287.139524] RBP: ffff88013fc63bb0 R08: ffffffff81d1bbc0 R09: 0000000000000000
[410287.139525] R10: 00000000000005a8 R11: 3a73555043206c6c R12: 000000000000b06a
[410287.139527] R13: ffffffff81d1bbc0 R14: 0000000000000400 R15: 0000000000000082
[410287.139532] FS:  00007f52d5e11780(0000) GS:ffff88013fc60000(0000) knlGS:0000000000000000
[410287.139534] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[410287.139535] CR2: 00007f52d5e30000 CR3: 0000000138aed000 CR4: 00000000000006e0
[410287.139547] Stack:
[410287.139549]  ffffffff00000003 0000000000000003 ffff88013fc63bd0 0000000000002710
[410287.139554]  0000000000000001 0000000000000000 ffffffff81d1bbe0 0000000000000087
[410287.139557]  ffff88013fc63bc0 ffffffff8104eb97 ffff88013fc63be0 ffffffff8104a4e4
[410287.139559] Call Trace:
[410287.139564]  <IRQ>
[410287.139570]  [<ffffffff8104eb97>] physflat_send_IPI_all+0x17/0x20
[410287.139581]  [<ffffffff8104a4e4>] arch_trigger_all_cpu_backtrace+0x74/0xb0
[410287.161341]  [<ffffffff810d5397>] print_other_cpu_stall+0x1f7/0x230
[410287.161353]  [<ffffffff810d5467>] check_cpu_stall.isra.50+0x97/0xb0
[410287.161356]  [<ffffffff810d54b8>] __rcu_pending+0x38/0x1d0
[410287.161359]  [<ffffffff810d635d>] rcu_check_callbacks+0xdd/0x190
[410287.192078]  [<ffffffff8107bd08>] update_process_times+0x48/0x80
[410287.419162]  [<ffffffff810e1c83>] tick_sched_handle.isra.14+0x33/0x70
[410287.419177]  [<ffffffff810e1dac>] tick_sched_timer+0x4c/0x80
[410287.451605]  [<ffffffff81094726>] __run_hrtimer+0x76/0x230
[410287.451626]  [<ffffffff810e1d60>] ? tick_nohz_handler+0xa0/0xa0
[410287.451631]  [<ffffffff81094eef>] hrtimer_interrupt+0xff/0x260
[410287.451756]  [<ffffffff8104875b>] local_apic_timer_interrupt+0x3b/0x60
[410287.451774]  [<ffffffff81786a95>] smp_apic_timer_interrupt+0x45/0x60
[410287.451777]  [<ffffffff817853dd>] apic_timer_interrupt+0x6d/0x80
[410287.451783]  [<ffffffff81071fa0>] ? __do_softirq+0x90/0x330
[410287.451786]  [<ffffffff810721b6>] ? __do_softirq+0x2a6/0x330
[410287.451792]  [<ffffffff810a5652>] ? __vtime_account_system+0x32/0x40
[410287.451796]  [<ffffffff8107254e>] irq_exit+0x11e/0x140
[410287.451798]  [<ffffffff81786a9a>] smp_apic_timer_interrupt+0x4a/0x60
[410287.451800]  [<ffffffff817853dd>] apic_timer_interrupt+0x6d/0x80
[410287.451802]  <EOI>
[410287.451820]  [<ffffffff8177afa0>] ? _raw_spin_lock+0x30/0x50
[410287.451833]  [<ffffffff811ecfdf>] evict+0x6f/0x190
[410287.451837]  [<ffffffff811ed8c0>] iput_final+0xe0/0x170
[410287.451839]  [<ffffffff811ed98e>] iput+0x3e/0x50
[410287.451844]  [<ffffffff811e7c88>] dentry_iput+0xa8/0xf0
[410287.451847]  [<ffffffff811e8c96>] __dentry_kill+0xb6/0x160
[410287.451849]  [<ffffffff811e92c1>] dput+0xb1/0x100
[410287.451852]  [<ffffffff811e0ae8>] SYSC_renameat2+0x428/0x4c0
[410287.643918]  [<ffffffff8111cdfc>] ? acct_account_cputime+0x1c/0x20
[410287.643943]  [<ffffffff8111cdfc>] ? acct_account_cputime+0x1c/0x20
[410287.643948]  [<ffffffff810a5469>] ? account_user_time+0x99/0xb0
[410287.643950]  [<ffffffff810a5a9d>] ? vtime_account_user+0x5d/0x70
[410287.643972]  [<ffffffff81021ee5>] ? syscall_trace_enter+0x165/0x280
[410287.643974]  [<ffffffff810220dd>] ? syscall_trace_leave+0xdd/0x150
[410287.643978]  [<ffffffff811e34ee>] SyS_renameat2+0xe/0x10
[410287.643980]  [<ffffffff811e352e>] SyS_rename+0x1e/0x20
[410287.643985]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410287.643987] Code: 0f 1f 00 f3 90 8b 04 25 00 73 5f ff f6 c4 10 75 f2 c1 e1 18 89 0c 25 10 73 5f ff 83 fb 02 44 89 f0 0f 45 c3 89 04 25 00 73 5f ff <eb> 9e 4c 89 ff 57 9d 66 66 90 66 90 48 83 c4 18 5b 41 5c 41 5d
[410287.644051] NMI backtrace for cpu 1
[410287.644063] CPU: 1 PID: 25014 Comm: atop Not tainted 3.15.1-031501-generic #201406161841
[410287.644064] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410287.644067] task: ffff880028dd1930 ti: ffff880024768000 task.ti: ffff880024768000
[410287.644069] RIP: 0010:[<ffffffff8177afad>]  [<ffffffff8177afad>] _raw_spin_lock+0x3d/0x50
[410287.644080] RSP: 0018:ffff88002476baf8  EFLAGS: 00000202
[410287.644081] RAX: 000000000000053b RBX: ffff8800b8f2e128 RCX: 0000000000005878
[410287.644083] RDX: 000000000000587a RSI: 000000000000587a RDI: ffffffff81c0b400
[410287.644084] RBP: ffff88002476baf8 R08: ffff88013fc37c30 R09: 0000000000000001
[410287.644085] R10: ffffc90000014000 R11: ffffc90000014000 R12: ffff880138454b90
[410287.644087] R13: ffffffff81828c60 R14: ffffffff81829278 R15: 0000000000000006
[410287.644091] FS:  00007f6d6c179740(0000) GS:ffff88013fc20000(0000) knlGS:0000000000000000
[410287.644093] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[410287.644094] CR2: 00007f6b980c8000 CR3: 000000002474c000 CR4: 00000000000006e0
[410287.644103] Stack:
[410287.644105]  ffff88002476bb18 ffffffff811ebc0d ffff88001b6b46c0 ffff8800b8f2e128
[410287.644108]  ffff88002476bb38 ffffffff811ee22d ffffffff811ea6e4 ffff88001b6b46c0
[410287.644110]  ffff88002476bb68 ffffffff8123c455 ffff88001b6b46c0 ffffffff81828c60
[410287.644112] Call Trace:
[410287.644120]  [<ffffffff811ebc0d>] inode_sb_list_add+0x1d/0x70
[410287.644123]  [<ffffffff811ee22d>] new_inode+0x2d/0x40
[410287.644126]  [<ffffffff811ea6e4>] ? __d_alloc+0x34/0x180
[410287.644130]  [<ffffffff8123c455>] proc_pid_make_inode+0x25/0x110
[410287.644132]  [<ffffffff8123c634>] proc_pident_instantiate+0x24/0xd0
[410287.644134]  [<ffffffff8123c77b>] proc_pident_lookup+0x9b/0xe0
[410287.644137]  [<ffffffff8123c81a>] proc_tgid_base_lookup+0x1a/0x20
[410287.644141]  [<ffffffff811dc03d>] lookup_real+0x1d/0x60
[410287.644143]  [<ffffffff811e1578>] lookup_open+0xc8/0x1d0
[410287.644145]  [<ffffffff811e1954>] do_last+0x2d4/0x7b0
[410287.644150]  [<ffffffff812ef976>] ? security_file_alloc+0x16/0x20
[410287.644153]  [<ffffffff811e1ef4>] path_openat+0xc4/0x4c0
[410287.644157]  [<ffffffff81186e44>] ? tlb_flush_mmu_free+0x34/0x60
[410287.644159]  [<ffffffff81187844>] ? tlb_flush_mmu+0x24/0x30
[410287.644162]  [<ffffffff811e2c63>] do_filp_open+0x43/0xa0
[410287.644164]  [<ffffffff811ef9cd>] ? __alloc_fd+0xcd/0x120
[410287.644168]  [<ffffffff811d1742>] do_sys_open+0x132/0x220
[410287.644172]  [<ffffffff810220dd>] ? syscall_trace_leave+0xdd/0x150
[410287.644174]  [<ffffffff811d184e>] SyS_open+0x1e/0x20
[410287.644178]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410287.644179] Code: 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 83 e8 01 74 0a 0f b7 0f 66 39 ca <75> f1 5d c3 66 66 66 90 66 66 90 eb da 66 0f 1f 44 00 00 66 66
[410287.644274] NMI backtrace for cpu 2
[410287.644296] CPU: 2 PID: 2654 Comm: psad Not tainted 3.15.1-031501-generic #201406161841
[410287.644298] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410287.644301] task: ffff8800ba0cb260 ti: ffff8800b4774000 task.ti: ffff8800b4774000
[410287.644304] RIP: 0010:[<ffffffff8177afaa>]  [<ffffffff8177afaa>] _raw_spin_lock+0x3a/0x50
[410287.644345] RSP: 0018:ffff8800b47778c8  EFLAGS: 00000202
[410287.644346] RAX: 0000000000006f6d RBX: ffff8800a8b89598 RCX: 0000000000005878
[410287.644348] RDX: 000000000000587c RSI: 000000000000587c RDI: ffffffff81c0b400
[410287.644349] RBP: ffff8800b47778c8 R08: ffff88013fc572c0 R09: 0000000000000000
[410287.644350] R10: ffff880026a375a0 R11: 0000000000000000 R12: ffff8800a8b89598
[410287.644352] R13: 0000000000000000 R14: 0000000000008000 R15: 0000000000000000
[410287.644357] FS:  00007fcc8b7e8740(0000) GS:ffff88013fc40000(0000) knlGS:0000000000000000
[410287.644359] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[410287.644360] CR2: 00007f3682ab7340 CR3: 00000000baed8000 CR4: 00000000000006e0
[410287.644372] Stack:
[410287.644374]  ffff8800b47778e8 ffffffff811ebc0d ffff8800a8b89400 ffff8800a8b89400
[410287.644380]  ffff8800b4777918 ffffffffa006e9da ffff8801021c94d8 ffff880137e1f000
[410287.644382]  ffff8801021c94d8 0000000000000000 ffff8800b47779a8 ffffffffa00aca4d
[410287.644385] Call Trace:
[410287.644409]  [<ffffffff811ebc0d>] inode_sb_list_add+0x1d/0x70
[410288.583763]  [<ffffffffa006e9da>] xfs_setup_inode+0x3a/0x330 [xfs]
[410288.777295]  [<ffffffffa00aca4d>] xfs_ialloc+0x4cd/0x6f0 [xfs]
[410288.777358]  [<ffffffffa00accda>] xfs_dir_ialloc+0x6a/0x280 [xfs]
[410288.839088]  [<ffffffffa00ab2bc>] ? xfs_ilock+0xdc/0x120 [xfs]
[410288.839152]  [<ffffffffa00ad440>] xfs_create+0x4d0/0x6c0 [xfs]
[410288.839168]  [<ffffffffa006de3b>] xfs_generic_create+0xdb/0x250 [xfs]
[410288.839183]  [<ffffffffa006dfe4>] xfs_vn_mknod+0x14/0x20 [xfs]
[410288.839209]  [<ffffffffa006e023>] xfs_vn_create+0x13/0x20 [xfs]
[410288.839216]  [<ffffffff811e143d>] vfs_create+0xed/0x160
[410288.839219]  [<ffffffff811e15ee>] lookup_open+0x13e/0x1d0
[410288.839245]  [<ffffffff811f2298>] ? __mnt_want_write+0x58/0x70
[410288.839247]  [<ffffffff811e1954>] do_last+0x2d4/0x7b0
[410288.839249]  [<ffffffff811e1ef4>] path_openat+0xc4/0x4c0
[410288.839270]  [<ffffffffa00ab634>] ? xfs_iunlock+0x114/0x120 [xfs]
[410288.839284]  [<ffffffffa0063fd7>] ? xfs_file_aio_read+0x1f7/0x2f0 [xfs]
[410288.839286]  [<ffffffff811e2c63>] do_filp_open+0x43/0xa0
[410288.839289]  [<ffffffff811ef9cd>] ? __alloc_fd+0xcd/0x120
[410288.839298]  [<ffffffff811d1742>] do_sys_open+0x132/0x220
[410288.839310]  [<ffffffff810220dd>] ? syscall_trace_leave+0xdd/0x150
[410288.839313]  [<ffffffff811d184e>] SyS_open+0x1e/0x20
[410288.839319]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410288.839321] Code: 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 83 e8 01 74 0a 0f b7 0f <66> 39 ca 75 f1 5d c3 66 66 66 90 66 66 90 eb da 66 0f 1f 44 00
[410288.839403] NMI backtrace for cpu 0
[410288.839431] CPU: 0 PID: 26553 Comm: xfs_db Not tainted 3.15.1-031501-generic #201406161841
[410288.839433] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410288.839435] task: ffff8800ba0ccb90 ti: ffff8800a8008000 task.ti: ffff8800a8008000
[410288.839438] RIP: 0010:[<ffffffff811fad4a>]  [<ffffffff811fad4a>] wait_sb_inodes+0x7a/0x120
[410288.863791] RSP: 0018:ffff8800a800bd58  EFLAGS: 00010206
[410288.863794] RAX: ffff8800136bbea0 RBX: ffff8800136bbd98 RCX: ffff88013fc11998
[410288.863796] RDX: 000000000000001c RSI: ffff8800ba0ccb90 RDI: ffff8800136bc220
[410288.863797] RBP: ffff8800a800bd88 R08: ffff8800a8008000 R09: 0000000000000001
[410288.863836] R10: dffa88fd27410620 R11: 0000000000000002 R12: ffff8800360228a0
[410288.863839] R13: ffff8800136bbe20 R14: ffff8800136bc2e8 R15: ffff880011925998
[410288.864238] FS:  00007fa610643780(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[410288.864241] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[410288.864242] CR2: 00007fe95be69000 CR3: 0000000011a5d000 CR4: 00000000000006f0
[410288.864325] Stack:
[410288.864327]  ffff8800a800bd58 ffff880036022800 ffff8800a800bdd8 ffff8800a800bd98
[410288.864334]  ffff88013ae6a080 ffff88013954cc00 ffff8800a800be18 ffffffff811fae93
[410288.864336]  7fffffffffffffff ffff880036022800 ffff8800abab7ca0 0000001000000001
[410288.864339] Call Trace:
[410288.864351]  [<ffffffff811fae93>] sync_inodes_sb+0xa3/0xd0
[410288.864354]  [<ffffffff8120222b>] __sync_filesystem+0x1b/0x60
[410288.864357]  [<ffffffff812022eb>] sync_filesystem+0x4b/0x70
[410288.877144]  [<ffffffff8120d20d>] fsync_bdev+0x2d/0x60
[410289.052866]  [<ffffffff8135837c>] blkdev_ioctl+0x18c/0x810
[410289.052885]  [<ffffffff8120b070>] block_ioctl+0x40/0x50
[410289.052908]  [<ffffffff811e4cf5>] do_vfs_ioctl+0x75/0x2c0
[410289.052920]  [<ffffffff81021f2d>] ? syscall_trace_enter+0x1ad/0x280
[410289.052923]  [<ffffffff810220dd>] ? syscall_trace_leave+0xdd/0x150
[410289.052926]  [<ffffffff8120bd8e>] ? blkdev_fsync+0x3e/0x50
[410289.052928]  [<ffffffff811e4fd1>] SyS_ioctl+0x91/0xb0
[410289.052947]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410289.052953] Code: 8d 00 00 00 0f 1f 44 00 00 4c 89 ef e8 c0 01 58 00 48 8b 83 08 01 00 00 49 39 c4 48 8d 98 f8 fe ff ff 74 6d 4c 8d ab 88 00 00 00 <4c> 8b 73 30 4c 89 ef e8 1a 02 58 00 f6 83 a0 00 00 00 38 75 c9
[410289.077524] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2211.030 msecs
[410473.734748] BUG: soft lockup - CPU#1 stuck for 67s! [postgres:2477]
[410473.734775] BUG: soft lockup - CPU#2 stuck for 67s! [postgres:2400]
[410473.734825] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs cpuid tcp_htcp sch_codel sch_fq_codel xt_multiport xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables ppdev vmw_balloon irda coretemp serio_raw crc_ccitt parport_pc vmwgfx ttm drm vmw_vmci mac_hid shpchp i2c_piix4 lp parport xfs libcrc32c psmouse floppy vmw_pvscsi vmxnet3
[410473.734829] CPU: 2 PID: 2400 Comm: postgres Not tainted 3.15.1-031501-generic #201406161841
[410473.734829] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 01/15/2013
[410473.734831] task: ffff8800b8ae64c0 ti: ffff880138b04000 task.ti: ffff880138b04000
[410473.734839] RIP: 0010:[<ffffffff8177afa2>]  [<ffffffff8177afa2>] _raw_spin_lock+0x32/0x50
[410473.734840] RSP: 0018:ffff880138b07c98  EFLAGS: 00000216
[410473.734841] RAX: 00000000000010dd RBX: ffffea0002bd71c0 RCX: 00000000000061ec
[410473.734842] RDX: 00000000000061f2 RSI: 00000000000061f2 RDI: ffffffff81c0b400
[410473.734842] RBP: ffff880138b07c98 R08: 0000000000000000 R09: 0000000000000000
[410473.734843] R10: ffff8800baa7f000 R11: 0000000000000000 R12: ffff8800360f8380
[410473.734843] R13: 00007f520000003f R14: ffffffffa00af2e6 R15: ffff880138b07cd8
[410473.734846] FS:  00007f52d5e11780(0000) GS:ffff88013fc40000(0000) knlGS:0000000000000000
[410473.734847] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[410473.734847] CR2: 00007f52d5e30000 CR3: 0000000138aed000 CR4: 00000000000006e0
[410473.734855] Stack:
[410473.734857]  ffff880138b07cc8 ffffffff811ecfdf 0000000000000000 ffff8801057a7d98
[410473.734858]  ffff8801057a7e20 ffffffffa00db720 ffff880138b07cf8 ffffffff811ed8c0
[410473.734860]  0000000000000000 ffff8801057a7d98 ffff8801057a7d98 ffff8800baa7f058
[410473.734860] Call Trace:
[410473.734866]  [<ffffffff811ecfdf>] evict+0x6f/0x190
[410473.742954]  [<ffffffff811ed8c0>] iput_final+0xe0/0x170
[410473.742960]  [<ffffffff811ed98e>] iput+0x3e/0x50
[410473.742965]  [<ffffffff811e7c88>] dentry_iput+0xa8/0xf0
[410473.742968]  [<ffffffff811e8c96>] __dentry_kill+0xb6/0x160
[410473.742971]  [<ffffffff811e92c1>] dput+0xb1/0x100
[410473.742974]  [<ffffffff811e0ae8>] SYSC_renameat2+0x428/0x4c0
[410473.743023]  [<ffffffff8111cdfc>] ? acct_account_cputime+0x1c/0x20
[410473.743027]  [<ffffffff8111cdfc>] ? acct_account_cputime+0x1c/0x20
[410473.743039]  [<ffffffff810a5469>] ? account_user_time+0x99/0xb0
[410473.743041]  [<ffffffff810a5a9d>] ? vtime_account_user+0x5d/0x70
[410473.743046]  [<ffffffff81021ee5>] ? syscall_trace_enter+0x165/0x280
[410473.743059]  [<ffffffff810d4063>] ? rcu_eqs_enter+0x73/0xa0
[410473.743062]  [<ffffffff811e34ee>] SyS_renameat2+0xe/0x10
[410473.743064]  [<ffffffff811e352e>] SyS_rename+0x1e/0x20
[410473.743069]  [<ffffffff8178487f>] tracesys+0xe1/0xe6
[410473.743069] Code: 89 e5 b8 00 00 02 00
[410473.743077] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs cpuid tcp_htcp sch_codel sch_fq_codel xt_multiport xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables ppdev vmw_balloon irda coretemp serio_raw crc_ccitt parport_pc vmwgfx ttm drm vmw_vmci mac_hid shpchp i2c_piix4 lp parport xfs libcrc32c psmouse floppy vmw_pvscsi vmxnet3
[410473.743127] CPU: 1 PID: 2477 Comm: postgres Not tainted 3.15.1-031501-generic #201406161841