[ 1258.519490] ========================================================= [ 1258.520014] [ INFO: possible irq lock inversion dependency detected ] [ 1258.520014] 3.14.0-rc7-default+ #122 Not tainted [ 1258.520014] --------------------------------------------------------- [ 1258.520014] kswapd0/33 just changed the state of lock: [ 1258.520014] (&delayed_node->mutex){+.+.-.}, at: [<ffffffffa0089d9f>] __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] but this lock took another, RECLAIM_FS-unsafe lock in the past: [ 1258.520014] (&found->groups_sem){+++++.} and interrupts could create inverse lock ordering between them. [ 1258.520014] [ 1258.520014] other info that might help us debug this: [ 1258.520014] Possible interrupt unsafe locking scenario: [ 1258.520014] [ 1258.520014] CPU0 CPU1 [ 1258.520014] ---- ---- [ 1258.520014] lock(&found->groups_sem); [ 1258.520014] local_irq_disable(); [ 1258.520014] lock(&delayed_node->mutex); [ 1258.520014] lock(&found->groups_sem); [ 1258.520014] <Interrupt> [ 1258.520014] lock(&delayed_node->mutex); [ 1258.520014] [ 1258.520014] *** DEADLOCK *** [ 1258.520014] [ 1258.520014] 2 locks held by kswapd0/33: [ 1258.520014] #0: (shrinker_rwsem){++++..}, at: [<ffffffff8113befd>] shrink_slab+0x3d/0x110 [ 1258.520014] #1: (&type->s_umount_key#37){.+.+..}, at: [<ffffffff8117cfe4>] grab_super_passive+0x44/0x90 [ 1258.520014] [ 1258.520014] the shortest dependencies between 2nd lock and 1st lock: [ 1258.520014] -> (&found->groups_sem){+++++.} ops: 24894 { [ 1258.520014] HARDIRQ-ON-W at: [ 1258.520014] [<ffffffff810af4a6>] __lock_acquire+0x7f6/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01edc>] down_write+0x5c/0xc0 [ 1258.520014] [<ffffffffa001f6a6>] __link_block_group+0x46/0x130 [btrfs] [ 1258.520014] [<ffffffffa0023411>] btrfs_read_block_groups+0x341/0x690 [btrfs] [ 1258.520014] [<ffffffffa0031c50>] open_ctree+0x1880/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] HARDIRQ-ON-R at: [ 1258.520014] [<ffffffff810af295>] __lock_acquire+0x5e5/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01e2c>] down_read+0x4c/0xa0 [ 1258.520014] [<ffffffffa002da32>] btrfs_calc_num_tolerated_disk_barrier_failures+0x142/0x240 [btrfs] [ 1258.520014] [<ffffffffa0031c6e>] open_ctree+0x189e/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] SOFTIRQ-ON-W at: [ 1258.520014] [<ffffffff810af4da>] __lock_acquire+0x82a/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01edc>] down_write+0x5c/0xc0 [ 1258.520014] [<ffffffffa001f6a6>] __link_block_group+0x46/0x130 [btrfs] [ 1258.520014] [<ffffffffa0023411>] btrfs_read_block_groups+0x341/0x690 [btrfs] [ 1258.520014] [<ffffffffa0031c50>] open_ctree+0x1880/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] SOFTIRQ-ON-R at: [ 1258.520014] [<ffffffff810af4da>] __lock_acquire+0x82a/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01e2c>] down_read+0x4c/0xa0 [ 1258.520014] [<ffffffffa002da32>] btrfs_calc_num_tolerated_disk_barrier_failures+0x142/0x240 [btrfs] [ 1258.520014] [<ffffffffa0031c6e>] open_ctree+0x189e/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] RECLAIM_FS-ON-W at: [ 1258.520014] [<ffffffff810b1c9c>] mark_held_locks+0x8c/0x170 [ 1258.520014] [<ffffffff810b24ca>] lockdep_trace_alloc+0x8a/0xd0 [ 1258.520014] [<ffffffff8116f957>] __kmalloc_track_caller+0x47/0x210 [ 1258.520014] [<ffffffff813cdf8b>] kvasprintf+0x5b/0x90 [ 1258.520014] [<ffffffff813c16da>] kobject_set_name_vargs+0x2a/0x70 [ 1258.520014] [<ffffffff813c206a>] kobject_add+0x5a/0xb0 [ 1258.520014] [<ffffffffa001f75d>] __link_block_group+0xfd/0x130 [btrfs] [ 1258.520014] [<ffffffffa0023411>] btrfs_read_block_groups+0x341/0x690 [btrfs] [ 1258.520014] [<ffffffffa0031c50>] open_ctree+0x1880/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] INITIAL USE at: [ 1258.520014] [<ffffffff810af004>] __lock_acquire+0x354/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01edc>] down_write+0x5c/0xc0 [ 1258.520014] [<ffffffffa001f6a6>] __link_block_group+0x46/0x130 [btrfs] [ 1258.520014] [<ffffffffa0023411>] btrfs_read_block_groups+0x341/0x690 [btrfs] [ 1258.520014] [<ffffffffa0031c50>] open_ctree+0x1880/0x2310 [btrfs] [ 1258.520014] [<ffffffffa00065db>] btrfs_mount+0x55b/0x860 [btrfs] [ 1258.520014] [<ffffffff8117d950>] mount_fs+0x20/0xe0 [ 1258.520014] [<ffffffff81199d96>] vfs_kern_mount+0x76/0x160 [ 1258.520014] [<ffffffff8119c53d>] do_mount+0x31d/0x970 [ 1258.520014] [<ffffffff8119cf10>] SyS_mount+0x90/0xe0 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] } [ 1258.520014] ... key at: [<ffffffffa00d06e0>] __key.40143+0x0/0xfffffffffffe2920 [btrfs] [ 1258.520014] ... acquired at: [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a01e2c>] down_read+0x4c/0xa0 [ 1258.520014] [<ffffffffa001ff78>] find_free_extent+0x7e8/0xbe0 [btrfs] [ 1258.520014] [<ffffffffa0020434>] btrfs_reserve_extent+0xa4/0x130 [btrfs] [ 1258.520014] [<ffffffffa0021903>] btrfs_alloc_free_block+0x103/0x4c0 [btrfs] [ 1258.520014] [<ffffffffa000f8f5>] split_leaf+0x135/0x7b0 [btrfs] [ 1258.520014] [<ffffffffa0010b21>] btrfs_search_slot+0x8d1/0xa00 [btrfs] [ 1258.520014] [<ffffffffa00123de>] btrfs_insert_empty_items+0x7e/0xe0 [btrfs] [ 1258.520014] [<ffffffffa008a744>] btrfs_insert_delayed_items+0x84/0x460 [btrfs] [ 1258.520014] [<ffffffffa008b457>] btrfs_async_run_delayed_root+0x107/0x210 [btrfs] [ 1258.520014] [<ffffffffa00658bc>] worker_loop+0xcc/0x5c0 [btrfs] [ 1258.520014] [<ffffffff8108747e>] kthread+0xee/0x110 [ 1258.520014] [<ffffffff81a0cbac>] ret_from_fork+0x7c/0xb0 [ 1258.520014] [ 1258.520014] -> (&delayed_node->mutex){+.+.-.} ops: 1060633 { [ 1258.520014] HARDIRQ-ON-W at: [ 1258.520014] [<ffffffff810af4a6>] __lock_acquire+0x7f6/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa008c569>] btrfs_delayed_update_inode+0x49/0x670 [btrfs] [ 1258.520014] [<ffffffffa003b76a>] btrfs_update_inode+0x6a/0x100 [btrfs] [ 1258.520014] [<ffffffffa0043e8e>] btrfs_create+0x16e/0x220 [btrfs] [ 1258.520014] [<ffffffff81188be9>] vfs_create+0x89/0xc0 [ 1258.520014] [<ffffffff811893b4>] do_last+0x794/0xd50 [ 1258.520014] [<ffffffff81189a37>] path_openat+0xc7/0x620 [ 1258.520014] [<ffffffff8118a44a>] do_filp_open+0x4a/0xa0 [ 1258.520014] [<ffffffff81178a0e>] do_sys_open+0x11e/0x230 [ 1258.520014] [<ffffffff81178b3e>] SyS_open+0x1e/0x20 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] SOFTIRQ-ON-W at: [ 1258.520014] [<ffffffff810af4da>] __lock_acquire+0x82a/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa008c569>] btrfs_delayed_update_inode+0x49/0x670 [btrfs] [ 1258.520014] [<ffffffffa003b76a>] btrfs_update_inode+0x6a/0x100 [btrfs] [ 1258.520014] [<ffffffffa0043e8e>] btrfs_create+0x16e/0x220 [btrfs] [ 1258.520014] [<ffffffff81188be9>] vfs_create+0x89/0xc0 [ 1258.520014] [<ffffffff811893b4>] do_last+0x794/0xd50 [ 1258.520014] [<ffffffff81189a37>] path_openat+0xc7/0x620 [ 1258.520014] [<ffffffff8118a44a>] do_filp_open+0x4a/0xa0 [ 1258.520014] [<ffffffff81178a0e>] do_sys_open+0x11e/0x230 [ 1258.520014] [<ffffffff81178b3e>] SyS_open+0x1e/0x20 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] IN-RECLAIM_FS-W at: [ 1258.520014] [<ffffffff810af31c>] __lock_acquire+0x66c/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa0089d9f>] __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffffa008ba64>] btrfs_remove_delayed_node+0x24/0x30 [btrfs] [ 1258.520014] [<ffffffffa0040c97>] btrfs_evict_inode+0x2a7/0x550 [btrfs] [ 1258.520014] [<ffffffff811953e8>] evict+0xb8/0x1c0 [ 1258.520014] [<ffffffff8119553f>] dispose_list+0x4f/0x60 [ 1258.520014] [<ffffffff8119663c>] prune_icache_sb+0x4c/0x60 [ 1258.520014] [<ffffffff8117d266>] super_cache_scan+0x126/0x190 [ 1258.520014] [<ffffffff8113a28e>] shrink_slab_node+0x14e/0x2c0 [ 1258.520014] [<ffffffff8113bf38>] shrink_slab+0x78/0x110 [ 1258.520014] [<ffffffff8113e8e5>] balance_pgdat+0x405/0x670 [ 1258.520014] [<ffffffff8113ecf8>] kswapd+0x1a8/0x4c0 [ 1258.520014] [<ffffffff8108747e>] kthread+0xee/0x110 [ 1258.520014] [<ffffffff81a0cbac>] ret_from_fork+0x7c/0xb0 [ 1258.520014] INITIAL USE at: [ 1258.520014] [<ffffffff810af004>] __lock_acquire+0x354/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa008c569>] btrfs_delayed_update_inode+0x49/0x670 [btrfs] [ 1258.520014] [<ffffffffa003b76a>] btrfs_update_inode+0x6a/0x100 [btrfs] [ 1258.520014] [<ffffffffa0043e8e>] btrfs_create+0x16e/0x220 [btrfs] [ 1258.520014] [<ffffffff81188be9>] vfs_create+0x89/0xc0 [ 1258.520014] [<ffffffff811893b4>] do_last+0x794/0xd50 [ 1258.520014] [<ffffffff81189a37>] path_openat+0xc7/0x620 [ 1258.520014] [<ffffffff8118a44a>] do_filp_open+0x4a/0xa0 [ 1258.520014] [<ffffffff81178a0e>] do_sys_open+0x11e/0x230 [ 1258.520014] [<ffffffff81178b3e>] SyS_open+0x1e/0x20 [ 1258.520014] [<ffffffff81a0cc52>] system_call_fastpath+0x16/0x1b [ 1258.520014] } [ 1258.520014] ... key at: [<ffffffffa00d4648>] __key.35420+0x0/0xfffffffffffde9b8 [btrfs] [ 1258.520014] ... acquired at: [ 1258.520014] [<ffffffff810addba>] check_usage_forwards+0xaa/0x120 [ 1258.520014] [<ffffffff810aea29>] mark_lock+0x1a9/0x430 [ 1258.520014] [<ffffffff810af31c>] __lock_acquire+0x66c/0x1fb0 [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa0089d9f>] __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffffa008ba64>] btrfs_remove_delayed_node+0x24/0x30 [btrfs] [ 1258.520014] [<ffffffffa0040c97>] btrfs_evict_inode+0x2a7/0x550 [btrfs] [ 1258.520014] [<ffffffff811953e8>] evict+0xb8/0x1c0 [ 1258.520014] [<ffffffff8119553f>] dispose_list+0x4f/0x60 [ 1258.520014] [<ffffffff8119663c>] prune_icache_sb+0x4c/0x60 [ 1258.520014] [<ffffffff8117d266>] super_cache_scan+0x126/0x190 [ 1258.520014] [<ffffffff8113a28e>] shrink_slab_node+0x14e/0x2c0 [ 1258.520014] [<ffffffff8113bf38>] shrink_slab+0x78/0x110 [ 1258.520014] [<ffffffff8113e8e5>] balance_pgdat+0x405/0x670 [ 1258.520014] [<ffffffff8113ecf8>] kswapd+0x1a8/0x4c0 [ 1258.520014] [<ffffffff8108747e>] kthread+0xee/0x110 [ 1258.520014] [<ffffffff81a0cbac>] ret_from_fork+0x7c/0xb0 [ 1258.520014] [ 1258.520014] [ 1258.520014] stack backtrace: [ 1258.520014] CPU: 0 PID: 33 Comm: kswapd0 Not tainted 3.14.0-rc7-default+ #122 [ 1258.520014] Hardware name: Intel Corporation Santa Rosa platform/Matanzas, BIOS TSRSCRB1.86C.0047.B00.0610170821 10/17/06 [ 1258.520014] ffffffff82c3aba0 ffff880070e8d668 ffffffff819fd603 0000000000000002 [ 1258.520014] ffffffff82c3aba0 ffff880070e8d6b8 ffffffff810adcc7 ffff880070e8d708 [ 1258.520014] ffffffff81f16d95 ffff880070e8d6d4 ffff880070e89070 ffff880070e8d6c8 [ 1258.520014] Call Trace: [ 1258.520014] [<ffffffff819fd603>] dump_stack+0x51/0x6e [ 1258.520014] [<ffffffff810adcc7>] print_irq_inversion_bug+0x1c7/0x210 [ 1258.520014] [<ffffffff810addba>] check_usage_forwards+0xaa/0x120 [ 1258.520014] [<ffffffff810de702>] ? __module_text_address+0x12/0x80 [ 1258.520014] [<ffffffff810add10>] ? print_irq_inversion_bug+0x210/0x210 [ 1258.520014] [<ffffffff810aea29>] mark_lock+0x1a9/0x430 [ 1258.520014] [<ffffffff810af31c>] __lock_acquire+0x66c/0x1fb0 [ 1258.520014] [<ffffffff810add10>] ? print_irq_inversion_bug+0x210/0x210 [ 1258.520014] [<ffffffff810ae4ff>] ? check_irq_usage+0x9f/0xf0 [ 1258.520014] [<ffffffff810b005b>] ? __lock_acquire+0x13ab/0x1fb0 [ 1258.520014] [<ffffffff81099ec5>] ? sched_clock_local+0x25/0x90 [ 1258.520014] [<ffffffff810add10>] ? print_irq_inversion_bug+0x210/0x210 [ 1258.520014] [<ffffffffa0089d9f>] ? __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffff810b1312>] lock_acquire+0x92/0x120 [ 1258.520014] [<ffffffffa0089d9f>] ? __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffff81a00dee>] mutex_lock_nested+0x6e/0x3a0 [ 1258.520014] [<ffffffffa0089d9f>] ? __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffff8109a158>] ? sched_clock_cpu+0xa8/0xd0 [ 1258.520014] [<ffffffffa0089d9f>] ? __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffff810ab65d>] ? lock_release_holdtime+0x3d/0x1c0 [ 1258.520014] [<ffffffffa0089d9f>] __btrfs_release_delayed_node+0x4f/0x220 [btrfs] [ 1258.520014] [<ffffffffa008ba64>] btrfs_remove_delayed_node+0x24/0x30 [btrfs] [ 1258.520014] [<ffffffffa0040c97>] btrfs_evict_inode+0x2a7/0x550 [btrfs] [ 1258.520014] [<ffffffff81a03e0b>] ? _raw_spin_unlock+0x2b/0x40 [ 1258.520014] [<ffffffff811953e8>] evict+0xb8/0x1c0 [ 1258.520014] [<ffffffff81196130>] ? insert_inode_locked+0x1a0/0x1a0 [ 1258.520014] [<ffffffff8119553f>] dispose_list+0x4f/0x60 [ 1258.520014] [<ffffffff8119663c>] prune_icache_sb+0x4c/0x60 [ 1258.520014] [<ffffffff8117d266>] super_cache_scan+0x126/0x190 [ 1258.520014] [<ffffffff8113a28e>] shrink_slab_node+0x14e/0x2c0 [ 1258.520014] [<ffffffff810ab65d>] ? lock_release_holdtime+0x3d/0x1c0 [ 1258.520014] [<ffffffff8113bf38>] shrink_slab+0x78/0x110 [ 1258.520014] [<ffffffff8113e8e5>] balance_pgdat+0x405/0x670 [ 1258.520014] [<ffffffff810b1f6d>] ? trace_hardirqs_on+0xd/0x10 [ 1258.520014] [<ffffffff8113ecf8>] kswapd+0x1a8/0x4c0 [ 1258.520014] [<ffffffff8113eb50>] ? balance_pgdat+0x670/0x670 [ 1258.520014] [<ffffffff810a4df0>] ? bit_waitqueue+0xc0/0xc0 [ 1258.520014] [<ffffffff8113eb50>] ? balance_pgdat+0x670/0x670 [ 1258.520014] [<ffffffff8108747e>] kthread+0xee/0x110 [ 1258.520014] [<ffffffff8109a196>] ? local_clock+0x16/0x30 [ 1258.520014] [<ffffffff81087390>] ? flush_kthread_worker+0x1e0/0x1e0 [ 1258.520014] [<ffffffff81a0cbac>] ret_from_fork+0x7c/0xb0 [ 1258.520014] [<ffffffff81087390>] ? flush_kthread_worker+0x1e0/0x1e0
Another report: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg32237.html
Fixed by https://patchwork.kernel.org/patch/3894781/