Created attachment 172781 [details] snapshot stability test Dear Maintainers, we can't get stable Linux kernel with XFS + NFS and LVM snapshots. Currently the most stable behaviour we noticed with Linux kernel 3.12.39 so it will be used as the reference. The kernel has applied patches for aufs support and following patches for XFS: * backported: ae687e58b3f09b1b3c0faf2cac8c27fbbefb5a48 xfs: use NOIO contexts for vm_map_ram - to fix one of the deadlock that occurs from time to time * ac8809f9ab01a73de1a47b5a37bd8dcca8712fb3 xfs: abort metadata writeback on permanent errors - which was suspected to fix currently reported issue but unfortunatelly it won't Test environment: One volume group with two logical volumes of size 15GB. Each logical volume has 10 snapshots of size 2GB. Most of the time these snapshots are overfilled (when they aren't system is stable). Both logical volumes contains XFS filesystem created with lazy-count=0 option. Logical volumes are mounted with options: defaults,usrquota,grpquota,nouuid,noatime,nodiratime. These mountpoints are exported as NFS shares. There is one client machine that runs fio read/write tests on both NFS shares to create load. Test description: Two background threads are running in the endless loop. One thread creates 50 files of size 100M on one mounted logical volume. Then the files are removed. The main test pseudocode looks like: do { for (i=0; i <20; ++i) { result = 0; if (i<10) { result += create_snapshot i on lv0 //lvconvert -s -c 512 lv0 snapshot_i sleep 5 result += mount_snapshot i on lv0 //mount -t xfs -o nouuid,noatime snapshot_i mountpoint_i } else { result += create_snapshot i on lv1 sleep 5 result += mount_snapshot i on lv1 } if (result) // if start or mount snapshot ends with error remove the snapshot remove_snapshot i //lvremove -sf snapshot_i for (i=0; i <20; ++i) { result = 0; if (i<10) { result += umount_snapshot i on lv0 //umount mountpoint_i sleep 5 result += remove_snapshot i on lv0 //lvremove -sf snapshot_i } else { result += umount_snapshot i on lv1 sleep 5 result += remove_snapshot i on lv1 } if (result) // if start or mount snapshot ends with error remove the snapshot remove_snapshot i } after a several snapshots there are endless messages in dmesg like: Mar 31 09:02:48 [kern.alert] kernel: [70323.609334] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 Mar 31 09:02:53 [kern.alert] kernel: [70328.655048] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 Mar 31 09:02:58 [kern.alert] kernel: [70333.670768] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 Mar 31 09:03:04 [kern.alert] kernel: [70338.696504] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 Mar 31 09:03:09 [kern.alert] kernel: [70343.722227] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 Mar 31 09:03:14 [kern.alert] kernel: [70348.737979] XFS (dm-42): metadata I/O error: block 0x0 (\"xfs_buf_iodone_callbacks\") error 5 numblks 1 dd and lvconvert processes are deadlocked. Blocked state: Mar 31 09:21:18 [kern.info] kernel: [71432.487036] SysRq : Show Blocked State Mar 31 09:21:18 [kern.info] kernel: [71432.543470] task PC stack pid father Mar 31 09:21:18 [kern.info] kernel: [71432.543476] kworker/1:1 D 0000000000000000 0 582 2 0x00000000 Mar 31 09:21:18 [kern.info] kernel: [71432.543483] Workqueue: xfs-eofblocks/dm-17 xfs_eofblocks_worker Mar 31 09:21:18 [kern.err] kernel: [71432.543484] ffff88020f03d978 0000000000000002 ffff88020f1cb900 ffff88020f03dfd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543486] ffff88020f03dfd8 ffff88020f03dfd8 ffff88020f510000 ffff88020f1cb900 Mar 31 09:21:18 [kern.err] kernel: [71432.543488] ffffffff83a561c0 0000000000000006 ffff88020f03d8f8 ffffffff810a64eb Mar 31 09:21:18 [kern.err] kernel: [71432.543490] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543494] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543497] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543499] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543501] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543503] [<ffffffff81134f6a>] __sb_start_write+0x16a/0x1a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543506] [<ffffffff812eec3c>] ? xfs_iunlock+0x4c/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543510] [<ffffffff81074e50>] ? __init_waitqueue_head+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543512] [<ffffffff812c0238>] xfs_trans_alloc+0x28/0x50 Mar 31 09:21:18 [kern.err] kernel: [71432.543514] [<ffffffff812a85e4>] xfs_free_eofblocks+0xf4/0x250 Mar 31 09:21:18 [kern.err] kernel: [71432.543516] [<ffffffff812b2f55>] xfs_inode_free_eofblocks+0x85/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543518] [<ffffffff812b17ff>] xfs_inode_ag_walk.isra.22+0x25f/0x360 Mar 31 09:21:18 [kern.err] kernel: [71432.543520] [<ffffffff812b1618>] ? xfs_inode_ag_walk.isra.22+0x78/0x360 Mar 31 09:21:18 [kern.err] kernel: [71432.543522] [<ffffffff812b2ed0>] ? xfs_inode_clear_eofblocks_tag+0xf0/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543525] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543527] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543530] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543532] [<ffffffff810a391d>] ? trace_hardirqs_off+0xd/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543534] [<ffffffff81083caf>] ? local_clock+0x6f/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543536] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543538] [<ffffffff812fbd12>] ? xfs_perag_get_tag+0x62/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543540] [<ffffffff812b2ed0>] ? xfs_inode_clear_eofblocks_tag+0xf0/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543542] [<ffffffff812fbd38>] ? xfs_perag_get_tag+0x88/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543543] [<ffffffff812fbcb0>] ? xfs_perag_get+0x80/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543545] [<ffffffff810a391d>] ? trace_hardirqs_off+0xd/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543547] [<ffffffff812b2ed0>] ? xfs_inode_clear_eofblocks_tag+0xf0/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543549] [<ffffffff812b2a23>] xfs_inode_ag_iterator_tag+0x73/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543551] [<ffffffff812b2cad>] xfs_icache_free_eofblocks+0x2d/0x30 Mar 31 09:21:18 [kern.err] kernel: [71432.543553] [<ffffffff812b2cca>] xfs_eofblocks_worker+0x1a/0x30 Mar 31 09:21:18 [kern.err] kernel: [71432.543556] [<ffffffff8106c70f>] process_one_work+0x1cf/0x450 Mar 31 09:21:18 [kern.err] kernel: [71432.543557] [<ffffffff8106c6a9>] ? process_one_work+0x169/0x450 Mar 31 09:21:18 [kern.err] kernel: [71432.543560] [<ffffffff8106d93b>] worker_thread+0x11b/0x370 Mar 31 09:21:18 [kern.err] kernel: [71432.543561] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543563] [<ffffffff8106d820>] ? manage_workers.isra.27+0x2a0/0x2a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543565] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543567] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543569] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543571] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543595] nfsd D 0000000000000001 0 15995 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543597] ffff88020c5df7c8 0000000000000002 ffff88020c568000 ffff88020c5dffd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543599] ffff88020c5dffd8 ffff88020c5dffd8 ffff8801e01e8000 ffff88020c568000 Mar 31 09:21:18 [kern.err] kernel: [71432.543602] ffff88020c5df708 ffffffff81083bed ffff88020c5df708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543604] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543605] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543607] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543609] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543611] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543613] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543614] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543616] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543617] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543619] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543621] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543622] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543624] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543626] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543628] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.543630] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543632] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543633] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543635] [<ffffffff81083caf>] ? local_clock+0x6f/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543637] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543639] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543641] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543642] [<ffffffff8174ac76>] ? _raw_spin_unlock+0x26/0x30 Mar 31 09:21:18 [kern.err] kernel: [71432.543644] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.543646] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543649] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543650] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543652] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543654] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543656] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543657] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543659] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543661] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543664] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.543666] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543669] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.543671] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.543672] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543674] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543676] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543678] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543679] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543680] nfsd D 0000000000000001 0 15996 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543682] ffff8801df6577c8 0000000000000002 ffff880002111300 ffff8801df657fd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543684] ffff8801df657fd8 ffff8801df657fd8 ffff88020f535f00 ffff880002111300 Mar 31 09:21:18 [kern.err] kernel: [71432.543686] ffff8801df657708 ffffffff81083bed ffff8801df657708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543688] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543689] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543691] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543693] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543694] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543696] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543697] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543699] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543700] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543702] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543703] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543705] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543706] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543708] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543710] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.543712] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543713] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543715] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543717] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543719] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543721] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543722] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.543724] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543726] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543728] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543729] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543731] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543732] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543734] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543735] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543737] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543739] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.543741] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543743] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.543745] [<ffffffff817245e1>] ? svc_recv+0x1/0x550 Mar 31 09:21:18 [kern.err] kernel: [71432.543746] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.543748] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543750] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543752] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543753] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543755] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543756] nfsd D 0000000000000002 0 15997 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543758] ffff8801df6637c8 0000000000000002 ffff880002115f00 ffff8801df663fd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543760] ffff8801df663fd8 ffff8801df663fd8 ffff88020f1cb900 ffff880002115f00 Mar 31 09:21:18 [kern.err] kernel: [71432.543761] ffff8801df663708 ffffffff81083bed ffff8801df663708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543763] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543765] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543766] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543768] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543770] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543771] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543773] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543774] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543776] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543777] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543779] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543780] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543782] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543784] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543785] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.543787] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543789] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543790] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543792] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543795] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543796] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543798] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543799] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543801] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.543803] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543805] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543806] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543808] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543810] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543811] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543813] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543814] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543816] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543817] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.543819] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543821] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.543823] [<ffffffff817245e1>] ? svc_recv+0x1/0x550 Mar 31 09:21:18 [kern.err] kernel: [71432.543825] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.543826] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543828] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543830] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543832] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543833] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543834] nfsd D 0000000000000001 0 15998 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543836] ffff8801df68b7c8 0000000000000002 ffff880002113900 ffff8801df68bfd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543838] ffff8801df68bfd8 ffff8801df68bfd8 ffff88020f4b3900 ffff880002113900 Mar 31 09:21:18 [kern.err] kernel: [71432.543840] ffff8801df68b708 ffffffff81083bed ffff8801df68b708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543842] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543843] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543845] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543846] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543848] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543850] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543851] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543853] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543854] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543855] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543857] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543858] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543860] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543862] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543863] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.543865] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543867] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543868] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543870] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543872] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543874] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543875] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543877] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.543879] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543881] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543882] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543884] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543886] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543887] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543888] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543890] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543892] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543893] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.543895] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543897] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.543899] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.543901] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543902] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543904] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543906] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543908] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543909] nfsd D 0000000000000000 0 15999 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543910] ffff8800b3d3d7c8 0000000000000002 ffff88020c5b2600 ffff8800b3d3dfd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543912] ffff8800b3d3dfd8 ffff8800b3d3dfd8 ffff88020f510000 ffff88020c5b2600 Mar 31 09:21:18 [kern.err] kernel: [71432.543914] ffff8800b3d3d708 ffffffff81083bed ffff8800b3d3d708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543916] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543918] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543919] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543921] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543922] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543924] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543925] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543927] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543928] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543930] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543931] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543933] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543934] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543936] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543938] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.543940] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.543941] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543943] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543945] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543946] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543948] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543950] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.543951] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.543953] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.543954] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.543956] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.543958] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.543959] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.543961] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543963] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.543964] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543965] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.543967] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543969] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.543971] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.543973] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.543974] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.543976] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.543978] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.543980] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543981] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.543983] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.info] kernel: [71432.543984] nfsd D 0000000000000003 0 16000 2 0x00000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543986] ffff8800b36097c8 0000000000000002 ffff8802005d2600 ffff8800b3609fd8 Mar 31 09:21:18 [kern.err] kernel: [71432.543988] ffff8800b3609fd8 ffff8800b3609fd8 ffff8801e02aa600 ffff8802005d2600 Mar 31 09:21:18 [kern.err] kernel: [71432.543989] ffff8800b3609708 ffffffff81083bed ffff8800b3609708 0000000000000000 Mar 31 09:21:18 [kern.err] kernel: [71432.543991] Call Trace: Mar 31 09:21:18 [kern.err] kernel: [71432.543993] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.543994] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.543996] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543998] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.543999] [<ffffffff810e73a0>] ? __lock_page+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.544001] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.544002] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.544004] [<ffffffff810e73a9>] sleep_on_page+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.544005] [<ffffffff81745b55>] __wait_on_bit_lock+0x55/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.544007] [<ffffffff810e71bc>] ? find_get_entry+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.544008] [<ffffffff810e7120>] ? find_get_pages_contig+0x1b0/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.544010] [<ffffffff810e7392>] __lock_page+0x62/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.544011] [<ffffffff81074ed0>] ? wake_atomic_t_function+0x40/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.544013] [<ffffffff8115e767>] __generic_file_splice_read+0x597/0x5d0 Mar 31 09:21:18 [kern.err] kernel: [71432.544015] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:18 [kern.err] kernel: [71432.544017] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:18 [kern.err] kernel: [71432.544018] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.544020] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.544022] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.544023] [<ffffffff81014f85>] ? native_sched_clock+0x15/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.544025] [<ffffffff810a8787>] ? __lock_acquire+0x467/0x20a0 Mar 31 09:21:18 [kern.err] kernel: [71432.544027] [<ffffffff8136a479>] ? lockref_put_or_lock+0x29/0x40 Mar 31 09:21:18 [kern.err] kernel: [71432.544028] [<ffffffff8115ccd0>] ? page_cache_pipe_buf_release+0x20/0x20 Mar 31 09:21:18 [kern.err] kernel: [71432.544030] [<ffffffff8115e7d9>] generic_file_splice_read+0x39/0x80 Mar 31 09:21:18 [kern.err] kernel: [71432.544032] [<ffffffff812eea85>] ? xfs_ilock+0xa5/0xb0 Mar 31 09:21:18 [kern.err] kernel: [71432.544033] [<ffffffff812adacc>] xfs_file_splice_read+0x9c/0xf0 Mar 31 09:21:18 [kern.err] kernel: [71432.544035] [<ffffffff8115cfa9>] do_splice_to+0x79/0xa0 Mar 31 09:21:18 [kern.err] kernel: [71432.544037] [<ffffffff8115d1bf>] splice_direct_to_actor+0x9f/0x1c0 Mar 31 09:21:18 [kern.err] kernel: [71432.544038] [<ffffffff8125f4f0>] ? nfsd_buffered_filldir+0x60/0x60 Mar 31 09:21:18 [kern.err] kernel: [71432.544039] [<ffffffff8125fa81>] nfsd_vfs_read.isra.17+0x121/0x170 Mar 31 09:21:18 [kern.err] kernel: [71432.544041] [<ffffffff812608f3>] nfsd_read+0x1e3/0x2c0 Mar 31 09:21:18 [kern.err] kernel: [71432.544043] [<ffffffff817193e2>] ? svcauth_unix_set_client+0x252/0x6b0 Mar 31 09:21:18 [kern.err] kernel: [71432.544044] [<ffffffff81269684>] nfsd3_proc_read+0xb4/0x100 Mar 31 09:21:18 [kern.err] kernel: [71432.544046] [<ffffffff8125baa1>] nfsd_dispatch+0xa1/0x1b0 Mar 31 09:21:18 [kern.err] kernel: [71432.544048] [<ffffffff81714e48>] svc_process+0x448/0x700 Mar 31 09:21:18 [kern.err] kernel: [71432.544050] [<ffffffff8125b3df>] nfsd+0xbf/0x130 Mar 31 09:21:18 [kern.err] kernel: [71432.544052] [<ffffffff8125b320>] ? nfsd_destroy+0xc0/0xc0 Mar 31 09:21:18 [kern.err] kernel: [71432.544053] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:18 [kern.err] kernel: [71432.544055] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:18 [kern.err] kernel: [71432.544057] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:18 [kern.err] kernel: [71432.544059] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 ...supressed nfsd tracebacks Mar 31 09:21:19 [kern.info] kernel: [71432.548437] dd D 0000000000000002 0 6937 12410 0x20020000 Mar 31 09:21:19 [kern.err] kernel: [71432.548440] ffff8801e0191e38 0000000000000002 ffff880008421300 ffff8801e0191fd8 Mar 31 09:21:19 [kern.err] kernel: [71432.548442] ffff8801e0191fd8 ffff8801e0191fd8 ffff88020f4f2600 ffff880008421300 Mar 31 09:21:19 [kern.err] kernel: [71432.548443] 0000000000000046 ffff880008421998 ffff8801e0191d98 ffffffff810a3a0c Mar 31 09:21:19 [kern.err] kernel: [71432.548445] Call Trace: Mar 31 09:21:19 [kern.err] kernel: [71432.548447] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:19 [kern.err] kernel: [71432.548449] [<ffffffff8174acba>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Mar 31 09:21:19 [kern.err] kernel: [71432.548451] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:19 [kern.err] kernel: [71432.548453] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:19 [kern.err] kernel: [71432.548454] [<ffffffff81134f6a>] __sb_start_write+0x16a/0x1a0 Mar 31 09:21:19 [kern.err] kernel: [71432.548456] [<ffffffff81084788>] ? vtime_account_user+0x58/0x70 Mar 31 09:21:19 [kern.err] kernel: [71432.548458] [<ffffffff8108e570>] ? cpuacct_charge+0x90/0x90 Mar 31 09:21:19 [kern.err] kernel: [71432.548460] [<ffffffff81074e50>] ? __init_waitqueue_head+0x60/0x60 Mar 31 09:21:19 [kern.err] kernel: [71432.548463] [<ffffffff811330c3>] vfs_write+0x1b3/0x1d0 Mar 31 09:21:19 [kern.err] kernel: [71432.548465] [<ffffffff81133470>] SyS_write+0x50/0xa0 Mar 31 09:21:19 [kern.err] kernel: [71432.548467] [<ffffffff81753ad9>] ia32_do_call+0x13/0x13 Mar 31 09:21:19 [kern.info] kernel: [71432.548469] lvconvert D 0000000000000000 0 8076 7349 0x20020000 Mar 31 09:21:19 [kern.err] kernel: [71432.548470] ffff880207dedaf8 0000000000000002 ffff8801cddecc00 ffff880207dedfd8 Mar 31 09:21:19 [kern.err] kernel: [71432.548472] ffff880207dedfd8 ffff880207dedfd8 ffff88020f511300 ffff8801cddecc00 Mar 31 09:21:19 [kern.err] kernel: [71432.548474] ffff880207deda28 ffffffff81014ff9 ffff880207deda48 ffffffff81083bed Mar 31 09:21:19 [kern.err] kernel: [71432.548476] Call Trace: Mar 31 09:21:19 [kern.err] kernel: [71432.548478] [<ffffffff81014ff9>] ? sched_clock+0x9/0x10 Mar 31 09:21:20 [kern.err] kernel: [71432.548480] [<ffffffff81083bed>] ? sched_clock_cpu+0xbd/0x110 Mar 31 09:21:20 [kern.err] kernel: [71432.548482] [<ffffffff810a391d>] ? trace_hardirqs_off+0xd/0x10 Mar 31 09:21:20 [kern.err] kernel: [71432.548483] [<ffffffff81083caf>] ? local_clock+0x6f/0x80 Mar 31 09:21:20 [kern.err] kernel: [71432.548485] [<ffffffff810a3a0c>] ? lock_release_holdtime.part.11+0x1c/0x1b0 Mar 31 09:21:20 [kern.err] kernel: [71432.548488] [<ffffffff81305ad3>] ? xfs_ail_push_all_sync+0x93/0xd0 Mar 31 09:21:20 [kern.err] kernel: [71432.548489] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:20 [kern.err] kernel: [71432.548491] [<ffffffff81305ad8>] xfs_ail_push_all_sync+0x98/0xd0 Mar 31 09:21:20 [kern.err] kernel: [71432.548493] [<ffffffff81074e50>] ? __init_waitqueue_head+0x60/0x60 Mar 31 09:21:20 [kern.err] kernel: [71432.548495] [<ffffffff81301d42>] xfs_log_quiesce+0x32/0x70 Mar 31 09:21:20 [kern.err] kernel: [71432.548497] [<ffffffff812bec36>] xfs_quiesce_attr+0x76/0xb0 Mar 31 09:21:20 [kern.err] kernel: [71432.548499] [<ffffffff812bee20>] xfs_fs_freeze+0x20/0x40 Mar 31 09:21:20 [kern.err] kernel: [71432.548500] [<ffffffff8113644c>] freeze_super+0xbc/0x140 Mar 31 09:21:20 [kern.err] kernel: [71432.548503] [<ffffffff81169f35>] freeze_bdev+0x85/0xd0 Mar 31 09:21:20 [kern.err] kernel: [71432.548505] [<ffffffff815eb5dc>] dm_suspend+0x8c/0x1e0 Mar 31 09:21:20 [kern.err] kernel: [71432.548507] [<ffffffff815f1480>] ? list_version_get_info+0xe0/0xe0 Mar 31 09:21:20 [kern.err] kernel: [71432.548509] [<ffffffff815f14e5>] dev_suspend+0x65/0x240 Mar 31 09:21:20 [kern.err] kernel: [71432.548511] [<ffffffff815f0dba>] ctl_ioctl+0x24a/0x510 Mar 31 09:21:20 [kern.err] kernel: [71432.548514] [<ffffffff815f10ad>] dm_compat_ctl_ioctl+0xd/0x20 Mar 31 09:21:20 [kern.err] kernel: [71432.548516] [<ffffffff81181a11>] compat_sys_ioctl+0xd1/0x1450 Mar 31 09:21:20 [kern.err] kernel: [71432.548518] [<ffffffff810e65de>] ? context_tracking_user_exit+0x5e/0xd0 Mar 31 09:21:20 [kern.err] kernel: [71432.548520] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:20 [kern.err] kernel: [71432.548521] [<ffffffff81019d1f>] ? syscall_trace_enter+0x1bf/0x1f0 Mar 31 09:21:20 [kern.err] kernel: [71432.548523] [<ffffffff81753ad9>] ia32_do_call+0x13/0x13 Mar 31 09:21:20 [kern.info] kernel: [71432.548525] kworker/u8:0 D 0000000000000000 0 24311 2 0x00000000 Mar 31 09:21:20 [kern.info] kernel: [71432.548528] Workqueue: writeback bdi_writeback_workfn (flush-253:18) Mar 31 09:21:20 [kern.err] kernel: [71432.548529] ffff88020ebff5d8 0000000000000002 ffff88020bfdb900 ffff88020ebfffd8 Mar 31 09:21:20 [kern.err] kernel: [71432.548531] ffff88020ebfffd8 ffff88020ebfffd8 ffff88020f512600 ffff88020bfdb900 Mar 31 09:21:20 [kern.err] kernel: [71432.548533] ffff88020ebff548 ffffffff810a64eb ffff88020ebff558 0000000600000007 Mar 31 09:21:20 [kern.err] kernel: [71432.548534] Call Trace: Mar 31 09:21:20 [kern.err] kernel: [71432.548536] [<ffffffff810a64eb>] ? mark_held_locks+0x6b/0x170 Mar 31 09:21:20 [kern.err] kernel: [71432.548538] [<ffffffff81346cc4>] ? blk_flush_plug_list+0x1f4/0x290 Mar 31 09:21:20 [kern.err] kernel: [71432.548540] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:20 [kern.err] kernel: [71432.548541] [<ffffffff81749174>] schedule+0x24/0x70 Mar 31 09:21:20 [kern.err] kernel: [71432.548543] [<ffffffff8174921a>] io_schedule+0x5a/0x80 Mar 31 09:21:20 [kern.err] kernel: [71432.548545] [<ffffffff813459c1>] get_request+0x3b1/0x580 Mar 31 09:21:20 [kern.err] kernel: [71432.548547] [<ffffffff81356431>] ? deadline_merge+0x71/0xa0 Mar 31 09:21:20 [kern.err] kernel: [71432.548549] [<ffffffff81074e50>] ? __init_waitqueue_head+0x60/0x60 Mar 31 09:21:20 [kern.err] kernel: [71432.548551] [<ffffffff81346e35>] blk_queue_bio+0x85/0x330 Mar 31 09:21:20 [kern.err] kernel: [71432.548552] [<ffffffff815ea129>] ? dm_request+0x39/0x1f0 Mar 31 09:21:20 [kern.err] kernel: [71432.548554] [<ffffffff81345222>] generic_make_request+0xc2/0x100 Mar 31 09:21:20 [kern.err] kernel: [71432.548556] [<ffffffff813452c7>] submit_bio+0x67/0x110 Mar 31 09:21:20 [kern.err] kernel: [71432.548558] [<ffffffff812a51ff>] xfs_submit_ioend_bio.isra.22+0x2f/0x40 Mar 31 09:21:20 [kern.err] kernel: [71432.548559] [<ffffffff812a530f>] xfs_submit_ioend+0xff/0x150 Mar 31 09:21:20 [kern.err] kernel: [71432.548561] [<ffffffff812a5711>] xfs_vm_writepage+0x3b1/0x540 Mar 31 09:21:20 [kern.err] kernel: [71432.548563] [<ffffffff810f1105>] __writepage+0x15/0x50 Mar 31 09:21:20 [kern.err] kernel: [71432.548565] [<ffffffff810f163e>] write_cache_pages+0x20e/0x400 Mar 31 09:21:20 [kern.err] kernel: [71432.548567] [<ffffffff810f10f0>] ? bdi_set_max_ratio+0x90/0x90 Mar 31 09:21:20 [kern.err] kernel: [71432.548569] [<ffffffff810f1878>] generic_writepages+0x48/0x70 Mar 31 09:21:20 [kern.err] kernel: [71432.548571] [<ffffffff812a42dd>] xfs_vm_writepages+0x4d/0x60 Mar 31 09:21:20 [kern.err] kernel: [71432.548573] [<ffffffff810f312e>] do_writepages+0x1e/0x30 Mar 31 09:21:20 [kern.err] kernel: [71432.548574] [<ffffffff8115a45e>] __writeback_single_inode+0x3e/0x130 Mar 31 09:21:20 [kern.err] kernel: [71432.548576] [<ffffffff8115b7cb>] writeback_sb_inodes+0x23b/0x390 Mar 31 09:21:20 [kern.err] kernel: [71432.548577] [<ffffffff8115b9b6>] __writeback_inodes_wb+0x96/0xc0 Mar 31 09:21:20 [kern.err] kernel: [71432.548579] [<ffffffff8115bbab>] wb_writeback+0x1cb/0x1e0 Mar 31 09:21:20 [kern.err] kernel: [71432.548581] [<ffffffff8115bc06>] ? wb_check_old_data_flush+0x46/0xa0 Mar 31 09:21:20 [kern.err] kernel: [71432.548582] [<ffffffff8115be69>] ? bdi_writeback_workfn+0x119/0x2b0 Mar 31 09:21:20 [kern.err] kernel: [71432.548584] [<ffffffff8115bea9>] bdi_writeback_workfn+0x159/0x2b0 Mar 31 09:21:20 [kern.err] kernel: [71432.548586] [<ffffffff8106c70f>] process_one_work+0x1cf/0x450 Mar 31 09:21:20 [kern.err] kernel: [71432.548588] [<ffffffff8106c6a9>] ? process_one_work+0x169/0x450 Mar 31 09:21:20 [kern.err] kernel: [71432.548590] [<ffffffff8106d93b>] worker_thread+0x11b/0x370 Mar 31 09:21:20 [kern.err] kernel: [71432.548591] [<ffffffff810a680d>] ? trace_hardirqs_on+0xd/0x10 Mar 31 09:21:20 [kern.err] kernel: [71432.548593] [<ffffffff8106d820>] ? manage_workers.isra.27+0x2a0/0x2a0 Mar 31 09:21:20 [kern.err] kernel: [71432.548595] [<ffffffff810743f6>] kthread+0xd6/0xe0 Mar 31 09:21:20 [kern.err] kernel: [71432.548597] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Mar 31 09:21:20 [kern.err] kernel: [71432.548598] [<ffffffff81751e68>] ret_from_fork+0x58/0x90 Mar 31 09:21:20 [kern.err] kernel: [71432.548600] [<ffffffff81074320>] ? __init_kthread_worker+0x70/0x70 Locks held: Mar 31 09:22:26 [kern.info] kernel: [71499.963466] SysRq : Show Locks Held Mar 31 09:22:26 [kern.err] kernel: [71500.014481] Mar 31 09:22:26 [kern.err] kernel: [71500.014481] Showing all locks held in the system: Mar 31 09:22:26 [kern.err] kernel: [71500.014488] 2 locks held by kworker/1:1/582: Mar 31 09:22:26 [kern.err] kernel: [71500.014489] #0: (xfs-eofblocks/%s){.+.+.+}, at: [<ffffffff8106c6a9>] process_one_work+0x169/0x450 Mar 31 09:22:26 [kern.err] kernel: [71500.014497] #1: ((&(&mp->m_eofblocks_work)->work)){+.+.+.}, at: [<ffffffff8106c6a9>] process_one_work+0x169/0x450 Mar 31 09:22:26 [kern.err] kernel: [71500.014516] 2 locks held by getty/21570: Mar 31 09:22:26 [kern.err] kernel: [71500.014516] #0: (&tty->ldisc_sem){++++++}, at: [<ffffffff8141b6af>] tty_ldisc_ref_wait+0x1f/0x50 Mar 31 09:22:26 [kern.err] kernel: [71500.014521] #1: (&ldata->atomic_read_lock){+.+...}, at: [<ffffffff81419132>] n_tty_read+0xb2/0xbb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014538] 3 locks held by bash/8511: Mar 31 09:22:26 [kern.err] kernel: [71500.014539] #0: (sb_writers#2){.+.+.+}, at: [<ffffffff811330c3>] vfs_write+0x1b3/0x1d0 Mar 31 09:22:26 [kern.err] kernel: [71500.014544] #1: (sysrq_key_table_lock){......}, at: [<ffffffff8142008d>] __handle_sysrq+0x2d/0x180 Mar 31 09:22:26 [kern.err] kernel: [71500.014548] #2: (tasklist_lock){.+.?..}, at: [<ffffffff810a47d2>] debug_show_all_locks+0x42/0x1e0 Mar 31 09:22:26 [kern.err] kernel: [71500.014555] 1 lock held by nfsd/15995: Mar 31 09:22:26 [kern.err] kernel: [71500.014556] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014561] 1 lock held by nfsd/15996: Mar 31 09:22:26 [kern.err] kernel: [71500.014562] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014565] 1 lock held by nfsd/15997: Mar 31 09:22:26 [kern.err] kernel: [71500.014566] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014569] 1 lock held by nfsd/15998: Mar 31 09:22:26 [kern.err] kernel: [71500.014570] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014573] 1 lock held by nfsd/15999: Mar 31 09:22:26 [kern.err] kernel: [71500.014574] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014577] 1 lock held by nfsd/16000: Mar 31 09:22:26 [kern.err] kernel: [71500.014578] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014581] 1 lock held by nfsd/16001: Mar 31 09:22:26 [kern.err] kernel: [71500.014582] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014585] 1 lock held by nfsd/16002: Mar 31 09:22:26 [kern.err] kernel: [71500.014586] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014589] 1 lock held by nfsd/16003: Mar 31 09:22:26 [kern.err] kernel: [71500.014590] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014593] 1 lock held by nfsd/16004: Mar 31 09:22:26 [kern.err] kernel: [71500.014594] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014597] 1 lock held by nfsd/16005: Mar 31 09:22:26 [kern.err] kernel: [71500.014598] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014601] 1 lock held by nfsd/16006: Mar 31 09:22:26 [kern.err] kernel: [71500.014602] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014605] 1 lock held by nfsd/16007: Mar 31 09:22:26 [kern.err] kernel: [71500.014606] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014609] 1 lock held by nfsd/16008: Mar 31 09:22:26 [kern.err] kernel: [71500.014610] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014613] 1 lock held by nfsd/16009: Mar 31 09:22:26 [kern.err] kernel: [71500.014614] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014617] 1 lock held by nfsd/16010: Mar 31 09:22:26 [kern.err] kernel: [71500.014617] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014621] 1 lock held by nfsd/16011: Mar 31 09:22:26 [kern.err] kernel: [71500.014621] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014625] 1 lock held by nfsd/16012: Mar 31 09:22:26 [kern.err] kernel: [71500.014625] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014629] 1 lock held by nfsd/16013: Mar 31 09:22:26 [kern.err] kernel: [71500.014629] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014633] 1 lock held by nfsd/16014: Mar 31 09:22:26 [kern.err] kernel: [71500.014633] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014636] 1 lock held by nfsd/16015: Mar 31 09:22:26 [kern.err] kernel: [71500.014637] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014640] 1 lock held by nfsd/16016: Mar 31 09:22:26 [kern.err] kernel: [71500.014641] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014644] 1 lock held by nfsd/16017: Mar 31 09:22:26 [kern.err] kernel: [71500.014645] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014648] 1 lock held by nfsd/16018: Mar 31 09:22:26 [kern.err] kernel: [71500.014649] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014652] 1 lock held by nfsd/16019: Mar 31 09:22:26 [kern.err] kernel: [71500.014653] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014656] 1 lock held by nfsd/16020: Mar 31 09:22:26 [kern.err] kernel: [71500.014657] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014660] 1 lock held by nfsd/16021: Mar 31 09:22:26 [kern.err] kernel: [71500.014661] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014664] 1 lock held by nfsd/16022: Mar 31 09:22:26 [kern.err] kernel: [71500.014665] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014668] 1 lock held by nfsd/16023: Mar 31 09:22:26 [kern.err] kernel: [71500.014669] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014672] 1 lock held by nfsd/16024: Mar 31 09:22:26 [kern.err] kernel: [71500.014673] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014676] 1 lock held by nfsd/16025: Mar 31 09:22:26 [kern.err] kernel: [71500.014677] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014680] 1 lock held by nfsd/16026: Mar 31 09:22:26 [kern.err] kernel: [71500.014681] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014684] 1 lock held by nfsd/16027: Mar 31 09:22:26 [kern.err] kernel: [71500.014685] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014688] 1 lock held by nfsd/16028: Mar 31 09:22:26 [kern.err] kernel: [71500.014689] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014692] 1 lock held by nfsd/16029: Mar 31 09:22:26 [kern.err] kernel: [71500.014693] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014696] 1 lock held by nfsd/16030: Mar 31 09:22:26 [kern.err] kernel: [71500.014696] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014700] 1 lock held by nfsd/16031: Mar 31 09:22:26 [kern.err] kernel: [71500.014700] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014704] 1 lock held by nfsd/16032: Mar 31 09:22:26 [kern.err] kernel: [71500.014704] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014707] 1 lock held by nfsd/16033: Mar 31 09:22:26 [kern.err] kernel: [71500.014708] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014711] 1 lock held by nfsd/16034: Mar 31 09:22:26 [kern.err] kernel: [71500.014712] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014715] 1 lock held by nfsd/16035: Mar 31 09:22:26 [kern.err] kernel: [71500.014716] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014719] 1 lock held by nfsd/16036: Mar 31 09:22:26 [kern.err] kernel: [71500.014720] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014723] 1 lock held by nfsd/16037: Mar 31 09:22:26 [kern.err] kernel: [71500.014724] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014727] 1 lock held by nfsd/16038: Mar 31 09:22:26 [kern.err] kernel: [71500.014728] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014731] 1 lock held by nfsd/16039: Mar 31 09:22:26 [kern.err] kernel: [71500.014731] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014735] 1 lock held by nfsd/16040: Mar 31 09:22:26 [kern.err] kernel: [71500.014735] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014739] 1 lock held by nfsd/16041: Mar 31 09:22:26 [kern.err] kernel: [71500.014739] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014743] 1 lock held by nfsd/16042: Mar 31 09:22:26 [kern.err] kernel: [71500.014743] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014747] 1 lock held by nfsd/16043: Mar 31 09:22:26 [kern.err] kernel: [71500.014747] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014751] 1 lock held by nfsd/16044: Mar 31 09:22:26 [kern.err] kernel: [71500.014751] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014754] 1 lock held by nfsd/16045: Mar 31 09:22:26 [kern.err] kernel: [71500.014755] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014758] 1 lock held by nfsd/16046: Mar 31 09:22:26 [kern.err] kernel: [71500.014759] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014762] 1 lock held by nfsd/16047: Mar 31 09:22:26 [kern.err] kernel: [71500.014763] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014766] 1 lock held by nfsd/16048: Mar 31 09:22:26 [kern.err] kernel: [71500.014767] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014770] 1 lock held by nfsd/16049: Mar 31 09:22:26 [kern.err] kernel: [71500.014771] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014774] 1 lock held by nfsd/16050: Mar 31 09:22:26 [kern.err] kernel: [71500.014775] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014778] 1 lock held by nfsd/16051: Mar 31 09:22:26 [kern.err] kernel: [71500.014779] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014782] 1 lock held by nfsd/16052: Mar 31 09:22:26 [kern.err] kernel: [71500.014783] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014786] 1 lock held by nfsd/16053: Mar 31 09:22:26 [kern.err] kernel: [71500.014786] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014790] 1 lock held by nfsd/16054: Mar 31 09:22:26 [kern.err] kernel: [71500.014790] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014794] 1 lock held by nfsd/16055: Mar 31 09:22:26 [kern.err] kernel: [71500.014794] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014798] 1 lock held by nfsd/16056: Mar 31 09:22:26 [kern.err] kernel: [71500.014798] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014801] 1 lock held by nfsd/16057: Mar 31 09:22:26 [kern.err] kernel: [71500.014802] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014805] 1 lock held by nfsd/16058: Mar 31 09:22:26 [kern.err] kernel: [71500.014806] #0: (&(&ip->i_iolock)->mr_lock#2){++++++}, at: [<ffffffff812eea85>] xfs_ilock+0xa5/0xb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014814] 2 locks held by bash/12056: Mar 31 09:22:26 [kern.err] kernel: [71500.014815] #0: (&tty->ldisc_sem){++++++}, at: [<ffffffff8141b6af>] tty_ldisc_ref_wait+0x1f/0x50 Mar 31 09:22:26 [kern.err] kernel: [71500.014817] #1: (&ldata->atomic_read_lock){+.+...}, at: [<ffffffff81419132>] n_tty_read+0xb2/0xbb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014824] 3 locks held by lvconvert/8076: Mar 31 09:22:26 [kern.err] kernel: [71500.014825] #0: (&md->suspend_lock){+.+.+.}, at: [<ffffffff815eb583>] dm_suspend+0x33/0x1e0 Mar 31 09:22:26 [kern.err] kernel: [71500.014829] #1: (&bdev->bd_fsfreeze_mutex){+.+.+.}, at: [<ffffffff81169edc>] freeze_bdev+0x2c/0xd0 Mar 31 09:22:26 [kern.err] kernel: [71500.014833] #2: (&type->s_umount_key#37){++++++}, at: [<ffffffff81136404>] freeze_super+0x74/0x140 Mar 31 09:22:26 [kern.err] kernel: [71500.014837] 2 locks held by bash/16248: Mar 31 09:22:26 [kern.err] kernel: [71500.014838] #0: (&tty->ldisc_sem){++++++}, at: [<ffffffff8141b6af>] tty_ldisc_ref_wait+0x1f/0x50 Mar 31 09:22:26 [kern.err] kernel: [71500.014841] #1: (&ldata->atomic_read_lock){+.+...}, at: [<ffffffff81419132>] n_tty_read+0xb2/0xbb0 Mar 31 09:22:26 [kern.err] kernel: [71500.014844] 3 locks held by kworker/u8:0/24311: Mar 31 09:22:26 [kern.err] kernel: [71500.014845] #0: (writeback){++++.+}, at: [<ffffffff8106c6a9>] process_one_work+0x169/0x450 Mar 31 09:22:26 [kern.err] kernel: [71500.014848] #1: ((&(&wb->dwork)->work)){+.+.+.}, at: [<ffffffff8106c6a9>] process_one_work+0x169/0x450 Mar 31 09:22:26 [kern.err] kernel: [71500.014851] #2: (&type->s_umount_key#37){++++++}, at: [<ffffffff8113657f>] grab_super_passive+0x3f/0x90 Mar 31 09:22:26 [kern.err] kernel: [71500.014855] Mar 31 09:22:26 [kern.err] kernel: [71500.014856] ============================================= Mar 31 09:22:26 [kern.err] kernel: [71500.014856] Full test script source is available as an attachment. To create structure and run test following command is used: ./snaptest.sh create <path_to_device> Best regards, Arkadiusz Bubała
I'm batch-closing all xfs bugs which are more than 1 year old, sorry about that. If you still have this issue on a current kernel, please retest and re-open with this information. Thanks, -Eric