Description of problem: I hit a kernel panic on xfs-linux "xfs-5.5-merge-6 + iomap-5.5-merge-6": [34623.023691] run fstests generic/475 at 2019-10-31 20:19:44 [34626.293952] XFS (dm-0): Mounting V5 Filesystem [34626.441140] XFS (dm-0): Ending clean mount [34626.483454] Mounted xfs file system at /mnt/xfstests/mnt2 supports timestamps until 2038 (0x7fffffff) [34628.679834] iomap_finish_ioend: 7 callbacks suppressed [34628.679840] dm-0: writeback error on inode 16797854, offset 1150976, sector 15762000 [34628.679905] dm-0: writeback error on inode 16797854, offset 2818048, sector 15761568 [34628.684120] dm-0: writeback error on inode 162, offset 176128, sector 21936 [34628.706668] dm-0: writeback error on inode 25166791, offset 1314816, sector 23616848 [34342.606123] dm-0: writeback error on inode 9220, offset 2793472, sector 181616 [34342.606215] dm-0: writeback error on inode 8429459, offset 212992, sector 7939416 [34342.638699] Buffer I/O error on dev dm-0, logical block 31457156, async page read [34342.672923] XFS (dm-0): log I/O error -5 [34342.704060] Buffer I/O error on dev dm-0, logical block 31457157, async page read [34342.739474] XFS (dm-0): xfs_do_force_shutdown(0x2) called from line 1297 of file fs/xfs/xfs_log.c. Return address = ffffffffc096045d [34342.771022] Buffer I/O error on dev dm-0, logical block 31457158, async page read [34342.803487] XFS (dm-0): Log I/O Error Detected. Shutting down filesystem [34342.803491] XFS (dm-0): Please unmount the filesystem and rectify the problem(s) [34342.805783] XFS (dm-0): log I/O error -5 [34342.837407] Buffer I/O error on dev dm-0, logical block 31457159, async page read [34347.295823] kasan: CONFIG_KASAN_INLINE enabled [34347.317883] kasan: GPF could be caused by NULL-ptr deref or user memory access [34347.355081] general protection fault: 0000 [#1] SMP KASAN PTI [34347.381157] CPU: 1 PID: 28596 Comm: fsstress Tainted: G B 5.4.0-rc3+ #1 [34347.416675] Hardware name: HP ProLiant ML150 Gen9/ProLiant ML150 Gen9, BIOS P95 10/17/2018 [34347.454030] RIP: 0010:iter_file_splice_write+0x63f/0xa90 [34347.477454] Code: 00 00 48 89 f8 48 c1 e8 03 80 3c 18 00 0f 85 61 03 00 00 48 8b 46 10 48 c7 46 10 00 00 00 00 48 8d 78 08 48 89 fa 48 c1 ea 03 <80> 3c 1a 00 0f 85 52 03 00 00 48 8b 40 08 48 89 ef e8 cb 87 7e 01 [34347.567386] RSP: 0018:ffff8881021478e8 EFLAGS: 00010202 [34347.594284] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000010 [34347.630173] RDX: 0000000000000001 RSI: ffff88803bb2f230 RDI: 0000000000000008 [34347.664386] RBP: ffff88804593f800 R08: fffff9400085f55f R09: fffff9400085f55f [34347.697446] R10: fffff9400085f55e R11: ffffea00042faaf7 R12: ffffed1008b27f27 [34347.729815] R13: ffffed1008b27f1f R14: 000000000000f991 R15: ffff88804593f8fc [34347.762054] FS: 00007fb54c31db80(0000) GS:ffff888111200000(0000) knlGS:0000000000000000 [34347.798497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34347.824248] CR2: 00007fa1677d97bc CR3: 000000010a6b0005 CR4: 00000000001606e0 [34347.861461] Call Trace: [34347.872401] ? __x64_sys_tee+0x220/0x220 [34347.890482] ? generic_file_splice_read+0x4f5/0x6c0 [34347.913129] ? add_to_pipe+0x330/0x330 [34347.930226] ? _cond_resched+0x15/0x30 [34347.947065] direct_splice_actor+0x107/0x1d0 [34347.966187] splice_direct_to_actor+0x2ed/0x7f0 [34347.986506] ? wakeup_pipe_readers+0x80/0x80 [34348.005665] ? do_splice_to+0x140/0x140 [34348.022821] ? security_file_permission+0x53/0x2b0 [34348.044361] do_splice_direct+0x158/0x250 [34348.062308] ? splice_direct_to_actor+0x7f0/0x7f0 [34348.083596] ? __sb_start_write+0x1c4/0x310 [34348.102277] vfs_copy_file_range+0x39c/0xa40 [34348.121542] ? __x64_sys_sendfile+0x1d0/0x1d0 [34348.141038] ? lockdep_hardirqs_on+0x590/0x590 [34348.160710] ? lock_downgrade+0x6d0/0x6d0 [34348.178716] ? lock_acquire+0x15a/0x3d0 [34348.196037] ? __might_fault+0xc4/0x1b0 [34348.213777] __x64_sys_copy_file_range+0x1e8/0x460 [34348.235365] ? __ia32_sys_copy_file_range+0x460/0x460 [34348.257992] ? __audit_syscall_exit+0x796/0xab0 [34348.278378] do_syscall_64+0x9f/0x4f0 [34348.294861] entry_SYSCALL_64_after_hwframe+0x49/0xbe [34348.317937] RIP: 0033:0x7fb54b80f99d [34348.334352] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bb 64 2c 00 f7 d8 64 89 01 48 [34348.423644] RSP: 002b:00007ffedb7d1ae8 EFLAGS: 00000246 ORIG_RAX: 0000000000000146 [34348.457684] RAX: ffffffffffffffda RBX: 00007ffedb7d1b38 RCX: 00007fb54b80f99d [34348.489609] RDX: 0000000000000004 RSI: 00007ffedb7d1b30 RDI: 0000000000000003 [34348.521710] RBP: 000000000001cb24 R08: 000000000001cb24 R09: 0000000000000000 [34348.554435] R10: 00007ffedb7d1b38 R11: 0000000000000246 R12: 00007ffedb7d1b30 [34348.586406] R13: 0000000000000003 R14: 0000000000000004 R15: 00000000000bfb4b [34348.618467] Modules linked in: dm_mod iTCO_wdt intel_rapl_msr iTCO_vendor_support intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf dax_pmem_compat device_dax dax_pmem_core pcspkr nd_pmem i2c_i801 lpc_ich ipmi_ssif hpilo hpwdt ipmi_si sg ioatdma ipmi_devintf dca sunrpc ipmi_msghandler acpi_tad acpi_power_meter vfat fat xfs libcrc32c sd_mod mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_algo_bit drm_vram_helper ttm ahci libahci drm libata crc32c_intel tg3 wmi [34348.869785] ---[ end trace 0c361151da993489 ]--- [34348.908928] RIP: 0010:iter_file_splice_write+0x63f/0xa90 [34348.932900] Code: 00 00 48 89 f8 48 c1 e8 03 80 3c 18 00 0f 85 61 03 00 00 48 8b 46 10 48 c7 46 10 00 00 00 00 48 8d 78 08 48 89 fa 48 c1 ea 03 <80> 3c 1a 00 0f 85 52 03 00 00 48 8b 40 08 48 89 ef e8 cb 87 7e 01 [34349.017307] RSP: 0018:ffff8881021478e8 EFLAGS: 00010202 [34349.040799] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000010 [34349.072993] RDX: 0000000000000001 RSI: ffff88803bb2f230 RDI: 0000000000000008 [34349.105063] RBP: ffff88804593f800 R08: fffff9400085f55f R09: fffff9400085f55f [34349.137064] R10: fffff9400085f55e R11: ffffea00042faaf7 R12: ffffed1008b27f27 [34349.169397] R13: ffffed1008b27f1f R14: 000000000000f991 R15: ffff88804593f8fc [34349.201528] FS: 00007fb54c31db80(0000) GS:ffff888111200000(0000) knlGS:0000000000000000 [34349.238784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [34349.264602] CR2: 00007fa1677d97bc CR3: 000000010a6b0005 CR4: 00000000001606e0 [34349.812107] XFS (dm-0): Unmounting Filesystem [34350.152935] XFS (dm-0): Mounting V5 Filesystem # cat generic/475.full meta-data=/dev/sdb4 isize=512 agcount=4, agsize=983040 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=3932160, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 device-mapper: remove ioctl on error-test failed: No such device or address Command failed. seed = 1572771114 seed = 1572585309 seed = 1572643818 seed = 1572855289 ... ... Sorry for this late bug report, I find this issue earlier, but I was stuck by other things, didn't have time to report it. I'll test on latest xfs to check if this issue still there.
Hit this panic again on mainline linux v5.4 (HEAD=c2da5bdc66a3) with xfs-linux(xfs-5.5-merge-15 + iomap-5.5-merge-11 + vfs-5.5-merge-1): [14842.571874] kasan: CONFIG_KASAN_INLINE enabled [14842.576866] kasan: GPF could be caused by NULL-ptr deref or user memory access [14842.584962] general protection fault: 0000 [#1] SMP KASAN PTI [14842.591387] CPU: 30 PID: 12775 Comm: fsstress Tainted: G B 5.4.0+ #1 [14842.599938] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 2.8.0 005/17/2018 [14842.608403] RIP: 0010:iter_file_splice_write+0x63f/0xa90 [14842.614341] Code: 00 00 48 89 f8 48 c1 e8 03 80 3c 18 00 0f 85 61 03 00 00 48 8b 46 10 48 c7 46 10 00 00 00 00 48 8d 78 08 48 89 fa 48 c1 ea 03 <80> 3c 1a 00 0f 85 52 03 00 00 48 8b 40 08 48 89 ef e8 5b 2e 7d 01 [14842.635305] RSP: 0018:ffff888b9935fc68 EFLAGS: 00010202 [14842.641142] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000010 [14842.649110] RDX: 0000000000000001 RSI: ffff888bb0d53208 RDI: 0000000000000008 [14842.657069] RBP: ffff888b713fb400 R08: fffff94005b3058f R09: fffff94005b3058f [14842.665038] R10: fffff94005b3058e R11: ffffea002d982c77 R12: ffffed116e27f6a7 [14842.673005] R13: ffffed116e27f69f R14: 000000000002ff84 R15: ffff888b713fb4fc [14842.680969] FS: 00007fcaa5972b80(0000) GS:ffff888115c00000(0000) knlGS:0000000000000000 [14842.690006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14842.696424] CR2: 000055a30542b000 CR3: 0000000b9fc8e005 CR4: 00000000001606e0 [14842.704393] Call Trace: [14842.707147] ? __x64_sys_tee+0x220/0x220 [14842.711535] ? lock_acquire+0x15a/0x3d0 [14842.715820] ? do_splice+0xb37/0x1110 [14842.719919] ? __sb_start_write+0x191/0x310 [14842.724592] ? __sb_start_write+0x1c4/0x310 [14842.729273] do_splice+0xa12/0x1110 [14842.733175] ? opipe_prep+0x300/0x300 [14842.737267] ? syscall_slow_exit_work+0x540/0x540 [14842.742523] ? __audit_syscall_exit+0x796/0xab0 [14842.747592] __x64_sys_splice+0x247/0x2d0 [14842.752080] do_syscall_64+0x9f/0x4f0 [14842.756177] entry_SYSCALL_64_after_hwframe+0x49/0xbe [14842.761817] RIP: 0033:0x7fcaa4e6b39b [14842.765810] Code: c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 c5 52 2c 00 49 89 ca 8b 00 85 c0 75 14 b8 13 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 75 c3 0f 1f 40 00 41 57 4d 89 c7 41 56 49 89 [14842.786771] RSP: 002b:00007fffbae7c5f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000113 [14842.795227] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fcaa4e6b39b [14842.803196] RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000005 [14842.811162] RBP: 00007fffbae7c648 R08: 000000000000cb23 R09: 0000000000000000 [14842.819133] R10: 00007fffbae7c648 R11: 0000000000000246 R12: 000000000000cb23 [14842.827104] R13: 000000000000cb23 R14: 000000000000cb23 R15: 0000000000000003 [14842.835082] Modules linked in: dm_mod intel_rapl_msr iTCO_wdt iTCO_vendor_support dcdbas intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf pcspkr dax_pmem_compat device_dax nd_pmem dax_pmem_core mxm_wmi cdc_ether lpc_ich usbnet mii mei_me mei ipmi_ssif sg ioatdma dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter sunrpc vfat fat ip_tables xfs sr_mod cdrom sd_mod mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_algo_bit drm_vram_helper ttm drm ahci bnx2x libahci libata mdio megaraid_sas libcrc32c tg3 crc32c_intel wmi [14842.902703] ---[ end trace dfcbf13626b906e6 ]--- [14843.053619] RIP: 0010:iter_file_splice_write+0x63f/0xa90 [14843.059564] Code: 00 00 48 89 f8 48 c1 e8 03 80 3c 18 00 0f 85 61 03 00 00 48 8b 46 10 48 c7 46 10 00 00 00 00 48 8d 78 08 48 89 fa 48 c1 ea 03 <80> 3c 1a 00 0f 85 52 03 00 00 48 8b 40 08 48 89 ef e8 5b 2e 7d 01 [14843.080528] RSP: 0018:ffff888b9935fc68 EFLAGS: 00010202 [14843.086372] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000010 [14843.094348] RDX: 0000000000000001 RSI: ffff888bb0d53208 RDI: 0000000000000008 [14843.102322] RBP: ffff888b713fb400 R08: fffff94005b3058f R09: fffff94005b3058f [14843.110297] R10: fffff94005b3058e R11: ffffea002d982c77 R12: ffffed116e27f6a7 [14843.118262] R13: ffffed116e27f69f R14: 000000000002ff84 R15: ffff888b713fb4fc [14843.126237] FS: 00007fcaa5972b80(0000) GS:ffff888115c00000(0000) knlGS:0000000000000000 [14843.135270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14843.141693] CR2: 000055a30542b000 CR3: 0000000b9fc8e005 CR4: 00000000001606e0
Could you please post the source line translations of the relevant functions? I don't have your kernel build.
(In reply to Darrick J. Wong from comment #2) > Could you please post the source line translations of the relevant > functions? I don't have your kernel build. I already removed this testing kernel build, and merged lots of new patches. But good news is I still can reproduce this issue[1] (by g/461 this time). I'll build the new kernel and post the source line translations of the relevant functions later. [ 4693.175856] run fstests generic/461 at 2019-12-04 21:46:00 [ 4693.694096] XFS (sda5): Mounting V5 Filesystem [ 4693.703963] XFS (sda5): Ending clean mount [ 4693.710992] xfs filesystem being mounted at /mnt/xfstests/mnt2 supports timestamps until 2038 (0x7fffffff) [ 4693.726744] XFS (sda5): User initiated shutdown received. Shutting down filesystem [ 4693.740549] XFS (sda5): Unmounting Filesystem [ 4693.895876] XFS (sda5): Mounting V5 Filesystem [ 4693.905492] XFS (sda5): Ending clean mount [ 4693.912655] xfs filesystem being mounted at /mnt/xfstests/mnt2 supports timestamps until 2038 (0x7fffffff) [ 4702.015718] restraintd[1391]: *** Current Time: Wed Dec 04 21:46:11 2019 Localwatchdog at: Fri Dec 06 20:32:11 2019 [ 4708.950866] XFS (sda5): User initiated shutdown received. Shutting down filesystem [ 4708.972833] kasan: CONFIG_KASAN_INLINE enabled [ 4708.977801] kasan: GPF could be caused by NULL-ptr deref or user memory access [ 4708.985889] general protection fault: 0000 [#1] SMP KASAN PTI [ 4708.992294] CPU: 0 PID: 19412 Comm: fsstress Not tainted 5.4.0+ #1 [ 4708.999190] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.2.10 03/09/2015 [ 4709.007655] RIP: 0010:iter_file_splice_write+0x668/0xa00 [ 4709.013584] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 97 02 00 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 49 89 f8 49 c1 e8 03 <41> 80 3c 18 00 0f 85 96 02 00 00 48 8b 52 08 4c 89 e7 41 83 c6 01 [ 4709.034540] RSP: 0018:ffff8887ca8bf8d8 EFLAGS: 00010202 [ 4709.040373] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: ffffffff93c2f280 [ 4709.048336] RDX: 0000000000000000 RSI: ffff8887fcd05000 RDI: 0000000000000008 [ 4709.056299] RBP: ffffed1102ae1ca7 R08: 0000000000000001 R09: fffff94000397e8f [ 4709.064262] R10: fffff94000397e8e R11: ffffea0001cbf477 R12: ffff88881570e400 [ 4709.072225] R13: 0000000000003000 R14: 0000000000000010 R15: ffffed1102ae1c9f [ 4709.080188] FS: 00007f89493b6b80(0000) GS:ffff888827a00000(0000) knlGS:0000000000000000 [ 4709.089217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4709.095628] CR2: 00007f89493b5000 CR3: 00000007ce162004 CR4: 00000000001606f0 [ 4709.103590] Call Trace: [ 4709.106327] ? __x64_sys_tee+0x220/0x220 [ 4709.110704] ? generic_file_splice_read+0x4f5/0x6c0 [ 4709.116148] ? add_to_pipe+0x370/0x370 [ 4709.120330] ? _cond_resched+0x15/0x30 [ 4709.124518] direct_splice_actor+0x107/0x1d0 [ 4709.129284] splice_direct_to_actor+0x32d/0x8a0 [ 4709.134342] ? wakeup_pipe_readers+0x80/0x80 [ 4709.139099] ? do_splice_to+0x140/0x140 [ 4709.143381] ? security_file_permission+0x53/0x2b0 [ 4709.148738] do_splice_direct+0x158/0x250 [ 4709.153212] ? splice_direct_to_actor+0x8a0/0x8a0 [ 4709.158464] ? __sb_start_write+0x1c4/0x310 [ 4709.163125] vfs_copy_file_range+0x39c/0xa40 [ 4709.167890] ? __x64_sys_sendfile+0x1d0/0x1d0 [ 4709.172753] ? lockdep_hardirqs_on+0x590/0x590 [ 4709.177706] ? lock_downgrade+0x6d0/0x6d0 [ 4709.182180] ? lock_acquire+0x15a/0x3d0 [ 4709.186459] ? __might_fault+0xc4/0x1a0 [ 4709.190754] __x64_sys_copy_file_range+0x1e8/0x460 [ 4709.196101] ? __ia32_sys_copy_file_range+0x460/0x460 [ 4709.201749] ? __audit_syscall_exit+0x796/0xab0 [ 4709.206810] do_syscall_64+0x9f/0x4f0 [ 4709.210897] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 4709.216534] RIP: 0033:0x7f89488a96fd [ 4709.220523] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5b 57 2c 00 f7 d8 64 89 01 48 [ 4709.241479] RSP: 002b:00007fff83524e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000146 [ 4709.249928] RAX: ffffffffffffffda RBX: 00007fff83524ee8 RCX: 00007f89488a96fd [ 4709.257891] RDX: 0000000000000004 RSI: 00007fff83524ee0 RDI: 0000000000000003 [ 4709.265854] RBP: 0000000000010fcc R08: 0000000000010fcc R09: 0000000000000000 [ 4709.273817] R10: 00007fff83524ee8 R11: 0000000000000246 R12: 00007fff83524ee0 [ 4709.281779] R13: 0000000000000003 R14: 0000000000000004 R15: 0000000000214da7 [ 4709.289746] Modules linked in: intel_rapl_msr intel_rapl_common iTCO_wdt iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp dcdbas coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf dax_pmem_compat device_dax nd_pmem dax_pmem_core pcspkr mei_me ipmi_ssif mei lpc_ich sg ipmi_si ipmi_devintf ipmi_msghandler rfkill sunrpc acpi_power_meter ip_tables xfs libcrc32c sd_mod mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_vram_helper lpfc drm_ttm_helper ttm nvmet_fc nvmet drm nvme_fc crc32c_intel nvme_fabrics ahci igb libahci nvme_core libata scsi_transport_fc megaraid_sas dca i2c_algo_bit wmi [ 4709.358683] ---[ end trace 2d7c5824fba18cef ]--- [ 4709.432470] RIP: 0010:iter_file_splice_write+0x668/0xa00 [ 4709.438415] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 97 02 00 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 49 89 f8 49 c1 e8 03 <41> 80 3c 18 00 0f 85 96 02 00 00 48 8b 52 08 4c 89 e7 41 83 c6 01 [ 4709.459386] RSP: 0018:ffff8887ca8bf8d8 EFLAGS: 00010202 [ 4709.465230] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: ffffffff93c2f280 [ 4709.473196] RDX: 0000000000000000 RSI: ffff8887fcd05000 RDI: 0000000000000008 [ 4709.481161] RBP: ffffed1102ae1ca7 R08: 0000000000000001 R09: fffff94000397e8f [ 4709.489138] R10: fffff94000397e8e R11: ffffea0001cbf477 R12: ffff88881570e400 [ 4709.497112] R13: 0000000000003000 R14: 0000000000000010 R15: ffffed1102ae1c9f [ 4709.505079] FS: 00007f89493b6b80(0000) GS:ffff888827a00000(0000) knlGS:0000000000000000 [ 4709.514110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4709.520534] CR2: 00007f89493b5000 CR3: 00000007ce162004 CR4: 00000000001606f0 [ 4715.584506] XFS (sda5): Unmounting Filesystem
(In reply to Zorro Lang from comment #3) > (In reply to Darrick J. Wong from comment #2) > > Could you please post the source line translations of the relevant > > functions? I don't have your kernel build. > > I already removed this testing kernel build, and merged lots of new patches. > But good news is I still can reproduce this issue[1] (by g/461 this time). > I'll build the new kernel and post the source line translations of the > relevant functions later. > > > [ 4693.175856] run fstests generic/461 at 2019-12-04 21:46:00 > [ 4693.694096] XFS (sda5): Mounting V5 Filesystem > [ 4693.703963] XFS (sda5): Ending clean mount > [ 4693.710992] xfs filesystem being mounted at /mnt/xfstests/mnt2 supports > timestamps until 2038 (0x7fffffff) > [ 4693.726744] XFS (sda5): User initiated shutdown received. Shutting down > filesystem > [ 4693.740549] XFS (sda5): Unmounting Filesystem > [ 4693.895876] XFS (sda5): Mounting V5 Filesystem > [ 4693.905492] XFS (sda5): Ending clean mount > [ 4693.912655] xfs filesystem being mounted at /mnt/xfstests/mnt2 supports > timestamps until 2038 (0x7fffffff) > [ 4702.015718] restraintd[1391]: *** Current Time: Wed Dec 04 21:46:11 2019 > Localwatchdog at: Fri Dec 06 20:32:11 2019 > [ 4708.950866] XFS (sda5): User initiated shutdown received. Shutting down > filesystem > [ 4708.972833] kasan: CONFIG_KASAN_INLINE enabled > [ 4708.977801] kasan: GPF could be caused by NULL-ptr deref or user memory > access > [ 4708.985889] general protection fault: 0000 [#1] SMP KASAN PTI > [ 4708.992294] CPU: 0 PID: 19412 Comm: fsstress Not tainted 5.4.0+ #1 > [ 4708.999190] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.2.10 > 03/09/2015 > [ 4709.007655] RIP: 0010:iter_file_splice_write+0x668/0xa00 # ./scripts/faddr2line vmlinux iter_file_splice_write+0x668 iter_file_splice_write+0x668/0xa00: pipe_buf_release at include/linux/pipe_fs_i.h:187 (inlined by) iter_file_splice_write at fs/splice.c:773 691 ssize_t 692 iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, 693 loff_t *ppos, size_t len, unsigned int flags) 694 { 695 struct splice_desc sd = { 696 .total_len = len, 697 .flags = flags, 698 .pos = *ppos, 699 .u.file = out, 700 }; 701 int nbufs = pipe->max_usage; 702 struct bio_vec *array = kcalloc(nbufs, sizeof(struct bio_vec), 703 GFP_KERNEL); 704 ssize_t ret; 705 706 if (unlikely(!array)) 707 return -ENOMEM; 708 709 pipe_lock(pipe); 710 711 splice_from_pipe_begin(&sd); 712 while (sd.total_len) { 713 struct iov_iter from; 714 unsigned int head = pipe->head; 715 unsigned int tail = pipe->tail; 716 unsigned int mask = pipe->ring_size - 1; 717 size_t left; 718 int n; 719 720 ret = splice_from_pipe_next(pipe, &sd); 721 if (ret <= 0) 722 break; 723 724 if (unlikely(nbufs < pipe->max_usage)) { 725 kfree(array); 726 nbufs = pipe->max_usage; 727 array = kcalloc(nbufs, sizeof(struct bio_vec), 728 GFP_KERNEL); 729 if (!array) { 730 ret = -ENOMEM; 731 break; 732 } 733 } 734 735 /* build the vector */ 736 left = sd.total_len; 737 for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++, n++) { 738 struct pipe_buffer *buf = &pipe->bufs[tail & mask]; 739 size_t this_len = buf->len; 740 741 if (this_len > left) 742 this_len = left; 743 744 ret = pipe_buf_confirm(pipe, buf); 745 if (unlikely(ret)) { 746 if (ret == -ENODATA) 747 ret = 0; 748 goto done; 749 } 750 751 array[n].bv_page = buf->page; 752 array[n].bv_len = this_len; 753 array[n].bv_offset = buf->offset; 754 left -= this_len; 755 } 756 757 iov_iter_bvec(&from, WRITE, array, n, sd.total_len - left); 758 ret = vfs_iter_write(out, &from, &sd.pos, 0); 759 if (ret <= 0) 760 break; 761 762 sd.num_spliced += ret; 763 sd.total_len -= ret; 764 *ppos = sd.pos; 765 766 /* dismiss the fully eaten buffers, adjust the partial one */ 767 tail = pipe->tail; 768 while (ret) { 769 struct pipe_buffer *buf = &pipe->bufs[tail & mask]; 770 if (ret >= buf->len) { 771 ret -= buf->len; 772 buf->len = 0; 773 pipe_buf_release(pipe, buf); 774 tail++; 775 pipe->tail = tail; 776 if (pipe->files) 777 sd.need_wakeup = true; 778 } else { 779 buf->offset += ret; 780 buf->len -= ret; 781 ret = 0; 782 } 783 } 784 } 785 done: 786 kfree(array); 787 splice_from_pipe_end(pipe, &sd); 788 789 pipe_unlock(pipe); 790 791 if (sd.num_spliced) 792 ret = sd.num_spliced; 793 794 return ret; 795 } And 181 static inline void pipe_buf_release(struct pipe_inode_info *pipe, 182 struct pipe_buffer *buf) 183 { 184 const struct pipe_buf_operations *ops = buf->ops; 185 186 buf->ops = NULL; 187 ops->release(pipe, buf); 188 } > [ 4709.013584] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 97 02 00 > 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 49 89 f8 49 c1 e8 03 <41> > 80 3c 18 00 0f 85 96 02 00 00 48 8b 52 08 4c 89 e7 41 83 c6 01 > [ 4709.034540] RSP: 0018:ffff8887ca8bf8d8 EFLAGS: 00010202 > [ 4709.040373] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: > ffffffff93c2f280 > [ 4709.048336] RDX: 0000000000000000 RSI: ffff8887fcd05000 RDI: > 0000000000000008 > [ 4709.056299] RBP: ffffed1102ae1ca7 R08: 0000000000000001 R09: > fffff94000397e8f > [ 4709.064262] R10: fffff94000397e8e R11: ffffea0001cbf477 R12: > ffff88881570e400 > [ 4709.072225] R13: 0000000000003000 R14: 0000000000000010 R15: > ffffed1102ae1c9f > [ 4709.080188] FS: 00007f89493b6b80(0000) GS:ffff888827a00000(0000) > knlGS:0000000000000000 > [ 4709.089217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 4709.095628] CR2: 00007f89493b5000 CR3: 00000007ce162004 CR4: > 00000000001606f0 > [ 4709.103590] Call Trace: > [ 4709.106327] ? __x64_sys_tee+0x220/0x220 > [ 4709.110704] ? generic_file_splice_read+0x4f5/0x6c0 > [ 4709.116148] ? add_to_pipe+0x370/0x370 > [ 4709.120330] ? _cond_resched+0x15/0x30 > [ 4709.124518] direct_splice_actor+0x107/0x1d0 > [ 4709.129284] splice_direct_to_actor+0x32d/0x8a0 > [ 4709.134342] ? wakeup_pipe_readers+0x80/0x80 > [ 4709.139099] ? do_splice_to+0x140/0x140 > [ 4709.143381] ? security_file_permission+0x53/0x2b0 > [ 4709.148738] do_splice_direct+0x158/0x250 > [ 4709.153212] ? splice_direct_to_actor+0x8a0/0x8a0 > [ 4709.158464] ? __sb_start_write+0x1c4/0x310 > [ 4709.163125] vfs_copy_file_range+0x39c/0xa40 > [ 4709.167890] ? __x64_sys_sendfile+0x1d0/0x1d0 > [ 4709.172753] ? lockdep_hardirqs_on+0x590/0x590 > [ 4709.177706] ? lock_downgrade+0x6d0/0x6d0 > [ 4709.182180] ? lock_acquire+0x15a/0x3d0 > [ 4709.186459] ? __might_fault+0xc4/0x1a0 > [ 4709.190754] __x64_sys_copy_file_range+0x1e8/0x460 > [ 4709.196101] ? __ia32_sys_copy_file_range+0x460/0x460 > [ 4709.201749] ? __audit_syscall_exit+0x796/0xab0 > [ 4709.206810] do_syscall_64+0x9f/0x4f0 > [ 4709.210897] entry_SYSCALL_64_after_hwframe+0x49/0xbe > [ 4709.216534] RIP: 0033:0x7f89488a96fd > [ 4709.220523] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 > 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> > 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5b 57 2c 00 f7 d8 64 89 01 48 > [ 4709.241479] RSP: 002b:00007fff83524e98 EFLAGS: 00000246 ORIG_RAX: > 0000000000000146 > [ 4709.249928] RAX: ffffffffffffffda RBX: 00007fff83524ee8 RCX: > 00007f89488a96fd > [ 4709.257891] RDX: 0000000000000004 RSI: 00007fff83524ee0 RDI: > 0000000000000003 > [ 4709.265854] RBP: 0000000000010fcc R08: 0000000000010fcc R09: > 0000000000000000 > [ 4709.273817] R10: 00007fff83524ee8 R11: 0000000000000246 R12: > 00007fff83524ee0 > [ 4709.281779] R13: 0000000000000003 R14: 0000000000000004 R15: > 0000000000214da7 > [ 4709.289746] Modules linked in: intel_rapl_msr intel_rapl_common iTCO_wdt > iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp dcdbas > coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul > ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf > dax_pmem_compat device_dax nd_pmem dax_pmem_core pcspkr mei_me ipmi_ssif mei > lpc_ich sg ipmi_si ipmi_devintf ipmi_msghandler rfkill sunrpc > acpi_power_meter ip_tables xfs libcrc32c sd_mod mgag200 drm_kms_helper > syscopyarea sysfillrect sysimgblt fb_sys_fops drm_vram_helper lpfc > drm_ttm_helper ttm nvmet_fc nvmet drm nvme_fc crc32c_intel nvme_fabrics ahci > igb libahci nvme_core libata scsi_transport_fc megaraid_sas dca i2c_algo_bit > wmi > [ 4709.358683] ---[ end trace 2d7c5824fba18cef ]--- > [ 4709.432470] RIP: 0010:iter_file_splice_write+0x668/0xa00 > [ 4709.438415] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 97 02 00 > 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 49 89 f8 49 c1 e8 03 <41> > 80 3c 18 00 0f 85 96 02 00 00 48 8b 52 08 4c 89 e7 41 83 c6 01 > [ 4709.459386] RSP: 0018:ffff8887ca8bf8d8 EFLAGS: 00010202 > [ 4709.465230] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: > ffffffff93c2f280 > [ 4709.473196] RDX: 0000000000000000 RSI: ffff8887fcd05000 RDI: > 0000000000000008 > [ 4709.481161] RBP: ffffed1102ae1ca7 R08: 0000000000000001 R09: > fffff94000397e8f > [ 4709.489138] R10: fffff94000397e8e R11: ffffea0001cbf477 R12: > ffff88881570e400 > [ 4709.497112] R13: 0000000000003000 R14: 0000000000000010 R15: > ffffed1102ae1c9f > [ 4709.505079] FS: 00007f89493b6b80(0000) GS:ffff888827a00000(0000) > knlGS:0000000000000000 > [ 4709.514110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 4709.520534] CR2: 00007f89493b5000 CR3: 00000007ce162004 CR4: > 00000000001606f0 > [ 4715.584506] XFS (sda5): Unmounting Filesystem
Hit this issue again on upstream mainline kernel 5.5.0-rc4, the HEAD commit is: commit fd6988496e79a6a4bdb514a4655d2920209eb85d Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sun Dec 29 15:29:16 2019 -0800 Linux 5.5-rc4 [31287.078801] kasan: CONFIG_KASAN_INLINE enabled [31287.141302] kasan: GPF could be caused by NULL-ptr deref or user memory access [31287.175211] general protection fault: 0000 [#1] SMP KASAN PTI [31287.202065] CPU: 18 PID: 14354 Comm: fsstress Not tainted 5.5.0-rc4+ #1 [31287.231860] Hardware name: HP ProLiant XL190r Gen9/ProLiant XL190r Gen9, BIOS U14 05/22/2018 [31287.270854] RIP: 0010:iter_file_splice_write+0x65f/0xa10 [31287.296139] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 e8 02 00 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 48 89 f9 48 c1 e9 03 <80> 3c 19 00 0f 85 80 02 00 00 48 8b 52 08 4c 89 ff 41 83 c5 01 e8 [31287.384468] RSP: 0018:ffff888074e778e0 EFLAGS: 00010202 [31287.409475] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000001 [31287.442886] RDX: 0000000000000000 RSI: ffff888045ddc000 RDI: 0000000000000008 [31287.475859] RBP: ffffed100909fa27 R08: fffff9400014fa97 R09: fffff9400014fa97 [31287.509173] R10: fffff9400014fa96 R11: ffffea0000a7d4b7 R12: 000000000000d38a [31287.543272] R13: 0000000000000010 R14: ffffed100909fa1f R15: ffff8880484fd000 [31287.577414] FS: 00007fa75d7f2b80(0000) GS:ffff888114200000(0000) knlGS:0000000000000000 [31287.616719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31287.645465] CR2: 00007ffb22049978 CR3: 000000010055c006 CR4: 00000000001606e0 [31287.681499] Call Trace: [31287.693418] ? __x64_sys_tee+0x220/0x220 [31287.712293] ? generic_file_splice_read+0x4f5/0x6c0 [31287.735384] ? add_to_pipe+0x370/0x370 [31287.753082] ? _cond_resched+0x15/0x30 [31287.770530] direct_splice_actor+0x107/0x1d0 [31287.790652] splice_direct_to_actor+0x32d/0x8a0 [31287.812108] ? wakeup_pipe_readers+0x80/0x80 [31287.831962] ? do_splice_to+0x140/0x140 [31287.849955] ? security_file_permission+0x53/0x2b0 [31287.872161] do_splice_direct+0x158/0x250 [31287.890904] ? splice_direct_to_actor+0x8a0/0x8a0 [31287.912841] ? __sb_start_write+0x1c4/0x310 [31287.932673] vfs_copy_file_range+0x39c/0xa40 [31287.952671] ? __x64_sys_sendfile+0x1d0/0x1d0 [31287.972959] ? lockdep_hardirqs_on+0x590/0x590 [31287.993801] ? lock_downgrade+0x6d0/0x6d0 [31288.012379] ? lock_acquire+0x15a/0x3d0 [31288.030130] ? __might_fault+0xc4/0x1a0 [31288.048482] __x64_sys_copy_file_range+0x1e8/0x460 [31288.070780] ? __ia32_sys_copy_file_range+0x460/0x460 [31288.094431] ? __audit_syscall_exit+0x796/0xab0 [31288.115526] do_syscall_64+0x9f/0x4f0 [31288.133481] entry_SYSCALL_64_after_hwframe+0x49/0xbe [31288.160230] RIP: 0033:0x7fa75cce36ed [31288.178391] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6b 57 2c 00 f7 d8 64 89 01 48 [31288.266254] RSP: 002b:00007ffe44247728 EFLAGS: 00000246 ORIG_RAX: 0000000000000146 [31288.302144] RAX: ffffffffffffffda RBX: 000000000000037d RCX: 00007fa75cce36ed [31288.335625] RDX: 0000000000000004 RSI: 00007ffe44247760 RDI: 0000000000000003 [31288.368948] RBP: 00000000006ec600 R08: 000000000001092c R09: 0000000000000000 [31288.402554] R10: 00007ffe44247768 R11: 0000000000000246 R12: 0000000000000003 [31288.435810] R13: 000000000001092c R14: 0000000000027b80 R15: 00000000001037f6 [31288.469094] Modules linked in: loop intel_rapl_msr iTCO_wdt intel_rapl_common iTCO_vendor_support sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf pcspkr nd_pmem dax_pmem_compat device_dax dax_pmem_core rfkill i2c_i801 lpc_ich hpilo hpwdt sunrpc ipmi_ssif ioatdma ipmi_si acpi_tad ipmi_devintf ipmi_msghandler acpi_power_meter ext4 mbcache jbd2 ip_tables xfs libcrc32c sd_mod sg mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_vram_helper drm_ttm_helper ttm drm ahci igb serio_raw crc32c_intel libahci libata tg3 dca i2c_algo_bit wmi [31288.752336] ---[ end trace 568a174fdf5e3acb ]--- [31288.782986] RIP: 0010:iter_file_splice_write+0x65f/0xa10 [31288.807838] Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 1a 00 0f 85 e8 02 00 00 48 8b 56 10 48 c7 46 10 00 00 00 00 48 8d 7a 08 48 89 f9 48 c1 e9 03 <80> 3c 19 00 0f 85 80 02 00 00 48 8b 52 08 4c 89 ff 41 83 c5 01 e8 [31288.895645] RSP: 0018:ffff888074e778e0 EFLAGS: 00010202 [31288.920068] RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000001 [31288.953407] RDX: 0000000000000000 RSI: ffff888045ddc000 RDI: 0000000000000008 [31288.986935] RBP: ffffed100909fa27 R08: fffff9400014fa97 R09: fffff9400014fa97 [31289.019794] R10: fffff9400014fa96 R11: ffffea0000a7d4b7 R12: 000000000000d38a [31289.052915] R13: 0000000000000010 R14: ffffed100909fa1f R15: ffff8880484fd000 [31289.086764] FS: 00007fa75d7f2b80(0000) GS:ffff888114200000(0000) knlGS:0000000000000000 [31289.124620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [31289.151387] CR2: 00007ffb22049978 CR3: 000000010055c006 CR4: 00000000001606e0 [31301.358103] XFS (sdb2): Unmounting Filesystem [31301.706524] XFS (sdb2): Mounting V5 Filesystem