Description: Inside qemu-VM, rsync in guest-VM-kernel emits traces when 85Gib of data is rsynced from src:vhdx/ntfs3 to dst:qcow2(dynamic)/ext4. Bug did not show up when dst was dd-raw-img or qcow2(fixed). Its difficult for me to ascertain whether this is qemu-bug or kernel-bug. So, I have also filed a bug in qemu. https://gitlab.com/qemu-project/qemu/-/issues/854 Steps to reproduce 1) Inside booted vm, use gdisk to partition /dev/sda1 if necessary 2) mkfs.ext4 /dev/sda1 -L fs_gkpics001 3) mkdir /mnt/a /mnt/b 4) mount -t ext4 /dev/sda1 /mnt/a 5) mount -t ntfs3 /dev/sdb2 /mnt/b 6) rsync testdata: (sdate=`date` ; echo "$sdate" ; cd /mnt/b ; rsync -avH ./photos001 /mnt/a | tee /tmp/rst.txt ; echo "$sdate" ; date ) 7) umount /mnt/a ; 8) mount -t ext4 /dev/sda1 /mnt/a 9) verify: (sdate=`date` ; echo "$sdate" ; cd /mnt/a/photos001 ; sha256sum -c ./find.CHECKSUM --quiet ; echo "$sdate" ; date ) 10) dmesg -w (in another pty) 11) umount /mnt/a ; umount /mnt/b; HOST * OS: fedora-35 * kernel: 5.17.0-0.rc2 (from koji https://koji.fedoraproject.org/koji/packageinfo?packageID=8) * qemu: 6.2.0-2.fc36.x86_64 (from koji https://koji.fedoraproject.org/koji/packageinfo?packageID=3685) GUEST * guest kernel: also 5.17.0-0.rc2 Livecd: * Fedora-Workstation-Live-x86_64-Rawhide-20220201.n.0.iso virtual-disk1 (sda): * dynamic qcow2 * fresh/blank, partition and filesystem created after booting livecd * qemu-img create -f qcow2 /mnt/a16/gkpics01.qcow2 99723771904 * located on an exfat partition virtual-disk2 (sdb): * vhdx with ntfs3 containing synthetic data generated as per script-lets included in comment in issue 727 filed against qemu * https://gitlab.com/qemu-project/qemu/-/issues/727#note_739930694 * located on an ntfs3 partition QEMU invocation: qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20220201.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15" Other Tests/Observations * bug does not happen when virtual disk is raw-img, or fixed qcow2, this indicates that bug may be timing related. * no traceback if rsync test is done directly on host: 5.17.0-0.rc2/ExFAT from ntfs3/sgdata * no traceback if rsync test is done directly on host: 5.17.0-0.rc2/ntfs3 from ntfs3/sgdata The kernel trace: [ 1020.645483] general protection fault, probably for non-canonical address 0xffa0b585e6584000: 0000 [#1] PREEMPT SMP PTI [ 1020.645497] CPU: 4 PID: 2470 Comm: rsync Not tainted 5.17.0-0.rc2.83.fc36.x86_64 #1 [ 1020.645514] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1020.645516] RIP: 0010:obj_cgroup_charge_pages+0x9b/0xe0 [ 1020.645526] Code: e8 ea 21 e2 ff 89 e8 5b 5d 41 5c 41 5d c3 e8 5c e5 e1 ff 48 8b 43 10 a8 03 75 29 65 48 ff 00 e8 cb 21 e2 ff eb 8b 48 8b 43 18 <f0> 48 83 28 01 75 cf 48 8b 43 18 48 8d 7b 10 48 8b 40 08 e8 1d 93 [ 1020.645529] RSP: 0018:ffffae2a01debbc8 EFLAGS: 00010206 [ 1020.645532] RAX: ffa0b585e6584000 RBX: ffffa0b5847cbfff RCX: 0000000000000000 [ 1020.645534] RDX: ffffa0b5d50cce80 RSI: 000000000000026d RDI: 0007ffffffffffff [ 1020.645536] RBP: 0000000000000000 R08: 0000000000001000 R09: 00000000000ff000 [ 1020.645539] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000008 [ 1020.645541] R13: ffffa0b585e65740 R14: ffffa0b585e65740 R15: 0000000000000001 [ 1020.645544] FS: 00007fa122790b80(0000) GS:ffffa0b5ffd00000(0000) knlGS:0000000000000000 [ 1020.645546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1020.645548] CR2: 00007fe215ede600 CR3: 0000000001122000 CR4: 00000000000006e0 [ 1020.645553] Call Trace: [ 1020.645563] <TASK> [ 1020.645569] __memcg_kmem_charge_page+0xf3/0x220 [ 1020.645573] __alloc_pages+0x102/0x210 [ 1020.645579] alloc_skb_with_frags+0x173/0x1c0 [ 1020.645586] sock_alloc_send_pskb+0x1db/0x230 [ 1020.645590] ? __check_object_size+0x46/0x140 [ 1020.645596] unix_stream_sendmsg+0x239/0x600 [ 1020.645606] sock_sendmsg+0x5b/0x60 [ 1020.645609] sock_write_iter+0x89/0xd0 [ 1020.645612] ? release_ds_buffers+0x60/0xe0 [ 1020.645616] new_sync_write+0x177/0x190 [ 1020.645622] ? release_ds_buffers+0x60/0xe0 [ 1020.645625] vfs_write+0x208/0x2a0 [ 1020.645629] ksys_write+0x8b/0xc0 [ 1020.645633] do_syscall_64+0x3a/0x80 [ 1020.645639] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1020.645648] RIP: 0033:0x7fa1228b9f97 [ 1020.645687] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 1020.645689] RSP: 002b:00007ffe2bc4baa8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1020.645692] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fa1228b9f97 [ 1020.645694] RDX: 0000000000008008 RSI: 00007fa12458e8c0 RDI: 0000000000000004 [ 1020.645696] RBP: 00007fa122fb1080 R08: 0000000000008008 R09: 0000000000000000 [ 1020.645697] R10: 00007ffe2bc4bb80 R11: 0000000000000246 R12: 00007ffe2bc4bc00 [ 1020.645699] R13: 0000000000000002 R14: 00007ffe2bc4bb00 R15: 00007ffe2bc4bb80 [ 1020.645703] </TASK> [ 1020.645706] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt virtio_gpu i2c_i801 intel_pmc_bxt iTCO_vendor_support i2c_smbus ppdev lpc_ich virtio_dma_buf parport_pc parport pcspkr joydev zram isofs squashfs virtio_net net_failover serio_raw failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 1020.646560] ---[ end trace 0000000000000000 ]--- [ 1020.646565] RIP: 0010:obj_cgroup_charge_pages+0x9b/0xe0 [ 1020.646573] Code: e8 ea 21 e2 ff 89 e8 5b 5d 41 5c 41 5d c3 e8 5c e5 e1 ff 48 8b 43 10 a8 03 75 29 65 48 ff 00 e8 cb 21 e2 ff eb 8b 48 8b 43 18 <f0> 48 83 28 01 75 cf 48 8b 43 18 48 8d 7b 10 48 8b 40 08 e8 1d 93 [ 1020.646576] RSP: 0018:ffffae2a01debbc8 EFLAGS: 00010206 [ 1020.646580] RAX: ffa0b585e6584000 RBX: ffffa0b5847cbfff RCX: 0000000000000000 [ 1020.646583] RDX: ffffa0b5d50cce80 RSI: 000000000000026d RDI: 0007ffffffffffff [ 1020.646585] RBP: 0000000000000000 R08: 0000000000001000 R09: 00000000000ff000 [ 1020.646587] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000008 [ 1020.646589] R13: ffffa0b585e65740 R14: ffffa0b585e65740 R15: 0000000000000001 [ 1020.646591] FS: 00007fa122790b80(0000) GS:ffffa0b5ffd00000(0000) knlGS:0000000000000000 [ 1020.646594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1020.646596] CR2: 00007fe215ede600 CR3: 0000000001122000 CR4: 00000000000006e0 References * https://gitlab.com/qemu-project/qemu/-/issues/854 * A similar bug, but different traceback happens on 5.15 kernels, but on fixed raw images or fixed qcow2: "fs/ntfs3: page_cache_ra_unbounded on rsync from ntfs3 to ext4" https://bugzilla.kernel.org/show_bug.cgi?id=215460
Does not happen on guest kernel 5.17.0-0.rc3 (no kernel-tracebacks / corruption) rsync proceeds to completion. sha256sum is uneventful. Host: 5.17.0-0.rc3.89.fc35.x86_64 (SB=selfbuilt from srpm) Guest: 5.17.0-0.rc3.89.fc36.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220208.n.0.iso qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: qcow2(dyn)/ext4 hdb: vhdx(dyn)/ntfs3/sgdata Done expansion test two times, fresh vm startup. So this was surely a kernel bug. Does this mean that this bug is confirm-ably fixed ? I saw a whole lot of mm code was committed to rc3. Is it possible to document by commenting in this bug report what, ie which commit may have fixed this bug for sure?
Now guest-kernel 5.17-rc4, using same previously described test, gives a traceback and rsync aborted after copying 451 MiB out of 85 GiB. The qcow2 had expanded to 25 GiB of data. Of 428 files that were rsync manged to make, only the last file that was in progress when abort happenned, was corrupted as expected. rsync: connection unexpectedly closed (34026 bytes) rsync error: error in rsync protocol data stream (code 12) at io.c(228) [generator=3.2.3] dst: qcow2(dyn)/ext4 host-kernel: 5.17.0-0.rc4.96.fc35.x86_64 (self built from srpm) guest-kernel: 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso https://koji.fedoraproject.org/koji/buildinfo?buildID=1919340 qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: qcow2(dyn)/ext4 hdb: vhdx(dyn)/ntfs3/sgdata [ 173.450812] sda: sda1 [ 174.512256] sda: sda1 [ 196.057995] ntfs3: Max link count 4000 [ 196.058001] ntfs3: Enabled Linux POSIX ACLs support [ 196.058003] ntfs3: Read-only LZX/Xpress compression included [ 204.340953] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quotamode: none. [ 364.982940] kworker/dying (78) used greatest stack depth: 12112 bytes left [ 1806.089002] general protection fault, probably for non-canonical address 0x2cc05801a22800: 0000 [#1] PREEMPT SMP PTI [ 1806.089016] CPU: 5 PID: 2691 Comm: rsync Not tainted 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 1806.089022] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1806.089025] RIP: 0010:drain_obj_stock+0xb3/0x2e0 [ 1806.089037] Code: 00 00 31 f6 48 c7 c7 e0 0c 19 a6 e8 d7 ab da ff e8 42 ec dc ff 5a 84 c0 0f 84 31 01 00 00 48 8b 45 00 a8 03 0f 85 ba 01 00 00 <65> 48 ff 08 e8 24 ec dc ff 84 c0 0f 84 d6 00 00 00 e8 77 62 dd ff [ 1806.089040] RSP: 0018:ffffadf382b83b60 EFLAGS: 00010246 [ 1806.089044] RAX: 002d1f8406822800 RBX: ffffa0d3fb3f20f0 RCX: 0000000000000001 [ 1806.089048] RDX: ffffffffa43c35c4 RSI: ffffffffa589dcc8 RDI: ffffffffa5854997 [ 1806.089050] RBP: ffffa0d389a3dfbf R08: 0000000000000029 R09: 00000000f0e32550 [ 1806.089052] R10: 00000000ffffffff R11: 00000000ffffffff R12: 0000000000000000 [ 1806.089054] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffa4ba7b11 [ 1806.089056] FS: 00007fe9db4eac40(0000) GS:ffffa0d3fb200000(0000) knlGS:0000000000000000 [ 1806.089059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1806.089061] CR2: 00007fb3ff084600 CR3: 00000000010ce000 CR4: 00000000000006e0 [ 1806.089069] Call Trace: [ 1806.089073] <TASK> [ 1806.089078] refill_obj_stock+0x66/0x230 [ 1806.089087] obj_cgroup_charge+0xca/0x140 [ 1806.089093] kmem_cache_alloc_node+0x92/0x4d0 [ 1806.089099] __alloc_skb+0x171/0x1d0 [ 1806.089106] alloc_skb_with_frags+0x4c/0x1c0 [ 1806.089112] sock_alloc_send_pskb+0x1db/0x230 [ 1806.089125] unix_stream_sendmsg+0x239/0x600 [ 1806.089135] sock_sendmsg+0x5b/0x60 [ 1806.089142] sock_write_iter+0x8c/0xe0 [ 1806.089147] ? intel_pmu_drain_pebs_nhm+0x1b0/0x5e0 [ 1806.089166] new_sync_write+0x17a/0x190 [ 1806.089173] ? intel_pmu_drain_pebs_nhm+0x1b0/0x5e0 [ 1806.089178] vfs_write+0x297/0x3a0 [ 1806.089184] ksys_write+0x97/0xd0 [ 1806.089189] do_syscall_64+0x3a/0x80 [ 1806.089199] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1806.089208] RIP: 0033:0x7fe9db614027 [ 1806.089305] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 1806.089308] RSP: 002b:00007fff78d5f0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1806.089312] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fe9db614027 [ 1806.089314] RDX: 0000000000008008 RSI: 00005575c045c8c0 RDI: 0000000000000004 [ 1806.089316] RBP: 00005575bf0f5080 R08: 0000000000008008 R09: 0000000000000000 [ 1806.089318] R10: 00007fff78d5f1a0 R11: 0000000000000246 R12: 00007fff78d5f220 [ 1806.089320] R13: 0000000000000002 R14: 00007fff78d5f120 R15: 00007fff78d5f1a0 [ 1806.089327] </TASK> [ 1806.089333] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr ppdev iTCO_wdt virtio_gpu intel_pmc_bxt iTCO_vendor_support i2c_i801 parport_pc i2c_smbus lpc_ich virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 1806.089438] ---[ end trace 0000000000000000 ]--- [ 1806.089442] RIP: 0010:drain_obj_stock+0xb3/0x2e0 [ 1806.089448] Code: 00 00 31 f6 48 c7 c7 e0 0c 19 a6 e8 d7 ab da ff e8 42 ec dc ff 5a 84 c0 0f 84 31 01 00 00 48 8b 45 00 a8 03 0f 85 ba 01 00 00 <65> 48 ff 08 e8 24 ec dc ff 84 c0 0f 84 d6 00 00 00 e8 77 62 dd ff [ 1806.089451] RSP: 0018:ffffadf382b83b60 EFLAGS: 00010246 [ 1806.089455] RAX: 002d1f8406822800 RBX: ffffa0d3fb3f20f0 RCX: 0000000000000001 [ 1806.089457] RDX: ffffffffa43c35c4 RSI: ffffffffa589dcc8 RDI: ffffffffa5854997 [ 1806.089460] RBP: ffffa0d389a3dfbf R08: 0000000000000029 R09: 00000000f0e32550 [ 1806.089462] R10: 00000000ffffffff R11: 00000000ffffffff R12: 0000000000000000 [ 1806.089465] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffa4ba7b11 [ 1806.089467] FS: 00007fe9db4eac40(0000) GS:ffffa0d3fb200000(0000) knlGS:0000000000000000 [ 1806.089470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1806.089473] CR2: 00007fb3ff084600 CR3: 00000000010ce000 CR4: 00000000000006e0 [ 1806.089479] note: rsync[2691] exited with preempt_count 1 [ 1806.089487] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:49 [ 1806.089490] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 2691, name: rsync [ 1806.089493] preempt_count: 0, expected: 0 [ 1806.089495] RCU nest depth: 1, expected: 0 [ 1806.089499] INFO: lockdep is turned off. [ 1806.089502] CPU: 5 PID: 2691 Comm: rsync Tainted: G D --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 1806.089507] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1806.089509] Call Trace: [ 1806.089515] <TASK> [ 1806.089517] dump_stack_lvl+0x5e/0x77 [ 1806.089524] __might_resched.cold+0x101/0x13c [ 1806.089528] exit_signals+0x1a/0x320 [ 1806.089534] do_exit+0x158/0xc20 [ 1806.089538] make_task_dead+0x51/0x60 [ 1806.089540] rewind_stack_and_make_dead+0x17/0x17 [ 1806.089545] RIP: 0033:0x7fe9db614027 [ 1806.089556] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 1806.089558] RSP: 002b:00007fff78d5f0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1806.089560] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fe9db614027 [ 1806.089561] RDX: 0000000000008008 RSI: 00005575c045c8c0 RDI: 0000000000000004 [ 1806.089563] RBP: 00005575bf0f5080 R08: 0000000000008008 R09: 0000000000000000 [ 1806.089564] R10: 00007fff78d5f1a0 R11: 0000000000000246 R12: 00007fff78d5f220 [ 1806.089565] R13: 0000000000000002 R14: 00007fff78d5f120 R15: 00007fff78d5f1a0 [ 1806.089569] </TASK> [root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15" qemu-system-x86_64: warning: hub 0 with no nics [root@sirius gana]# ls -l /mnt/a16/gkpics01.qcow2 -rwxr-xr-x. 1 root root 2569994240 Feb 17 20:44 /mnt/a16/gkpics01.qcow2 --------- The guest-kernel 5.17-rc4 also crashes catastrophically using the same previously described test with kernel trace-back after copying about 39 GiB of data. However, the expanding VHDX of qemu may itself be another complicating factor. dst: vhdx(dyn) host-kernel: 5.17.0-0.rc4.96.fc35.x86_64 (self built from srpm) guest-kernel: 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: vhdx(dyn)/ntfs3 hdb: vhdx(dyn)/ntfs3/sgdata [ 25.428640] NET: Registered PF_QIPCRTR protocol family [ 40.488169] systemd-journald[922]: Time jumped backwards, rotating. [ 44.186316] rfkill: input handler disabled [ 344.362616] kworker/dying (81) used greatest stack depth: 12160 bytes left [ 345.299943] ntfs3: Max link count 4000 [ 345.299952] ntfs3: Enabled Linux POSIX ACLs support [ 345.299955] ntfs3: Read-only LZX/Xpress compression included [ 361.226526] kworker/dying (107) used greatest stack depth: 12112 bytes left [ 472.620198] sched: RT throttling activated [ 1259.505021] kworker/dying (1842) used greatest stack depth: 11680 bytes left [ 1831.815183] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 1831.815199] CPU: 5 PID: 2459 Comm: rsync Not tainted 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 1831.815205] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1831.815207] RIP: 0010:__schedule+0x498/0x1560 [ 1831.815254] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 b9 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 1831.815257] RSP: 0018:ffffb36180193b48 EFLAGS: 00010096 [ 1831.815261] RAX: ffff9cd64036b480 RBX: ffff9cd6bb3f5ec0 RCX: 00000000c0000100 [ 1831.815264] RDX: 0000000000000000 RSI: 00000000929dcc40 RDI: 00000000c0000100 [ 1831.815266] RBP: ffffb36180193bc8 R08: 000001ff708c4b47 R09: 0000000000000000 [ 1831.815269] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9cd64036b480 [ 1831.815271] R13: ffff9cd65107b480 R14: ffffb36180193b90 R15: ffff9cd6bb3f5ec0 [ 1831.815274] FS: 00007f38929dcc40(0000) GS:ffff9cd6bb200000(0000) knlGS:0000000000000000 [ 1831.815277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1831.815279] CR2: 00007fb9674bc300 CR3: 0000000001122000 CR4: 00000000000006e0 [ 1831.815285] Call Trace: [ 1831.815287] <TASK> [ 1831.815291] ? lock_is_held_type+0xea/0x140 [ 1831.815295] ? kvm_sched_clock_read+0x14/0x40 [ 1831.815319] schedule+0x4e/0xc0 [ 1831.815323] io_schedule+0x47/0x70 [ 1831.815326] folio_wait_bit_common+0x13e/0x430 [ 1831.815349] ? dio_warn_stale_pagecache.part.0+0x50/0x50 [ 1831.815353] filemap_get_pages+0x631/0x6e0 [ 1831.815356] ? lock_is_held_type+0xea/0x140 [ 1831.815359] filemap_read+0xbb/0x340 [ 1831.815365] new_sync_read+0x104/0x180 [ 1831.815380] vfs_read+0x129/0x1c0 [ 1831.815384] ksys_read+0x5c/0xd0 [ 1831.815388] do_syscall_64+0x3a/0x80 [ 1831.815400] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1831.815404] RIP: 0033:0x7f3892b05f82 [ 1831.815442] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 7a c5 0b 00 e8 45 f5 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24 [ 1831.815444] RSP: 002b:00007fff3cf3c3f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1831.815447] RAX: ffffffffffffffda RBX: 0000559649fdfb10 RCX: 00007f3892b05f82 [ 1831.815448] RDX: 0000000000040000 RSI: 000055964a1d2990 RDI: 0000000000000003 [ 1831.815449] RBP: 0000000000040000 R08: 000000000005065b R09: 0000000000040000 [ 1831.815451] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000000000 [ 1831.815452] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000040000 [ 1831.815456] </TASK> [ 1831.815458] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev virtio_gpu intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus parport_pc lpc_ich virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 1831.815537] ---[ end trace 0000000000000000 ]--- [ 1831.815540] RIP: 0010:__schedule+0x498/0x1560 [ 1831.815546] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 b9 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 1831.815548] RSP: 0018:ffffb36180193b48 EFLAGS: 00010096 [ 1831.815551] RAX: ffff9cd64036b480 RBX: ffff9cd6bb3f5ec0 RCX: 00000000c0000100 [ 1831.815552] RDX: 0000000000000000 RSI: 00000000929dcc40 RDI: 00000000c0000100 [ 1831.815555] RBP: ffffb36180193bc8 R08: 000001ff708c4b47 R09: 0000000000000000 [ 1831.815557] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9cd64036b480 [ 1831.815559] R13: ffff9cd65107b480 R14: ffffb36180193b90 R15: ffff9cd6bb3f5ec0 [ 1831.815561] FS: 00007f38929dcc40(0000) GS:ffff9cd6bb200000(0000) knlGS:0000000000000000 [ 1831.815564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1831.815566] CR2: 00007fb9674bc300 CR3: 0000000001122000 CR4: 00000000000006e0 [ 1831.815570] note: rsync[2459] exited with preempt_count 2 [ 1880.592386] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/7:1:55] [ 1880.592397] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev virtio_gpu intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus parport_pc lpc_ich virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 1880.592502] irq event stamp: 148390 [ 1880.592507] hardirqs last enabled at (148389): [<ffffffffb9e96064>] _raw_spin_unlock_irq+0x24/0x50 [ 1880.592517] hardirqs last disabled at (148390): [<ffffffffb9e8dffd>] __schedule+0xbbd/0x1560 [ 1880.592523] softirqs last enabled at (147340): [<ffffffffb90f2a63>] __irq_exit_rcu+0xf3/0x170 [ 1880.592640] softirqs last disabled at (147333): [<ffffffffb90f2a63>] __irq_exit_rcu+0xf3/0x170 [ 1880.592645] CPU: 7 PID: 55 Comm: kworker/7:1 Tainted: G D --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 1880.592649] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1880.592656] Workqueue: events netstamp_clear [ 1880.592680] RIP: 0010:smp_call_function_many_cond+0x124/0x3d0 [ 1880.592711] Code: 49 8b 76 08 e8 6d 85 5f 00 3b 05 b7 ad 2d 02 89 c7 73 21 48 63 c7 49 8b 0e 48 03 0c c5 e0 fa 96 ba 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb cc 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e [ 1880.592714] RSP: 0018:ffffb36180217d38 EFLAGS: 00000202 [ 1880.592718] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9cd6bb4002a0 [ 1880.592721] RDX: 0000000000000001 RSI: 0000000000000005 RDI: 0000000000000005 [ 1880.592723] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 1880.592725] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000007 [ 1880.592727] R13: 0000000000000000 R14: ffff9cd6bbbf7300 R15: 0000000000000008 [ 1880.592730] FS: 0000000000000000(0000) GS:ffff9cd6bba00000(0000) knlGS:0000000000000000 [ 1880.592733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1880.592736] CR2: 00007fb9674c1800 CR3: 000000010e028000 CR4: 00000000000006e0 [ 1880.592741] Call Trace: [ 1880.592745] <TASK> [ 1880.592751] ? optimize_nops+0x110/0x110 [ 1880.592786] ? optimize_nops+0x110/0x110 [ 1880.592790] on_each_cpu_cond_mask+0x3c/0x70 [ 1880.592796] ? netif_receive_skb_list_internal+0x48/0x3a0 [ 1880.592815] text_poke_bp_batch+0xda/0x330 [ 1880.592822] text_poke_finish+0x1b/0x30 [ 1880.592826] arch_jump_label_transform_apply+0x18/0x30 [ 1880.592830] static_key_enable_cpuslocked+0x59/0x90 [ 1880.592837] static_key_enable+0x16/0x20 [ 1880.592841] process_one_work+0x29a/0x5f0 [ 1880.592861] worker_thread+0x4f/0x390 [ 1880.592866] ? process_one_work+0x5f0/0x5f0 [ 1880.592870] kthread+0xf5/0x120 [ 1880.592892] ? kthread_complete_and_exit+0x20/0x20 [ 1880.592897] ret_from_fork+0x22/0x30 [ 1880.592919] </TASK> [ 1888.528505] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [llvmpipe-0:1520] [ 1888.528516] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev virtio_gpu intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus parport_pc lpc_ich virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 1888.528604] irq event stamp: 3546894 [ 1888.528607] hardirqs last enabled at (3546893): [<ffffffffb9e83720>] syscall_enter_from_user_mode+0x20/0x70 [ 1888.528616] hardirqs last disabled at (3546894): [<ffffffffb9e8dffd>] __schedule+0xbbd/0x1560 [ 1888.528623] softirqs last enabled at (3546882): [<ffffffffb90f2a63>] __irq_exit_rcu+0xf3/0x170 [ 1888.528628] softirqs last disabled at (3546731): [<ffffffffb90f2a63>] __irq_exit_rcu+0xf3/0x170 [ 1888.528633] CPU: 2 PID: 1520 Comm: llvmpipe-0 Tainted: G D L --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 1888.528637] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 1888.528640] RIP: 0010:smp_call_function_many_cond+0x124/0x3d0 [ 1888.528648] Code: 49 8b 76 08 e8 6d 85 5f 00 3b 05 b7 ad 2d 02 89 c7 73 21 48 63 c7 49 8b 0e 48 03 0c c5 e0 fa 96 ba 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb cc 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e [ 1888.528652] RSP: 0018:ffffb361828fbce8 EFLAGS: 00000202 [ 1888.528655] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9cd6bb800200 [ 1888.528658] RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000006 [ 1888.528661] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 1888.528663] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 1888.528665] R13: ffff9cd6ba7f5640 R14: ffff9cd6ba7f7300 R15: 0000000000000008 [ 1888.528668] FS: 00007fb952a48640(0000) GS:ffff9cd6ba600000(0000) knlGS:0000000000000000 [ 1888.528671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1888.528674] CR2: 00007fb9674c1000 CR3: 0000000107030000 CR4: 00000000000006e0 [ 1888.528681] Call Trace: [ 1888.528684] <TASK> [ 1888.528691] ? leave_mm+0x40/0x40 [ 1888.528877] ? leave_mm+0x40/0x40 [ 1888.528882] on_each_cpu_cond_mask+0x3c/0x70 [ 1888.528891] flush_tlb_mm_range+0x144/0x1b0 [ 1888.528897] tlb_finish_mmu+0x113/0x1a0 [ 1888.528920] unmap_region+0xbd/0xe0 [ 1888.528942] ? rcu_read_lock_sched_held+0x10/0x70 [ 1888.528961] ? lock_acquire+0x26c/0x2d0 [ 1888.528993] __do_munmap+0x1fd/0x540 [ 1888.529001] __vm_munmap+0x67/0xf0 [ 1888.529007] __x64_sys_munmap+0x17/0x20 [ 1888.529012] do_syscall_64+0x3a/0x80 [ 1888.529019] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1888.529024] RIP: 0033:0x7fb96e81b8ab [ 1888.529047] Code: 8b 15 71 75 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3d 75 0e 00 f7 d8 64 89 01 48 [ 1888.529051] RSP: 002b:00007fb952a46ed8 EFLAGS: 00000213 ORIG_RAX: 000000000000000b [ 1888.529055] RAX: ffffffffffffffda RBX: 000055f7c77af260 RCX: 00007fb96e81b8ab [ 1888.529057] RDX: 0000000000000000 RSI: 0000000000140000 RDI: 00007fb96739b000 [ 1888.529060] RBP: 0000000000000000 R08: 000055f7c78a8ee0 R09: 000000000001ed3e [ 1888.529062] R10: 0000000000000000 R11: 0000000000000213 R12: 000055f7c78a7920 [ 1888.529065] R13: 000055f7c7cdb720 R14: 000055f7c78a8ee0 R15: 00007fb952248000 [ 1888.529073] </TASK> [root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso" -hda "/mnt/a16/gkpics01.vhdx" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15" qemu-system-x86_64: warning: hub 0 with no nics [root@sirius gana]# ls -l /mnt/a16/gkpics01.vhdx -rwxr-xr-x. 1 root root 39766196224 Feb 17 19:52 /mnt/a16/gkpics01.vhdx
The guest-kernel 5.17-rc4, using the same described test, upon using a fixed raw img, also crashed catastrophically with kernel trace-back happening fairly early into the copy after starting rsync. dst: raw img [root@sirius gana]# dd if=/dev/zero of=/mnt/a16/gkpics01.raw.img bs=1M count=95104 95104+0 records in 95104+0 records out host-kernel: 5.17.0-0.rc4.96.fc35.x86_64 (self built from srpm) guest-kernel: 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: raw/ext4 hdb: vhdx(dyn)/ntfs3/sgdata [ 107.530891] ntfs3: Max link count 4000 [ 107.530900] ntfs3: Enabled Linux POSIX ACLs support [ 107.530903] ntfs3: Read-only LZX/Xpress compression included [ 114.911302] sda: sda1 [ 115.925788] sda: sda1 [ 142.952551] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quotamode: none. [ 321.211961] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 321.211976] CPU: 0 PID: 2509 Comm: rsync Not tainted 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 321.211982] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 321.211984] RIP: 0010:__schedule+0x498/0x1560 [ 321.211992] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 b0 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 321.211996] RSP: 0018:ffffb05bc0d97b48 EFLAGS: 00010096 [ 321.211999] RAX: ffffffffb2032a00 RBX: ffff9d43741f5ec0 RCX: 00000000c0000100 [ 321.212002] RDX: 0000000000000000 RSI: 00000000b9037c40 RDI: 00000000c0000100 [ 321.212004] RBP: ffffb05bc0d97bc8 R08: 0000005a3161981b R09: 0000000000000000 [ 321.212006] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffffb2032a00 [ 321.212009] R13: ffff9d4351f03480 R14: ffffb05bc0d97b90 R15: ffff9d43741f5ec0 [ 321.212011] FS: 00007f6ab9037c40(0000) GS:ffff9d4374000000(0000) knlGS:0000000000000000 [ 321.212014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 321.212016] CR2: 00007fd86def2900 CR3: 000000007f602000 CR4: 00000000000006f0 [ 321.212021] Call Trace: [ 321.212024] <TASK> [ 321.212027] ? lock_is_held_type+0xea/0x140 [ 321.212032] ? kvm_sched_clock_read+0x14/0x40 [ 321.212042] schedule+0x4e/0xc0 [ 321.212046] io_schedule+0x47/0x70 [ 321.212050] folio_wait_bit_common+0x13e/0x430 [ 321.212058] ? dio_warn_stale_pagecache.part.0+0x50/0x50 [ 321.212064] filemap_get_pages+0x631/0x6e0 [ 321.212070] ? lock_is_held_type+0xea/0x140 [ 321.212076] filemap_read+0xbb/0x340 [ 321.212086] new_sync_read+0x104/0x180 [ 321.212096] vfs_read+0x129/0x1c0 [ 321.212102] ksys_read+0x5c/0xd0 [ 321.212108] do_syscall_64+0x3a/0x80 [ 321.212114] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 321.212119] RIP: 0033:0x7f6ab9160f82 [ 321.212146] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 7a c5 0b 00 e8 45 f5 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24 [ 321.212149] RSP: 002b:00007ffee7873648 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 321.212153] RAX: ffffffffffffffda RBX: 000055641f22c8c0 RCX: 00007f6ab9160f82 [ 321.212156] RDX: 0000000000040000 RSI: 000055641f99eaf0 RDI: 0000000000000003 [ 321.212158] RBP: 0000000000040000 R08: 0000000000051ee2 R09: 0000000000040000 [ 321.212160] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000000000 [ 321.212163] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000040000 [ 321.212182] </TASK> [ 321.212187] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr ppdev iTCO_wdt intel_pmc_bxt iTCO_vendor_support virtio_gpu i2c_i801 lpc_ich i2c_smbus parport_pc virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 321.212280] ---[ end trace 0000000000000000 ]--- [ 321.212282] RIP: 0010:__schedule+0x498/0x1560 [ 321.212287] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 b0 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 321.212289] RSP: 0018:ffffb05bc0d97b48 EFLAGS: 00010096 [ 321.212292] RAX: ffffffffb2032a00 RBX: ffff9d43741f5ec0 RCX: 00000000c0000100 [ 321.212295] RDX: 0000000000000000 RSI: 00000000b9037c40 RDI: 00000000c0000100 [ 321.212297] RBP: ffffb05bc0d97bc8 R08: 0000005a3161981b R09: 0000000000000000 [ 321.212299] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffffb2032a00 [ 321.212301] R13: ffff9d4351f03480 R14: ffffb05bc0d97b90 R15: ffff9d43741f5ec0 [ 321.212303] FS: 00007f6ab9037c40(0000) GS:ffff9d4374000000(0000) knlGS:0000000000000000 [ 321.212305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 321.212308] CR2: 00007fd86def2900 CR3: 000000007f602000 CR4: 00000000000006f0 [ 321.212312] note: rsync[2509] exited with preempt_count 2 [ 386.215801] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 636.718666] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [llvmpipe-0:1529] [ 636.718687] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr ppdev iTCO_wdt intel_pmc_bxt iTCO_vendor_support virtio_gpu i2c_i801 lpc_ich i2c_smbus parport_pc virtio_dma_buf parport joydev pcspkr zram isofs squashfs serio_raw virtio_net net_failover failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 636.718752] irq event stamp: 952280 [ 636.718754] hardirqs last enabled at (952279): [<ffffffffb0e83720>] syscall_enter_from_user_mode+0x20/0x70 [ 636.718763] hardirqs last disabled at (952280): [<ffffffffb0e8dffd>] __schedule+0xbbd/0x1560 [ 636.718767] softirqs last enabled at (952264): [<ffffffffb00f2a63>] __irq_exit_rcu+0xf3/0x170 [ 636.718772] softirqs last disabled at (952257): [<ffffffffb00f2a63>] __irq_exit_rcu+0xf3/0x170 [ 636.718775] CPU: 3 PID: 1529 Comm: llvmpipe-0 Tainted: G D --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 636.718779] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 636.718781] RIP: 0010:smp_call_function_many_cond+0x124/0x3d0 [ 636.718787] Code: 49 8b 76 08 e8 6d 85 5f 00 3b 05 b7 ad 2d 02 89 c7 73 21 48 63 c7 49 8b 0e 48 03 0c c5 e0 fa 96 b1 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb cc 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e [ 636.718790] RSP: 0018:ffffb05bc2783ba0 EFLAGS: 00000202 [ 636.718792] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9d4374600220 [ 636.718795] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000001 [ 636.718796] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 636.718797] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 [ 636.718798] R13: ffff9d4374df5640 R14: ffff9d4374df7300 R15: 0000000000000008 [ 636.718800] FS: 00007fd858eae640(0000) GS:ffff9d4374c00000(0000) knlGS:0000000000000000 [ 636.718802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 636.718803] CR2: 00007fd86df45800 CR3: 0000000107056000 CR4: 00000000000006e0 [ 636.718807] Call Trace: [ 636.718810] <TASK> [ 636.718814] ? leave_mm+0x40/0x40 [ 636.718820] ? leave_mm+0x40/0x40 [ 636.718823] on_each_cpu_cond_mask+0x3c/0x70 [ 636.718827] flush_tlb_mm_range+0x144/0x1b0 [ 636.718831] flush_tlb_batched_pending+0x3c/0x60 [ 636.718834] unmap_page_range+0x6a9/0x1040 [ 636.718841] unmap_vmas+0xb3/0x130 [ 636.718844] unmap_region+0x94/0xe0 [ 636.718848] ? rcu_read_lock_sched_held+0x10/0x70 [ 636.718852] ? rcu_read_lock_sched_held+0x10/0x70 [ 636.718854] ? lock_acquire+0x26c/0x2d0 [ 636.718858] __do_munmap+0x1fd/0x540 [ 636.718862] __vm_munmap+0x67/0xf0 [ 636.718866] __x64_sys_munmap+0x17/0x20 [ 636.718868] do_syscall_64+0x3a/0x80 [ 636.718872] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 636.718876] RIP: 0033:0x7fd874ddb8ab [ 636.718900] Code: 8b 15 71 75 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3d 75 0e 00 f7 d8 64 89 01 48 [ 636.718901] RSP: 002b:00007fd858eaced8 EFLAGS: 00000213 ORIG_RAX: 000000000000000b [ 636.718903] RAX: ffffffffffffffda RBX: 0000556a3dd76cc0 RCX: 00007fd874ddb8ab [ 636.718905] RDX: 0000000000000000 RSI: 0000000000140000 RDI: 00007fd86de2e000 [ 636.718906] RBP: 0000000000000000 R08: 0000556a3de486b0 R09: 000000000000a09c [ 636.718907] R10: 0000000000000000 R11: 0000000000000213 R12: 0000556a3de470f0 [ 636.718909] R13: 0000556a3e27c240 R14: 0000556a3de486b0 R15: 00007fd8586ae000 [ 636.718912] </TASK> [root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso" -drive "file=/mnt/a16/gkpics01.raw.img,format=raw,index=0,media=disk" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15" qemu-system-x86_64: warning: hub 0 with no nics [root@sirius gana]# ls -l /mnt/a16/gkpics01.raw.img -rwxr-xr-x. 1 root root 99723771904 Feb 17 22:30 /mnt/a16/gkpics01.raw.img ----------------------- The guest-kernel 5.17-rc4, using the same described test, upon using a fixed qcow2 virtual disk gave a traceback and crash/freeze, which happened early into the copy after starting rsync. dst: qcow2(fixed) [root@sirius gana]# qemu-img create -f qcow2 -o preallocation=falloc /mnt/a16/gkpics01.qcow2 99723771904 Formatting '/mnt/a16/gkpics01.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=falloc compression_type=zlib size=99723771904 lazy_refcounts=off refcount_bits=16 host-kernel: 5.17.0-0.rc4.96.fc35.x86_64 (self built from srpm) guest-kernel: 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: qcow(fixed)/ext4 hdb: vhdx(dyn)/ntfs3/sgdata [ 112.934180] ntfs3: Max link count 4000 [ 112.934195] ntfs3: Enabled Linux POSIX ACLs support [ 112.934198] ntfs3: Read-only LZX/Xpress compression included [ 125.888823] sda: sda1 [ 126.906228] sda: sda1 [ 147.792846] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quotamode: none. [ 355.557716] kworker/dying (107) used greatest stack depth: 12160 bytes left [ 456.689395] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 456.689407] CPU: 6 PID: 3528 Comm: rsync Not tainted 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 456.689410] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 456.689412] RIP: 0010:__schedule+0x498/0x1560 [ 456.689429] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 89 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 456.689431] RSP: 0018:ffffa585c2cc7b48 EFLAGS: 00010096 [ 456.689434] RAX: ffff9a4cc03bb480 RBX: ffff9a4d3b7f5ec0 RCX: 00000000c0000100 [ 456.689435] RDX: 0000000000000000 RSI: 0000000083e33c40 RDI: 00000000c0000100 [ 456.689437] RBP: ffffa585c2cc7bc8 R08: 00000080285da038 R09: 0000000000000000 [ 456.689438] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9a4cc03bb480 [ 456.689439] R13: ffff9a4cc6b7b480 R14: ffffa585c2cc7b90 R15: ffff9a4d3b7f5ec0 [ 456.689441] FS: 00007f3583e33c40(0000) GS:ffff9a4d3b600000(0000) knlGS:0000000000000000 [ 456.689443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 456.689445] CR2: 00007f91010d8200 CR3: 000000010687c000 CR4: 00000000000006e0 [ 456.689448] Call Trace: [ 456.689450] <TASK> [ 456.689453] ? lock_is_held_type+0xea/0x140 [ 456.689456] ? kvm_sched_clock_read+0x14/0x40 [ 456.689467] schedule+0x4e/0xc0 [ 456.689471] io_schedule+0x47/0x70 [ 456.689474] folio_wait_bit_common+0x13e/0x430 [ 456.689483] ? dio_warn_stale_pagecache.part.0+0x50/0x50 [ 456.689487] filemap_get_pages+0x631/0x6e0 [ 456.689490] ? lock_is_held_type+0xea/0x140 [ 456.689493] filemap_read+0xbb/0x340 [ 456.689499] new_sync_read+0x104/0x180 [ 456.689520] vfs_read+0x129/0x1c0 [ 456.689524] ksys_read+0x5c/0xd0 [ 456.689528] do_syscall_64+0x3a/0x80 [ 456.689534] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 456.689538] RIP: 0033:0x7f3583f5cf82 [ 456.689560] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 7a c5 0b 00 e8 45 f5 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24 [ 456.689562] RSP: 002b:00007ffccfe2bda8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 456.689564] RAX: ffffffffffffffda RBX: 000055756c83edd0 RCX: 00007f3583f5cf82 [ 456.689566] RDX: 0000000000040000 RSI: 000055756cab29b0 RDI: 0000000000000003 [ 456.689567] RBP: 0000000000040000 R08: 000000000005fa54 R09: 0000000000040000 [ 456.689570] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000000000 [ 456.689572] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000040000 [ 456.689576] </TASK> [ 456.689577] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev intel_pmc_bxt iTCO_vendor_support i2c_i801 virtio_gpu i2c_smbus virtio_dma_buf lpc_ich parport_pc parport pcspkr joydev zram isofs squashfs virtio_net net_failover serio_raw failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 456.689638] ---[ end trace 0000000000000000 ]--- [ 456.689640] RIP: 0010:__schedule+0x498/0x1560 [ 456.689644] Code: 48 0d 00 00 48 8d 78 18 48 c7 c6 90 d8 e8 89 e8 4e 02 2e ff 0f 1f 44 00 00 4c 89 63 10 4c 89 ef 4c 89 e6 e8 da 43 17 ff 48 89 <c7> e8 62 9b 29 ff 48 83 c4 58 5b 41 5c 41 5d 41 5e 41 5f 5d c3 65 [ 456.689648] RSP: 0018:ffffa585c2cc7b48 EFLAGS: 00010096 [ 456.689650] RAX: ffff9a4cc03bb480 RBX: ffff9a4d3b7f5ec0 RCX: 00000000c0000100 [ 456.689652] RDX: 0000000000000000 RSI: 0000000083e33c40 RDI: 00000000c0000100 [ 456.689653] RBP: ffffa585c2cc7bc8 R08: 00000080285da038 R09: 0000000000000000 [ 456.689655] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9a4cc03bb480 [ 456.689656] R13: ffff9a4cc6b7b480 R14: ffffa585c2cc7b90 R15: ffff9a4d3b7f5ec0 [ 456.689658] FS: 00007f3583e33c40(0000) GS:ffff9a4d3b600000(0000) knlGS:0000000000000000 [ 456.689660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 456.689661] CR2: 00007f91010d8200 CR3: 000000010687c000 CR4: 00000000000006e0 [ 456.689664] note: rsync[3528] exited with preempt_count 2 [ 480.750941] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [llvmpipe-0:2682] [ 480.750952] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev intel_pmc_bxt iTCO_vendor_support i2c_i801 virtio_gpu i2c_smbus virtio_dma_buf lpc_ich parport_pc parport pcspkr joydev zram isofs squashfs virtio_net net_failover serio_raw failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 480.751017] irq event stamp: 964200 [ 480.751021] hardirqs last enabled at (964199): [<ffffffff89e83720>] syscall_enter_from_user_mode+0x20/0x70 [ 480.751027] hardirqs last disabled at (964200): [<ffffffff89e8dffd>] __schedule+0xbbd/0x1560 [ 480.751031] softirqs last enabled at (963776): [<ffffffff890f2a63>] __irq_exit_rcu+0xf3/0x170 [ 480.751035] softirqs last disabled at (963755): [<ffffffff890f2a63>] __irq_exit_rcu+0xf3/0x170 [ 480.751038] CPU: 3 PID: 2682 Comm: llvmpipe-0 Tainted: G D --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 480.751041] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 480.751042] RIP: 0010:smp_call_function_many_cond+0x124/0x3d0 [ 480.751048] Code: 49 8b 76 08 e8 6d 85 5f 00 3b 05 b7 ad 2d 02 89 c7 73 21 48 63 c7 49 8b 0e 48 03 0c c5 e0 fa 96 8a 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb cc 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e [ 480.751050] RSP: 0018:ffffa585c2d7fce8 EFLAGS: 00000202 [ 480.751052] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9a4d3a000220 [ 480.751054] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000 [ 480.751055] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 480.751056] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 [ 480.751057] R13: ffff9a4d3abf5640 R14: ffff9a4d3abf7300 R15: 0000000000000008 [ 480.751059] FS: 00007f90da0bd640(0000) GS:ffff9a4d3aa00000(0000) knlGS:0000000000000000 [ 480.751061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 480.751062] CR2: 00007f91010d0700 CR3: 0000000109608000 CR4: 00000000000006e0 [ 480.751066] Call Trace: [ 480.751069] <TASK> [ 480.751072] ? leave_mm+0x40/0x40 [ 480.751077] ? leave_mm+0x40/0x40 [ 480.751080] on_each_cpu_cond_mask+0x3c/0x70 [ 480.751084] flush_tlb_mm_range+0x144/0x1b0 [ 480.751088] tlb_finish_mmu+0x113/0x1a0 [ 480.751094] unmap_region+0xbd/0xe0 [ 480.751101] ? rcu_read_lock_sched_held+0x10/0x70 [ 480.751105] ? lock_acquire+0x26c/0x2d0 [ 480.751111] __do_munmap+0x1fd/0x540 [ 480.751116] __vm_munmap+0x67/0xf0 [ 480.751120] __x64_sys_munmap+0x17/0x20 [ 480.751122] do_syscall_64+0x3a/0x80 [ 480.751127] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 480.751132] RIP: 0033:0x7f9107f4a8ab [ 480.751147] Code: 8b 15 71 75 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3d 75 0e 00 f7 d8 64 89 01 48 [ 480.751151] RSP: 002b:00007f90da0bbed8 EFLAGS: 00000213 ORIG_RAX: 000000000000000b [ 480.751154] RAX: ffffffffffffffda RBX: 000055d6cc862420 RCX: 00007f9107f4a8ab [ 480.751155] RDX: 0000000000000000 RSI: 0000000000140000 RDI: 00007f9100faf000 [ 480.751156] RBP: 0000000000000000 R08: 000055d6cb3545b0 R09: 0000000000008e5c [ 480.751158] R10: 0000000000000000 R11: 0000000000000213 R12: 000055d6cb352ff0 [ 480.751159] R13: 000055d6cb770ca0 R14: 000055d6cb3545b0 R15: 00007f90d98bd000 [ 480.751163] </TASK> [ 508.750572] watchdog: BUG: soft lockup - CPU#3 stuck for 48s! [llvmpipe-0:2682] [ 508.750581] Modules linked in: ntfs3 uinput snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr iTCO_wdt ppdev intel_pmc_bxt iTCO_vendor_support i2c_i801 virtio_gpu i2c_smbus virtio_dma_buf lpc_ich parport_pc parport pcspkr joydev zram isofs squashfs virtio_net net_failover serio_raw failover qemu_fw_cfg sunrpc be2iscsi bnx2i cnic uio cxgb4icxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcplibiscsi_tcp libiscsi scsi_transport_iscsi loop ipmi_devintf ipmi_msghandler fuse [ 508.750651] irq event stamp: 964200 [ 508.750652] hardirqs last enabled at (964199): [<ffffffff89e83720>] syscall_enter_from_user_mode+0x20/0x70 [ 508.750660] hardirqs last disabled at (964200): [<ffffffff89e8dffd>] __schedule+0xbbd/0x1560 [ 508.750664] softirqs last enabled at (963776): [<ffffffff890f2a63>] __irq_exit_rcu+0xf3/0x170 [ 508.750669] softirqs last disabled at (963755): [<ffffffff890f2a63>] __irq_exit_rcu+0xf3/0x170 [ 508.750673] CPU: 3 PID: 2682 Comm: llvmpipe-0 Tainted: G D L --------- --- 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64 #1 [ 508.750676] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 [ 508.750678] RIP: 0010:smp_call_function_many_cond+0x124/0x3d0 [ 508.750685] Code: 49 8b 76 08 e8 6d 85 5f 00 3b 05 b7 ad 2d 02 89 c7 73 21 48 63 c7 49 8b 0e 48 03 0c c5 e0 fa 96 8a 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb cc 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e [ 508.750687] RSP: 0018:ffffa585c2d7fce8 EFLAGS: 00000202 [ 508.750690] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9a4d3a000220 [ 508.750692] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000 [ 508.750694] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 508.750696] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 [ 508.750697] R13: ffff9a4d3abf5640 R14: ffff9a4d3abf7300 R15: 0000000000000008 [ 508.750700] FS: 00007f90da0bd640(0000) GS:ffff9a4d3aa00000(0000) knlGS:0000000000000000 [ 508.750702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 508.750704] CR2: 00007f91010d0700 CR3: 0000000109608000 CR4: 00000000000006e0 [ 508.750709] Call Trace: [ 508.750712] <TASK> [ 508.750716] ? leave_mm+0x40/0x40 [ 508.750722] ? leave_mm+0x40/0x40 [ 508.750725] on_each_cpu_cond_mask+0x3c/0x70 [ 508.750731] flush_tlb_mm_range+0x144/0x1b0 [ 508.750736] tlb_finish_mmu+0x113/0x1a0 [ 508.750740] unmap_region+0xbd/0xe0 [ 508.750746] ? rcu_read_lock_sched_held+0x10/0x70 [ 508.750750] ? lock_acquire+0x26c/0x2d0 [ 508.750756] __do_munmap+0x1fd/0x540 [ 508.750761] __vm_munmap+0x67/0xf0 [ 508.750776] __x64_sys_munmap+0x17/0x20 [ 508.750780] do_syscall_64+0x3a/0x80 [ 508.750785] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 508.750789] RIP: 0033:0x7f9107f4a8ab [ 508.750805] Code: 8b 15 71 75 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3d 75 0e 00 f7 d8 64 89 01 48 [ 508.750807] RSP: 002b:00007f90da0bbed8 EFLAGS: 00000213 ORIG_RAX: 000000000000000b [ 508.750810] RAX: ffffffffffffffda RBX: 000055d6cc862420 RCX: 00007f9107f4a8ab [ 508.750812] RDX: 0000000000000000 RSI: 0000000000140000 RDI: 00007f9100faf000 [ 508.750817] RBP: 0000000000000000 R08: 000055d6cb3545b0 R09: 0000000000008e5c [ 508.750819] R10: 0000000000000000 R11: 0000000000000213 R12: 000055d6cb352ff0 [ 508.750821] R13: 000055d6cb770ca0 R14: 000055d6cb3545b0 R15: 00007f90d98bd000 [ 508.750827] </TASK> [root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15" qemu-system-x86_64: warning: hub 0 with no nics [root@sirius gana]# ls -l /mnt/a16/gkpics01.qcow2 # nothing to see in fixed size virtual disk file -rwxr-xr-x. 1 root root 99739238400 Feb 17 22:53 /mnt/a16/gkpics01.qcow2
Addendum/Correction to comment-2 and comment-3: Perhaps this might not be a mm/paging only bug, and may be due to ntfs3-paragon-kernel-driver, needing to catch-up to changes in the mm/paging. I mounted the src disk using ntfs3g-tuxera-fuseblk driver instead of the ntfs3-paragon-kernel driver. rsync/verify succeeds in this case. host-kernel: 5.17.0-0.rc4.96.fc35.x86_64 (self built from srpm) guest-kernel: 5.17.0-0.rc4.20220216gitc5d9ae265b10.98.fc37.x86_64, Fedora-Workstation-Live-x86_64-Rawhide-20220217.n.0.iso https://koji.fedoraproject.org/koji/buildinfo?buildID=1919340 qemu: qemu-system-x86-6.2.0-2.fc35.1.x86_64 (SB) hda: qcow2(dyn)/ext4 hdb: vhdx(dyn)/ntfs3g-fuseblk/sgdata