Environment: kernel: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next commit: 39428f6e kernel version:6.4.0-rc2 Qemu: https://gitlab.com/qemu-project/qemu.git branch: master commit: 7efd6542 Host OS: CentOS stream 9 Guest OS: CentOS stream 9 CPU: Architecture: x86_64 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) Platinum Bug detail description: Create VF of X710 NIC on host, create VM with VF passthrough, there is a probability that the mac of VF in guest is inconsistent with it in host, the VF in guest gets a random mac. Reproduce steps: 1.Create two VFs of NIC X710: [root@spr-2s2 ~]# lspci -k -s 98:00.0 98:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 Kernel driver in use: i40e Kernel modules: i40e [root@spr-2s2 ~]# echo 2 > /sys/bus/pci/devices/0000:98:00.0/sriov_numvfs 2. Check mac and driver of the 2 VFs [root@spr-2s2 xf]# ip address 14: ens28f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 32:40:f7:6a:dc:8a brd ff:ff:ff:ff:ff:ff altname enp152s0f0v0 15: ens28f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 6e:bd:a8:ee:83:c4 brd ff:ff:ff:ff:ff:ff altname enp152s0f0v1 [root@spr-2s2 xf]# ethtool -i ens28f0v0 driver: iavf version: 6.4.0-rc2 bus-info: 0000:98:02.0 [root@spr-2s2 xf]# ethtool -i ens28f0v1 driver: iavf version: 6.4.0-rc2 bus-info: 0000:98:02.1 [root@spr-2s2 xf]# lspci -k -s 98:02.0 98:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: iavf Kernel modules: iavf [root@spr-2s2 xf]# lspci -k -s 98:02.1 98:02.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: iavf Kernel modules: iavf 3. Unbind the VFs from iavf driver and bind them to vfio-pci [root@spr-2s2 xf]# echo 0000:98:02.0 > /sys/bus/pci/devices/0000\:98\:02.0/driver/unbind [root@spr-2s2 xf]# echo 0000:98:02.1 > /sys/bus/pci/devices/0000\:98\:02.1/driver/unbind [root@spr-2s2 xf]# modprobe vfio-pci [root@spr-2s2 xf]# lspci -n -s 98:02.0 98:02.0 0200: 8086:154c (rev 01) [root@spr-2s2 xf]# echo 8086 154c > /sys/bus/pci/drivers/vfio-pci/new_id [root@spr-2s2 xf]# lspci -k -s 98:02.0 98:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: vfio-pci Kernel modules: iavf [root@spr-2s2 xf]# lspci -k -s 98:02.1 98:02.1 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: vfio-pci Kernel modules: iavf 4.Create guest with these 2 VFs passthrough qemu-system-x86_64 -accel kvm -m 4096 -cpu host -drive file=centos9.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0,bootindex=0 -smp 4 -device vfio-pci,host=98:02.0 -net none -device vfio-pci,host=98:02.1 -net none -daemonize -vnc :5 5. Check the mac and ip address of the these 2 VFs in guest [root@localhost ~]# ip ad 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether c2:4e:37:99:fc:ff brd ff:ff:ff:ff:ff:ff altname enp0s4 inet 192.168.111.27/20 brd 192.168.111.255 scope global dynamic noprefixroute ens4 valid_lft 278sec preferred_lft 278sec inet6 fe80::ac6c:bd39:6513:d29/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 6e:bd:a8:ee:83:c4 brd ff:ff:ff:ff:ff:ff altname enp0s5 inet 192.168.111.26/20 brd 192.168.111.255 scope global dynamic noprefixroute ens5 valid_lft 276sec preferred_lft 276sec inet6 fe80::7a32:aa2c:c2ee:8d5d/64 scope link noprefixroute valid_lft forever preferred_lft forever The mac of VF1 in host is 32:40:f7:6a:dc:8a, but it in guest is c2:4e:37:99:fc:ff, VF2 gets the same mac 6e:bd:a8:ee:83:c4 The guest dmesg about iavf: [ 3.516623] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver [ 3.517302] Copyright (c) 2013 - 2018 Intel Corporation. [ 3.588858] ppdev: user-space parallel port driver [ 3.590703] iavf 0000:00:04.0: Invalid MAC address 00:00:00:00:00:00, using random [ 3.592382] iavf 0000:00:04.0: Multiqueue Enabled: Queue pair count = 4 [ 3.593697] iavf 0000:00:05.0: Multiqueue Enabled: Queue pair count = 4 [ 3.594086] iavf 0000:00:04.0: MAC address: c2:4e:37:99:fc:ff [ 3.595297] iavf 0000:00:04.0: GRO is enabled [ 3.595932] iavf 0000:00:05.0: MAC address: 6e:bd:a8:ee:83:c4 [ 3.596503] iavf 0000:00:05.0: GRO is enabled [ 3.598333] iavf 0000:00:04.0 ens4: renamed from eth0 [ 3.606666] XFS (vda2): Mounting V5 Filesystem 8736b23e-ddde-4cca-9166-2623d2e57e5a [ 3.606987] iavf 0000:00:05.0 ens5: renamed from eth1 [ 3.614474] XFS (vda2): Ending clean mount [ 3.692511] RPC: Registered named UNIX socket transport module. [ 3.693055] RPC: Registered udp transport module. [ 3.693665] RPC: Registered tcp transport module. [ 3.694243] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 4.125995] iavf 0000:00:04.0 ens4: NIC Link is Up Speed is 10 Gbps Full Duplex [ 4.126628] IPv6: ADDRCONF(NETDEV_CHANGE): ens4: link becomes ready [ 4.158025] iavf 0000:00:05.0 ens5: NIC Link is Up Speed is 10 Gbps Full Duplex [ 4.158648] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready Currect Result: There is a probability that the mac of VF in guest is inconsistent with it in host Expected Result When passthrough NIC VF to guest, the VF in guest should always get the same mac as it in host.
This issue is not found from this commit, it has been going on for some days, we found kernel 6.0.0-rc7 is good, but not sure when it started from.
(In reply to Chen, Fan from comment #1) > This issue is not found from this commit, it has been going on for some > days, we found kernel 6.0.0-rc7 is good, but not sure when it started from. Can you please then perform bisection between 6.0-rc7 and 6.4-rc2?
We bisect and found the first bad commit, I don't understand if this commit is intend to do so? Make the mac of VF in VM different from it in host? We think they use the same mac is better for users, with the same mac, the mac of VF is known for us before creating VM, then we can get IP address of the VF interface in VM from mac without using vnc or other UI or serial port. How can we make the mac of VF keep the same in host and VM now? commit ceb29474bbbc377e11be3a70589a0005305e4fc3 Author: Sylwester Dziedziuch <sylwesterx.dziedziuch@intel.com> Date: Thu Mar 30 10:00:22 2023 -0700 i40e: Add support for VF to specify its primary MAC address Currently in the i40e driver there is no implementation of different MAC address handling depending on whether it is a legacy or primary. Introduce new checks for VF to be able to specify its primary MAC address based on the VIRTCHNL_ETHER_ADDR_PRIMARY type. Primary MAC address are treated differently compared to legacy ones in a scenario where: 1. If a unicast MAC is being added and it's specified as VIRTCHNL_ETHER_ADDR_PRIMARY, then replace the current default_lan_addr.addr. 2. If a unicast MAC is being deleted and it's type is specified as VIRTCHNL_ETHER_ADDR_PRIMARY, then zero the hw_lan_addr.addr. Actually, we also tried to use "ip link set ens28f0 vf 0 mac <mac>" to fix a mac for VF, it works in host, but we create VM with this VF passthrough, it has a high probability failed to assign the mac to VF in vm, and the interface failed yo get IP either, error dmesg in VM: [ 3.037955] iavf 0000:00:04.0: Invalid MAC address 00:00:00:00:00:00, using random [ 3.039523] iavf 0000:00:04.0: Multiqueue Enabled: Queue pair count = 4 [ 3.040466] iavf 0000:00:04.0: MAC address: 3a:38:ca:66:f9:65 [ 3.040980] iavf 0000:00:04.0: GRO is enabled [ 3.042240] iavf 0000:00:05.0: Multiqueue Enabled: Queue pair count = 4 [ 3.043232] iavf 0000:00:05.0: MAC address: 36:91:96:9d:5d:05 [ 3.043770] iavf 0000:00:05.0: GRO is enabled [ 3.044401] iavf 0000:00:04.0 ens4: renamed from eth0 [ 3.049199] iavf 0000:00:05.0 ens5: renamed from eth1 [ 3.072576] ppdev: user-space parallel port driver [ 3.094077] XFS (vda2): Mounting V5 Filesystem 8736b23e-ddde-4cca-9166-2623d2e57e5a [ 3.102245] XFS (vda2): Ending clean mount [ 3.550242] iavf 0000:00:04.0: Failed to add MAC filter, error IAVF_ERR_NVM [ 3.617176] iavf 0000:00:04.0 ens4: NIC Link is Up Speed is 10 Gbps Full Duplex [ 3.618173] IPv6: ADDRCONF(NETDEV_CHANGE): ens4: link becomes ready [ 3.628189] iavf 0000:00:05.0 ens5: NIC Link is Up Speed is 10 Gbps Full Duplex [ 3.629039] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready Fix this issue will be very helpful for us. I failed to CC the primary author sylwesterx.dziedziuch@intel.com and mateusz.palczewski@intel.com in bugliza. Thanks, Fan
Add mateusz.palczewski@intel.com and mateusz.palczewski@intel.com -----Original Message----- From: bugzilla-daemon@kernel.org <bugzilla-daemon@kernel.org> Sent: Wednesday, June 21, 2023 9:26 PM To: kvm@vger.kernel.org Subject: [Bug 217558] In KVM guest with VF of X710 NIC passthrough, the mac address of VF is inconsistent with it in host https://bugzilla.kernel.org/show_bug.cgi?id=217558 --- Comment #3 from Chen, Fan (farrah.chen@intel.com) --- We bisect and found the first bad commit, I don't understand if this commit is intend to do so? Make the mac of VF in VM different from it in host? We think they use the same mac is better for users, with the same mac, the mac of VF is known for us before creating VM, then we can get IP address of the VF interface in VM from mac without using vnc or other UI or serial port. How can we make the mac of VF keep the same in host and VM now? commit ceb29474bbbc377e11be3a70589a0005305e4fc3 Author: Sylwester Dziedziuch <sylwesterx.dziedziuch@intel.com> Date: Thu Mar 30 10:00:22 2023 -0700 i40e: Add support for VF to specify its primary MAC address Currently in the i40e driver there is no implementation of different MAC address handling depending on whether it is a legacy or primary. Introduce new checks for VF to be able to specify its primary MAC address based on the VIRTCHNL_ETHER_ADDR_PRIMARY type. Primary MAC address are treated differently compared to legacy ones in a scenario where: 1. If a unicast MAC is being added and it's specified as VIRTCHNL_ETHER_ADDR_PRIMARY, then replace the current default_lan_addr.addr. 2. If a unicast MAC is being deleted and it's type is specified as VIRTCHNL_ETHER_ADDR_PRIMARY, then zero the hw_lan_addr.addr. Actually, we also tried to use "ip link set ens28f0 vf 0 mac <mac>" to fix a mac for VF, it works in host, but we create VM with this VF passthrough, it has a high probability failed to assign the mac to VF in vm, and the interface failed yo get IP either, error dmesg in VM: [ 3.037955] iavf 0000:00:04.0: Invalid MAC address 00:00:00:00:00:00, using random [ 3.039523] iavf 0000:00:04.0: Multiqueue Enabled: Queue pair count = 4 [ 3.040466] iavf 0000:00:04.0: MAC address: 3a:38:ca:66:f9:65 [ 3.040980] iavf 0000:00:04.0: GRO is enabled [ 3.042240] iavf 0000:00:05.0: Multiqueue Enabled: Queue pair count = 4 [ 3.043232] iavf 0000:00:05.0: MAC address: 36:91:96:9d:5d:05 [ 3.043770] iavf 0000:00:05.0: GRO is enabled [ 3.044401] iavf 0000:00:04.0 ens4: renamed from eth0 [ 3.049199] iavf 0000:00:05.0 ens5: renamed from eth1 [ 3.072576] ppdev: user-space parallel port driver [ 3.094077] XFS (vda2): Mounting V5 Filesystem 8736b23e-ddde-4cca-9166-2623d2e57e5a [ 3.102245] XFS (vda2): Ending clean mount [ 3.550242] iavf 0000:00:04.0: Failed to add MAC filter, error IAVF_ERR_NVM [ 3.617176] iavf 0000:00:04.0 ens4: NIC Link is Up Speed is 10 Gbps Full Duplex [ 3.618173] IPv6: ADDRCONF(NETDEV_CHANGE): ens4: link becomes ready [ 3.628189] iavf 0000:00:05.0 ens5: NIC Link is Up Speed is 10 Gbps Full Duplex [ 3.629039] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready Fix this issue will be very helpful for us. I failed to CC the primary author sylwesterx.dziedziuch@intel.com and mateusz.palczewski@intel.com in bugliza. Thanks, Fan -- You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Thank you very much for reporting this AFAIR the patch from Sylwester was response to one of the critical issue We will take a look at this issue and see what can be done here Many thanks and regards, Patryk Piotrowski Intel Technology Poland
Hello Fan, Could you please try to reproduce the issue with the newest net next Kernel? I'm asking due to reason that the issue is not present there on our setups Thank you, Patryk Piotrowski Intel Technology Poland
Hello Patryk We can reproduce this issue on the latest commit in next branch of https://git.kernel.org/pub/scm/virt/kvm/kvm.git, actually we struggled with this problem for a long time, no fix found in this repo, so we bisect and found the first commit. Do you mean we can try the latest commit in master branch of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git? Thanks Fan
Hi Patryk We reproduced this issue with the latest kernel commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master a92b7d26 kernel version: 6.4.0_rc7 When you reproduce, did you create multiple VFs and passthrough all of them to KVM guest? I found that if we only passthrough one VF to guest, the mac is the same as it in host, but when passthrough 2 or more VFs, sometimes only the first VF use the same mac as it in host, other VFs' mac are random; sometimes all the VFs' mac are random.
Created attachment 304564 [details] [Patch]Fix inconsistent mac address of VF Proposal fix inconsistent mac address of VF after remove interface on VM.
Hello Fan, I reproduced issue on the latest commit in dev-queue branch https://git.kernel.org/pub/scm/linux/kernel/git/tnguy/net-queue.git/log/drivers/net/ethernet/intel/iavf?h=dev-queue Could you please try to reproduce the issue with this patch in Attachments added to above version kernel ?
Thanks Radoslaw, looks like your patch can fix this issue, I used above kernel with this patch, tried 5 times to reproduce, can't reproduce, mac of VF is the same in host and guest, got expected result.
Hello, The patch is already present on dev queue and it's waiting for "tested by" tag IWL link: https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230807125940.985494-1-radoslawx.tyl@intel.com/
Hi Patryk, Do I need to add "tested by"? I cannot find the email.
No, no - it's awaiting for "tested by" tag which is going to be provided by our VAL team once they verify the patch Once we will have this tag we will proceed with upstream and apply the patch to proper trees. I will keep you informed
Hi Patryk, Got it, thanks, could you help to add me as "Reported by"? Thanks Fan
Fixed with https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c8de44b577eb540e8bfea55afe1d0904bb571b7a