Created attachment 256769 [details] TCP stream over IPv4 We see regression on intel card(XL710) with i40e driver. Regression is about ~45% on TCP_STREAM and TCP_MAERTS test for IPv4 and IPv6. Regression was first visible in kernel-4.12.0-0.rc1. Test was running between two servers with i40e card on each side. The test was using netperf benchmark. All the results you can find in attachments. Which commit/change could cause this regression? Any hints?
Created attachment 256771 [details] TCP stream over IPv6
Created attachment 256773 [details] TCP maerts IPv4
Created attachment 256775 [details] TCP maerts IPv6
We run test with this command : netperf -P0 -cC -t TCP_STREAM -l 30 -L <src_ip> -H <dst_ip> -T ,0 -T 0, -- -m <msg_size>
Created attachment 256807 [details] ethtool -S before test
Created attachment 256809 [details] ethtool -S after test
So one thing we may want to look at double checking is if you might have an MTU mismatch somewhere in your network. I notice there are Rx length errors being reported. Do you have a system somewhere that might be submitting a frame that is larger than what would be expected? I know I have seen issues with MTU mismatch cause issues in the past and the new i40e driver would be more susceptible to that now since we changed the Rx buffer allocation scheme. You might try testing with wireshark running to verify if you have TSO/GSO in use in your test environment or not. You may be losing TSO/GSO due to an MTU mismatch and that might cause the performance regression you are seeing since TCP can work around the issue by limiting the send window size to something less than MSS if I am not mistaken.
On additional point that was brought to my attention is that the firmware image on your NICs seems a bit old. You may want to take a look at updating the firmware image just to avoid potentially tripping over any of the known issues that have been fixed since your version of the firmware.
I checked MTU once again. MTU is 1500 on NICs and 1514 on switch, which is in the middle of testing path. We don't have any system that might produce frame with larger MTU. All ports on switch are set to MTU 1514. TSO/GSO was used, while test was running. I saw output of tcpdump and there was message length higher then mtu (about 65k). Then I tried to run test with TSO and GSO turned off. Throughput was ~%5.5Gbps and message length in tcpdump output was 1460. I tried set priv-flag to legacy-rx off and it had no impact to throughput. I also tried disable adaptive interrupt moderation and set Rx usecs and Tx usecs to specific values on both servers and no impact as well. I updated firmware on NICs and ran CI test once again. I will let you know once the test is finished and I have the result adaptive interrupt moderation
*I updated firmware on NICs and ran CI test once again. I will let you know once the test is finished and I have the result.