Bug 195923

Summary: [i40e] regression on TCP stream and TCP maerts
Product: Drivers Reporter: Adrian Tomasov (atomasov)
Component: NetworkAssignee: drivers_network (drivers_network)
Status: RESOLVED PATCH_ALREADY_AVAILABLE    
Severity: normal CC: alexander.duyck, atomasov, hladky.jiri
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 4.12.0-0.rc2 Subsystem:
Regression: No Bisected commit-id:
Attachments: TCP stream over IPv4
TCP stream over IPv6
TCP maerts IPv4
TCP maerts IPv6
ethtool -S before test
ethtool -S after test

Description Adrian Tomasov 2017-05-30 12:00:52 UTC
Created attachment 256769 [details]
TCP stream over IPv4

We see regression on intel card(XL710) with i40e driver. Regression is about ~45% on TCP_STREAM and TCP_MAERTS test for IPv4 and IPv6. Regression was first visible in kernel-4.12.0-0.rc1. 

Test was running between two servers with i40e card on each side. The test was using netperf benchmark.

All the results you can find in attachments.

Which commit/change could cause this regression? Any hints?
Comment 1 Adrian Tomasov 2017-05-30 12:01:49 UTC
Created attachment 256771 [details]
TCP stream over IPv6
Comment 2 Adrian Tomasov 2017-05-30 12:03:38 UTC
Created attachment 256773 [details]
TCP maerts IPv4
Comment 3 Adrian Tomasov 2017-05-30 12:04:06 UTC
Created attachment 256775 [details]
TCP maerts IPv6
Comment 4 Adrian Tomasov 2017-05-30 12:10:36 UTC
We run test with this command : 
netperf -P0 -cC -t TCP_STREAM  -l 30  -L <src_ip>  -H <dst_ip>  -T ,0  -T 0, -- -m <msg_size>
Comment 5 Adrian Tomasov 2017-05-31 13:39:13 UTC
Created attachment 256807 [details]
ethtool -S before test
Comment 6 Adrian Tomasov 2017-05-31 13:39:38 UTC
Created attachment 256809 [details]
ethtool -S after test
Comment 7 Alexander Duyck 2017-05-31 21:40:14 UTC
So one thing we may want to look at double checking is if you might have an MTU mismatch somewhere in your network. I notice there are Rx length errors being reported. Do you have a system somewhere that might be submitting a frame that is larger than what would be expected?

I know I have seen issues with MTU mismatch cause issues in the past and the new i40e driver would be more susceptible to that now since we changed the Rx buffer allocation scheme. You might try testing with wireshark running to verify if you have TSO/GSO in use in your test environment or not. You may be losing TSO/GSO due to an MTU mismatch and that might cause the performance regression you are seeing since TCP can work around the issue by limiting the send window size to something less than MSS if I am not mistaken.
Comment 8 Alexander Duyck 2017-06-01 20:08:03 UTC
On additional point that was brought to my attention is that the firmware image on your NICs seems a bit old. You may want to take a look at updating the firmware image just to avoid potentially tripping over any of the known issues that have been fixed since your version of the firmware.
Comment 9 Adrian Tomasov 2017-06-05 16:26:02 UTC
I checked MTU once again. MTU is 1500 on NICs and 1514 on switch, which is in the middle of testing path. We don't have any system that might produce frame with larger MTU. All ports on switch are set to MTU 1514.

TSO/GSO was used, while test was running.  I saw output of tcpdump and there was message length higher then mtu (about 65k). Then I tried to run test with TSO and GSO turned off. Throughput was ~%5.5Gbps and message length in tcpdump output was 1460.

I tried set priv-flag to legacy-rx off and it had no impact to throughput.

I also tried disable adaptive interrupt moderation and set Rx usecs and Tx usecs to specific values on both servers and no impact as well.

I updated firmware on NICs and ran CI test once again. I will let you know once the test is finished and I have the result adaptive interrupt moderation
Comment 10 Adrian Tomasov 2017-06-05 16:26:48 UTC
*I updated firmware on NICs and ran CI test once again. I will let you know once the test is finished and I have the result.