Bug 209913

Summary: Low upload speed
Product: Drivers Reporter: Fibonacci (cliffburton11)
Component: network-wireless-intelAssignee: Default virtual assignee for network-wireless-intel (drivers_network-wireless-intel)
Status: RESOLVED IMPLEMENTED    
Severity: normal CC: andsmir, benland100, devin.tuchsen, fkrueger, hancockrwd, hugh.welles, jan.wrona9, janek+kernel, jmarcet, kevinkar731, kunal.bhat2001, lilydjwg, martin, oofnik, percy, robgri, streetwalkermc, t.clastres, tiago, world.root, youling257
Priority: P1    
Hardware: Intel   
OS: Linux   
Kernel Version: 5.9.1 Subsystem:
Regression: No Bisected commit-id:
Attachments: adds regression for ax200 upload

Description Fibonacci 2020-10-28 01:41:09 UTC
Hello,


My upload speed has reduced to about 50Mbps when I upgraded my kernel to 5.9.1.

When I boot with 5.8.16 kernel, my upload speed is back to ~450Mbps.

Download speed is the same in both cases ~940Mbps


Hardware environment : Intel ax200 connected to a 802.11ax network.
Distribution : Latest Manjaro
Comment 1 Fibonacci 2020-10-30 00:32:37 UTC
The bug is still present in 5.10rc1
Someone else have the same exact bug with an Intel Corporation Wireless-AC 9560 [Jefferson Peak ] (rev 10)
Comment 2 Kunal Bhat 2020-11-07 03:15:37 UTC
Facing this issue on 5.9.4. I'm using the Intel 9260 Wireless card.
Comment 3 Fibonacci 2020-11-07 14:24:17 UTC
The issue is still present on 5.9.6
No problem on 5.8.18 but unfortunately it's EOL now.
Comment 4 Arvin 2020-11-08 03:47:18 UTC
Same issue on 5.9.y (Arch Linux) using an Intel AX200 card. Managed to narrow down the problem to the following commit:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=3d5b459ba0e3788ab471e8cb98eee89964a9c5e8

Full upload speed is restored after doing a revert. Consequently, I also found that disabling TSO on my wireless card on a clean 5.9.6 kernel also works:

ethtool -K wlp3s0 tso off

My other PC with a Realtek RT8812BU USB wireless adapter has no such regression in 5.9, so it seems it might actually be an Intel driver issue.
Comment 5 Fibonacci 2020-11-08 15:12:26 UTC
I confirm that disabling TSO restores full upload speed on 5.9.6
hardware : Intel AX200
Comment 6 Emmanuel Grumbach 2020-11-12 10:39:28 UTC
*** Bug 210157 has been marked as a duplicate of this bug. ***
Comment 7 Benjamin Land 2020-11-19 18:44:53 UTC
Disabling TSO also fixes this issue on 5.9.9 for:
Intel Corporation Wireless-AC 9560 [Jefferson Peak ] (rev 10)
Comment 8 Fibonacci 2020-11-19 19:50:52 UTC
As Arvin mentionned, reverting that commit or just disabling TSO works.
Tested on 5.9.y and 5.10rc
Comment 9 hugh.welles 2020-11-23 04:07:14 UTC
I am new to bugzilla and arch in general. 

Just here to report that I had this same problem. Fixed by downgrading to 5.8.14. 

https://bbs.archlinux.org/viewtopic.php?pid=1935313#p1935313

Will try turning TSO off, but am a hobbyist with no understanding of what TSO is/does. 

lspci -vv output: http://ix.io/2Fac
Comment 10 hugh.welles 2020-11-24 21:17:47 UTC
can confirm update to 5.9.10 then disable tso -> return to 57MB/s upload speed to home server
Comment 11 Jo 2020-12-03 08:12:38 UTC
Same here on Kernel 5.9.12 with Intel Corporation Wi-Fi 6 AX200 (rev 1a).
The workaround in Comment #4 (turning DSO off) works for me too.
Comment 12 Martin 2020-12-26 21:58:53 UTC
I can also confirm this, disabling TSO works.


Intel(R) Wireless-AC 9560, REV=0x318

$ uname -a
Linux ntb 5.9.9-arch1-1 #1 SMP PREEMPT Wed, 18 Nov 2020 19:52:04 +0000 x86_64 GNU/Linux

$ iperf -c x.x.x.253 -d -t 30   
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to x.x.x.253, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  4] local x.x.x.x.232 port 59330 connected with x.x.x.253 port 5001
[  5] local x.x.x.x.232 port 5001 connected with x.x.x.253 port 53598
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-30.0 sec   843 MBytes   236 Mbits/sec
[  4]  0.0-30.6 sec  16.2 MBytes  4.45 Mbits/sec
$ 
$ sudo ethtool -K wlo1 tso off
$ 
$ iperf -c cubie2 -d -t 30
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to x.x.x.253, TCP port 5001
TCP window size:  136 KByte (default)
------------------------------------------------------------
[  4] local x.x.x.232 port 59376 connected with x.x.x.253 port 5001
[  5] local x.x.x.232 port 5001 connected with x.x.x.253 port 53602
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec   114 MBytes  32.0 Mbits/sec
[  5]  0.0-30.0 sec   688 MBytes   192 Mbits/sec
Comment 13 youling257 2021-01-02 19:09:44 UTC
I have the same problem,
kernel 5.9.16, ax200 upload speed only 10MB/S
kernel 5.8.18, ax200 upload speed can 80MB/S
Comment 14 Benjamin Land 2021-01-03 19:16:29 UTC
Following up, this seems to be fixed in 5.10.3+ and possibly a few revisions before that. Fairly sure that all of the 5.9.x kernels exhibited the issue reported.
Comment 15 Benjamin Land 2021-01-03 19:19:33 UTC
Spoke too soon - 5.10.4 again needs TSO disabled for full upload performance.
Comment 16 youling257 2021-01-04 03:25:36 UTC
Created attachment 294475 [details]
adds regression for  ax200 upload

https://www.mail-archive.com/netdev@vger.kernel.org/msg379947.html
Comment 17 Fibonacci 2021-01-06 23:46:04 UTC
The issue seems to be fixed in 5.10.5 
Can someone please confirm ? Thanks
Comment 18 Robert Grimm 2021-01-07 11:20:28 UTC
(In reply to Fibonacci from comment #17)
> The issue seems to be fixed in 5.10.5 
> Can someone please confirm ? Thanks

No, for me it persists on 5.10.5 with AX200.
Comment 19 Robert Grimm 2021-01-09 09:20:58 UTC
On Mon, 04 January 2021, you wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=209913
>
> youling257@gmail.com changed:
>
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>  Attachment #294475 [details]|0                           |1
>            is patch|                            |
>

Applied this patch on vanilla 5.10.5 sources and can confirm that this patch
fixes the regression.

Robert
Comment 20 Jo 2021-01-09 22:47:55 UTC
Robert, you mean *reverting* the patch fixed it, right?
Comment 21 Fibonacci 2021-01-09 23:12:33 UTC
Hi,

I don't know if archlinux or manjaro team applied a patch or something but I'm not facing this issue anymore since 5.10.5

https://i.imgur.com/UGGh0Wx.png
Comment 22 Robert Grimm 2021-01-10 09:54:55 UTC
(In reply to Jo from comment #20)
> Robert, you mean *reverting* the patch fixed it, right?

No, I included the attached patch (see comment #16) and the changes solved the Problem.

Robert
Comment 23 Robert Hancock 2021-01-16 01:54:38 UTC
What's the status of this fix? I am seeing this regression on any 5.9+ kernels on Intel 9260 as well unless TSO is turned off. I don't see any commit in mainline currently that appears to reference this.
Comment 24 Nephiller 2021-01-17 17:44:17 UTC
I think the Intel I225-V controller is having similar issues. Changing to different kernels and disabling TSO doesn't seem to fix download speeds.
Comment 25 Fibonacci 2021-01-17 19:14:09 UTC
(In reply to Robert Hancock from comment #23)
> What's the status of this fix? I am seeing this regression on any 5.9+
> kernels on Intel 9260 as well unless TSO is turned off. I don't see any
> commit in mainline currently that appears to reference this.

The issue is still present, even on vanilla 5.11-rc3

Manjaro team applied the patch in comment #16 for 5.10.y release which fixed the issue on their distro.
Comment 26 Johannes Berg 2021-01-26 10:43:35 UTC
The patch is now on its way to mainline/stable:

https://patchwork.kernel.org/project/netdevbpf/patch/20210125150949.619309-1-eric.dumazet@gmail.com/
Comment 27 Johannes Berg 2021-01-26 10:44:13 UTC
*** Bug 210789 has been marked as a duplicate of this bug. ***
Comment 28 Fibonacci 2021-02-01 12:01:26 UTC
The issue is fixed in 5.11-rc6. The patch in Attachment #294475 [details] made its way to mainline.
Comment 29 Robert Hancock 2021-02-02 00:50:39 UTC
The patch is also in the stable queue and should hopefully make it into 5.10.13.
Comment 30 Fibonacci 2021-02-08 14:00:44 UTC
The issue is totally fixed since 5.11-rc6. 
Changed status to resolved, Thanks everybody.