Bug 213377

Summary: Poor receive speeds using e1000e driver on Intel I219-V
Product: Drivers Reporter: Roland Sommer (rol+kernel)
Component: NetworkAssignee: drivers_network (drivers_network)
Status: NEW ---    
Severity: normal CC: akinzler, ccristian, dustin.bensing, kai.heng.feng, kane, kevin.bowling, kiran.telukunta, msalle, robin+kernel, royale, rudewire, somebodyhere2, tarkasteve, telihax7jjd, terentev, vitaly.lifshits
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 5.8, 5.10, 5.11 Subsystem:
Regression: No Bisected commit-id:
Attachments: dmesg from one of the test installs using 5.10er kernel on ubuntu 20.04
dmesg from test boot without additional kernel parameters, 5.11.10 from ubuntu mainline ppa
dmesg from test boot without additional kernel parameters, 5.10.42 built from git linux-stable
speed test with/without usb keys
disable ASPM (PCIe link power management) for CNL (CannonLake/TigerLake) chips
test patch
attachment-11265-0.html

Description Roland Sommer 2021-06-09 06:47:38 UTC
Created attachment 297253 [details]
dmesg from one of the test installs using 5.10er kernel on ubuntu 20.04

Hello,

I encountered very poor receive speeds using the e1000e driver for wired ethernet connection via I219-V internal network card.

I did file an upstream distribution bug at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1927925 but was requested to file a kernel bug against this.

So far I tested kernel versions 5.8, 5.10 and 5.11, mainly on ubuntu, 5.11 on fedora, so this does not seem to be distribution specific.

When booting without any special kernel parameters, receive speeds are very poor up to a complete hang. ethtool reports increasing errors at rx_errors and rx_crc_errors.

When booting the same kernel(s) with "intel_idle.max_cstate=1", everything works as expected.

I was not able to spot any conspicuous log messages (dmesg, syslog etc.) until now.

I'll attach a sample dmesg output from one of the test runs.

Quotation from the upstream bug report:
OK, so we know two things now:
1) The issue happens when the CPU reaches deeper power saving state, likely PC10.
2) Temporarily increase CPU usage via cpu_latency_qos_add_request() to disable C-State doesn't help.
Comment 1 Vitaly Lifshits 2021-06-09 16:04:07 UTC
Hello Roland,

I suspect that the issue is related to a patch in the pmc_core driver.

Can you try to reproduce this issue with vanilla kernel 5.11.10 or below (for example the long-term kernel 5.10.42)?
Comment 2 Roland Sommer 2021-06-10 09:07:24 UTC
Created attachment 297285 [details]
dmesg from test boot without additional kernel parameters, 5.11.10 from ubuntu mainline ppa

Booted and tested 5.11.10 from the ubuntu mainline ppa without additional kernel parameters - same behaviour as before.

I'm currently building 5.10.42 from git/linux-stable for further testing.
Comment 3 Roland Sommer 2021-06-10 13:34:25 UTC
Created attachment 297293 [details]
dmesg from test boot without additional kernel parameters, 5.10.42 built from git linux-stable

I did build the kernel from git/linux-stable, v5.10.42. The behaviour is the same. Without the additional kernel parameter, receive speeds are slow and ethtool reports errors on rx_errors and rx_crc_errors.
Comment 4 Roland Sommer 2021-06-11 07:59:48 UTC
Ok, things are getting really weird.

I did re-run the speed-tests with various kernels I already tested and for some reason even when booting with intel_idle.max_cstate=1, my networking was broken with ethtool reporting errors. All I did was to install everything I needed to build the vanilla kernel and some sensor stuff for fan-control.

I have removed all the newly installed packages, removed every module-setting added from/for lm-sensors or thinkfan and re-tested the kernels - the parameter had no effect anymore.

What I did next:
- I booted a live stick (Ubuntu 20.04.2.0), using ubuntu kernel 5.8.0-43 and speeds seem to be fine without any kernel parameter
- After I did this clean boot, I booted the updated ubuntu kernel 5.8.0-55 with the max_cstate parameter and speeds where fine again
- I rebooted the same kernel without the parameter and the errors returned
- I installed the vanilla 5.10.42 kernel, rebooted without the kernel parameter and the errors persisted
- I rebooted with the added max_cstate-parameter, and speeds are fine, no errors in ethtool
- I did reinstall lm-sensors and thinkfan, ran the detection with all defaults, adding coretemp to /etc/modules etc.
- Everything still seems back to the way it was, with the kernel parameter fixing the receive-problem.

I'm really curious what could have caused this weird behaviour. If there is any way to produce more useable debug information, I would like to help.
Comment 5 Roland Sommer 2021-06-11 13:05:20 UTC
To rule out any mistakes while building on my side, I started over with a fresh build of v5.10.42 (commit 65859eca4dff1af0db5e36d1cfbac15b834c6a65). I booted with and without the additional kernel parameter, this time without any network errors.

I retried using the ubuntu kernel 5.10.0-1029-oem which is based on 5.10.35. This kernel leads to rx-errors when not booted with the additional parameter.

I will now build 5.10.35 from git to see if there is any difference.
Comment 6 Roland Sommer 2021-06-11 15:07:58 UTC
vanilla kernel 5.10.35 with "intel_idle.max_cstate=1" is working.
vanilla kernel 5.10.35 without yields rx errors on the interface.

I'm suspecting that the working test with 5.10.42 was by chance, maybe the errors pop up later sometimes, because I did a reboot-cycle with 5.10.42 again and now the behaviour was as with 5.10.35: without parameters I encounter rx errors, with max_cstate=1 everything seems fine.
Comment 7 Roland Sommer 2021-06-11 20:34:24 UTC
I did one more vanilla build using v5.12.9, the behaviour stays the same, no errors with "intel_idle.max_cstate=1", rx errors without.
Comment 8 David Kraeutmann 2021-06-12 14:58:36 UTC
This significantly reduced the impact of the bug on an I219-LM Tiger Lake machine running 5.11.0-18 (Ubuntu), but iperf3 still shows weird results:

In send direction, everything is fine (both TCP and UDP).

In receive direction, it starts losing packets left and right after a few seconds -- 0% packet loss for the first 7-10 seconds of an iperf3 invoke, then spikes to 50% or so.

ethtool -S shows minor CRC errors -- 132 RX errors, 66 CRC errors with millions of packets sent.

Without intel_idle.max_cstate=1, the receive results are always lossy (send is identical) -- about 1-10% loss throughout. CRC errors remain on the same level.

Predictably, this absolutely dumpsters TCP speed.


I'm not sure if this is related, but since intel_idle.max_cstate=1 changed how this manifests, I think this might be.
Comment 9 Roland Sommer 2021-06-18 14:05:48 UTC
I just did firmware updates (1.36) and re-tested vanilla kernel 5.12.9 - now I'm back to "does not work even with intel_idle.max_cstate=1". I did boot several times, always the same. I did not change anything else since the last test runs.
Comment 10 Roland Sommer 2021-06-22 10:27:22 UTC
After updating to UEFI BIOS 1.36 (N34ET36W) and Embedded Controller Version 1.33 (N34HT33W), none of the already tested kernels work now, with or without intel_idle.max_cstate=1.
Comment 11 Roland Sommer 2021-06-22 13:25:37 UTC
This is getting ridiculous: I removed the custom kernel builds to free up space for a new distribution kernel. After booting the distribution kernel (which is 5.10.0-1032-oem), I was able to get a good connection using the kernel parameter and then retested with previously failing kernels (just a few hours ago) and now I get good connection speeds when using intel_idle.max_cstate=1 again.

If the hardware had not previously been tested on windows (without any errors), I would bet this must be some hardware problem because this seems so random. A specific behaviour is reproduceable for some time, and suddenly the behaviour changes after minor or unrelated changes (like removing some custom kernel builds like this time).
Comment 12 Kiran K Telukunta 2021-06-24 23:15:54 UTC
setting the parameter in /etc/default/grub as
GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1"

and ran grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg 

does not improve the condition of Ethernet for me.

Or am I doing something wrong?
Comment 13 Roland Sommer 2021-06-28 07:31:24 UTC
I discovered the next oddity: For the time being, I can reproduce the following behaviour in the currently running system:

1. Transfer of large file via wget from internal network is horribly slow or getting stack completely
2. I put in an USB-Stick and reissue the same wget, transfer is fast
3. I remove the USB stick, wget is slow again

I did this several times in a row, the USB stick is not involved in any way, just inserted. The issued wget looks like

wget -O /dev/null http://storage/largefile

Transfer speed was around 93MB/s while the USB stick was inserted, 5GB transfered in around 50sec.
Comment 14 Roland Sommer 2021-06-30 05:37:22 UTC
Just for reference, a similar bug has been posted to the netdev mailing list by another user. https://www.spinics.net/lists/netdev/msg743850.html
Comment 15 Robin Jarry 2021-06-30 07:56:31 UTC
Created attachment 297675 [details]
speed test with/without usb keys

Hi Roland,

I have the exact same problem with a I219-LM network card. I tested with a 5.10 (debian testing) and a 5.12 (tag v5.12.13) that I compiled myself.

Out of curiosity, I tried with different USB devices (mouse, yubikey) and the TCP transfer rate seems only to improve with mass storage usb devices.
Comment 16 Roland Sommer 2021-07-07 14:24:54 UTC
The workaround from https://bugzilla.kernel.org/show_bug.cgi?id=213651 also seems to fix the problem. Just tested and got good rx speeds afterwards.
Comment 17 akinzler 2021-08-19 08:42:59 UTC
Created attachment 298359 [details]
disable ASPM (PCIe link power management) for CNL (CannonLake/TigerLake) chips

@Roland: could you try my patch. On my Tiger Lake system it fixes the problem, but the problem was broken ASPM
Comment 18 Roland Sommer 2021-08-19 11:41:50 UTC
@Andreas against which kernel version should I try this patch?
Comment 19 Roland Sommer 2021-08-19 15:12:47 UTC
I tried against linux-stabe 5.13.12 without any success, receive speeds are still slow (if I did no mistake while building).
Comment 20 akinzler 2021-08-20 13:28:21 UTC
I developed the patch for 5.12.19, but it should work for 5.13.12 too.
Comment 21 Kiran K Telukunta 2021-08-20 13:56:28 UTC
I have 5.13.9-200.fc34.x86_64 and still get the RX errors

enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

inet 192.168.178.33  netmask 255.255.255.0  broadcast 192.168.178.255

        inet6 fd00::6ede:5e4d:3a7d:342b  prefixlen 64  scopeid 0x0<global>

        inet6 fe80::99b2:d9d1:233b:e70f  prefixlen 64  scopeid 0x20<link>

        ether xxxxxxxxxxxxx  txqueuelen 1000  (Ethernet)

        RX packets 4064  bytes 4188271 (3.9 MiB)

        RX errors 190  dropped 97  overruns 0  frame 95

        TX packets 16261  bytes 1315382 (1.2 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        device interrupt 16  memory 0xbed80000-beda0000
Comment 22 Roland Sommer 2021-09-09 10:26:53 UTC
@Andreas after being back from holidays, I re-ran the test with kernel 5.12.19 and your patch, but rx-speed ist still broken.
Comment 23 Kiran K Telukunta 2021-10-01 13:11:14 UTC
Tried to see if it was working. Looks like it is not resolved in 5.13.19-200.fc34.x86_64 kernel. 



enp0s31f6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        ether xxxxxxxxxxxxx  txqueuelen 1000  (Ethernet)

        RX packets 31038  bytes 19012398 (18.1 MiB)

        RX errors 926  dropped 138  overruns 0  frame 464

        TX packets 51132  bytes 59055699 (56.3 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


        device interrupt 16  memory 0xbed80000-beda0000
Comment 24 Roland Sommer 2021-10-01 13:56:57 UTC
The patches from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1927925/comments/93 did fix the problem using 5.15-rc3 as base kernel.
Comment 25 Steve Smith 2021-11-21 08:40:10 UTC
I'm still seeing this issue with a build of kernel 5.15.3. Adding the MEI power control override fixes it still. This is on a NUC7i3BNH with e1000e driver, I219-V (rev 21).
Comment 26 Kiran K Telukunta 2021-12-02 22:40:17 UTC
Is this happening in Kernel 5.13? Because initially I had issue in the Fedora when  I was using the kernel 5.13 and then now I see it is happening in Ubuntu and it has 5.13 Kernel now.
Comment 27 Dustin Bensing 2021-12-03 12:27:55 UTC
Found this issue here after recently getting constant disconnections on a Dell Latitude 5421 (11 Gen i7-11850h) with I219-LM on Kernel 5.15.5 but was probably introduced earlier (using wifi most of the time) and was wondering if this is connected to the issues discussed here (and maybe introduced by fixes applied).
Comment 28 ChrisO 2021-12-16 13:20:15 UTC
I can confirm e1000e connects at 1gb but speeds are limited to 10mbps MAX throughput on Fedora 5.15.7-200.fc35.x86_64 that was just pushed out last night and all prior Fedora 35 kernels are affected ( 5.14 ) 

00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 05)
        DeviceName:  Onboard LAN
        Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard


I have tried all the fixes above with no resolution.
Comment 29 Mike 2022-01-05 10:33:13 UTC
I also have a problem with the network card in Ubuntu 21.10. I found out that no RX CRC errors occur when the C-State(Package) option is set to maximum C6 in UEFI. With the option "auto" or above C6 the network adapter gets RX CRC errors(ip -s -s link show dev eno1).

00:1f.6 Ethernet controller : Intel Corporation Ethernet Connection (10) I219-V (rev 11)
	DeviceName: Onboard - Ethernet
	Subsystem: Micro-Star International Co., Ltd. [MSI] Ethernet Connection (10) I219-V
	Kernel driver in use: e1000e
	Kernel modules: e1000e

cat /sys/devices/system/cpu/cpuidle/current_driver
intel_idle
cat /sys/module/intel_idle/parameters/max_cstate
9
Comment 30 Mike 2022-01-12 18:39:56 UTC
I think I have found the error in my case.
I have enabled the C-State(Packages) max. C10 again in the bios. With Powertop 2.14-1(Debian Sid) I found out that the communication controller is causing the errors.

In Ubuntu 21.10 - 00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11)

Debian Sid Kernel 5.15.0-2-amd64 #1 SMP Debian 5.15.5-2 (2021-12-18) x86_64 GNU/Linux
lspci -vv
    00:16.0 Communication controller: Intel Corporation Device 43e0 (rev 11)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7d22
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 132
	Region 0: Memory at a123d000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: mei_me
	Kernel modules: mei_me
	
Now I have deactivated the energy saving function for the communication controller with powertop and I have created a service with which it is automatically loaded after system start.

echo 'on' > '/sys/bus/pci/devices/0000:00:16.0/power/control';

No crc error has appeared since then. In the kernel the option was preset to auto(echo 'auto' > '/sys/bus/pci/devices/0000:00:16.0/power/control';).
Comment 31 Evgeny Terentev 2022-02-19 00:47:29 UTC
i have a similar problem

The problem is solved by installing the kernel 5.4

I have this configuration
1) vps (openvpn server) (iroute 192.168.1.0 255.255.255.0)
2) router (openvpn client) (192.168.1.1)
3) intel nuc BOXNUC8i7BEH2 (192.168.1.3) + Ubuntu 20.04 HWE kernel 5.13 + (iperf3 -s)

speed degradation

vps:~# iperf3 -t 400 -c 192.168.1.3
Connecting to host 192.168.1.3, port 5201
[  5] local 10.8.0.1 port 47218 connected to 192.168.1.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.31 MBytes  19.4 Mbits/sec    0    455 KBytes       
[  5]   1.00-2.00   sec  5.67 MBytes  47.6 Mbits/sec   11    258 KBytes       
[  5]   2.00-3.00   sec  2.59 MBytes  21.7 Mbits/sec   11    174 KBytes       
[  5]   3.00-4.00   sec  1.48 MBytes  12.4 Mbits/sec   20    114 KBytes       
[  5]   4.00-5.00   sec   757 KBytes  6.21 Mbits/sec   27   74.9 KBytes       
[  5]   5.00-6.00   sec   757 KBytes  6.21 Mbits/sec    3   59.2 KBytes       
[  5]   6.00-7.00   sec   757 KBytes  6.20 Mbits/sec    0   65.7 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    4   52.6 KBytes       
[  5]   8.00-9.00   sec   757 KBytes  6.21 Mbits/sec    4   43.4 KBytes       
[  5]   9.00-10.00  sec   757 KBytes  6.20 Mbits/sec    0   52.6 KBytes       
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    6   30.2 KBytes       
[  5]  11.00-12.00  sec   757 KBytes  6.22 Mbits/sec    0   36.8 KBytes       
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec    0   43.4 KBytes       
[  5]  13.00-14.00  sec   757 KBytes  6.20 Mbits/sec    9   39.4 KBytes       
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec    0   46.0 KBytes       
[  5]  15.00-16.00  sec   757 KBytes  6.20 Mbits/sec    0   48.6 KBytes       
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec   14   42.1 KBytes 


with option -R speed normal

vps:~# iperf3 -t 400 -c 192.168.1.3 -R
Connecting to host 192.168.1.3, port 5201
Reverse mode, remote host 192.168.1.3 is sending
[  5] local 10.8.0.1 port 47222 connected to 192.168.1.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.66 MBytes  47.5 Mbits/sec                  
[  5]   1.00-2.00   sec  8.02 MBytes  67.3 Mbits/sec                  
[  5]   2.00-3.00   sec  14.4 MBytes   120 Mbits/sec                  
[  5]   3.00-4.00   sec  14.4 MBytes   121 Mbits/sec                  
[  5]   4.00-5.00   sec  14.9 MBytes   125 Mbits/sec                  
[  5]   5.00-6.00   sec  16.7 MBytes   140 Mbits/sec                  
[  5]   6.00-7.00   sec  11.6 MBytes  97.7 Mbits/sec 


At first I thought that the problem was in the I219-V, but after I connected the TP-LINK UE300 via USB, the problem did not go away

Connect nuc to router with wifi works well too.

Connect raspberry pi4 ubuntu kernel 5.13 to router works well too.

filename:       /lib/modules/5.4.0-100-generic/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
version:        3.2.6-k
license:        GPL v2
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     ACC5F4757D98B547BEECB61

filename:       /lib/modules/5.13.0-30-generic/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
license:        GPL v2
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     470A11DF18838A3C0FDC468
Comment 32 sal9k 2022-06-08 19:00:41 UTC
What is the current status of this bug report, has it been fixed on a particular Kernel version?

I have an Intel NUC7i5 w/ the I291-V, Ubuntu 22.04 LTS, kernel 5.15.0-35-generic, that has (slowly) accumulating rx_missed_errors (as reported with ethtool -S), and slow iperf3 speeds on 1GbE LAN (~300 Mbps).  The steps outlined in comment 30 of this thread, 5.15.0-35-generic, https://bugzilla.kernel.org/show_bug.cgi?id=213377#c30, of disabling the power management on PCI device "00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)" fixed the issue, and now no rx_missed_errors, and full ~940 Mbps iperf3 speeds.

Is this an e1000e driver, mei_mi kernel driver, or other issue?  Incidentally, I had the same issue on Ubuntu 20.04 as well.  Only the comment 30 fix worked for me.
Comment 33 Kai-Heng Feng 2022-06-09 01:14:55 UTC
sal(In reply to sal9k from comment #32)
> What is the current status of this bug report, has it been fixed on a
> particular Kernel version?
> 
> I have an Intel NUC7i5 w/ the I291-V, Ubuntu 22.04 LTS, kernel
> 5.15.0-35-generic, that has (slowly) accumulating rx_missed_errors (as
> reported with ethtool -S), and slow iperf3 speeds on 1GbE LAN (~300 Mbps). 
> The steps outlined in comment 30 of this thread, 5.15.0-35-generic,
> https://bugzilla.kernel.org/show_bug.cgi?id=213377#c30, of disabling the
> power management on PCI device "00:16.0 Communication controller: Intel
> Corporation Sunrise Point-LP CSME HECI #1 (rev 21)" fixed the issue, and now
> no rx_missed_errors, and full ~940 Mbps iperf3 speeds.
> 
> Is this an e1000e driver, mei_mi kernel driver, or other issue? 
> Incidentally, I had the same issue on Ubuntu 20.04 as well.  Only the
> comment 30 fix worked for me.

Can you please attach `lspci -nn` here? Thanks.
Comment 34 sal9k 2022-06-09 02:25:38 UTC
(In reply to Kai-Heng Feng from comment #33)
> sal(In reply to sal9k from comment #32)
> > What is the current status of this bug report, has it been fixed on a
> > particular Kernel version?
> > 
> > I have an Intel NUC7i5 w/ the I291-V, Ubuntu 22.04 LTS, kernel
> > 5.15.0-35-generic, that has (slowly) accumulating rx_missed_errors (as
> > reported with ethtool -S), and slow iperf3 speeds on 1GbE LAN (~300 Mbps). 
> > The steps outlined in comment 30 of this thread, 5.15.0-35-generic,
> > https://bugzilla.kernel.org/show_bug.cgi?id=213377#c30, of disabling the
> > power management on PCI device "00:16.0 Communication controller: Intel
> > Corporation Sunrise Point-LP CSME HECI #1 (rev 21)" fixed the issue, and
> now
> > no rx_missed_errors, and full ~940 Mbps iperf3 speeds.
> > 
> > Is this an e1000e driver, mei_mi kernel driver, or other issue? 
> > Incidentally, I had the same issue on Ubuntu 20.04 as well.  Only the
> > comment 30 fix worked for me.
> 
> Can you please attach `lspci -nn` here? Thanks.

$ lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5904] (rev 03)
00:02.0 VGA compatible controller [0300]: Intel Corporation Iris Plus Graphics 640 [8086:5926] (rev 06)
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:9d2f] (rev 21)
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:9d31] (rev 21)
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-LP CSME HECI #1 [8086:9d3a] (rev 21)
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] [8086:9d03] (rev 21)
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #1 [8086:9d10] (rev f1)
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #9 [8086:9d18] (rev f1)
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point LPC Controller/eSPI Controller [8086:9d4e] (rev 21)
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-V [8086:15d8] (rev 21)
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
Comment 35 Kai-Heng Feng 2022-06-09 03:04:06 UTC
Created attachment 301131 [details]
test patch

Please give this one a try.
Comment 36 Cristian Mihail Craciunescu 2023-08-12 18:38:58 UTC
(In reply to Kai-Heng Feng from comment #35)
> Created attachment 301131 [details]
> test patch
> 
> Please give this one a try.

Not working for me with kernel 6.1 which seams to include your patch:
lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b71] (rev 0c)
00:02.0 VGA compatible controller [0300]: Intel Corporation CometLake-U GT2 [UHD Graphics] [8086:9b41] (rev 02)
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake Thermal Subsytem [8086:02f9]
00:14.0 USB controller [0c03]: Intel Corporation Comet Lake PCH-LP USB 3.1 xHCI Host Controller [8086:02ed]
00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH-LP Shared SRAM [8086:02ef]
00:14.3 Network controller [0280]: Intel Corporation Comet Lake PCH-LP CNVi WiFi [8086:02f0]
00:15.0 Serial bus controller [0c80]: Intel Corporation Serial IO I2C Host Controller [8086:02e8]
00:15.2 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH-LP LPSS: I2C Controller #2 [8086:02ea]
00:16.0 Communication controller [0780]: Intel Corporation Comet Lake Management Engine Interface [8086:02e0]
00:17.0 SATA controller [0106]: Intel Corporation Comet Lake SATA AHCI Controller [8086:02d3]
00:1c.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #5 [8086:02bc] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #9 [8086:02b0] (rev f0)
00:1d.5 PCI bridge [0604]: Intel Corporation Device [8086:02b5] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Comet Lake PCH-LP LPC Premium Controller/eSPI Controller [8086:0284]
00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH-LP SMBus Host Controller [8086:02a3]
00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake SPI (flash) Controller [8086:02a4]
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (10) I219-V [8086:0d4f]
01:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 2C 2018] [8086:15e7] (rev 06)
02:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 2C 2018] [8086:15e7] (rev 06)
02:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 2C 2018] [8086:15e7] (rev 06)
02:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 2C 2018] [8086:15e7] (rev 06)
03:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 2C 2018] [8086:15e8] (rev 06)
39:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 2C 2018] [8086:15e9] (rev 06)
3a:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. Device [2646:501d]
3b:00.0 SD Host controller [0805]: Genesys Logic, Inc GL9755 SD Host Controller [17a0:9755]

ethtool -S eno1 | grep -i error
     rx_errors: 2261
     tx_errors: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 1257
     rx_frame_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     rx_csum_offload_errors: 0
     uncorr_ecc_errors: 0
     corr_ecc_errors: 0
Comment 37 Kai-Heng Feng 2023-10-16 06:08:18 UTC
(In reply to Cristian Mihail Craciunescu from comment #36)
> (In reply to Kai-Heng Feng from comment #35)
> > Created attachment 301131 [details]
> > test patch
> > 
> > Please give this one a try.
> 
> Not working for me with kernel 6.1 which seams to include your patch:

That was a long time ago. Can you please try newer mainline kernel?
Comment 38 amir.avivi 2023-10-16 06:08:39 UTC
Created attachment 305224 [details]
attachment-11265-0.html

Thanks for your email; I am Out of Office.
If you have urgent issues, please talk to my manager, shmuel.ben-nisan@intel.com.

Thanks,
Amir