Bug 215437 - [r8169] Lots of Rx errors with RTL8111/8168/8411 (rev 06)
Summary: [r8169] Lots of Rx errors with RTL8111/8168/8411 (rev 06)
Status: RESOLVED UNREPRODUCIBLE
Alias: None
Product: Drivers
Classification: Unclassified
Component: Network (show other bugs)
Hardware: x86-64 Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-12-29 14:52 UTC by Patrick Silva
Modified: 2022-11-17 16:32 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.15.11
Subsystem:
Regression: No
Bisected commit-id:


Attachments
full kernel log (161.71 KB, text/plain)
2021-12-29 14:52 UTC, Patrick Silva
Details

Description Patrick Silva 2021-12-29 14:52:59 UTC
Created attachment 300182 [details]
full kernel log

Hi

I see lots of Rx errors in the kernel log when using r8169 driver on Arch Linux.
I do not see these errors when using r8168 driver.

$ uname -a
Linux Arch-PC 5.15.11-arch2-1 #1 SMP PREEMPT Wed, 22 Dec 2021 09:23:54 +0000 x86_64 GNU/Linux

$ sudo lspci -v
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
	Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet
	Flags: bus master, fast devsel, latency 0, IRQ 18
	I/O ports at e000 [size=256]
	Memory at f7c00000 (64-bit, non-prefetchable) [size=4K]
	Memory at f0000000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Endpoint, MSI 01
	Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
	Capabilities: [d0] Vital Product Data
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
	Kernel driver in use: r8169
	Kernel modules: r8169, r8168


$ sudo dmesg -Tx|grep r8169
kern  :warn  : [qua dez 29 10:10:51 2021] r8169 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
kern  :info  : [qua dez 29 10:10:51 2021] libphy: r8169: probed
kern  :info  : [qua dez 29 10:10:51 2021] r8169 0000:04:00.0 eth0: RTL8168evl/8111evl, fc:aa:14:fc:4b:1b, XID 2c9, IRQ 30
kern  :info  : [qua dez 29 10:10:51 2021] r8169 0000:04:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
kern  :info  : [qua dez 29 10:10:54 2021] r8169 0000:04:00.0 enp4s0: renamed from eth0
kern  :info  : [qua dez 29 10:11:10 2021] r8169 0000:04:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
kern  :info  : [qua dez 29 10:11:11 2021] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
kern  :info  : [qua dez 29 10:11:12 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 10:11:15 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 10:11:15 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :warn  : [qua dez 29 10:13:18 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:15:57 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:21:16 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:24:03 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:26:08 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:27:48 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:46:11 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:46:17 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 10:54:48 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :warn  : [qua dez 29 11:16:39 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352c4174
kern  :warn  : [qua dez 29 11:19:08 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 392cc118
kern  :warn  : [qua dez 29 11:20:40 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac5ee
kern  :info  : [qua dez 29 11:21:20 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:21:21 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:21:24 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:21:25 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:21:25 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:21:25 2021] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
kern  :info  : [qua dez 29 11:21:25 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:21:29 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:21:29 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:24:24 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:24:25 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:24:25 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:24:28 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:24:28 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:24:28 2021] RTL8211E Gigabit Ethernet r8169-0-400:00: attached PHY driver (mii_bus:phy_addr=r8169-0-400:00, irq=MAC)
kern  :info  : [qua dez 29 11:24:29 2021] r8169 0000:04:00.0 enp4s0: Link is Down
kern  :info  : [qua dez 29 11:24:32 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :info  : [qua dez 29 11:24:32 2021] r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
kern  :warn  : [qua dez 29 11:30:23 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac171
kern  :warn  : [qua dez 29 11:30:33 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac143
kern  :warn  : [qua dez 29 11:32:09 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352c4177
kern  :warn  : [qua dez 29 11:37:26 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 392cc165
kern  :warn  : [qua dez 29 11:37:35 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352cc174
kern  :warn  : [qua dez 29 11:41:37 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac20f
kern  :warn  : [qua dez 29 11:45:47 2021] r8169 0000:04:00.0 enp4s0: Rx ERROR. status = 352ac143
Comment 1 Heiner Kallweit 2021-12-29 17:10:39 UTC
r8168 by default just doesn't print rx errors. If you use ethtool msglvl to enable rx error logging, you should see the same errors with r8168. With r8168, even if it doesn't print rx errors, you should see the rx errors under /sys/class/net/enp4s0/statistics.

The status value indicates CRC errors. For whatever reason the checksum of received packets doesn't match. You could use wireshark or a similar tool to further analyze the issue. Or, to rule out a physical issue, try the easiest first: switch the cable.
Comment 2 Patrick Silva 2021-12-29 22:44:02 UTC
Tried another network cable and the Rx errors persist.
With r8168 driver I have much less Rx errors. I got the outputs below
after ~4 hours using r8168 driver with the initial network cable.

$ sudo ethtool -S enp4s0
NIC statistics:
     tx_packets: 972174
     rx_packets: 1365445
     tx_errors: 23
     rx_errors: 5
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 1364929
     broadcast: 98
     multicast: 418
     tx_aborted: 0
     tx_underrun: 0

$ ifconfig enp4s0
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.50  netmask 255.255.255.0  broadcast 192.168.0.255
        ether fc:aa:14:fc:4b:1b  txqueuelen 1000  (Ethernet)
        RX packets 1388034  bytes 1343088660 (1.2 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 984102  bytes 431299882 (411.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 30  base 0xb000

Weirdly the values of /sys/class/net/enp4s0/statistics/rx_errors
and /sys/class/net/enp4s0/statistics/rx_crc_errors remain 0 (zero).


I do not know anything about wireshark. I will read about it later.
Thanks.
Comment 3 Patrick Silva 2022-01-19 15:46:40 UTC
I'm still having many Rx errors with r8169, now running kernel 5.16.1,
 and have no idea how to get helpful information with wireshark.
Sometimes even internet browsing is horrible and
Firefox takes ~15 seconds to fully load websites.
Right now I'm using r8168 with the same network cable and
rx error logging enabled with this command:

$ sudo ethtool -s enp4s0 msglvl rx_err on

$ sudo ethtool enp4s0
Settings for enp4s0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Half 1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: Unknown
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000073 (115)
                               drv probe ifdown ifup rx_err
	Link detected: yes


dmesg log and the commands below do not report any Rx error.

$ ip -s -s link show enp4s0
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fc:aa:14:fc:4b:1b brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     516589996  467039      0       1       0     900 
    RX errors:  length    crc   frame    fifo overrun           
                     0      0       0       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      28829051  282192      0       1       0       0 
    TX errors: aborted   fifo  window heartbt transns
                     0      0       0       0     104 

$ sudo ethtool -S enp4s0
NIC statistics:
     tx_packets: 285342
     rx_packets: 472224
     tx_errors: 60
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 471240
     broadcast: 79
     multicast: 905
     tx_aborted: 0
     tx_underrun: 0

$ ifconfig enp4s0
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.50  netmask 255.255.255.0  broadcast 192.168.0.255
        ether fc:aa:14:fc:4b:1b  txqueuelen 1000  (Ethernet)
        RX packets 476660  bytes 525878530 (501.5 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 288003  bytes 29285844 (27.9 MiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0
        device interrupt 28  base 0x9000  

However, auto-negotiation does not work reliably with r8168.
Sometimes my network connects at 10 Mbps after login.
I'm seriously thinking to replace this realtek adapter
with an intel pci/pci-e adapter. I'm tired of so many problems with realtek.
Comment 4 Heiner Kallweit 2022-01-19 15:58:09 UTC
The autonegotiation downshift indicates that the problem is on the physical side, e.g. cabling or RJ45 port problem on either side. There's no indication of a driver problem.
Comment 5 Patrick Silva 2022-11-08 15:08:08 UTC
Could RAM failures cause RX errors? Memtest have detected RAM failures on my machine.
Another thing I have noticed is that the RX errors do not occur when the computer is idle,
even if there is network activity (by streaming audio over network, for example).
Comment 6 Heiner Kallweit 2022-11-09 09:34:06 UTC
If your RAM is faulty, then best replace it and then re-check whether RX errors persist.
Comment 7 Stephen Hemminger 2022-11-09 16:45:06 UTC
Check your power supply, if you have a weak power supply it can cause memory, CPU and network errors.
Comment 8 Stephen Hemminger 2022-11-09 16:45:46 UTC
Closing this as probable hardware issues.
Comment 9 Patrick Silva 2022-11-09 21:33:11 UTC
(In reply to Stephen Hemminger from comment #7)
> Check your power supply, if you have a weak power supply it can cause
> memory, CPU and network errors.

Power supply is good quality, manufactured by Seasonic, enough for my weak hardware. No rx error occurs when I connect a laptop and the affected computer directly, without my modem/router between them. I will ask my ISP provider to replace my modem/router. Thanks.
Comment 10 Patrick Silva 2022-11-17 16:32:57 UTC
I just want to confirm that replacing the modem/router solved the RX errors.
Thank you all.

Note You need to log in before you can comment on or make changes to this bug.