Bug 217362

Summary: Low performance on Realtek RTL8125 2.5Gb NIC using r8169 driver
Product: Networking Reporter: Jon (jon780)
Component: OtherAssignee: Stephen Hemminger (stephen)
Status: RESOLVED ANSWERED    
Severity: normal CC: hkallweit1
Priority: P3    
Hardware: All   
OS: Linux   
Kernel Version: Subsystem:
Regression: No Bisected commit-id:

Description Jon 2023-04-23 15:36:07 UTC 
Poor performance on receive-side only when using RTL8125 NIC with the in-tree r8169 driver.  The performance problems occur only in the receive direction. If I reverse the sender I can get the full 2.5Gb (2.35Gb+overhead) over the link.  From my client if I run iperf3 to other hosts on the same switch I see 2.5Gb in both directions.  The Realtek maintained driver (r8125) supposedly fixes the issue from what I've read but unfortunately I'm unable to compile it on my OS (Rocky 8.7).  

$ iperf3 -c server
[  5] local 10.200.16.20 port 58358 connected to 10.200.16.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   176 MBytes  1.47 Gbits/sec  1113   41.0 KBytes       
[  5]   1.00-2.00   sec   169 MBytes  1.42 Gbits/sec  1070   46.7 KBytes       
[  5]   2.00-3.00   sec   176 MBytes  1.47 Gbits/sec  1287   49.5 KBytes       
[  5]   3.00-4.00   sec   172 MBytes  1.44 Gbits/sec  1121   35.4 KBytes       
[  5]   4.00-5.00   sec   176 MBytes  1.47 Gbits/sec  1105   33.9 KBytes       
[  5]   5.00-6.00   sec   167 MBytes  1.40 Gbits/sec  1100   39.6 KBytes       
[  5]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec  1134   46.7 KBytes       
[  5]   7.00-8.00   sec   173 MBytes  1.45 Gbits/sec  1147   33.9 KBytes       
[  5]   8.00-9.00   sec   167 MBytes  1.40 Gbits/sec  966   46.7 KBytes       
[  5]   9.00-10.00  sec   177 MBytes  1.48 Gbits/sec  1164   41.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.69 GBytes  1.45 Gbits/sec  11207             sender
[  5]   0.00-10.04  sec  1.69 GBytes  1.44 Gbits/sec                  receiver


And now we just reverse the sender:
$ iperf3 -Rc server
[  5] local 10.200.16.20 port 41788 connected to 10.200.16.40 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   277 MBytes  2.32 Gbits/sec                  
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   8.00-9.00   sec   278 MBytes  2.33 Gbits/sec                  
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  2.73 GBytes  2.33 Gbits/sec    4             sender
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  receiver


Kernel version:
Linux ultrix 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Tue Apr 4 22:38:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

modinfo on r8169:
filename:       /lib/modules/4.18.0-425.19.2.el8_7.x86_64/kernel/drivers/net/ethernet/realtek/r8169.ko.xz
firmware:       rtl_nic/rtl8125b-2.fw
firmware:       rtl_nic/rtl8125a-3.fw
firmware:       rtl_nic/rtl8107e-2.fw
firmware:       rtl_nic/rtl8107e-1.fw
firmware:       rtl_nic/rtl8168fp-3.fw
firmware:       rtl_nic/rtl8168h-2.fw
firmware:       rtl_nic/rtl8168h-1.fw
firmware:       rtl_nic/rtl8168g-3.fw
firmware:       rtl_nic/rtl8168g-2.fw
firmware:       rtl_nic/rtl8106e-2.fw
firmware:       rtl_nic/rtl8106e-1.fw
firmware:       rtl_nic/rtl8411-2.fw
firmware:       rtl_nic/rtl8411-1.fw
firmware:       rtl_nic/rtl8402-1.fw
firmware:       rtl_nic/rtl8168f-2.fw
firmware:       rtl_nic/rtl8168f-1.fw
firmware:       rtl_nic/rtl8105e-1.fw
firmware:       rtl_nic/rtl8168e-3.fw
firmware:       rtl_nic/rtl8168e-2.fw
firmware:       rtl_nic/rtl8168e-1.fw
firmware:       rtl_nic/rtl8168d-2.fw
firmware:       rtl_nic/rtl8168d-1.fw
license:        GPL
softdep:        pre: realtek
description:    RealTek RTL-8169 Gigabit Ethernet driver
author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
rhelversion:    8.7
srcversion:     97902AC339128804EACE041
alias:          pci:v000010ECd00003000sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008125sv*sd*bc*sc*i*
alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
alias:          pci:v000010FFd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008162sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008161sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
alias:          pci:v000010ECd00002600sv*sd*bc*sc*i*
alias:          pci:v000010ECd00002502sv*sd*bc*sc*i*
depends:        
intree:         Y
name:           r8169
vermagic:       4.18.0-425.19.2.el8_7.x86_64 SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Rocky kernel signing key
sig_key:        31:6B:5A:38:DC:7F:D6:69:81:FF:0E:A0:F3:A0:40:1F:9E:9D:02:27
sig_hashalgo:   sha256
signature:      97:97:FA:A9:56:9A:59:3A:E4:FB:BC:02:09:7A:AE:6A:0D:82:95:2A:
                31:09:9C:78:AD:62:54:D2:73:82:8E:C0:D6:7A:D8:A5:16:F4:3B:FF:
                A0:DB:61:04:9D:DC:3C:CD:61:E0:92:F9:81:FC:87:EF:C0:BC:BA:A7:
                60:24:59:4A:C5:B5:2F:51:F5:97:C2:18:18:18:42:5A:E1:90:5F:78:
                14:75:C8:67:D0:3D:03:93:E5:98:D8:48:B2:8E:64:B4:73:DA:3C:A5:
                6D:7D:24:79:2C:76:34:21:3F:35:5D:E8:93:5F:6B:FD:13:13:6F:9C:
                C3:F6:F2:0A:25:6C:1D:14:FF:23:84:87:94:01:F9:3F:C9:65:6B:4F:
                F3:44:65:F7:21:03:D9:9C:5F:60:A5:3B:2D:91:E3:BE:88:49:3B:2A:
                97:41:81:6C:87:0A:47:C3:E9:46:FD:15:85:8E:07:A8:6C:5E:69:45:
                E9:A1:4B:68:89:04:73:00:35:61:85:7C:A2:5B:7F:FC:C6:DB:7B:A2:
                8B:E6:36:1C:09:7A:02:8E:C5:72:5D:1C:F4:78:A9:C7:42:84:0B:E1:
                B6:5C:A4:B5:F8:AB:1F:FA:1B:51:E9:A3:78:17:90:BC:5B:03:CA:AC:
                C5:34:BB:4C:EE:CE:75:BE:15:F2:C6:C6:8C:6F:A8:5B:E0:38:7F:6F:
                9F:18:63:5C:F0:2D:50:26:BD:DD:A2:D3:AF:C0:CA:64:82:6D:F7:39:
                AD:5A:C3:87:88:57:8F:F6:28:15:2F:F4:86:85:F7:71:BA:E8:9E:6B:
                DB:9C:10:B1:5D:90:77:47:27:DD:F4:68:C8:68:E3:60:BE:C4:FC:CD:
                B2:C0:B4:B0:44:F4:1B:37:6E:37:55:39:17:EE:00:AE:5A:51:8F:02:
                09:AF:A7:59:DF:E1:E6:09:A0:50:6D:51:41:11:E0:9D:D9:68:4A:43:
                B3:8A:4E:00:80:A7:4C:B4:7C:7B:00:1E:F1:AA:2D:F6:1D:DE:B7:07:
                F6:50:6A:AF

lshw -C network output:
  *-network                 
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: enp2s0
       version: 05
       serial: 78:72:64:40:6b:53
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=4.18.0-425.19.2.el8_7.x86_64 duplex=full firmware=rtl8125b-2_0.0.2 07/13/20 ip=10.200.16.43 latency=0 link=yes multicast=yes port=MII
       resources: irq:18 ioport:3000(size=256) memory:7ff00000-7ff0ffff memory:7ff10000-7ff13fff
  *-network
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0
       version: 05
       serial: 78:72:64:40:6b:54
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=4.18.0-425.19.2.el8_7.x86_64 firmware=rtl8125b-2_0.0.2 07/13/20 latency=0 link=no multicast=yes port=MII
       resources: irq:19 ioport:2000(size=256) memory:7fe00000-7fe0ffff memory:7fe10000-7fe13fff


ethtool output:
Settings for enp2s0:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
                                             10000baseT/Full
                                             2500baseT/Full
                                             5000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Auto-negotiation: on
        master-slave cfg: preferred slave
        master-slave status: slave
        Port: MII
        PHYAD: 0
        Transceiver: external
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes
Comment 1 Jon 2023-04-23 15:44:04 UTC 
Output from lspci -vvv -s 02:00.0

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: I/O ports at 3000 [size=256]
        Region 2: Memory at 7ff00000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at 7ff10000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 01
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
                Not readable
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [168 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178 v1] Transaction Processing Hints
                No steering table available
        Capabilities: [204 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Capabilities: [20c v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=81920ns
                L1SubCtl2: T_PwrOn=150us
        Capabilities: [21c v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169
Comment 2 Jon 2023-04-23 16:16:58 UTC 
ethtool -k enp2s0. I have tried with generic-receive-offload both off and on.  Latter was the default.

Features for enp2s0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
rx-udp-gro-forwarding: off
rx-gro-list: off
tls-hw-rx-offload: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
Comment 3 Heiner Kallweit 2023-04-30 22:44:35 UTC 
Kernel version 4.18.0-425.19.2.el8_7.x86_64?
Mainline added RTL8125 support with 5.9. Apparently your distro backported supported for RTL8125, so you should ask for support there.
Alternatively re-test with a recent mainline kernel.
Comment 4 Jon 2023-05-05 19:44:44 UTC 
Thanks, I opened a bug report at Red Hat.  Feel free to close this.
Comment 5 Jon 2023-05-05 19:45:14 UTC 
Vendor issue