Poor performance on receive-side only when using RTL8125 NIC with the in-tree r8169 driver. The performance problems occur only in the receive direction. If I reverse the sender I can get the full 2.5Gb (2.35Gb+overhead) over the link. From my client if I run iperf3 to other hosts on the same switch I see 2.5Gb in both directions. The Realtek maintained driver (r8125) supposedly fixes the issue from what I've read but unfortunately I'm unable to compile it on my OS (Rocky 8.7). $ iperf3 -c server [ 5] local 10.200.16.20 port 58358 connected to 10.200.16.40 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 176 MBytes 1.47 Gbits/sec 1113 41.0 KBytes [ 5] 1.00-2.00 sec 169 MBytes 1.42 Gbits/sec 1070 46.7 KBytes [ 5] 2.00-3.00 sec 176 MBytes 1.47 Gbits/sec 1287 49.5 KBytes [ 5] 3.00-4.00 sec 172 MBytes 1.44 Gbits/sec 1121 35.4 KBytes [ 5] 4.00-5.00 sec 176 MBytes 1.47 Gbits/sec 1105 33.9 KBytes [ 5] 5.00-6.00 sec 167 MBytes 1.40 Gbits/sec 1100 39.6 KBytes [ 5] 6.00-7.00 sec 175 MBytes 1.47 Gbits/sec 1134 46.7 KBytes [ 5] 7.00-8.00 sec 173 MBytes 1.45 Gbits/sec 1147 33.9 KBytes [ 5] 8.00-9.00 sec 167 MBytes 1.40 Gbits/sec 966 46.7 KBytes [ 5] 9.00-10.00 sec 177 MBytes 1.48 Gbits/sec 1164 41.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.69 GBytes 1.45 Gbits/sec 11207 sender [ 5] 0.00-10.04 sec 1.69 GBytes 1.44 Gbits/sec receiver And now we just reverse the sender: $ iperf3 -Rc server [ 5] local 10.200.16.20 port 41788 connected to 10.200.16.40 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 277 MBytes 2.32 Gbits/sec [ 5] 1.00-2.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 7.00-8.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 8.00-9.00 sec 278 MBytes 2.33 Gbits/sec [ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.04 sec 2.73 GBytes 2.33 Gbits/sec 4 sender [ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver Kernel version: Linux ultrix 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Tue Apr 4 22:38:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux modinfo on r8169: filename: /lib/modules/4.18.0-425.19.2.el8_7.x86_64/kernel/drivers/net/ethernet/realtek/r8169.ko.xz firmware: rtl_nic/rtl8125b-2.fw firmware: rtl_nic/rtl8125a-3.fw firmware: rtl_nic/rtl8107e-2.fw firmware: rtl_nic/rtl8107e-1.fw firmware: rtl_nic/rtl8168fp-3.fw firmware: rtl_nic/rtl8168h-2.fw firmware: rtl_nic/rtl8168h-1.fw firmware: rtl_nic/rtl8168g-3.fw firmware: rtl_nic/rtl8168g-2.fw firmware: rtl_nic/rtl8106e-2.fw firmware: rtl_nic/rtl8106e-1.fw firmware: rtl_nic/rtl8411-2.fw firmware: rtl_nic/rtl8411-1.fw firmware: rtl_nic/rtl8402-1.fw firmware: rtl_nic/rtl8168f-2.fw firmware: rtl_nic/rtl8168f-1.fw firmware: rtl_nic/rtl8105e-1.fw firmware: rtl_nic/rtl8168e-3.fw firmware: rtl_nic/rtl8168e-2.fw firmware: rtl_nic/rtl8168e-1.fw firmware: rtl_nic/rtl8168d-2.fw firmware: rtl_nic/rtl8168d-1.fw license: GPL softdep: pre: realtek description: RealTek RTL-8169 Gigabit Ethernet driver author: Realtek and the Linux r8169 crew <netdev@vger.kernel.org> rhelversion: 8.7 srcversion: 97902AC339128804EACE041 alias: pci:v000010ECd00003000sv*sd*bc*sc*i* alias: pci:v000010ECd00008125sv*sd*bc*sc*i* alias: pci:v00000001d00008168sv*sd00002410bc*sc*i* alias: pci:v00001737d00001032sv*sd00000024bc*sc*i* alias: pci:v000016ECd00000116sv*sd*bc*sc*i* alias: pci:v00001259d0000C107sv*sd*bc*sc*i* alias: pci:v00001186d00004302sv*sd*bc*sc*i* alias: pci:v00001186d00004300sv*sd*bc*sc*i* alias: pci:v00001186d00004300sv00001186sd00004B10bc*sc*i* alias: pci:v000010ECd00008169sv*sd*bc*sc*i* alias: pci:v000010FFd00008168sv*sd*bc*sc*i* alias: pci:v000010ECd00008168sv*sd*bc*sc*i* alias: pci:v000010ECd00008167sv*sd*bc*sc*i* alias: pci:v000010ECd00008162sv*sd*bc*sc*i* alias: pci:v000010ECd00008161sv*sd*bc*sc*i* alias: pci:v000010ECd00008136sv*sd*bc*sc*i* alias: pci:v000010ECd00008129sv*sd*bc*sc*i* alias: pci:v000010ECd00002600sv*sd*bc*sc*i* alias: pci:v000010ECd00002502sv*sd*bc*sc*i* depends: intree: Y name: r8169 vermagic: 4.18.0-425.19.2.el8_7.x86_64 SMP mod_unload modversions sig_id: PKCS#7 signer: Rocky kernel signing key sig_key: 31:6B:5A:38:DC:7F:D6:69:81:FF:0E:A0:F3:A0:40:1F:9E:9D:02:27 sig_hashalgo: sha256 signature: 97:97:FA:A9:56:9A:59:3A:E4:FB:BC:02:09:7A:AE:6A:0D:82:95:2A: 31:09:9C:78:AD:62:54:D2:73:82:8E:C0:D6:7A:D8:A5:16:F4:3B:FF: A0:DB:61:04:9D:DC:3C:CD:61:E0:92:F9:81:FC:87:EF:C0:BC:BA:A7: 60:24:59:4A:C5:B5:2F:51:F5:97:C2:18:18:18:42:5A:E1:90:5F:78: 14:75:C8:67:D0:3D:03:93:E5:98:D8:48:B2:8E:64:B4:73:DA:3C:A5: 6D:7D:24:79:2C:76:34:21:3F:35:5D:E8:93:5F:6B:FD:13:13:6F:9C: C3:F6:F2:0A:25:6C:1D:14:FF:23:84:87:94:01:F9:3F:C9:65:6B:4F: F3:44:65:F7:21:03:D9:9C:5F:60:A5:3B:2D:91:E3:BE:88:49:3B:2A: 97:41:81:6C:87:0A:47:C3:E9:46:FD:15:85:8E:07:A8:6C:5E:69:45: E9:A1:4B:68:89:04:73:00:35:61:85:7C:A2:5B:7F:FC:C6:DB:7B:A2: 8B:E6:36:1C:09:7A:02:8E:C5:72:5D:1C:F4:78:A9:C7:42:84:0B:E1: B6:5C:A4:B5:F8:AB:1F:FA:1B:51:E9:A3:78:17:90:BC:5B:03:CA:AC: C5:34:BB:4C:EE:CE:75:BE:15:F2:C6:C6:8C:6F:A8:5B:E0:38:7F:6F: 9F:18:63:5C:F0:2D:50:26:BD:DD:A2:D3:AF:C0:CA:64:82:6D:F7:39: AD:5A:C3:87:88:57:8F:F6:28:15:2F:F4:86:85:F7:71:BA:E8:9E:6B: DB:9C:10:B1:5D:90:77:47:27:DD:F4:68:C8:68:E3:60:BE:C4:FC:CD: B2:C0:B4:B0:44:F4:1B:37:6E:37:55:39:17:EE:00:AE:5A:51:8F:02: 09:AF:A7:59:DF:E1:E6:09:A0:50:6D:51:41:11:E0:9D:D9:68:4A:43: B3:8A:4E:00:80:A7:4C:B4:7C:7B:00:1E:F1:AA:2D:F6:1D:DE:B7:07: F6:50:6A:AF lshw -C network output: *-network description: Ethernet interface product: RTL8125 2.5GbE Controller vendor: Realtek Semiconductor Co., Ltd. physical id: 0 bus info: pci@0000:02:00.0 logical name: enp2s0 version: 05 serial: 78:72:64:40:6b:53 capacity: 1Gbit/s width: 64 bits clock: 33MHz capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=4.18.0-425.19.2.el8_7.x86_64 duplex=full firmware=rtl8125b-2_0.0.2 07/13/20 ip=10.200.16.43 latency=0 link=yes multicast=yes port=MII resources: irq:18 ioport:3000(size=256) memory:7ff00000-7ff0ffff memory:7ff10000-7ff13fff *-network description: Ethernet interface product: RTL8125 2.5GbE Controller vendor: Realtek Semiconductor Co., Ltd. physical id: 0 bus info: pci@0000:03:00.0 logical name: enp3s0 version: 05 serial: 78:72:64:40:6b:54 capacity: 1Gbit/s width: 64 bits clock: 33MHz capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=4.18.0-425.19.2.el8_7.x86_64 firmware=rtl8125b-2_0.0.2 07/13/20 latency=0 link=no multicast=yes port=MII resources: irq:19 ioport:2000(size=256) memory:7fe00000-7fe0ffff memory:7fe10000-7fe13fff ethtool output: Settings for enp2s0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 2500Mb/s Duplex: Full Auto-negotiation: on master-slave cfg: preferred slave master-slave status: slave Port: MII PHYAD: 0 Transceiver: external Supports Wake-on: pumbg Wake-on: d Link detected: yes
Output from lspci -vvv -s 02:00.0 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) Subsystem: Realtek Semiconductor Co., Ltd. Device 0123 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 18 Region 0: I/O ports at 3000 [size=256] Region 2: Memory at 7ff00000 (64-bit, non-prefetchable) [size=64K] Region 4: Memory at 7ff10000 (64-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x1 (ok) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp+ ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [b0] MSI-X: Enable+ Count=32 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00000800 Capabilities: [d0] Vital Product Data Not readable Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [168 v1] Device Serial Number 01-00-00-00-68-4c-e0-00 Capabilities: [178 v1] Transaction Processing Hints No steering table available Capabilities: [204 v1] Latency Tolerance Reporting Max snoop latency: 3145728ns Max no snoop latency: 3145728ns Capabilities: [20c v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=81920ns L1SubCtl2: T_PwrOn=150us Capabilities: [21c v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Kernel driver in use: r8169 Kernel modules: r8169
ethtool -k enp2s0. I have tried with generic-receive-offload both off and on. Latter was the default. Features for enp2s0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: off generic-segmentation-offload: off [requested on] generic-receive-offload: off large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-gre-csum-segmentation: off [fixed] tx-ipxip4-segmentation: off [fixed] tx-ipxip6-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-udp_tnl-csum-segmentation: off [fixed] tx-gso-partial: off [fixed] tx-tunnel-remcsum-segmentation: off [fixed] tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] tx-udp-segmentation: off [fixed] tx-gso-list: off [fixed] rx-udp-gro-forwarding: off rx-gro-list: off tls-hw-rx-offload: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off [fixed] esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off [fixed] tls-hw-tx-offload: off [fixed] rx-gro-hw: off [fixed] tls-hw-record: off [fixed]
Kernel version 4.18.0-425.19.2.el8_7.x86_64? Mainline added RTL8125 support with 5.9. Apparently your distro backported supported for RTL8125, so you should ask for support there. Alternatively re-test with a recent mainline kernel.
Thanks, I opened a bug report at Red Hat. Feel free to close this.
Vendor issue