I should clarify that the router is not doing any actual routing here, it only acts as a VLAN-capable switch, encapsulating the Internet uplink (untagged) into one of VLANs (tagged).

Some more details, the device is:

[780834.487733] r8169 0000:04:00.0 eth0: RTL8168evl/8111evl, xxxxxxxx, XID 2c9, IRQ 32
[780834.487735] r8169 0000:04:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]

And I see that the r8168 vendor driver only enables flow control if Jumbo Frames are not in use:

                //flow control
                if (dev->mtu <= ETH_DATA_LEN)
                        auto_nego |= ADVERTISE_PAUSE_CAP|ADVERTISE_PAUSE_ASYM;

Here jumbo frames are enabled, and the mentioned Gigabit traffic would indeed consist of those.

...and somehow, disabling EEE helps lower the severity of this issue by an order of magnitude.

 ethtool --set-eee rt0 eee off

Now the Internet upload stream is no longer impacted at all, but the LAN transfers are still being slowed down. Before there would be an effect on both.

Simultaneous upload to:  Internet     LAN                 

Realtek (EEE on):        unusable     unusable - both connections stall
Realtek (EEE off):       94 Mbit      120-400 Mbit, couple of drops to 4 and 33 Mbit

Intel (EEE unsupported): 94 Mbit      600-700 Mbit

AR8327 (at least certain revisions) has known compatibility issues with EEE. Therefore e.g. in OpenWRT EEE is disabled per default for this chipset.

Unfortunately Realtek doen't release any NIC datasheets, therefore I don't know whether pause (sym or asym) is supported for jumbo frames. But you're right, the vendor driver (apart from r8168 also r8125) doesn't use pause with jumbo frames. Can you switch to a standard MTU to check whether the issue gone then?

Regarding the ethtool question:
r8169 doesn't support getting or setting pause settings. That's what "operation not supported" refers to, it doesn't say anything about whether pause is active or not.

You can give the following a try, like the vendor driver it disables pause advertisement in jumbo mode.


diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index eb6da93ac..2a7a9ca55 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -2318,6 +2318,7 @@ static void r8168b_1_hw_jumbo_disable(struct rtl8169_private *tp)
 static void rtl_jumbo_config(struct rtl8169_private *tp)
 {
 	bool jumbo = tp->dev->mtu > ETH_DATA_LEN;
+	struct phy_device *phydev = tp->phydev;
 	int readrq = 4096;
 
 	rtl_unlock_config_regs(tp);
@@ -2358,6 +2359,20 @@ static void rtl_jumbo_config(struct rtl8169_private *tp)
 
 	if (pci_is_pcie(tp->pci_dev) && tp->supports_gmii)
 		pcie_set_readrq(tp->pci_dev, readrq);
+
+	/* Chip doesn't support pause in jumbo mode */
+	if (jumbo) {
+		linkmode_clear_bit(ETHTOOL_LINK_MODE_Pause_BIT,
+				   phydev->advertising);
+		linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT,
+				   phydev->advertising);
+	} else {
+		linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT,
+				 phydev->advertising);
+		linkmode_set_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT,
+				 phydev->advertising);
+	}
+	phy_start_aneg(phydev);
 }
 
 DECLARE_RTL_COND(rtl_chipcmd_cond)
@@ -4619,8 +4634,6 @@ static int r8169_phy_connect(struct rtl8169_private *tp)
 	if (!tp->supports_gmii)
 		phy_set_max_speed(phydev, SPEED_100);
 
-	phy_support_asym_pause(phydev);
-
 	phy_attached_info(phydev);
 
 	return 0;
-- 
2.31.1

On Sat, 10 Apr 2021 21:46:50 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> --- Comment #4 from Heiner Kallweit (hkallweit1@gmail.com) ---
> AR8327 (at least certain revisions) has known compatibility issues with EEE.
> Therefore e.g. in OpenWRT EEE is disabled per default for this chipset.

Actually my router runs OpenWRT 17.01.7 and I did not manually enable EEE
there. I see the port status as:

	enable_eee: 0
	igmp_snooping: 0
	pvid: 0
	link: port:3 link:up speed:1000baseT full-duplex txflow rxflow eee100 eee1000 auto

If it's not enabled on the switch side, it seems weird that toggling it on the
NIC side affects things so much.

> Can you switch to a standard MTU to check whether the issue gone then?

Switching to MTU 1500 reduces the issue severity with EEE on, so it's about
the same as with EEE off, with none of the previous catastrophic stalling.

With EEE off though, there's not really any difference whether jumbo frames are
used or not. So of course I'll pick keeping jumbo frames over keeping EEE.

Also it seems that setting the actual NIC to MTU 1500 is not making a
difference over just limiting MTU of the used VLAN to 1500 (and keeping the
NIC itself at MTU 9000). So in this case any positive effect appears to be not
from reducing the device MTU in the driver, but just from sending smaller
packets than before.

(In reply to rm from comment #6)
> On Sat, 10 Apr 2021 21:46:50 +0000
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > --- Comment #4 from Heiner Kallweit (hkallweit1@gmail.com) ---
> > AR8327 (at least certain revisions) has known compatibility issues with
> EEE.
> > Therefore e.g. in OpenWRT EEE is disabled per default for this chipset.
> 
> Actually my router runs OpenWRT 17.01.7 and I did not manually enable EEE
> there. I see the port status as:
> 
>       enable_eee: 0
>       igmp_snooping: 0
>       pvid: 0
>       link: port:3 link:up speed:1000baseT full-duplex txflow rxflow eee100
> eee1000 auto
> 
> If it's not enabled on the switch side, it seems weird that toggling it on
> the
> NIC side affects things so much.
> 
The port status includes eee100 and eee1000, therefore EEE seems to be active. Looks like enable_eee setting is ignored.

On Sun, 11 Apr 2021 11:36:46 +0000
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=212617
> 
> --- Comment #5 from Heiner Kallweit (hkallweit1@gmail.com) ---
> You can give the following a try, like the vendor driver it disables pause
> advertisement in jumbo mode.

Thanks, I'll try it some time later when I set up a newer series build, as
it doesn't apply to the 5.4 kernel I'm testing with at the moment.

> diff --git a/drivers/net/ethernet/realtek/r8169_main.c
> b/drivers/net/ethernet/realtek/r8169_main.c
> index eb6da93ac..2a7a9ca55 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -2318,6 +2318,7 @@ static void r8168b_1_hw_jumbo_disable(struct
> rtl8169_private *tp)
>  static void rtl_jumbo_config(struct rtl8169_private *tp)
>  {
>         bool jumbo = tp->dev->mtu > ETH_DATA_LEN;
> +       struct phy_device *phydev = tp->phydev;
>         int readrq = 4096;
> 
>         rtl_unlock_config_regs(tp);
> @@ -2358,6 +2359,20 @@ static void rtl_jumbo_config(struct rtl8169_private
> *tp)
> 
>         if (pci_is_pcie(tp->pci_dev) && tp->supports_gmii)
>                 pcie_set_readrq(tp->pci_dev, readrq);
> +
> +       /* Chip doesn't support pause in jumbo mode */
> +       if (jumbo) {
> +               linkmode_clear_bit(ETHTOOL_LINK_MODE_Pause_BIT,
> +                                  phydev->advertising);
> +               linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT,
> +                                  phydev->advertising);
> +       } else {
> +               linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT,
> +                                phydev->advertising);
> +               linkmode_set_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT,
> +                                phydev->advertising);
> +       }
> +       phy_start_aneg(phydev);
>  }
> 
>  DECLARE_RTL_COND(rtl_chipcmd_cond)
> @@ -4619,8 +4634,6 @@ static int r8169_phy_connect(struct rtl8169_private
> *tp)
>         if (!tp->supports_gmii)
>                 phy_set_max_speed(phydev, SPEED_100);
> 
> -       phy_support_asym_pause(phydev);
> -
>         phy_attached_info(phydev);
> 
>         return 0;

Turns out the patch does not apply to 5.10 or 5.11 either, and I'm not really comfortable running the RC or mainline kernels on my main machine.

If you think it's still something worth trying (given the other observations I posted), could you make a version of the patch for 5.4?

Or even a smaller one that just disables pause altogether, since we can already try the case of "MTU 1500, pause enabled" as is, and the counterpart remaining to be tested is "MTU 9000, pause disabled".

This one applies on 5.4 and disables pause support unconditionally.


diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index bd8decc54..15ab0cad6 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -6368,7 +6368,8 @@ static int r8169_phy_connect(struct rtl8169_private *tp)
 	if (!tp->supports_gmii)
 		phy_set_max_speed(phydev, SPEED_100);
 
-	phy_support_asym_pause(phydev);
+	linkmode_clear_bit(ETHTOOL_LINK_MODE_Pause_BIT, phydev->advertising);
+	linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, phydev->advertising);
 
 	phy_attached_info(phydev);
 
-- 
2.31.1

Thank you. It is *so* much better with this patch!

Now getting a stable and steady 840 Mbit upload to LAN during a concurrent 94 Mbit upload to the Internet.

And now it does not matter whether EEE is on or off, it's all fine even with EEE on.

To clarify, it never worked this well before (with pause on), not even with EEE off and MTU 1500.

So while pause indeed better be disabled with jumbo frames since the vendor driver does that, on my setup it looks like pause is not contributing in a positive way even with the standard MTU. Maybe it does in some other scenarios though, or when paired with a different switch.

Fixed with 453a77894efa ("r8169: don't advertise pause in jumbo mode").

Thanks!