(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sat, 6 Jun 2009 17:43:46 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=13467
> 
>            Summary: Cannot set larger mtu on vlan then on underlying
>                     untagged device
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 2.6.27.24-170.2.68.fc10.i686
>           Platform: All
>         OS/Version: Linux
>               Tree: Fedora
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: acme@ghostprotocols.net
>         ReportedBy: tomek@jot23.org
>         Regression: No
> 
> 
> I want to create vlan interface which uses jumbo frames while underlying
> network device is still using standerd 1500 bytes frames. So I tried:
> 
> tkepczyx-mobl1:~# ip link show dev eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
> qlen 100
>     link/ether 00:0d:60:79:e0:9d brd ff:ff:ff:ff:ff:ff
> tkepczyx-mobl1:~# vconfig add eth0 12
> Added VLAN with VID == 12 to IF -:eth0:-
> tkepczyx-mobl1:~# ip link set dev eth0.12 mtu 9000
> RTNETLINK answers: Numerical result out of range
> 
> So I made sure that eth0 supports 9000 bytes mtu:
> 
> tkepczyx-mobl1:~# ip link set dev eth0 mtu 9000
> tkepczyx-mobl1:~# ip link show dev eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP
> qlen 100
>     link/ether 00:0d:60:79:e0:9d brd ff:ff:ff:ff:ff:ff
> 
> It apparently does. So I gave it another try:
> 
> tkepczyx-mobl1:~# ip link set dev eth0.12 mtu 9000
> tkepczyx-mobl1:~# ip link show dev eth0.12
> 4: eth0.12@eth0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN
>     link/ether 00:0d:60:79:e0:9d brd ff:ff:ff:ff:ff:ff
> 
> Now it worked. So lets decrease mtu on eth0:
> 
> tkepczyx-mobl1:~# ip link set dev eth0 mtu 1500
> tkepczyx-mobl1:~# ip link show dev eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
> qlen 100
>     link/ether 00:0d:60:79:e0:9d brd ff:ff:ff:ff:ff:ff
> tkepczyx-mobl1:~# ip link show dev eth0.12
> 4: eth0.12@eth0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN
>     link/ether 00:0d:60:79:e0:9d brd ff:ff:ff:ff:ff:ff
> 
> It also worked.
> Now - if there is a reason which prevents me to increase mtu on vlan over mtu
> on underlyng device, the very same reason should have prevented me from
> decreasing mtu on underlying eth0 below mtu on eth0.12 vlan. This is a fault.
> 
> I started digging a bit in sources and there is one thing which rings a bell:
> vlan_dev_change_mtu function in net/8021q/vlan_dev.c:
> 
> static int vlan_dev_change_mtu(struct net_device *dev, int new_mtu)
> {
>         /* TODO: gotta make sure the underlying layer can handle it,
>          * maybe an IFF_VLAN_CAPABLE flag for devices?
>          */
>         if (vlan_dev_info(dev)->real_dev->mtu < new_mtu)
>                 return -ERANGE;
> 
>         dev->mtu = new_mtu;
> 
>         return 0;
> }
> 
> My guess is that check in if() statement above is against currently
> configured
> mtu while it probably should check what physically device can transmit.
>

Andrew Morton <akpm@linux-foundation.org> wrote:
>
>>            Summary: Cannot set larger mtu on vlan then on underlying
>>                     untagged device

Please try this patch.

vlan: Propagate physical MTU changes

When the physical MTU changes we want to ensure that all existing
VLAN device MTUs do not exceed the new underlying MTU.  This patch
adds that propagation.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
index fe64908..6d37b7e 100644
--- a/net/8021q/vlan.c
+++ b/net/8021q/vlan.c
@@ -468,6 +468,19 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event,
 		}
 		break;
 
+	case NETDEV_CHANGEMTU:
+		for (i = 0; i < VLAN_GROUP_ARRAY_LEN; i++) {
+			vlandev = vlan_group_get_device(grp, i);
+			if (!vlandev)
+				continue;
+
+			if (vlandev->mtu <= dev->mtu)
+				continue;
+
+			dev_set_mtu(vlandev, dev->mtu);
+		}
+		break;
+
 	case NETDEV_FEAT_CHANGE:
 		/* Propagate device features to underlying device */
 		for (i = 0; i < VLAN_GROUP_ARRAY_LEN; i++) {

Cheers,

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Mon, 6 Jul 2009 18:19:50 +0800

> Andrew Morton <akpm@linux-foundation.org> wrote:
>>
>>>            Summary: Cannot set larger mtu on vlan then on underlying
>>>                     untagged device
> 
> Please try this patch.
> 
> vlan: Propagate physical MTU changes
> 
> When the physical MTU changes we want to ensure that all existing
> VLAN device MTUs do not exceed the new underlying MTU.  This patch
> adds that propagation.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Please can we have some test results for this patch?  It's been
more than a week since this patch and request for testing was
posted.

Thanks.

I will not be able to test by the end of July.
Secondly - is there any particular reason why tagged network interface cannot have larger MTU then plain (untagged) network device? I am looking into a scenario where untagged vlan uses standard ethernet frames (MTU=1500) and tagged vlan uses jumbo frames. This patch won't allow me to do that for no apparent reason.

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Mon, 6 Jul 2009 18:19:50 +0800

> vlan: Propagate physical MTU changes
> 
> When the physical MTU changes we want to ensure that all existing
> VLAN device MTUs do not exceed the new underlying MTU.  This patch
> adds that propagation.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Applied to net-next-2.6, thanks.

Is the original issue solved? Can this bug be closed?

It depends how you define the original issue.
1. For me the problem no 1 is that I can't set vlan with MTU larger then the original untagged interface. I do not believe (but haven't checked) this was solved or even justified that things should work this way.
2. For others the issue was that kernel allowed to lower the MTU on untagged original interface below MTU on tagged vlan interface. I believe this was solved but I haven't checked this (but I can if you expect me to do that)

I speak as a network engineer who specializes in large data center LANs, not a sysadmin or developer.

Having varying MTUs between VLAN tags (including the native/untagged VLAN) is a perfectly valid configuration.  The problem is that Linux is using the base interface MTU as both the physical interface MTU as well as the layer 3 MTU of the IP placed there.  Ideally these would be two different values.  For example, here is a Cisco configuration:


interface vlan100
 description public
 ip address 10.0.0.1 255.255.255.0
 mtu 1500

interface vlan200
 description private
 ip address 192.168.0.1 255.255.255.0
 mtu 9000

interface gigabitethernet1
 description to linux server
 switchport
 switchport mode trunk
 switchport trunk native vlan 100
 mtu 9216


Notice that the two layer 3 MTUs are independent of the physical port's MTU.  All that matters is that a) the physical port MTU be at least big enough to support the highest MTU that the layer 3 interfaces might generate, and b) all layer 3 interfaces in a VLAN must share the exact same MTU.

I am unable to connect a Linux server to this network configuration above because of the issue described in this bug.  I need the IP on eth0 to be transmitting 1500 MTU packets, and I need the IP on eth0.200 to be transmitting 9000 MTU packets.

Windows (using HP NIC drivers for VLAN tagging) doesn't have this problem.  The way it avoids the issue is by never putting IPs on the base interface.  Instead, the native/untagged VLAN is also created as a subinterface, and then there is a configuration field on the physical NIC to set which VLAN (or no VLAN) should be untagged.  I have a windows server connected to the network above without issue.

It seems to me that the simplest solution would be to just decouple MTU between the base interface and all tagged subinterfaces.  The other option would be to add an additional setting for "native_vlan=x" and have users place their native VLAN IP on a subinterface instead of the base interface, so that eth0 just becomes the physical port and no longer has a layer 3 config (as Windows does).

I had intended to use a configuration similar to the above as the standard way I connect all new customers to my network.  The idea was that the Internet facing VLAN (MTU 1500 of course) would be untagged and would "just work" even if you don't configure tagging on the server.  Then, users who need additional VLANs (some of which require jumbo frames) could simply add them as tags.  My hope was to avoid needing to add NICs and cables for additional VLANs that might be added in the future.

While creating the proof of concept in a lab I had no problems setting Windows up, but found Linux unable to support this.  While trying to figure out why, I found this bug report.  If current Linux kernels can't support jumbo frames on tags without forcing jumbo frames onto the native VLAN, I guess I'm going to have to avoid using the native VLAN at all when connecting to Linux servers.  This means switch ports won't "just work" for new servers being installed that don't have tagging configured yet.  This also means sysadmins won't be able to configure IPs on servers without first finding out the VLAN.

It's all certainly something that can be worked around by avoiding the native vlan, but it increases the support team workload to have to jump through unnecessary hoops to make things work.