Bug 215675 - Dummy interface is not working properly under VRF, is droping the income data
Summary: Dummy interface is not working properly under VRF, is droping the income data
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 enhancement
Assignee: Stephen Hemminger
URL: https://github.com/EasyNetDev/linux-m...
Keywords:
Depends on:
Blocks:
 
Reported: 2022-03-11 19:53 UTC by Easynet
Modified: 2022-03-12 11:18 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.16.X 5.15.x 5.10.x
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Easynet 2022-03-11 19:53:36 UTC
Hi all,

I was playing around with dummy interface to use it as an additional loopback interface in my system.
I notice that if I'm creating a VRF using:

ip link add internet type vrf table 2
ip link set internet up
ip route add blackhole default metric 4278198272 table 2

Adding some IPs:

ip address add 10.0.0.1/32 dev lo
ip address add 10.2.0.1/32 dev internet

Then add route-leaks:

# ip route show
10.2.0.1 nhid 28 dev internet proto bgp metric 20 

# ip route show table 2
10.0.0.1 nhid 32 dev lo proto bgp metric 20 

Seems to work fine.
Ping from VRF internet:
# ip vrf exec internet /usr/bin/ping 10.0.0.1 -I 10.2.0.1
ING 10.0.0.1 (10.0.0.1) from 10.2.0.1 : 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.069 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.026/0.047/0.069/0.021 ms

Ping from GRT / default:
# /usr/bin/ping 10.2.0.1 -I 10.0.0.1
PING 10.2.0.1 (10.2.0.1) from 10.0.0.1 : 56(84) bytes of data.
64 bytes from 10.2.0.1: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 10.2.0.1: icmp_seq=2 ttl=64 time=0.051 ms
^C
--- 10.2.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1023ms
rtt min/avg/max/mdev = 0.034/0.042/0.051/0.008 ms



But if I'm adding 2 more dummy interfaces:

ip link add lo0 type dummy
ip link set lo0 up

ip link add lo1 type dummy
ip link set lo1 master internet
ip link set lo1 up

And add some IPs:
ip address add 10.0.0.2/32 dev lo0
ip address add 10.2.0.2/32 dev lo1

Adding route-leaks:
# ip route show
10.2.0.2 nhid 48 dev lo1 proto bgp metric 20 

# ip route show table 2
10.0.0.2 nhid 44 dev lo0 proto bgp metric 20 

Then something is not working well:

root@VM:/opt/Kitts/frr# ip vrf exec internet /usr/bin/ping 10.0.0.2 -I 10.2.0.2
PING 10.0.0.2 (10.0.0.2) from 10.2.0.2 : 56(84) bytes of data.
^C
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1022ms


# /usr/bin/ping 10.2.0.2 -I 10.0.0.2
PING 10.2.0.2 (10.2.0.2) from 10.0.0.2 : 56(84) bytes of data.
^C
--- 10.2.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms

Even if I'm trying to ping the interface of my VM still not working using as source dummy interface:

# ip route show
10.2.0.1 nhid 28 dev internet proto bgp metric 20 
10.2.0.2 nhid 48 dev lo1 proto bgp metric 20 
192.168.105.0/24 dev ens33 proto kernel scope link src 192.168.105.203 metric 100 

# ip route show table 2
# ip route show table 2
blackhole default metric 4278198272 
10.0.0.1 nhid 32 dev lo proto bgp metric 20 
10.0.0.2 nhid 44 dev lo0 proto bgp metric 20 
local 10.2.0.1 dev internet proto kernel scope host src 10.2.0.1 
broadcast 10.2.0.1 dev internet proto kernel scope link src 10.2.0.1 
local 10.2.0.2 dev lo1 proto kernel scope host src 10.2.0.2 
broadcast 10.2.0.2 dev lo1 proto kernel scope link src 10.2.0.2 
192.168.105.0/24 nhid 33 dev ens33 proto bgp metric 20 

Some test with tcpdump shows that the packages are going out from VRF and returning back, but the interface is not responging back:

# tcpdump -nvli any 'host 192.168.105.203 and (host 10.2.0.2 or host 10.2.0.1)'
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
21:50:25.130727 lo    In  IP (tos 0x0, ttl 64, id 29146, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.0.2 > 192.168.105.203: ICMP echo request, id 42070, seq 1, length 64
21:50:25.130738 lo1   Out IP (tos 0x0, ttl 64, id 13146, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.105.203 > 10.2.0.2: ICMP echo reply, id 42070, seq 1, length 64


If I'm using the "internet" interface as source is working and the tcpdump is the same:

21:51:14.251462 lo    In  IP (tos 0x0, ttl 64, id 23812, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.0.1 > 192.168.105.203: ICMP echo request, id 44870, seq 1, length 64
21:51:14.251473 internet Out IP (tos 0x0, ttl 64, id 26291, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.105.203 > 10.2.0.1: ICMP echo reply, id 44870, seq 1, length 64


I was thinking there is an issue with the dummy driver that is droping the income data. I tried to create another driver based on loopback and vrf interface, but without success.

There is a why to add such feature to dummy interface to be able to work properly under VRF?
Comment 1 Easynet 2022-03-11 22:04:45 UTC
Hi,

After some modification to my driver I was able to make it working.
If this feature looks promising as a patch or as a new driver to Linux kernel, please take a look to my driver here: 

https://github.com/EasyNetDev/linux-multi-loopback

I'm not a good programmer in for Linux Kernel, but I tried as much as possible to understand the logics behind of the VRF, loopback and dummy driver to be able to implement this additional loopback driver.

I did an detailed explanation in README.md the strange behavior of dummy driver under VRF with a more complete scenario using FRRouting.

I think there are more pleople trying to find a way to add more loopback interfaces in their Linux router and they will face this issue.
Comment 2 David Ahern 2022-03-12 04:27:18 UTC
Does `ip -s li sh` show tx drops for the dummy interface? Guessing so. dummy interfaces just drop packets; they do not have a xmit function.
Comment 3 Easynet 2022-03-12 11:16:37 UTC
Hi David,

Indeed the dummy interface has xmit to drop the packets. But is very interesting that is working well in default VRF.
I checked the drops of the interfaces and are 0:

# ip -s li sh lo0
32: lo0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 92:be:12:4b:4d:f6 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
             0       0      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
             0       0      0       0       0       0

# ip -s li sh lo1
33: lo1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master internet state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 46:f7:a2:61:2d:02 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
             0       0      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
         14406     147      0       0       0       0

lo0 is i default VRF and lo1 is in internet VRF.
For this reason I had to create this additional driver to be able to increase the number of loopbacks in the system and to be able to use them normaly inside of VRFs.
Comment 4 Easynet 2022-03-12 11:18:27 UTC
Actually there is where dummy interface is supose to drop the packats:

static netdev_tx_t dummy_xmit(struct sk_buff *skb, struct net_device *dev)
{
	dev_lstats_add(dev, skb->len);

	skb_tx_timestamp(skb);
	dev_kfree_skb(skb);
	return NETDEV_TX_OK;
}

static const struct net_device_ops dummy_netdev_ops = {
	.ndo_init		= dummy_dev_init,
	.ndo_uninit		= dummy_dev_uninit,
	.ndo_start_xmit		= dummy_xmit,
	.ndo_validate_addr	= eth_validate_addr,
	.ndo_set_rx_mode	= set_multicast_list,
	.ndo_set_mac_address	= eth_mac_addr,
	.ndo_get_stats64	= dummy_get_stats64,
	.ndo_change_carrier	= dummy_change_carrier,
};

Note You need to log in before you can comment on or make changes to this bug.