Bug 215599 - System freeze after add mirror rule from traffic control [BUG: soft lockup ]
Summary: System freeze after add mirror rule from traffic control [BUG: soft lockup ]
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords: trivial
Depends on:
Blocks:
 
Reported: 2022-02-12 21:16 UTC by Alexey Boldyrev
Modified: 2022-03-16 00:07 UTC (History)
2 users (show)

See Also:
Kernel Version: 5.16.5-arch1-1
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Alexey Boldyrev 2022-02-12 21:16:42 UTC
The rule:
tc qdisc add dev eth1 handle ffff: ingress
tc filter add dev eth1 parent 1: flower src_mac 22:22:22:22:22:22 action mirred ingress mirror dev gretap-test0

System freezes after creating the rule from container.
Comment 1 Alexey Boldyrev 2022-02-12 21:33:18 UTC
The is not correct command. Correct:
```
tc qdisc add dev eth1 handle 1: root htb default 1
tc filter add dev eth1 parent 1: flower src_mac 22:22:22:22:22:22 action mirred ingress mirror dev gretap-test0
```
Comment 2 Cong Wang 2022-02-14 23:54:37 UTC
What do you mean by freeze? Did you see any kernel stack trace before it is frozen? If not, can you enable kernel soft lockup detector and maybe netconsole or kdump if it is a kernel crash?
Comment 3 Alexey Boldyrev 2022-02-16 16:56:59 UTC
(In reply to Cong Wang from comment #2)
> What do you mean by freeze? Did you see any kernel stack trace before it is
> frozen? If not, can you enable kernel soft lockup detector and maybe
> netconsole or kdump if it is a kernel crash?

1. Kernel panic message not receiveng from netconsole
2. System crash after:
tc filter add dev gretap-test0 parent ffff: matchall action mirred egress redirect dev bridge0    
tc filter add dev bridge0 parent 1: matchall  action mirred ingress mirror dev gretap-test0
Comment 4 Alexey Boldyrev 2022-02-16 22:00:51 UTC
[  476.059875] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [swapper/0:0]
[  504.059883] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [swapper/0:0]
Comment 5 Alexey Boldyrev 2022-02-16 22:29:31 UTC
Soft lookup from:
```
http://45.147.198.155/p/paste.txt
```
Comment 6 Cong Wang 2022-02-17 04:47:27 UTC
Looks like you get packets looping between devices. How do you config gretap-test0 and bridge0?
Comment 7 Alexey Boldyrev 2022-02-17 12:05:59 UTC
(In reply to Cong Wang from comment #6)
> Looks like you get packets looping between devices. How do you config
> gretap-test0 and bridge0?

Config is standard.
Creating the interfaces from command.

ip link add dev bridge0 up mtu 16384 type bridge
ip link add dev gretap-test0 mtu 8954 up type gretap local <local-ip> remote <remote-ip> key 1
Comment 8 Peilin Ye 2022-03-15 23:53:29 UTC
Hi Alexey,

I'm following up on this issue, but I failed to reproduce the freeze or crash.
What exactly is your setup?  What are you trying to do with the bridge and gretap device?

By looking at your commands:

> 2. System crash after:
> tc filter add dev gretap-test0 parent ffff: matchall action mirred egress
> redirect dev bridge0

This redirects gretap-test0's ingress (ffff:) to bridge0's egress;

> tc filter add dev bridge0 parent 1: matchall action mirred ingress mirror
> dev gretap-test0

This mirrors bridge0's egress back to gretap-test0's ingress.
So I _guess_ this creates a loop.  What is your intention here?

Could you provide a complete script?  For example, the following script creates network namespaces "alice" and "bob", and creates an L2 GRE tunnel between them.  veth devices are connected using a bridge in the default namespace.  I know this is probably different from your setup: it seems that your bridge and gretap devices are in the same namespace.

Could you modify/complete it so that it reproduces the freeze or crash on a fresh machine?  It would be very helpful:

ip netns add alice
ip netns add bob

ip link add veth1 type veth peer name br-veth1
ip link add veth2 type veth peer name br-veth2

ip link set veth1 netns alice
ip link set veth2 netns bob

ip -n alice addr add 192.168.1.11/24 dev veth1
ip -n bob addr add 192.168.1.12/24 dev veth2

ip link add name br1 type bridge
ip link set br1 up

ip link set br-veth1 up
ip -n alice link set veth1 up
ip link set br-veth2 up
ip -n bob link set veth2 up

# connect veth devices using a bridge
ip link set br-veth1 master br1
ip link set br-veth2 master br1

ip addr add 192.168.1.10/24 brd + dev br1

# set up an L2 GRE tunnel
ip -n alice link add gretap1 type gretap \
    local 192.168.1.11 remote 192.168.1.12
ip -n bob link add gretap2 type gretap \
    local 192.168.1.12 remote 192.168.1.11

# configure GRE endpoint IPv4 addresses
ip -n alice addr add 10.0.0.1/24 dev gretap1
ip -n alice link set gretap1 up
ip -n bob addr add 10.0.0.2/24 dev gretap2
ip -n bob link set gretap2 up
Comment 9 Alexey Boldyrev 2022-03-16 00:07:06 UTC
ip netns add alice
ip netns add bob

ip link add veth1 type veth peer name br-veth1
ip link add veth2 type veth peer name br-veth2

ip link set veth1 netns alice
ip link set veth2 netns bob

ip -n alice addr add 192.168.1.11/24 dev veth1
ip -n bob addr add 192.168.1.12/24 dev veth2

ip link add name br1 type bridge
ip link set br1 up

ip link set br-veth1 up
ip -n alice link set veth1 up
ip link set br-veth2 up
ip -n bob link set veth2 up

# connect veth devices using a bridge
ip link set br-veth1 master br1
ip link set br-veth2 master br1

ip addr add 192.168.1.10/24 brd + dev br1

# set up an L2 GRE tunnel
ip -n alice link add gretap1 type gretap \
    local 192.168.1.11 remote 192.168.1.12
ip -n bob link add gretap2 type gretap \
    local 192.168.1.12 remote 192.168.1.11

# configure GRE endpoint IPv4 addresses
ip -n alice addr add 10.0.0.1/24 dev gretap1
ip -n alice link set gretap1 up


ip -n bob link set gretap2 up
ip netns exec bob tc qdisc add dev veth2 handle 1: root htb default 1
tc filter add dev eth1 parent 1: flower protocol ip action mirred ingress mirror dev gretap-test0

Note You need to log in before you can comment on or make changes to this bug.