I have a system that's performing heavy SNMP monitoring on various devices over IPv6. After upgrading it from 4.11.5 to 4.12.2 last week, I noticed this weekend that all UDP packets received by the system were being silently dropped. The problem happened this weekend, after the system had been running for about 2 days with no issue, and without any human intervention at the time (i.e. it broke on its own - no configuration changes were made). I verified UDP replies were coming in via tcpdump, and checked netfilter and conntrack to make sure nothing was blocking it (indeed netfilter/conntrack is able to see the traffic, indicating the problem is deeper in the network stack). There is no relevant or even unusual output whatsoever in dmesg. This affects ALL received UDPv4/UDPv6 packets, even traffic on a loopback interface: opening a pair of Python sessions and trying to send UDP traffic back and forth on 127.0.0.1 results in the traffic getting dropped as well. ICMP and TCP are unaffected. My gut says it's a race condition somewhere in the UDP receive queue(s). What puzzles me is why heavy UDPv6 load caused the kernel to shut out all UDPv4 traffic too.
A couple more bits of information: 1) Watching /proc/net/snmp{,6} shows UDP(6)'s InErrors increment each time a packet is received on a valid port. 2) The system does still respond with an ICMP error when a datagram destined to an invalid port is received.
None of the other SNMP counters nor `sk_drops` get increased when a packet is dropped; this means the fault should be somewhere either in `__udp4_lib_rcv` or `__udp_queue_rcv_skb`, as those are the only functions that can increment `UDP_MIB_INERRORS` without also incrementing either `sk_drops`, `UDP_MIB_CSUMERRORS`, or `UDP_MIB_RCVBUFERRORS` as well. Since these functions deal with adding the page to the memory queue, I decided to try increasing the UDP rmem max via the udp_mem sysctl, and it began working again. Therefore this issue is very likely a memory leak in the receive queue, which was probably introduced by 850cbaddb52dfd4e0c7cabe2c168dd34b44ae0b9.
But commit 850cbaddb52dfd4e0c7cabe2c168dd34b is merged in v4.10-rc1, not 4.11 or 4.12.
It's entirely possible I was just "lucky" and never encountered the set of circumstances required to trigger the bug until recently (I hadn't been using that node for SNMP-over-IPv6 for long, and it was a TON of UDP traffic once I started). It's also equally possible that 850cbad isn't the culprit commit and I'm barking up the wrong tree. I couldn't see any other commit that seemed to fit the bill for what would affect UDPv4 from heavy UDPv6 traffic, but then again I'm not as familiar with the commit history in that part of the kernel tree as some others. :)
There is a leak in the UDP code path triggered by the ipv6 early demux introduced in 4.12. When an udp v6 packet reaches __udp6_lib_lookup_skb() via the early demux path, an UDP socket reference count is leaked and and the UDP socket destructor is never called. If there are some packets queued there, some amount of accounted memory is leaked, too. In the long run this will cause the effect you are experiencing. I'll submit a patch soon.
(In reply to Paolo Abeni from comment #5) > There is a leak in the UDP code path triggered by the ipv6 early demux > introduced in 4.12. > > When an udp v6 packet reaches __udp6_lib_lookup_skb() via the early demux > path, an UDP socket reference count is leaked and and the UDP socket > destructor is never called. If there are some packets queued there, some > amount of accounted memory is leaked, too. > > In the long run this will cause the effect you are experiencing. In the meanwhile, as a workaround, you can disable UDP early demux with: sysctl -w net.ipv4.udp_early_demux=0