Bug 196469
Summary: | New UDP memory accounting schema leaks receive queue pages; eventually kills receiving altogether | ||
---|---|---|---|
Product: | Networking | Reporter: | Sam Edwards (CFSworks) |
Component: | IPV4 | Assignee: | Stephen Hemminger (stephen) |
Status: | NEW --- | ||
Severity: | normal | CC: | pabeni, xiyou.wangcong |
Priority: | P1 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | 4.12.2 | Subsystem: | |
Regression: | Yes | Bisected commit-id: |
Description
Sam Edwards
2017-07-24 20:52:01 UTC
A couple more bits of information: 1) Watching /proc/net/snmp{,6} shows UDP(6)'s InErrors increment each time a packet is received on a valid port. 2) The system does still respond with an ICMP error when a datagram destined to an invalid port is received. None of the other SNMP counters nor `sk_drops` get increased when a packet is dropped; this means the fault should be somewhere either in `__udp4_lib_rcv` or `__udp_queue_rcv_skb`, as those are the only functions that can increment `UDP_MIB_INERRORS` without also incrementing either `sk_drops`, `UDP_MIB_CSUMERRORS`, or `UDP_MIB_RCVBUFERRORS` as well. Since these functions deal with adding the page to the memory queue, I decided to try increasing the UDP rmem max via the udp_mem sysctl, and it began working again. Therefore this issue is very likely a memory leak in the receive queue, which was probably introduced by 850cbaddb52dfd4e0c7cabe2c168dd34b44ae0b9. But commit 850cbaddb52dfd4e0c7cabe2c168dd34b is merged in v4.10-rc1, not 4.11 or 4.12. It's entirely possible I was just "lucky" and never encountered the set of circumstances required to trigger the bug until recently (I hadn't been using that node for SNMP-over-IPv6 for long, and it was a TON of UDP traffic once I started). It's also equally possible that 850cbad isn't the culprit commit and I'm barking up the wrong tree. I couldn't see any other commit that seemed to fit the bill for what would affect UDPv4 from heavy UDPv6 traffic, but then again I'm not as familiar with the commit history in that part of the kernel tree as some others. :) There is a leak in the UDP code path triggered by the ipv6 early demux introduced in 4.12. When an udp v6 packet reaches __udp6_lib_lookup_skb() via the early demux path, an UDP socket reference count is leaked and and the UDP socket destructor is never called. If there are some packets queued there, some amount of accounted memory is leaked, too. In the long run this will cause the effect you are experiencing. I'll submit a patch soon. (In reply to Paolo Abeni from comment #5) > There is a leak in the UDP code path triggered by the ipv6 early demux > introduced in 4.12. > > When an udp v6 packet reaches __udp6_lib_lookup_skb() via the early demux > path, an UDP socket reference count is leaked and and the UDP socket > destructor is never called. If there are some packets queued there, some > amount of accounted memory is leaked, too. > > In the long run this will cause the effect you are experiencing. In the meanwhile, as a workaround, you can disable UDP early demux with: sysctl -w net.ipv4.udp_early_demux=0 |