Bug 199963 - UDP rx_queue incorrect calculation in /proc/net/udp
Summary: UDP rx_queue incorrect calculation in /proc/net/udp
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-06-07 13:21 UTC by trevor.francis
Modified: 2018-06-09 19:25 UTC (History)
1 user (show)

See Also:
Kernel Version: Kernels >= 4.15
Subsystem:
Regression: No
Bisected commit-id:


Attachments
Graph of UDP rx queue lengths during iperf test. (58.67 KB, image/png)
2018-06-09 19:25 UTC, Doug Smythies
Details

Description trevor.francis 2018-06-07 13:21:23 UTC
since upgrading to any kernel >= 4.15 the rx_queue in /proc/net/udp is now reporting a queue, regardless of system load and regardless of what applications are running on it. The tx_queue is always 0, but rx_queue has seemingly random spikes of udp queueing. This is observed across hundreds of servers with either varying or no workload.

netstat -nl|grep ^udp
udp 4352 0 0.0.0.0:68 0.0.0.0:*

sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
14645: 3500007F:0035 00000000:0000 07 00000000:0000C900 00:00000000 00000000 101 0 3367 2 ffff8da177fdcc00 0
Comment 1 Doug Smythies 2018-06-07 18:54:44 UTC
I bisected the kernel and got:

0d4a6608f68c7532dcbfec2ea1150c9761767d03 is the first bad commit
commit 0d4a6608f68c7532dcbfec2ea1150c9761767d03
Author: Paolo Abeni <pabeni@redhat.com>
Date:   Tue Sep 19 12:11:43 2017 +0200

    udp: do rmem bulk free even if the rx sk queue is empty

However, I also found this:
https://www.mail-archive.com/netdev@vger.kernel.org/msg239743.html
and see that Paolo is already on it.
Comment 2 Doug Smythies 2018-06-08 23:12:37 UTC
I am not on the netdev e-mail list, but anyhow saw a patch posted there. I tried it on top of kernel 4.17, and it seems to work. I do not know how to create some real RX queue, so didn't (couldn't) test that.

Reference:
https://marc.info/?l=linux-netdev&m=152845055527577&w=2
Comment 3 Doug Smythies 2018-06-09 19:25:57 UTC
Created attachment 276437 [details]
Graph of UDP rx queue lengths during iperf test.

Using iperf, I was able to creates non-zero reported UDP Rx queues. The graph compares a 5 minute test with Kernel 4.4, 4.17, and 4.17 + the proposed patch. The results are as expected.

Note You need to log in before you can comment on or make changes to this bug.