Bug 99671 - glibc deadlock in __check_pf() presumed due to missing netlink response
Summary: glibc deadlock in __check_pf() presumed due to missing netlink response
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-06-08 21:20 UTC by David Woodhouse
Modified: 2016-02-03 12:16 UTC (History)
2 users (show)

See Also:
Kernel Version: 4.0
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description David Woodhouse 2015-06-08 21:20:23 UTC
I keep seeing multithreaded processes deadlocked on glibc's __check_pf function, with the culprit stuck like this:

#0  0x0000003841302edd in recvmsg () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000384131f8ee in __check_pf (pid=-52599, fd=20) at ../sysdeps/unix/sysv/linux/check_pf.c:166
#2  0x000000384131f8ee in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f5b137fdc32, seen_ipv6=seen_ipv6@entry=0x7f5b137fdc33, in6ai=in6ai@entry=0x7f5b137fdc40, in6ailen=in6ailen@entry=0x7f5b137fdc48) at ../sysdeps/unix/sysv/linux/check_pf.c:324

I don't understand why the glibc code is waiting; even if netlink responses are dropped, it should get ENOBUFS and bail out.

This started happening to me, sporadically, when I updated my Fedora 22 beta installation to the 4.0 kernel. It seemed to go away when I reverted to 3.19, and came back again when I went back to 4.0. Filed in Red Hat bugzilla as https://bugzilla.redhat.com/show_bug.cgi?id=1209433 and reported upstream at  https://www.marc.info/?l=linux-netdev&m=142849461211078&w=3 and https://www.marc.info/?l=linux-netdev&m=142954024310299&w=3

Others seem to have seen it on earlier kernels though:
https://github.com/nahi/httpclient/issues/232
Comment 2 David Woodhouse 2015-06-11 00:23:02 UTC
I'm running a kernel with http://patchwork.ozlabs.org/patch/473041/ now. I don't have a reliable reproducer but it usually happens within a day or two. If it lasts a week on this kernel, I'll call it fixed. (Thanks Eric Dumazet for pointing it out).
Comment 3 Konstantin Khlebnikov 2015-06-26 10:59:28 UTC
I've found couple bugs in glibc:

1) this function ignores NLMSG_ERROR message. In this case kernel don't sent NLMSG_DONE at all

2) check (nlmh->nlmsg_pid != pid) isn't safe: in case of pid collision (for example several pid-namespace in one net-namespace) kernel binds netlink socket to some random (and negative?) pid.


Kernel sends -ECONNREFUSED because it cannot find socket in hash
and looks like this code is broken differently in 3.17..3.19 and in 4.0..
Comment 4 David Woodhouse 2015-06-26 13:10:20 UTC
Neither of those were actually causing the problem in my case though; we were getting *no* messages back.

I did see it again after applying the patch mentioned in comment 2. After *also* applying a version of the patch from https://patchwork.ozlabs.org/patch/473049/ I think it finally seems to have gone away.
Comment 5 Konstantin Khlebnikov 2015-06-26 13:29:57 UTC
I've found race in kernels 3.17 .. 3.19
Fix: https://patchwork.ozlabs.org/patch/488736/

In my case I see NLMSG_ERROR message in receive buffer, glibc ignores it and wants more.

Note You need to log in before you can comment on or make changes to this bug.