I keep seeing multithreaded processes deadlocked on glibc's __check_pf function, with the culprit stuck like this: #0 0x0000003841302edd in recvmsg () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000384131f8ee in __check_pf (pid=-52599, fd=20) at ../sysdeps/unix/sysv/linux/check_pf.c:166 #2 0x000000384131f8ee in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f5b137fdc32, seen_ipv6=seen_ipv6@entry=0x7f5b137fdc33, in6ai=in6ai@entry=0x7f5b137fdc40, in6ailen=in6ailen@entry=0x7f5b137fdc48) at ../sysdeps/unix/sysv/linux/check_pf.c:324 I don't understand why the glibc code is waiting; even if netlink responses are dropped, it should get ENOBUFS and bail out. This started happening to me, sporadically, when I updated my Fedora 22 beta installation to the 4.0 kernel. It seemed to go away when I reverted to 3.19, and came back again when I went back to 4.0. Filed in Red Hat bugzilla as https://bugzilla.redhat.com/show_bug.cgi?id=1209433 and reported upstream at https://www.marc.info/?l=linux-netdev&m=142849461211078&w=3 and https://www.marc.info/?l=linux-netdev&m=142954024310299&w=3 Others seem to have seen it on earlier kernels though: https://github.com/nahi/httpclient/issues/232
This is the glibc code in question: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/check_pf.c;h=162606d7;hb=glibc-2.21#l166
I'm running a kernel with http://patchwork.ozlabs.org/patch/473041/ now. I don't have a reliable reproducer but it usually happens within a day or two. If it lasts a week on this kernel, I'll call it fixed. (Thanks Eric Dumazet for pointing it out).
I've found couple bugs in glibc: 1) this function ignores NLMSG_ERROR message. In this case kernel don't sent NLMSG_DONE at all 2) check (nlmh->nlmsg_pid != pid) isn't safe: in case of pid collision (for example several pid-namespace in one net-namespace) kernel binds netlink socket to some random (and negative?) pid. Kernel sends -ECONNREFUSED because it cannot find socket in hash and looks like this code is broken differently in 3.17..3.19 and in 4.0..
Neither of those were actually causing the problem in my case though; we were getting *no* messages back. I did see it again after applying the patch mentioned in comment 2. After *also* applying a version of the patch from https://patchwork.ozlabs.org/patch/473049/ I think it finally seems to have gone away.
I've found race in kernels 3.17 .. 3.19 Fix: https://patchwork.ozlabs.org/patch/488736/ In my case I see NLMSG_ERROR message in receive buffer, glibc ignores it and wants more.