Bug 99671

Summary:	glibc deadlock in __check_pf() presumed due to missing netlink response
Product:	Networking	Reporter:	David Woodhouse (dwmw2)
Component:	Other	Assignee:	Stephen Hemminger (stephen)
Status:	NEW ---
Severity:	normal	CC:	fweimer, koct9i
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	4.0	Subsystem:
Regression:	No	Bisected commit-id:

Description David Woodhouse 2015-06-08 21:20:23 UTC

I keep seeing multithreaded processes deadlocked on glibc's __check_pf function, with the culprit stuck like this:

#0  0x0000003841302edd in recvmsg () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000384131f8ee in __check_pf (pid=-52599, fd=20) at ../sysdeps/unix/sysv/linux/check_pf.c:166
#2  0x000000384131f8ee in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f5b137fdc32, seen_ipv6=seen_ipv6@entry=0x7f5b137fdc33, in6ai=in6ai@entry=0x7f5b137fdc40, in6ailen=in6ailen@entry=0x7f5b137fdc48) at ../sysdeps/unix/sysv/linux/check_pf.c:324

I don't understand why the glibc code is waiting; even if netlink responses are dropped, it should get ENOBUFS and bail out.

This started happening to me, sporadically, when I updated my Fedora 22 beta installation to the 4.0 kernel. It seemed to go away when I reverted to 3.19, and came back again when I went back to 4.0. Filed in Red Hat bugzilla as https://bugzilla.redhat.com/show_bug.cgi?id=1209433 and reported upstream at  https://www.marc.info/?l=linux-netdev&m=142849461211078&w=3 and https://www.marc.info/?l=linux-netdev&m=142954024310299&w=3

Others seem to have seen it on earlier kernels though:
https://github.com/nahi/httpclient/issues/232

Comment 1 David Woodhouse 2015-06-08 21:21:37 UTC

This is the glibc code in question: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/check_pf.c;h=162606d7;hb=glibc-2.21#l166

Comment 2 David Woodhouse 2015-06-11 00:23:02 UTC

I'm running a kernel with http://patchwork.ozlabs.org/patch/473041/ now. I don't have a reliable reproducer but it usually happens within a day or two. If it lasts a week on this kernel, I'll call it fixed. (Thanks Eric Dumazet for pointing it out).

Comment 3 Konstantin Khlebnikov 2015-06-26 10:59:28 UTC

I've found couple bugs in glibc:

1) this function ignores NLMSG_ERROR message. In this case kernel don't sent NLMSG_DONE at all

2) check (nlmh->nlmsg_pid != pid) isn't safe: in case of pid collision (for example several pid-namespace in one net-namespace) kernel binds netlink socket to some random (and negative?) pid.


Kernel sends -ECONNREFUSED because it cannot find socket in hash
and looks like this code is broken differently in 3.17..3.19 and in 4.0..

Comment 4 David Woodhouse 2015-06-26 13:10:20 UTC

Neither of those were actually causing the problem in my case though; we were getting *no* messages back.

I did see it again after applying the patch mentioned in comment 2. After *also* applying a version of the patch from https://patchwork.ozlabs.org/patch/473049/ I think it finally seems to have gone away.

Comment 5 Konstantin Khlebnikov 2015-06-26 13:29:57 UTC

I've found race in kernels 3.17 .. 3.19
Fix: https://patchwork.ozlabs.org/patch/488736/

In my case I see NLMSG_ERROR message in receive buffer, glibc ignores it and wants more.