Most recent kernel where this bug did *NOT* occur: Unknown Distribution: SUSE 10 Pro Hardware Environment: Intel(R) Pentium(R) 4 CPU 2.80GHz, 32 bit, 512 Megs of RAM Software Environment: SUSE 10, 2.6.17-rt7-default with ingo molnar RT patch Problem Description: Established tcp network connections are not shown in /proc/net/tcp (not displayed in netstat). I began to notice this when I would use netstat to display all established connections. There were a few connections not shown as ESTABLISHED by netstat, yet my application did not report a connection broken with the remote device. I logged into the remote device and observed the connection was ESTABLISHED on the remote side. In addition, I verified the transfer of data to the remote device. I searched through the proc file system to find out more info and it seems that my application still held the socket file descriptor as opened. #### LOCAL MACHINE WITH BUG# 10.1.2.71 is IP of remote device note connection not established local:/ # netstat -a | grep 10.1.2.71 ## NOTE IP of local device local:/ # ifconfig eth1 eth1 Link encap:Ethernet HWaddr 00:03:2D:09:0E:6F inet addr:10.1.2.10 Bcast:10.1.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:558434 errors:0 dropped:0 overruns:0 frame:0 TX packets:264478 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:37082257 (35.3 Mb) TX bytes:19489061 (18.5 Mb) Interrupt:18 # ##telnet into remote machine local:/ # telnet 10.1.2.71 Trying 10.1.2.71... Connected to 10.1.2.71. Escape character is '^]'. MontaVista(R) Linux(R) Professional Edition 3.1 Linux/ppc 2.4.20_mvl31-405gp_eval (none) login: root Password: Linux (none) 2.4.20_mvl31-405gp_eval #474 Tue Dec 27 16:37:25 PST 2005 ppc unknown MontaVista(R) Linux(R) Professional Edition 3.1 BusyBox v0.60.3 (2004.01.09-22:55+0000) Built-in shell (ash) Enter 'help' for a list of built-in commands. ######## Remote machine. Note connection established on port 1400 with local machine # netstat -a | grep 10.1.2.10 tcp 0 0 10.1.2.71:telnet 10.1.2.10:51555 ESTABLISHED tcp 0 0 10.1.2.71:1400 10.1.2.10:60380 ESTABLISHED ###### Back on LOCAL machine with bug ### Note established connection to IP 10.1.2.71 on port 1400 not shown. However, other established connections shown to other devices running the same remote server software. local:/proc/2774/fd # grep 0A02010A /proc/net/tcp | more 64: 0A02010A:EBDC 4702010A:0578 01 00000000:00000000 00:00000000 00000000 0 0 7477 1 d8ba1980 201 40 10 2 100 70: 0A02010A:A547 4802010A:0578 01 00000000:00000000 00:00000000 00000000 0 0 7478 1 d8ba1400 201 40 10 2 100 #### Display open inodes ### I grep for the inode of the established connections from above. My application will open the sockets to the remote devices sequentially. ml_alc10:/proc/2774/fd # ls -l | grep socket | grep 747[0-9] lrwx------ 1 root root 64 Mar 7 10:37 154 -> socket:[7471] lrwx------ 1 root root 64 Mar 7 10:37 155 -> socket:[7472] lrwx------ 1 root root 64 Mar 7 10:37 158 -> socket:[7473] lrwx------ 1 root root 64 Mar 7 10:37 159 -> socket:[7474] lrwx------ 1 root root 64 Mar 7 10:37 160 -> socket:[7475] lrwx------ 1 root root 64 Mar 7 10:37 161 -> socket:[7476] lrwx------ 1 root root 64 Mar 7 10:37 162 -> socket:[7477] lrwx------ 1 root root 64 Mar 7 10:37 163 -> socket:[7478] lrwx------ 1 root root 64 Mar 7 10:37 164 -> socket:[7479] ## Check if socket inode 7479 exists in /proc/net/tcp local:/proc/2774/fd # grep 7479 /proc/net/tcp | more local:/proc/2774/fd # ls -l | grep 7479 lrwx------ 1 root root 64 Mar 7 10:37 164 -> socket:[7479] local:/proc/2774/fd # grep 7479 /proc/net/tcp | more local:/proc/2774/fd # #### NOTE Very Strange behavior. The connection will intermittently be shown in /proc/net/tcp with the same local port number and inode. This leads me to believe that the connection persists throughout and that the kernel is failing to accurately display all established socket connections. Please advise on how display all ESTABLISHED connections correctly. Thanks.
Reply-To: akpm@linux-foundation.org Begin forwarded message: Date: Thu, 8 Mar 2007 07:53:57 -0800 From: bugme-daemon@bugzilla.kernel.org To: bugme-new@lists.osdl.org Subject: [Bugme-new] [Bug 8151] New: Established connections not displayed in /proc/tcp/netstat http://bugzilla.kernel.org/show_bug.cgi?id=8151 Summary: Established connections not displayed in /proc/tcp/netstat Kernel Version: 2.6.17-rt7-default #3 PREEMPT Status: NEW Severity: normal Owner: acme@conectiva.com.br Submitter: getitupsidelines@yahoo.com CC: getitupsidelines@yahoo.com Most recent kernel where this bug did *NOT* occur: Unknown Distribution: SUSE 10 Pro Hardware Environment: Intel(R) Pentium(R) 4 CPU 2.80GHz, 32 bit, 512 Megs of RAM Software Environment: SUSE 10, 2.6.17-rt7-default with ingo molnar RT patch Problem Description: Established tcp network connections are not shown in /proc/net/tcp (not displayed in netstat). I began to notice this when I would use netstat to display all established connections. There were a few connections not shown as ESTABLISHED by netstat, yet my application did not report a connection broken with the remote device. I logged into the remote device and observed the connection was ESTABLISHED on the remote side. In addition, I verified the transfer of data to the remote device. I searched through the proc file system to find out more info and it seems that my application still held the socket file descriptor as opened. #### LOCAL MACHINE WITH BUG# 10.1.2.71 is IP of remote device note connection not established local:/ # netstat -a | grep 10.1.2.71 ## NOTE IP of local device local:/ # ifconfig eth1 eth1 Link encap:Ethernet HWaddr 00:03:2D:09:0E:6F inet addr:10.1.2.10 Bcast:10.1.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:558434 errors:0 dropped:0 overruns:0 frame:0 TX packets:264478 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:37082257 (35.3 Mb) TX bytes:19489061 (18.5 Mb) Interrupt:18 # ##telnet into remote machine local:/ # telnet 10.1.2.71 Trying 10.1.2.71... Connected to 10.1.2.71. Escape character is '^]'. MontaVista(R) Linux(R) Professional Edition 3.1 Linux/ppc 2.4.20_mvl31-405gp_eval (none) login: root Password: Linux (none) 2.4.20_mvl31-405gp_eval #474 Tue Dec 27 16:37:25 PST 2005 ppc unknown MontaVista(R) Linux(R) Professional Edition 3.1 BusyBox v0.60.3 (2004.01.09-22:55+0000) Built-in shell (ash) Enter 'help' for a list of built-in commands. ######## Remote machine. Note connection established on port 1400 with local machine # netstat -a | grep 10.1.2.10 tcp 0 0 10.1.2.71:telnet 10.1.2.10:51555 ESTABLISHED tcp 0 0 10.1.2.71:1400 10.1.2.10:60380 ESTABLISHED ###### Back on LOCAL machine with bug ### Note established connection to IP 10.1.2.71 on port 1400 not shown. However, other established connections shown to other devices running the same remote server software. local:/proc/2774/fd # grep 0A02010A /proc/net/tcp | more 64: 0A02010A:EBDC 4702010A:0578 01 00000000:00000000 00:00000000 00000000 0 0 7477 1 d8ba1980 201 40 10 2 100 70: 0A02010A:A547 4802010A:0578 01 00000000:00000000 00:00000000 00000000 0 0 7478 1 d8ba1400 201 40 10 2 100 #### Display open inodes ### I grep for the inode of the established connections from above. My application will open the sockets to the remote devices sequentially. ml_alc10:/proc/2774/fd # ls -l | grep socket | grep 747[0-9] lrwx------ 1 root root 64 Mar 7 10:37 154 -> socket:[7471] lrwx------ 1 root root 64 Mar 7 10:37 155 -> socket:[7472] lrwx------ 1 root root 64 Mar 7 10:37 158 -> socket:[7473] lrwx------ 1 root root 64 Mar 7 10:37 159 -> socket:[7474] lrwx------ 1 root root 64 Mar 7 10:37 160 -> socket:[7475] lrwx------ 1 root root 64 Mar 7 10:37 161 -> socket:[7476] lrwx------ 1 root root 64 Mar 7 10:37 162 -> socket:[7477] lrwx------ 1 root root 64 Mar 7 10:37 163 -> socket:[7478] lrwx------ 1 root root 64 Mar 7 10:37 164 -> socket:[7479] ## Check if socket inode 7479 exists in /proc/net/tcp local:/proc/2774/fd # grep 7479 /proc/net/tcp | more local:/proc/2774/fd # ls -l | grep 7479 lrwx------ 1 root root 64 Mar 7 10:37 164 -> socket:[7479] local:/proc/2774/fd # grep 7479 /proc/net/tcp | more local:/proc/2774/fd # #### NOTE Very Strange behavior. The connection will intermittently be shown in /proc/net/tcp with the same local port number and inode. This leads me to believe that the connection persists throughout and that the kernel is failing to accurately display all established socket connections. Please advise on how display all ESTABLISHED connections correctly. Thanks. ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
The data in /proc/net/tcp is not atomic. If there are lots of connections it is easily possible for a list to change during the traversal and some entries will be missed. Is this problem occurring on an active machine with connections coming and going? or is it a standalone test/workstation?
Hello and thanks for the response. The machine is active. Of course connections can occur at anytime the application is running but for the specific sockets that are "missing" from the /proc/net/tcp file the connection is proven to persists for hours or days at a time.
Andrew Morton <akpm@linux-foundation.org> wrote: > > Date: Thu, 8 Mar 2007 07:53:57 -0800 > From: bugme-daemon@bugzilla.kernel.org > To: bugme-new@lists.osdl.org > Subject: [Bugme-new] [Bug 8151] New: Established connections not displayed in /proc/tcp/netstat Please try running 'ss -nt' to see if it displays the missing connections. If it does then we can confirm a problem with /proc, otherwise the you might have a TCP problem instead. Note that with a recent kernel you may have to build/load the tcp_diag module before 'ss -nt' will work. Cheers,
This is confusing. Your example shows # grep 0A02010A /proc/net/tcp | more 64: 0A02010A:EBDC 4702010A:0578 01 00000000:00000000 00:00000000 00000000 0 0 7477 1 d8ba1980 201 40 10 2 100 and you say "established connection to IP 10.1.2.71 on port 1400 not shown". But 4702010A:0578 really is 10.1.2.71, port 1400 (0x0578 in hex), so it _is_ shown. The remainder of the report - you list some open sockets of your application, and then go looking for the one with inode 7479 in /proc/net/tcp. Have you confirmed prior to this that the socket is indeed a TCP socket? What does lsof -p <pid-of-your-application> report?
ss -nt does not show the established socket. In addition, lsof -p shows "can't identify protocol" for my missing sockets. So If understand correctly: as far as the system is concerned the missing sockets are not tcp sockets (according to lsof, even though they are according to everything else I see) therefore, they are not listed in /proc/net/tcp and netstat. So unless im missing something here I assume this is NOT a kernel bug and should therefore be reported on another board. Please advise. Thanks
The lsof error message could mean that lsof is broken, or that the kernel indeed isn't printing all sockets. Can you please try the following: - compile the program attached below, put the binary in /tmp/fdsock - pick an application where you believe netstat et al don't show connected sockets properly. Run the app from the command line, and note the PID - Attach to the running process using gdb: gdb /path/to/my/application <pid> - Within gdb, run p system("/tmp/fdsock") This should display a listing of all file handles the application has open, with an indication of whether it's a socket, tty, etc. Note that the output goes to the tty of the application you have attached to, not the terminal you're running gdb in. For sockets, fdsock will also print out the local address, and - if connected - the peer's address, like this: socket inet/stream 192.168.5.32:58131 connected to 192:168.5.1:22 flags 802 (RDWR,NONBLOCK on fd 3 Compare that to the output of /proc/net/tcp, netstat, etc.
Created attachment 10773 [details] fdsock test program Well, it helps to actually attach the test program, not just talk about doing it :-)
Maybe the missing sockets are Unix domain, not TCP at all.
Please give us some feedback, is this still a bug or is it a non-problem.
Please reopen this bug if: - it is still present with kernel 2.6.22 and - you can provide the requested information.