Distribution: archlinux Problem Description: 'cat /proc/net/ip_conntrack' systematically returns No space left on device. After having emptied the entries with 'conntrack -F' (conntrack v0.9.15), /proc/net/ip_conntrack is indeed empty but fills up and is full again when re-accessing the net. 'conntrack -E --buffer-size 1000000' returns: NOTICE: Netlink socket buffer size has been set to 2000000 bytes. WARNING: We have hit ENOBUFS! We are losing events. This message means that the current netlink socket buffer size is too small. Please, check --buffer-size in conntrack(8) manpage. conntrack v0.9.15 (conntrack-tools): Operation failed: No buffer space available Attached you can find the output of 'iptables -nvL' and dmesg, which shows a nice call trace for 'WARNING: at mm/page_alloc.c:1990 __alloc_pages_nodemask+0x4be/0x600()'. Remark: iptables rules are generated using vuurmuur firewall, but other archlinux users have reported the problem without using vuurmuur. Just let me know if you need more info or any actions from my side. Kind Regards
Created attachment 47232 [details] Output of iptables -nvL and dmesg
Here is how I can reproduce the case: 1. Stop iptables and remove all iptable and conntrack modules (ip_* & nf_*) 2. Start iptables and empty rules - Here is are the rules at this step: [root@laptop /home/dobedo]$ iptables -nvL Chain INPUT (policy ACCEPT 1446 packets, 1194K bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 1463 packets, 234K bytes) pkts bytes target prot opt in out source destination - Here is the list of modules at this step: [root@laptop /home/dobedo]$ lsmod | grep -E "ip_|nf_" nf_conntrack_netlink 15541 0 nf_conntrack 48578 1 nf_conntrack_netlink ip_tables 9108 1 iptable_filter ip_queue 4194 0 nfnetlink 2302 2 nf_conntrack_netlink,nfnetlink_queue nf_defrag_ipv4 1007 0 x_tables 11530 10 iptable_filter,ip_tables,xt_CLASSIFY,xt_TCPMSS,xt_NFQUEUE,xt_mac,xt_mark,xt_limit,xt_length,xt_tcpudp - conntrack is not yet loaded at this step: [root@laptop /home/dobedo]$ cat /proc/net/ip_conntrack cat: /proc/net/ip_conntrack: No such file or directory 3. Load conntrack modules, which trigger the creation of the file in /proc [root@laptop /home/dobedo]$ modprobe nf_conntrack_ipv4 [root@laptop /home/dobedo]$ cat /proc/net/ip_conntrack cat: /proc/net/ip_conntrack: No space left on device 4. Empty the conntrack table: [root@laptop /home/dobedo]$ conntrack -F conntrack v0.9.15 (conntrack-tools): connection tracking table has been emptied. [root@laptop /home/dobedo]$ cat /proc/net/ip_conntrack 5. Display e.g. a website in my web browser 6. Try again [root@laptop /home/dobedo]$ cat /proc/net/ip_conntrack cat: /proc/net/ip_conntrack: No space left on device ...also, this is what I get with conntrack tool: [root@laptop /home/dobedo]$ conntrack -E WARNING: We have hit ENOBUFS! We are losing events. This message means that the current netlink socket buffer size is too small. Please, check --buffer-size in conntrack(8) manpage. conntrack v0.9.15 (conntrack-tools): Operation failed: No buffer space available
...and I forgot to mention also the dmesg result after case reproduced: ctnetlink: unregistering from nfnetlink. ip_tables: (C) 2000-2006 Netfilter Core Team nf_conntrack version 0.5.0 (16384 buckets, 65536 max) ctnetlink v0.93: registering with nfnetlink. ------------[ cut here ]------------ WARNING: at mm/page_alloc.c:1990 __alloc_pages_nodemask+0x4be/0x600() Hardware name: Inspiron 1520 Modules linked in: nf_conntrack_ipv4 nf_conntrack_netlink nf_conntrack iptable_filter ip_tables rfcomm sco bnep l2cap crc16 fuse xt_CLASSIFY xt_TCPMSS xt_NFQUEUE xt_mac xt_mark xt_limit xt_length xt_tcpudp nfnetlink_queue ip_queue nfnetlink nf_defrag_ipv4 x_tables coretemp nvidia(P) arc4 ecb iwl3945 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_hda_codec_idt iwlcore snd_pcm_oss snd_mixer_oss mac80211 cfg80211 cpufreq_ondemand joydev dell_wmi sparse_keymap snd_hda_intel snd_hda_codec snd_hwdep snd_pcm b44 acpi_cpufreq intel_agp snd_timer firewire_ohci uvcvideo freq_table ssb firewire_core pcmcia snd sdhci_pci intel_gtt videodev i2c_i801 dell_laptop pcmcia_core sdhci soundcore mii mmc_core crc_itu_t btusb v4l1_compat snd_page_alloc bluetooth usbhid video psmouse wmi i2c_core agpgart output hid shpchp ac battery rfkill thermal iTCO_wdt iTCO_vendor_support button dcdbas processor pci_hotplug sg serio_raw mperf evdev kvm_intel kvm ext3 jbd mbcache uhci_hcd ehci_hcd usbcore sr_mod cdrom sd_mod ata_piix ahci libahci ata_generic pata_acpi libata scsi_mod [last unloaded: ipt_REJECT] Pid: 7990, comm: xchat Tainted: P 2.6.37-ARCH #1 Call Trace: [<c1044b7d>] warn_slowpath_common+0x6d/0xa0 [<c10ca2ee>] ? __alloc_pages_nodemask+0x4be/0x600 [<c10ca2ee>] ? __alloc_pages_nodemask+0x4be/0x600 [<c1044bcd>] warn_slowpath_null+0x1d/0x20 [<c10ca2ee>] __alloc_pages_nodemask+0x4be/0x600 [<c10ca447>] __get_free_pages+0x17/0x30 [<c10fa1d8>] __kmalloc_track_caller+0x178/0x200 [<c10f92b9>] ? __kmalloc+0x159/0x200 [<c103db8e>] ? dequeue_task_fair+0x3e/0x50 [<c1272f19>] ? __alloc_skb+0x29/0x100 [<fb84e03b>] ? ctnetlink_conntrack_event+0xeb/0x790 [nf_conntrack_netlink] [<c1272f43>] __alloc_skb+0x53/0x100 [<fb84e03b>] ctnetlink_conntrack_event+0xeb/0x790 [nf_conntrack_netlink] [<c1315388>] ? _raw_spin_unlock_bh+0x18/0x20 [<fb835c16>] nf_ct_deliver_cached_events+0x86/0xd0 [nf_conntrack] [<fb86f408>] ipv4_confirm+0xd8/0x170 [nf_conntrack_ipv4] [<c129f13d>] nf_iterate+0x4d/0x80 [<c12a97b0>] ? ip_finish_output+0x0/0x300 [<c12a97b0>] ? ip_finish_output+0x0/0x300 [<c129f215>] nf_hook_slow+0xa5/0x100 [<c12a97b0>] ? ip_finish_output+0x0/0x300 [<c12aa4e2>] ip_output+0xc2/0xe0 [<c12a97b0>] ? ip_finish_output+0x0/0x300 [<c12a88f0>] ? dst_output+0x0/0x10 [<c12a9bfb>] ip_local_out+0x1b/0x20 [<c12a9d4c>] ip_queue_xmit+0x14c/0x3f0 [<c11129d0>] ? __pollwait+0x0/0xd0 [<c12c1d47>] ? __tcp_v4_send_check+0x47/0xe0 [<c12be643>] tcp_transmit_skb+0x363/0x800 [<c1112aa0>] ? pollwake+0x0/0x60 [<c12bf0ba>] tcp_write_xmit+0xfa/0x950 [<c12bf977>] __tcp_push_pending_frames+0x27/0x80 [<c12b2b05>] tcp_sendmsg+0x7b5/0xa80 [<c12d22f7>] inet_sendmsg+0x57/0xb0 [<c126accf>] sock_sendmsg+0xef/0x110 [<c126c358>] sys_sendto+0x108/0x150 [<c110302c>] ? do_sync_read+0x9c/0xd0 [<c126c3d6>] sys_send+0x36/0x40 [<c126ce48>] sys_socketcall+0x1d8/0x2a0 [<c11aa65e>] ? copy_to_user+0x2e/0x50 [<c100391f>] sysenter_do_call+0x12/0x28 [<c1310000>] ? generic_processor_info+0x18/0x14f ---[ end trace a73667eead5b71e9 ]---
Am 11.02.2011 21:59, schrieb bugzilla-daemon@bugzilla.kernel.org: > https://bugzilla.kernel.org/show_bug.cgi?id=28862 Does this patch fix the issue?
Created attachment 47892 [details] Fix return values of netfilter sysfs This one, apparently bugzilla doesn't add attachments to the report.
Hi, The patch helps partially. Now, I can read the pseudo file /proc/net/ip_conntrack => no ENOSPC anymore (good) but the data read doesn't contain newlines anymore (not good). It's all on one line. Maybe the ENOSPC is used in one way or another to trigger the newline. I'll try to check that further tonight. Cheers
On 17.02.2011 10:06, bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=28862 > > > --- Comment #6 from valere monseur <valere.monseur@ymail.com> 2011-02-17 > 09:06:37 --- > Hi, > > The patch helps partially. > > Now, I can read the pseudo file /proc/net/ip_conntrack => no ENOSPC anymore > (good) but the data read doesn't contain newlines anymore (not good). It's > all > on one line. I'll look into this. Please post a few lines of the output you're getting.
Created attachment 48172 [details] output of /proc/net/ip_conntrack after patch (no newline)
Am 17.02.2011 20:00, schrieb bugzilla-daemon@bugzilla.kernel.org: > https://bugzilla.kernel.org/show_bug.cgi?id=28862 > > > > > > --- Comment #8 from valere monseur <valere.monseur@ymail.com> 2011-02-17 > 19:00:31 --- > Created an attachment (id=48172) > --> (https://bugzilla.kernel.org/attachment.cgi?id=48172) > output of /proc/net/ip_conntrack after patch (no newline) > There's something very odd about your setup, apparently the space provided to the conntrack seq-file callbacks isn't even enough for a single entry. The "use=" part should follow "mark=", but not even the first entry contains this. Are you running a mainline kernel or does it include any patches?
Created attachment 48202 [details] Things done Well, 2 things: - First, I'm using 2.6.37 to which the current archlinux patches have been applied. Location: ftp://ftp.archlinux.org/other/kernel26/patch-2.6.37-4-ARCH.bz2 - Second, in fact I've been unable to apply exactly the patch you provide me so I've made the changes "by hand" in the 2 files...probably a bad idea I know. I've attached the file mypatch.tar.gz containing the changes I've made if you want to have a look. Have I broken the stuff ? Do you want me to use 2.6.37 with only your patch ? Something special to do to apply it ? Thanks
Am 17.02.2011 20:50, schrieb bugzilla-daemon@bugzilla.kernel.org: > Well, 2 things: > - First, I'm using 2.6.37 to which the current archlinux patches have been > applied. Location: > ftp://ftp.archlinux.org/other/kernel26/patch-2.6.37-4-ARCH.bz2 That one looks fine. > - Second, in fact I've been unable to apply exactly the patch you provide me > so > I've made the changes "by hand" in the 2 files...probably a bad idea I know. > I've attached the file mypatch.tar.gz containing the changes I've made if you > want to have a look. That patch is hard to read because of reindentation, but the relevant parts looks fine. > Have I broken the stuff ? > Do you want me to use 2.6.37 with only your patch ? > Something special to do to apply it ? That would be useful I think. I think the patch should apply cleanly, but in case it doesn't, just change the seq file functions to unconditionally return 0.
Created attachment 48342 [details] /proc/net/ip_conntrack output Hi, Sorry for the reindentation, I turned it off in my editor now. I've recompiled 2.6.37 with only your patch, which applies correctly btw (when I take correct actions). Please, find the result in attach. Result is still the same: no newlines. Any idea? Thanks
I'll run a test now, keep you posted.
Created attachment 48382 [details] test: where does it 'goto release' ? Hi, I've made a quick and ugly test case to know where it does the 'goto release', which seems to be the reason why there is no newline (indeed, the newline is only written at the very last seq_printf, meaning that any 'goto release' performed before will cause the generation of a line without newline). The attachment contains these files: the 2 modified sources, the .config file used (which is the one of the distro - unmodified), the outpur result of /proc/net/ip_conntrack. What we can see is that there is this call performed from ct_seq_show function: if (ct_show_secctx(s, ct)) goto release; and then this call is done from ct_show_secctx function: ret = security_secid_to_secctx(ct->secmark, &secctx, &len); if (ret) return ret; ...and it returns! Meaning that back into ct_seq_show function if will perform the 'goto release' and then the newline is not written at all. Ok, now my stupid patch doesn't tell me the return value of security_secid_to_secctx but I could modify the patch to get it, if needed. Do you have any idea why security_secid_to_secctx returns a value and what does it mean ? is it a kernel bug or a misconfiguration ? Thanks in advance Cheers
Damned, this looks quite similar to what I have here: http://gitorious.org/sched_deadline/linux-26/commit/cba85b532e4aabdb97f44c18987d45141fd93faa as I have also no security module but NF_CONNTRACK_SECMARK enabled. Can you check and let me know ?
Closing as obsolete, if this is still seen with modern kernels please re-open and update