Most recent kernel where this bug did not occur: don't know
Distribution: gentoo
Hardware Environment: Opteron
Software Environment: nf_conntrack 0.5.0
Problem Description:

We rely on netfilter's ESTABLISHED,CONNECTED connection tracking for our iptables rules.
We have changed the value of two kernel tunables to prevent packets being dropped when the connection tracking table gets full.

Nasty message from dmesg:

nf_conntrack version 0.5.0 (8192 buckets, 65536 max)
bash: page allocation failure. order:10, mode:0xd0
 [<c013991b>] __alloc_pages+0x275/0x286
 [<c0139958>] __get_free_pages+0x2c/0x3a
 [<f88b05c3>] alloc_hashtable+0x2f/0x6a [nf_conntrack]
 [<f88b065b>] set_hashsize+0x3d/0x110 [nf_conntrack]
 [<c014e97b>] do_filp_open+0x32/0x39
 [<c01278aa>] param_attr_store+0x14/0x1d
 [<c0127896>] param_attr_store+0x0/0x1d
 [<c01278f5>] module_attr_store+0x24/0x29
 [<c017fd16>] sysfs_write_file+0x9b/0xc5
 [<c017fc7b>] sysfs_write_file+0x0/0xc5
 [<c0150264>] vfs_write+0xa8/0x154
 [<c0150871>] sys_write+0x41/0x67
 [<c0102b76>] sysenter_past_esp+0x5f/0x85
 =======================
Mem-info:
DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd: 169   Cold: hi:   62, btch:  15 usd:  53
CPU    1: Hot: hi:  186, btch:  31 usd:  79   Cold: hi:   62, btch:  15 usd:  60
CPU    2: Hot: hi:  186, btch:  31 usd: 145   Cold: hi:   62, btch:  15 usd:  58
CPU    3: Hot: hi:  186, btch:  31 usd:  52   Cold: hi:   62, btch:  15 usd:  48
HighMem per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd: 160   Cold: hi:   62, btch:  15 usd:  12
CPU    1: Hot: hi:  186, btch:  31 usd:  84   Cold: hi:   62, btch:  15 usd:   2
CPU    2: Hot: hi:  186, btch:  31 usd:  21   Cold: hi:   62, btch:  15 usd:  12
CPU    3: Hot: hi:  186, btch:  31 usd: 130   Cold: hi:   62, btch:  15 usd:  12
Active:258342 inactive:103609 dirty:3110 writeback:0 unstable:0 free:569156 slab:66033 mapped:2773 pagetables:968
DMA free:3888kB min:68kB low:84kB high:100kB active:0kB inactive:4kB present:16256kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 873 3921
Normal free:12328kB min:3744kB low:4680kB high:5616kB active:358004kB inactive:209268kB present:894080kB pages_scanned:34 all_unreclaimable? no
lowmem_reserve[]: 0 0 24383
HighMem free:2260408kB min:512kB low:3780kB high:7052kB active:675236kB inactive:205292kB present:3121092kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 108*4kB 64*8kB 10*16kB 5*32kB 1*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3888kB
Normal: 1681*4kB 366*8kB 33*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 12388kB
HighMem: 3586*4kB 3014*8kB 406*16kB 7975*32kB 7829*64kB 3308*128kB 962*256kB 178*512kB 28*1024kB 5*2048kB 161*4096kB = 2260408kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 2000084kB
Total swap = 2000084kB
Free swap:       2000084kB
1015792 pages of RAM
786416 pages of HIGHMEM
9244 reserved pages
311964 pages shared
0 pages swap cached
3110 pages dirty
0 pages writeback
2773 pages mapped
65487 pages slab
968 pages pagetables
nf_conntrack: falling back to vmalloc.

Steps to reproduce:

echo 524288 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
echo 524288 > /sys/module/nf_conntrack/parameters/hashsize
/etc/init.d/iptables start