Bug 13561 - swapper: page allocation failure. order:0, mode:0x20
swapper: page allocation failure. order:0, mode:0x20
Status: RESOLVED OBSOLETE
Product: Memory Management
Classification: Unclassified
Component: Other
All Linux
: P1 normal
Assigned To: Andrew Morton
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-17 18:41 UTC by Nigel Kukard
Modified: 2012-11-20 17:02 UTC (History)
3 users (show)

See Also:
Kernel Version: 2.6.39.3
Tree: Mainline
Regression: No


Attachments

Description Nigel Kukard 2009-06-17 18:41:22 UTC
AMD X2 dual processor system with 4Gb RAM

# lspci 
00:00.0 Host bridge: ATI Technologies Inc RS690 Host Bridge
00:02.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Graphics Port 0)
00:07.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 3)
00:12.0 SATA controller: ATI Technologies Inc SB600 Non-Raid-5 SATA
00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0)
00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1)
00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2)
00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3)
00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4)
00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI)
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 14)
00:14.1 IDE interface: ATI Technologies Inc SB600 IDE
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: nVidia Corporation G94 [GeForce 9600 GT] (rev a1)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)


# lsmod
Module                  Size  Used by
isofs                  31544  0 
zlib_inflate           15416  1 isofs
psmouse                42484  0 
uhci_hcd               25512  0 
r8169                  33580  0 
mii                     4760  1 r8169
ext2                   65464  1 
nvram                   7684  0 
loop                   15252  0 
usbhid                 43184  0 
hid                    40688  1 usbhid
nvidiafb               43064  0 
fb                     45832  1 nvidiafb
fb_ddc                  2136  1 nvidiafb
backlight               5632  1 nvidiafb
i2c_algo_bit            5772  1 nvidiafb
cfbcopyarea             3688  1 nvidiafb
vgastate                8664  1 nvidiafb
cfbimgblt               2616  1 nvidiafb
cfbfillrect             3864  1 nvidiafb
snd_hda_codec_realtek   243548  1 
nvidia              10257288  36 
snd_hda_intel          26856  0 
snd_hda_codec          59112  2 snd_hda_codec_realtek,snd_hda_intel
snd_pcsp               10684  0 
ohci_hcd               35348  0 
snd_hwdep               7840  1 snd_hda_codec
snd_pcm                68128  3 snd_hda_intel,snd_hda_codec,snd_pcsp
ssb                    38044  1 ohci_hcd
rtc_cmos               13624  0 
fan                     3976  0 
rtc_core               19748  1 rtc_cmos
snd_timer              21160  1 snd_pcm
pcmcia                 26080  1 ssb
ehci_hcd               49652  0 
rtc_lib                 2664  1 rtc_core
thermal                18920  0 
pcmcia_core            32916  2 ssb,pcmcia
processor              45024  1 thermal
snd                    58408  7 snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_pcsp,snd_hwdep,snd_pcm,snd_timer
usbcore               170800  5 uhci_hcd,usbhid,ohci_hcd,ehci_hcd
i2c_piix4              12808  0 
k8temp                  4888  0 
thermal_sys            12240  3 fan,thermal,processor
bitrev                  1944  2 r8169,nvidiafb
button                  7256  0 
i2c_core               29888  5 nvidiafb,fb_ddc,i2c_algo_bit,nvidia,i2c_piix4
hwmon                   3304  2 k8temp,thermal_sys
snd_page_alloc          8904  2 snd_hda_intel,snd_pcm
crc32                   3944  2 r8169,pcmcia
evdev                  10720  6 
sg                     21304  0 
ext3                  124696  2 
jbd                    47152  1 ext3
mbcache                 7980  2 ext2,ext3
dm_mod                 55744  0 
sr_mod                 15044  1 
sd_mod                 30184  5 
crc_t10dif              1848  1 sd_mod
ide_cd_mod             30584  0 
cdrom                  32712  2 sr_mod,ide_cd_mod
ide_gd_mod             24336  0 
ahci                   33832  4 
atiixp                  4140  0 
ide_core              100036  3 ide_cd_mod,ide_gd_mod,atiixp
pata_atiixp             5576  0 
pata_acpi               4920  0 
libata                157244  3 ahci,pata_atiixp,pata_acpi



Kernel 2.6.29.4 compiled 64-bit x86_64.  The -1.0  is just a build number on my side to identify which kernel I built.

Any info you need at all, please let me know.

[ 1884.639134] swapper: page allocation failure. order:0, mode:0x20
[ 1884.639136] Pid: 0, comm: swapper Tainted: P           2.6.29.4-1.0 #1
[ 1884.639137] Call Trace:
[ 1884.639139]  <IRQ>  [<ffffffff8028828b>] __alloc_pages_internal+0x41e/0x43f
[ 1884.639146]  [<ffffffff802ac183>] alloc_pages_current+0xb9/0xc2
[ 1884.639149]  [<ffffffff802b08eb>] new_slab+0xcf/0x28c
[ 1884.639151]  [<ffffffff802b0d14>] __slab_alloc+0x200/0x3e2
[ 1884.639154]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
[ 1884.639157]  [<ffffffff804974cc>] ? _spin_lock_irqsave+0x28/0x31
[ 1884.639160]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
[ 1884.639162]  [<ffffffff802b12cf>] kmem_cache_alloc_node+0x7c/0xc6
[ 1884.639165]  [<ffffffff8024b8a4>] ? __mod_timer+0xb3/0xc5
[ 1884.639168]  [<ffffffff803d9e3b>] __alloc_skb+0x42/0x131
[ 1884.639171]  [<ffffffff8041892e>] tcp_send_ack+0x2b/0x112
[ 1884.639173]  [<ffffffff80415c53>] __tcp_ack_snd_check+0x65/0x7d
[ 1884.639176]  [<ffffffff80416906>] tcp_rcv_established+0x7d7/0x926
[ 1884.639179]  [<ffffffff8041dfb3>] tcp_v4_do_rcv+0x1b1/0x35e
[ 1884.639181]  [<ffffffff8041e606>] tcp_v4_rcv+0x4a6/0x785
[ 1884.639185]  [<ffffffff80402a2d>] ip_local_deliver_finish+0x177/0x25a
[ 1884.639187]  [<ffffffff80402b82>] ip_local_deliver+0x72/0x7a
[ 1884.639190]  [<ffffffff804025e3>] ip_rcv_finish+0x32b/0x345
[ 1884.639192]  [<ffffffff80402879>] ip_rcv+0x27c/0x2b9
[ 1884.639195]  [<ffffffff803e05b0>] netif_receive_skb+0x471/0x496
[ 1884.639201]  [<ffffffffa0ca07c0>] rtl8169_rx_interrupt+0x362/0x43d [r8169]
[ 1884.639205]  [<ffffffffa0ca38b3>] rtl8169_poll+0x3f/0x1fe [r8169]
[ 1884.639208]  [<ffffffff803de82c>] net_rx_action+0xae/0x19c
[ 1884.639212]  [<ffffffff8024745d>] __do_softirq+0x8a/0x139
[ 1884.639214]  [<ffffffff8021259c>] call_softirq+0x1c/0x30
[ 1884.639217]  [<ffffffff80213d78>] do_softirq+0x44/0x8f
[ 1884.639219]  [<ffffffff80247153>] irq_exit+0x3f/0x7e
[ 1884.639222]  [<ffffffff80213ff3>] do_IRQ+0xc3/0xe4
[ 1884.639224]  [<ffffffff80211d13>] ret_from_intr+0x0/0x29
[ 1884.639225]  <EOI>  [<ffffffff8022704e>] ? native_safe_halt+0x6/0x8
[ 1884.639230]  [<ffffffff802180c2>] ? default_idle+0x2e/0x4b
[ 1884.639233]  [<ffffffff8021830c>] ? c1e_idle+0x109/0x110
[ 1884.639235]  [<ffffffff802590c9>] ? atomic_notifier_call_chain+0x13/0x15
[ 1884.639239]  [<ffffffff8021022e>] ? cpu_idle+0x59/0x9a
[ 1884.639242]  [<ffffffff804925d6>] ? start_secondary+0x254/0x25b
[ 1884.639244] Mem-Info:
[ 1884.639244] Node 0 DMA per-cpu:
[ 1884.639246] CPU    0: hi:    0, btch:   1 usd:   0
[ 1884.639248] CPU    1: hi:    0, btch:   1 usd:   0
[ 1884.639249] Node 0 DMA32 per-cpu:
[ 1884.639251] CPU    0: hi:  186, btch:  31 usd: 181
[ 1884.639253] CPU    1: hi:  186, btch:  31 usd: 207
[ 1884.639254] Node 0 Normal per-cpu:
[ 1884.639256] CPU    0: hi:  186, btch:  31 usd: 162
[ 1884.639257] CPU    1: hi:  186, btch:  31 usd: 178
[ 1884.639260] Active_anon:18228 active_file:11269 inactive_anon:6552
[ 1884.639261]  inactive_file:913548 unevictable:120 dirty:69982 writeback:194 unstable:0
[ 1884.639262]  free:4387 slab:39717 mapped:7378 pagetables:1235 bounce:0
[ 1884.639264] Node 0 DMA free:11692kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15284kB pages_scanned:0 all_unreclaimable? yes
[ 1884.639269] lowmem_reserve[]: 0 3255 4012 4012
[ 1884.639271] Node 0 DMA32 free:5412kB min:6560kB low:8200kB high:9840kB active_anon:60476kB inactive_anon:12224kB active_file:28720kB inactive_file:2987624kB unevictable:0kB present:3333216kB pages_scanned:0 all_unreclaimable? no
[ 1884.639276] lowmem_reserve[]: 0 0 757 757
[ 1884.639279] Node 0 Normal free:444kB min:1524kB low:1904kB high:2284kB active_anon:12436kB inactive_anon:13984kB active_file:16356kB inactive_file:666568kB unevictable:480kB present:775680kB pages_scanned:0 all_unreclaimable? no
[ 1884.639284] lowmem_reserve[]: 0 0 0 0
[ 1884.639286] Node 0 DMA: 1*4kB 1*8kB 2*16kB 2*32kB 1*64kB 2*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11692kB
[ 1884.639293] Node 0 DMA32: 397*4kB 12*8kB 6*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 5460kB
[ 1884.639300] Node 0 Normal: 73*4kB 3*8kB 0*16kB 0*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 444kB
[ 1884.639307] 925361 total pagecache pages
[ 1884.639308] 0 pages in swap cache
[ 1884.639309] Swap cache stats: add 32, delete 32, find 30/30
[ 1884.639311] Free swap  = 1048568kB
[ 1884.639312] Total swap = 1048568kB
[ 1884.649121] 1048560 pages RAM
[ 1884.649121] 43140 pages reserved
[ 1884.649121] 938822 pages shared
[ 1884.649121] 82468 pages non-shared
Comment 1 Andrew Morton 2009-06-17 19:16:15 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 17 Jun 2009 18:41:24 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=13561
> 
>            Summary: swapper: page allocation failure. order:0, mode:0x20
>            Product: Memory Management
> 
> ...
>
> [ 1884.639134] swapper: page allocation failure. order:0, mode:0x20
> [ 1884.639136] Pid: 0, comm: swapper Tainted: P           2.6.29.4-1.0 #1
> [ 1884.639137] Call Trace:
> [ 1884.639139]  <IRQ>  [<ffffffff8028828b>] __alloc_pages_internal+0x41e/0x43f
> [ 1884.639146]  [<ffffffff802ac183>] alloc_pages_current+0xb9/0xc2
> [ 1884.639149]  [<ffffffff802b08eb>] new_slab+0xcf/0x28c
> [ 1884.639151]  [<ffffffff802b0d14>] __slab_alloc+0x200/0x3e2
> [ 1884.639154]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
> [ 1884.639157]  [<ffffffff804974cc>] ? _spin_lock_irqsave+0x28/0x31
> [ 1884.639160]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
> [ 1884.639162]  [<ffffffff802b12cf>] kmem_cache_alloc_node+0x7c/0xc6
> [ 1884.639165]  [<ffffffff8024b8a4>] ? __mod_timer+0xb3/0xc5
> [ 1884.639168]  [<ffffffff803d9e3b>] __alloc_skb+0x42/0x131
> [ 1884.639171]  [<ffffffff8041892e>] tcp_send_ack+0x2b/0x112
> [ 1884.639173]  [<ffffffff80415c53>] __tcp_ack_snd_check+0x65/0x7d
> [ 1884.639176]  [<ffffffff80416906>] tcp_rcv_established+0x7d7/0x926
> [ 1884.639179]  [<ffffffff8041dfb3>] tcp_v4_do_rcv+0x1b1/0x35e
> [ 1884.639181]  [<ffffffff8041e606>] tcp_v4_rcv+0x4a6/0x785
> [ 1884.639185]  [<ffffffff80402a2d>] ip_local_deliver_finish+0x177/0x25a
> [ 1884.639187]  [<ffffffff80402b82>] ip_local_deliver+0x72/0x7a
> [ 1884.639190]  [<ffffffff804025e3>] ip_rcv_finish+0x32b/0x345
> [ 1884.639192]  [<ffffffff80402879>] ip_rcv+0x27c/0x2b9
> [ 1884.639195]  [<ffffffff803e05b0>] netif_receive_skb+0x471/0x496
> [ 1884.639201]  [<ffffffffa0ca07c0>] rtl8169_rx_interrupt+0x362/0x43d [r8169]
> [ 1884.639205]  [<ffffffffa0ca38b3>] rtl8169_poll+0x3f/0x1fe [r8169]
> [ 1884.639208]  [<ffffffff803de82c>] net_rx_action+0xae/0x19c
> [ 1884.639212]  [<ffffffff8024745d>] __do_softirq+0x8a/0x139
> [ 1884.639214]  [<ffffffff8021259c>] call_softirq+0x1c/0x30
> [ 1884.639217]  [<ffffffff80213d78>] do_softirq+0x44/0x8f
> [ 1884.639219]  [<ffffffff80247153>] irq_exit+0x3f/0x7e
> [ 1884.639222]  [<ffffffff80213ff3>] do_IRQ+0xc3/0xe4
> [ 1884.639224]  [<ffffffff80211d13>] ret_from_intr+0x0/0x29
> [ 1884.639225]  <EOI>  [<ffffffff8022704e>] ? native_safe_halt+0x6/0x8
> [ 1884.639230]  [<ffffffff802180c2>] ? default_idle+0x2e/0x4b
> [ 1884.639233]  [<ffffffff8021830c>] ? c1e_idle+0x109/0x110
> [ 1884.639235]  [<ffffffff802590c9>] ? atomic_notifier_call_chain+0x13/0x15
> [ 1884.639239]  [<ffffffff8021022e>] ? cpu_idle+0x59/0x9a
> [ 1884.639242]  [<ffffffff804925d6>] ? start_secondary+0x254/0x25b

yep, that's OK.  The kernel was excessively low on memory and the
network driver was unable to allocate a page for a received packet. 
The packet will just be dropped and everything should recover.  If you
get a lot of these warnings (one per minute?) then increasing the value
in /proc/sys/vm/min_free_kbytes should help.


Dave, we get quite a few reports of this nature, especially from e1000
(grr).  Do you think we could/should suppress the warning, by
sprinkling a few __GFP_NOWARNs in the right places?  It doesn't seem
like it's being very useful?
Comment 2 Nigel Kukard 2009-06-17 19:31:35 UTC
>
> On Wed, 17 Jun 2009 18:41:24 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
>
>   
>> http://bugzilla.kernel.org/show_bug.cgi?id=13561
>>
>>            Summary: swapper: page allocation failure. order:0, mode:0x20
>>            Product: Memory Management
>>
>> ...
>>
>> [ 1884.639134] swapper: page allocation failure. order:0, mode:0x20
>> [ 1884.639136] Pid: 0, comm: swapper Tainted: P           2.6.29.4-1.0 #1
>> [ 1884.639137] Call Trace:
>> [ 1884.639139]  <IRQ>  [<ffffffff8028828b>] __alloc_pages_internal+0x41e/0x43f
>> [ 1884.639146]  [<ffffffff802ac183>] alloc_pages_current+0xb9/0xc2
>> [ 1884.639149]  [<ffffffff802b08eb>] new_slab+0xcf/0x28c
>> [ 1884.639151]  [<ffffffff802b0d14>] __slab_alloc+0x200/0x3e2
>> [ 1884.639154]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
>> [ 1884.639157]  [<ffffffff804974cc>] ? _spin_lock_irqsave+0x28/0x31
>> [ 1884.639160]  [<ffffffff803d9e3b>] ? __alloc_skb+0x42/0x131
>> [ 1884.639162]  [<ffffffff802b12cf>] kmem_cache_alloc_node+0x7c/0xc6
>> [ 1884.639165]  [<ffffffff8024b8a4>] ? __mod_timer+0xb3/0xc5
>> [ 1884.639168]  [<ffffffff803d9e3b>] __alloc_skb+0x42/0x131
>> [ 1884.639171]  [<ffffffff8041892e>] tcp_send_ack+0x2b/0x112
>> [ 1884.639173]  [<ffffffff80415c53>] __tcp_ack_snd_check+0x65/0x7d
>> [ 1884.639176]  [<ffffffff80416906>] tcp_rcv_established+0x7d7/0x926
>> [ 1884.639179]  [<ffffffff8041dfb3>] tcp_v4_do_rcv+0x1b1/0x35e
>> [ 1884.639181]  [<ffffffff8041e606>] tcp_v4_rcv+0x4a6/0x785
>> [ 1884.639185]  [<ffffffff80402a2d>] ip_local_deliver_finish+0x177/0x25a
>> [ 1884.639187]  [<ffffffff80402b82>] ip_local_deliver+0x72/0x7a
>> [ 1884.639190]  [<ffffffff804025e3>] ip_rcv_finish+0x32b/0x345
>> [ 1884.639192]  [<ffffffff80402879>] ip_rcv+0x27c/0x2b9
>> [ 1884.639195]  [<ffffffff803e05b0>] netif_receive_skb+0x471/0x496
>> [ 1884.639201]  [<ffffffffa0ca07c0>] rtl8169_rx_interrupt+0x362/0x43d [r8169]
>> [ 1884.639205]  [<ffffffffa0ca38b3>] rtl8169_poll+0x3f/0x1fe [r8169]
>> [ 1884.639208]  [<ffffffff803de82c>] net_rx_action+0xae/0x19c
>> [ 1884.639212]  [<ffffffff8024745d>] __do_softirq+0x8a/0x139
>> [ 1884.639214]  [<ffffffff8021259c>] call_softirq+0x1c/0x30
>> [ 1884.639217]  [<ffffffff80213d78>] do_softirq+0x44/0x8f
>> [ 1884.639219]  [<ffffffff80247153>] irq_exit+0x3f/0x7e
>> [ 1884.639222]  [<ffffffff80213ff3>] do_IRQ+0xc3/0xe4
>> [ 1884.639224]  [<ffffffff80211d13>] ret_from_intr+0x0/0x29
>> [ 1884.639225]  <EOI>  [<ffffffff8022704e>] ? native_safe_halt+0x6/0x8
>> [ 1884.639230]  [<ffffffff802180c2>] ? default_idle+0x2e/0x4b
>> [ 1884.639233]  [<ffffffff8021830c>] ? c1e_idle+0x109/0x110
>> [ 1884.639235]  [<ffffffff802590c9>] ? atomic_notifier_call_chain+0x13/0x15
>> [ 1884.639239]  [<ffffffff8021022e>] ? cpu_idle+0x59/0x9a
>> [ 1884.639242]  [<ffffffff804925d6>] ? start_secondary+0x254/0x25b
>>     
>
> yep, that's OK.  The kernel was excessively low on memory and the
> network driver was unable to allocate a page for a received packet. 
>   

Even though the box had 4Gbyte RAM with nothing running but console? 
maybe I"m mistaking the amount of RAM for the amount of memory in a
buffer used for network IO?

-N


> The packet will just be dropped and everything should recover.  If you
> get a lot of these warnings (one per minute?) then increasing the value
> in /proc/sys/vm/min_free_kbytes should help.
>
>
> Dave, we get quite a few reports of this nature, especially from e1000
> (grr).  Do you think we could/should suppress the warning, by
> sprinkling a few __GFP_NOWARNs in the right places?  It doesn't seem
> like it's being very useful?
>
>
>
Comment 3 Andrew Morton 2009-06-17 19:46:23 UTC
On Wed, 17 Jun 2009 19:29:25 +0000
Nigel Kukard <nkukard@lbsd.net> wrote:

> > yep, that's OK.  The kernel was excessively low on memory and the
> > network driver was unable to allocate a page for a received packet. 
> >   
> 
> Even though the box had 4Gbyte RAM with nothing running but console? 

yep.

> maybe I"m mistaking the amount of RAM for the amount of memory in a
> buffer used for network IO?

[ 1884.639264] Node 0 DMA free:11692kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15284kB pages_scanned:0 all_unreclaimable? yes

[ 1884.639271] Node 0 DMA32 free:5412kB min:6560kB low:8200kB high:9840kB active_anon:60476kB inactive_anon:12224kB active_file:28720kB inactive_file:2987624kB unevictable:0kB present:3333216kB pages_scanned:0 all_unreclaimable? no

[ 1884.639279] Node 0 Normal free:444kB min:1524kB low:1904kB high:2284kB active_anon:12436kB inactive_anon:13984kB active_file:16356kB inactive_file:666568kB unevictable:480kB present:775680kB pages_scanned:0 all_unreclaimable? no

Pretty much all your memory was used by pagecache.  The DMA32 zone is
below the minimum threshold.  The VM has decided that there's no memory
left for networking.  It has retained a little bit of free memory so
the block device drivers can still perform writeout, to free up
additional memory.

What is _supposed_ to happen (and usually does) it that kswapd will see
that the machine is getting short on memory and will work to free some
up, thus making it available to networking receive.  But sometimes
kswapd can't keep up.
Comment 4 Nigel Kukard 2009-06-17 20:02:35 UTC
>   
>> maybe I"m mistaking the amount of RAM for the amount of memory in a
>> buffer used for network IO?
>>     
>
> [ 1884.639264] Node 0 DMA free:11692kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15284kB pages_scanned:0 all_unreclaimable? yes
>
> [ 1884.639271] Node 0 DMA32 free:5412kB min:6560kB low:8200kB high:9840kB active_anon:60476kB inactive_anon:12224kB active_file:28720kB inactive_file:2987624kB unevictable:0kB present:3333216kB pages_scanned:0 all_unreclaimable? no
>
> [ 1884.639279] Node 0 Normal free:444kB min:1524kB low:1904kB high:2284kB active_anon:12436kB inactive_anon:13984kB active_file:16356kB inactive_file:666568kB unevictable:480kB present:775680kB pages_scanned:0 all_unreclaimable? no
>
> Pretty much all your memory was used by pagecache.  The DMA32 zone is
> below the minimum threshold.  The VM has decided that there's no memory
> left for networking.  It has retained a little bit of free memory so
> the block device drivers can still perform writeout, to free up
> additional memory.
>
> What is _supposed_ to happen (and usually does) it that kswapd will see
> that the machine is getting short on memory and will work to free some
> up, thus making it available to networking receive.  But sometimes
> kswapd can't keep up.
>   
Understood .... I did copy about 2.5Tb of files from 1 SATA drive to
another and was busy trasferring 50Gb to the box from Gbit ethernet.

Cool, thanks again guys. Just thought I'd report to make sure its
something trivial and not a serious issue.  :)

-N
Comment 5 Sebastien Koechlin 2011-08-10 08:28:29 UTC
Probably same problem here with a 2.6.39.3 kernel and a rtl8169 NIC (and order=1).

Got this stack trace in the logs on an Atom with 4GB of memory, no swap activity but high LVM+Raid1 disk activity (got an error on ext4 later). Network activity was about 5KB/s according to MRTG (5 minutes granularity).

The RX count error of the network card was not increased.

eth0      Link encap:Ethernet  HWaddr 1c:6f:65:5d:01:c3
          inet adr:192.168.0.2  Bcast:192.168.0.255  Masque:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6192264 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8650686 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000
          RX bytes:3144978240 (2.9 GiB)  TX bytes:9084870934 (8.4 GiB)
          Interruption:41


[640793.018739] EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
[647233.722234] kworker/0:0: page allocation failure. order:1, mode:0x20
[647233.722337] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39.3.skc2 #23
[647233.722429] Call Trace:
[647233.722509]  <IRQ>  [<ffffffff81070dff>] __alloc_pages_nodemask+0x6e7/0x721
[647233.722626]  [<ffffffff81096821>] cache_alloc_refill+0x264/0x4a6
[647233.722723]  [<ffffffff81435392>] ? __nf_ct_refresh_acct+0x5d/0xc6
[647233.722819]  [<ffffffff81096593>] kmem_cache_alloc+0x4c/0x76
[647233.722915]  [<ffffffff813f1fab>] sk_prot_alloc+0x2a/0x9a
[647233.723011]  [<ffffffff813f20d3>] sk_clone+0x1d/0x266
[647233.723108]  [<ffffffff8146b78b>] inet_csk_clone+0x13/0x84
[647233.723205]  [<ffffffff8147f5b9>] tcp_create_openreq_child+0x24/0x43a
[647233.723305]  [<ffffffff8147dfd8>] tcp_v4_syn_recv_sock+0x57/0x1e2
[647233.723403]  [<ffffffff8147f444>] tcp_check_req+0x241/0x392
[647233.723499]  [<ffffffff8149ed26>] ? nf_nat_fn+0x12e/0x144
[647233.723595]  [<ffffffff8147d646>] tcp_v4_do_rcv+0xfc/0x236
[647233.723693]  [<ffffffff8147dc7b>] tcp_v4_rcv+0x4fb/0x801
[647233.723788]  [<ffffffff8146343e>] ip_local_deliver_finish+0x125/0x1d0
[647233.723887]  [<ffffffff8146355b>] ip_local_deliver+0x72/0x79
[647233.723984]  [<ffffffff8146308f>] ip_rcv_finish+0x27f/0x2a9
[647233.724080]  [<ffffffff814632f3>] ip_rcv+0x23a/0x260
[647233.724174]  [<ffffffff813fca52>] __netif_receive_skb+0x441/0x473
[647233.724270]  [<ffffffff813fcc94>] netif_receive_skb+0x67/0x6e
[647233.724363]  [<ffffffff813fcd79>] napi_skb_finish+0x24/0x3b
[647233.724457]  [<ffffffff813fd22b>] napi_gro_receive+0xa8/0xad
[647233.724550]  [<ffffffff812a3c96>] rtl8169_rx_interrupt+0x2e0/0x354
[647233.724645]  [<ffffffff812a5dcc>] rtl8169_poll+0x33/0x181
[647233.724748]  [<ffffffff812a565b>] ? rtl8169_interrupt+0x28c/0x2f1
[647233.724858]  [<ffffffff813fd327>] net_rx_action+0x6a/0x16f
[647233.724967]  [<ffffffff810368d0>] __do_softirq+0x83/0x114
[647233.725068]  [<ffffffff814e46cc>] call_softirq+0x1c/0x30
[647233.725172]  [<ffffffff810033fb>] do_softirq+0x33/0x68
[647233.725276]  [<ffffffff8103676d>] irq_exit+0x3f/0x88
[647233.725368]  [<ffffffff81002ce2>] do_IRQ+0x98/0xaf
[647233.725459]  [<ffffffff814e3253>] common_interrupt+0x13/0x13
[647233.725551]  <EOI>  [<ffffffff811e1511>] ? intel_idle+0xc3/0xe9
[647233.725655]  [<ffffffff811e14f4>] ? intel_idle+0xa6/0xe9
[647233.725747]  [<ffffffff8136eec7>] cpuidle_idle_call+0x94/0xcd
[647233.725842]  [<ffffffff81001314>] cpu_idle+0xa7/0xdd
[647233.725934]  [<ffffffff81b54ac7>] start_secondary+0x1e5/0x1ec
[647233.726026] Mem-Info:
[647233.726110] DMA per-cpu:
[647233.726191] CPU    0: hi:    0, btch:   1 usd:   0
[647233.726279] CPU    1: hi:    0, btch:   1 usd:   0
[647233.726366] CPU    2: hi:    0, btch:   1 usd:   0
[647233.726453] CPU    3: hi:    0, btch:   1 usd:   0
[647233.726539] DMA32 per-cpu:
[647233.726620] CPU    0: hi:  186, btch:  31 usd:  39
[647233.726708] CPU    1: hi:  186, btch:  31 usd: 172
[647233.726795] CPU    2: hi:  186, btch:  31 usd:   5
[647233.726884] CPU    3: hi:  186, btch:  31 usd:  40
[647233.726970] Normal per-cpu:
[647233.727051] CPU    0: hi:  186, btch:  31 usd:  61
[647233.727143] CPU    1: hi:  186, btch:  31 usd:  71
[647233.727231] CPU    2: hi:  186, btch:  31 usd:  26
[647233.727319] CPU    3: hi:  186, btch:  31 usd:  20
[647233.727414] active_anon:177616 inactive_anon:73202 isolated_anon:0
[647233.727418]  active_file:217661 inactive_file:241205 isolated_file:0
[647233.727422]  unevictable:0 dirty:452 writeback:0 unstable:0
[647233.727426]  free:89576 slab_reclaimable:198275 slab_unreclaimable:8385
[647233.727430]  mapped:11597 shmem:8406 pagetables:1207 bounce:0
[647233.727874] DMA free:15924kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15700kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[647233.728400] lowmem_reserve[]: 0 2990 4000 4000
[647233.728513] DMA32 free:321236kB min:50324kB low:62904kB high:75484kB active_anon:585268kB inactive_anon:162624kB active_file:631132kB inactive_file:713800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3061856kB mlocked:0kB dirty:1332kB writeback:0kB mapped:26016kB shmem:28272kB slab_reclaimable:616544kB slab_unreclaimable:10116kB kernel_stack:272kB pagetables:1608kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[647233.729049] lowmem_reserve[]: 0 0 1010 1010
[647233.729158] Normal free:21144kB min:16996kB low:21244kB high:25492kB active_anon:125196kB inactive_anon:130184kB active_file:239512kB inactive_file:251020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:0kB dirty:476kB writeback:0kB mapped:20372kB shmem:5352kB slab_reclaimable:176556kB slab_unreclaimable:23424kB kernel_stack:1296kB pagetables:3220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[647233.729720] lowmem_reserve[]: 0 0 0 0
[647233.729815] DMA: 1*4kB 0*8kB 1*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15924kB
[647233.730025] DMA32: 80216*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 320864kB
[647233.730229] Normal: 4492*4kB 141*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 21144kB
[647233.730432] 471748 total pagecache pages
[647233.730516] 4414 pages in swap cache
[647233.730603] Swap cache stats: add 8778, delete 4364, find 3043263/3043783
[647233.730696] Free swap  = 503124kB
[647233.730777] Total swap = 524280kB
[647233.732143] 1048560 pages RAM
[647233.732143] 37520 pages reserved
[647233.732143] 310276 pages shared
[647233.732143] 637616 pages non-shared
Comment 6 Alan 2012-11-20 17:02:39 UTC
Closing as obsolete, if this is still seen with modern kernels please re-open and update

Note You need to log in before you can comment on or make changes to this bug.