Bug 9771 - page allocation failure. order:1, mode:0x4020
Summary: page allocation failure. order:1, mode:0x4020
Status: CLOSED OBSOLETE
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-01-17 14:55 UTC by Puzin, Dimitri
Modified: 2012-05-17 15:28 UTC (History)
5 users (show)

See Also:
Kernel Version: 2.6.24-rc8-git1
Subsystem:
Regression: No
Bisected commit-id:


Attachments
Kernel config (44.29 KB, application/octet-stream)
2008-01-17 14:57 UTC, Puzin, Dimitri
Details

Description Puzin, Dimitri 2008-01-17 14:55:55 UTC
Latest working kernel version: 2.6.24-rc7-git2

Earliest failing kernel version: 2.6.24-rc8-git1

Distribution: Debian GNU/Linux 4.0r2 (Etch) x86_64

Hardware Environment: Dual Opteron 252/w 4+2GB RAM on Tyan S2882-D

Software Environment: aes-256-cbc crypted LVs on a soft-RAID-5, additionally, NFS fixes for 2.6.24-rc8 from client.linux-nfs.org were applied. Kernel config attached. 

Problem Description: OOPS

Steps to reproduce: This is the first occurence, the system  has now 23:51:13 up 21:48 uptime. The previous kernel has been running a couple of days without any problems.

The system logs following:

kernel: kcryptd: page allocation failure. order:1, mode:0x4020
kernel: Pid: 2723, comm: kcryptd Not tainted 2.6.24-rc8-git1-nfs16012008 #1
kernel:
kernel: Call Trace:
kernel: <IRQ>  [<ffffffff8025f8c7>] __alloc_pages+0x2e2/0x2fb
kernel: [<ffffffff8027c913>] new_slab+0x63/0x22a
kernel: [<ffffffff8027cc8c>] __slab_alloc+0x1b2/0x3ad
kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43
kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43
kernel: [<ffffffff8027dcc8>] __kmalloc_node_track_caller+0x8c/0xc8
kernel: [<ffffffff803467c8>] __alloc_skb+0x69/0x132
kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43
kernel: [<ffffffff881ccb67>] :tg3:tg3_alloc_rx_skb+0xc4/0x150
kernel: [<ffffffff881d4669>] :tg3:tg3_poll+0x393/0x91c
kernel: [<ffffffff8034d1d7>] net_rx_action+0x9e/0x18c
kernel: [<ffffffff80235ead>] __do_softirq+0x55/0xc3
kernel: [<ffffffff8021ca64>] ack_apic_level+0x10/0xd9
kernel: [<ffffffff8020cfcc>] call_softirq+0x1c/0x28
kernel: [<ffffffff8020ec0e>] do_softirq+0x2c/0x7d
kernel: [<ffffffff80235e13>] irq_exit+0x3f/0x84
kernel: [<ffffffff8020ee45>] do_IRQ+0xb6/0xd5
kernel: [<ffffffff8020c351>] ret_from_intr+0x0/0xa
kernel: <EOI>  [<ffffffff8830c835>] :aes_x86_64:enc128+0x4c3/0x80e
kernel: [<ffffffff88305395>] :cbc:crypto_cbc_encrypt+0xef/0x140
kernel: [<ffffffff8830502c>] :cbc:xor_128+0x0/0x17
kernel: [<ffffffff8830d705>] :aes_x86_64:aes_encrypt+0x0/0x5
kernel: [<ffffffff8838c73f>] :dm_crypt:crypt_convert_scatterlist+0x7b/0xc7
kernel: [<ffffffff8838c8ae>] :dm_crypt:crypt_convert+0x123/0x15d
kernel: [<ffffffff8838cabd>] :dm_crypt:kcryptd_do_crypt+0x1d5/0x253
kernel: [<ffffffff8838c8e8>] :dm_crypt:kcryptd_do_crypt+0x0/0x253
kernel: [<ffffffff8023fbb5>] run_workqueue+0x7f/0x10b
kernel: [<ffffffff80240557>] worker_thread+0x90/0x9b
kernel: [<ffffffff802434cc>] autoremove_wake_function+0x0/0x2e
kernel: [<ffffffff802404c7>] worker_thread+0x0/0x9b
kernel: [<ffffffff802433a2>] kthread+0x47/0x75
kernel: [<ffffffff8020cc58>] child_rip+0xa/0x12
kernel: [<ffffffff8024335b>] kthread+0x0/0x75
kernel: [<ffffffff8020cc4e>] child_rip+0x0/0x12
kernel:
kernel: Mem-info:
kernel: Node 0 DMA per-cpu:
kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
kernel: Node 0 DMA32 per-cpu:
kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 159   Cold: hi:   62, btch:  15 usd:  60
kernel: CPU    1: Hot: hi:  186, btch:  31 usd:  83   Cold: hi:   62, btch:  15 usd:  36
kernel: Node 1 DMA32 per-cpu:
kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  76   Cold: hi:   62, btch:  15 usd:   8
kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 170   Cold: hi:   62, btch:  15 usd:  50
kernel: Node 1 Normal per-cpu:
kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 185   Cold: hi:   62, btch:  15 usd:  10
kernel: CPU    1: Hot: hi:  186, btch:  31 usd:  14   Cold: hi:   62, btch:  15 usd:  47
kernel: Active:74680 inactive:1373896 dirty:83344 writeback:10472 unstable:0
kernel: free:4303 slab:42709 mapped:7248 pagetables:1906 bounce:0
kernel: Node 0 DMA free:8024kB min:12kB low:12kB high:16kB active:0kB inactive:4308kB present:9096kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 2004 2004 2004
kernel: Node 0 DMA32 free:1684kB min:3332kB low:4164kB high:4996kB active:35424kB inactive:1931932kB present:2052320kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 0
kernel: Node 1 DMA32 free:6320kB min:3144kB low:3928kB high:4716kB active:15392kB inactive:1839036kB present:1937344kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 2020 2020
kernel: Node 1 Normal free:1700kB min:3356kB low:4192kB high:5032kB active:247904kB inactive:1720308kB present:2068480kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 0
kernel: Node 0 DMA: 25*4kB 7*8kB 12*16kB 8*32kB 4*64kB 2*128kB 3*256kB 2*512kB 1*1024kB 2*2048kB 0*4096kB = 8028kB
kernel: Node 0 DMA32: 279*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1756kB
kernel: Node 1 DMA32: 52*4kB 185*8kB 45*16kB 10*32kB 6*64kB 6*128kB 2*256kB 2*512kB 1*1024kB 0*2048kB 0*4096kB = 6440kB
kernel: Node 1 Normal: 31*4kB 30*8kB 7*16kB 11*32kB 8*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2620kB
kernel: Swap cache: add 39, delete 39, find 0/0, race 0+0
kernel: Free swap  = 1048416kB
kernel: Total swap = 1048568kB
kernel: Free swap:       1048416kB
kernel: 1540096 pages of RAM
kernel: 22480 reserved pages
kernel: 911826 pages shared
kernel: 0 pages swap cached
Comment 1 Puzin, Dimitri 2008-01-17 14:57:26 UTC
Created attachment 14494 [details]
Kernel config
Comment 2 KOSAKI Motohiro 2008-03-16 22:04:25 UTC
altanative easy reproduce method:

% ./hackbench 105 process 1000

this parameter mean consume all physical memory, but no use any swap.
(please adjust your test environment)

I guess the cause of bug is current linux VM can't memory reclaim under interrupt context.
thus, when interrupt happend under memory reclaim running, it cause kmalloc(GFP_ATOMIC) failure.

I working on fixed it now.
but it need a bit long time...
Comment 3 Natalie Protasevich 2008-06-02 18:03:12 UTC
Hi Kosaki, thanks for the report and your effort. Please keep us updates on your work.
Comment 4 Erik Andr 2009-12-30 14:40:39 UTC
Is this still an issue with a recent kernel?

Note You need to log in before you can comment on or make changes to this bug.