Latest working kernel version: 2.6.24-rc7-git2 Earliest failing kernel version: 2.6.24-rc8-git1 Distribution: Debian GNU/Linux 4.0r2 (Etch) x86_64 Hardware Environment: Dual Opteron 252/w 4+2GB RAM on Tyan S2882-D Software Environment: aes-256-cbc crypted LVs on a soft-RAID-5, additionally, NFS fixes for 2.6.24-rc8 from client.linux-nfs.org were applied. Kernel config attached. Problem Description: OOPS Steps to reproduce: This is the first occurence, the system has now 23:51:13 up 21:48 uptime. The previous kernel has been running a couple of days without any problems. The system logs following: kernel: kcryptd: page allocation failure. order:1, mode:0x4020 kernel: Pid: 2723, comm: kcryptd Not tainted 2.6.24-rc8-git1-nfs16012008 #1 kernel: kernel: Call Trace: kernel: <IRQ> [<ffffffff8025f8c7>] __alloc_pages+0x2e2/0x2fb kernel: [<ffffffff8027c913>] new_slab+0x63/0x22a kernel: [<ffffffff8027cc8c>] __slab_alloc+0x1b2/0x3ad kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43 kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43 kernel: [<ffffffff8027dcc8>] __kmalloc_node_track_caller+0x8c/0xc8 kernel: [<ffffffff803467c8>] __alloc_skb+0x69/0x132 kernel: [<ffffffff8034739c>] __netdev_alloc_skb+0x29/0x43 kernel: [<ffffffff881ccb67>] :tg3:tg3_alloc_rx_skb+0xc4/0x150 kernel: [<ffffffff881d4669>] :tg3:tg3_poll+0x393/0x91c kernel: [<ffffffff8034d1d7>] net_rx_action+0x9e/0x18c kernel: [<ffffffff80235ead>] __do_softirq+0x55/0xc3 kernel: [<ffffffff8021ca64>] ack_apic_level+0x10/0xd9 kernel: [<ffffffff8020cfcc>] call_softirq+0x1c/0x28 kernel: [<ffffffff8020ec0e>] do_softirq+0x2c/0x7d kernel: [<ffffffff80235e13>] irq_exit+0x3f/0x84 kernel: [<ffffffff8020ee45>] do_IRQ+0xb6/0xd5 kernel: [<ffffffff8020c351>] ret_from_intr+0x0/0xa kernel: <EOI> [<ffffffff8830c835>] :aes_x86_64:enc128+0x4c3/0x80e kernel: [<ffffffff88305395>] :cbc:crypto_cbc_encrypt+0xef/0x140 kernel: [<ffffffff8830502c>] :cbc:xor_128+0x0/0x17 kernel: [<ffffffff8830d705>] :aes_x86_64:aes_encrypt+0x0/0x5 kernel: [<ffffffff8838c73f>] :dm_crypt:crypt_convert_scatterlist+0x7b/0xc7 kernel: [<ffffffff8838c8ae>] :dm_crypt:crypt_convert+0x123/0x15d kernel: [<ffffffff8838cabd>] :dm_crypt:kcryptd_do_crypt+0x1d5/0x253 kernel: [<ffffffff8838c8e8>] :dm_crypt:kcryptd_do_crypt+0x0/0x253 kernel: [<ffffffff8023fbb5>] run_workqueue+0x7f/0x10b kernel: [<ffffffff80240557>] worker_thread+0x90/0x9b kernel: [<ffffffff802434cc>] autoremove_wake_function+0x0/0x2e kernel: [<ffffffff802404c7>] worker_thread+0x0/0x9b kernel: [<ffffffff802433a2>] kthread+0x47/0x75 kernel: [<ffffffff8020cc58>] child_rip+0xa/0x12 kernel: [<ffffffff8024335b>] kthread+0x0/0x75 kernel: [<ffffffff8020cc4e>] child_rip+0x0/0x12 kernel: kernel: Mem-info: kernel: Node 0 DMA per-cpu: kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 kernel: Node 0 DMA32 per-cpu: kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 60 kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 83 Cold: hi: 62, btch: 15 usd: 36 kernel: Node 1 DMA32 per-cpu: kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 76 Cold: hi: 62, btch: 15 usd: 8 kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 170 Cold: hi: 62, btch: 15 usd: 50 kernel: Node 1 Normal per-cpu: kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 185 Cold: hi: 62, btch: 15 usd: 10 kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 47 kernel: Active:74680 inactive:1373896 dirty:83344 writeback:10472 unstable:0 kernel: free:4303 slab:42709 mapped:7248 pagetables:1906 bounce:0 kernel: Node 0 DMA free:8024kB min:12kB low:12kB high:16kB active:0kB inactive:4308kB present:9096kB pages_scanned:0 all_unreclaimable? no kernel: lowmem_reserve[]: 0 2004 2004 2004 kernel: Node 0 DMA32 free:1684kB min:3332kB low:4164kB high:4996kB active:35424kB inactive:1931932kB present:2052320kB pages_scanned:0 all_unreclaimable? no kernel: lowmem_reserve[]: 0 0 0 0 kernel: Node 1 DMA32 free:6320kB min:3144kB low:3928kB high:4716kB active:15392kB inactive:1839036kB present:1937344kB pages_scanned:0 all_unreclaimable? no kernel: lowmem_reserve[]: 0 0 2020 2020 kernel: Node 1 Normal free:1700kB min:3356kB low:4192kB high:5032kB active:247904kB inactive:1720308kB present:2068480kB pages_scanned:0 all_unreclaimable? no kernel: lowmem_reserve[]: 0 0 0 0 kernel: Node 0 DMA: 25*4kB 7*8kB 12*16kB 8*32kB 4*64kB 2*128kB 3*256kB 2*512kB 1*1024kB 2*2048kB 0*4096kB = 8028kB kernel: Node 0 DMA32: 279*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1756kB kernel: Node 1 DMA32: 52*4kB 185*8kB 45*16kB 10*32kB 6*64kB 6*128kB 2*256kB 2*512kB 1*1024kB 0*2048kB 0*4096kB = 6440kB kernel: Node 1 Normal: 31*4kB 30*8kB 7*16kB 11*32kB 8*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2620kB kernel: Swap cache: add 39, delete 39, find 0/0, race 0+0 kernel: Free swap = 1048416kB kernel: Total swap = 1048568kB kernel: Free swap: 1048416kB kernel: 1540096 pages of RAM kernel: 22480 reserved pages kernel: 911826 pages shared kernel: 0 pages swap cached
Created attachment 14494 [details] Kernel config
altanative easy reproduce method: % ./hackbench 105 process 1000 this parameter mean consume all physical memory, but no use any swap. (please adjust your test environment) I guess the cause of bug is current linux VM can't memory reclaim under interrupt context. thus, when interrupt happend under memory reclaim running, it cause kmalloc(GFP_ATOMIC) failure. I working on fixed it now. but it need a bit long time...
Hi Kosaki, thanks for the report and your effort. Please keep us updates on your work.
Is this still an issue with a recent kernel?