Created attachment 47442 [details] dmesg 2.6.38-rc4 [274762.344172] kswapd0: page allocation failure. order:1, mode:0x4020 [274762.344178] Pid: 26, comm: kswapd0 Not tainted 2.6.38-020638rc4-generic #201102081010 [274762.344180] Call Trace: [274762.344183] <IRQ> [<ffffffff8110f213>] ? __alloc_pages_slowpath+0x553/0x720 [274762.344196] [<ffffffff8110beff>] ? zone_watermark_ok+0x1f/0x30 [274762.344201] [<ffffffff8110f573>] ? __alloc_pages_nodemask+0x193/0x1d0 [274762.344205] [<ffffffff815a8884>] ? _raw_spin_lock_irqsave+0x34/0x50 [274762.344210] [<ffffffff81144a93>] ? alloc_pages_current+0xa3/0x110 [274762.344216] [<ffffffff8114e245>] ? allocate_slab+0x175/0x180 [274762.344219] [<ffffffff8114e840>] ? new_slab+0x30/0x170 [274762.344223] [<ffffffff8114fdb9>] ? __slab_alloc+0x1e9/0x250 [274762.344235] [<ffffffffa01090c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [274762.344239] [<ffffffff81151428>] ? __kmalloc_node_track_caller+0x188/0x1e0 [274762.344244] [<ffffffffa01090c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [274762.344248] [<ffffffff814ab22f>] ? __alloc_skb+0x7f/0x170 [274762.344252] [<ffffffffa01090c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [274762.344260] [<ffffffffa022e884>] ? ath_rx_tasklet+0x294/0x600 [ath9k] [274762.344266] [<ffffffffa022b2c8>] ? ath9k_tasklet+0xb8/0x180 [ath9k] [274762.344270] [<ffffffff8106b3ba>] ? tasklet_action+0x6a/0x110 [274762.344273] [<ffffffff8106ad40>] ? __do_softirq+0xb0/0x1c0 [274762.344278] [<ffffffff8102fcfa>] ? ack_apic_level+0x7a/0xf0 [274762.344282] [<ffffffff8100cf1c>] ? call_softirq+0x1c/0x30 [274762.344285] [<ffffffff8100e875>] ? do_softirq+0x65/0xa0 [274762.344288] [<ffffffff8106aed5>] ? irq_exit+0x85/0x90 [274762.344292] [<ffffffff815b0cf9>] ? do_IRQ+0x69/0xe0 [274762.344295] [<ffffffff815a8f13>] ? ret_from_intr+0x0/0x15 [274762.344297] <EOI> [<ffffffff815a88d9>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [274762.344303] [<ffffffff811484c1>] ? isolate_freepages+0x111/0x1a0 [274762.344306] [<ffffffff81148680>] ? compaction_alloc+0x60/0x70 [274762.344309] [<ffffffff811534c8>] ? unmap_and_move+0x48/0x2f0 [274762.344312] [<ffffffff81153815>] ? migrate_pages+0xa5/0x140 [274762.344315] [<ffffffff81148620>] ? compaction_alloc+0x0/0x70 [274762.344318] [<ffffffff81148cc0>] ? compact_zone+0xa0/0x150 [274762.344321] [<ffffffff81148fca>] ? compact_zone_order+0xca/0x100 [274762.344326] [<ffffffff8111a05b>] ? balance_pgdat+0x4eb/0x5d0 [274762.344330] [<ffffffff8111a26b>] ? kswapd+0x12b/0x1a0 [274762.344333] [<ffffffff8111a140>] ? kswapd+0x0/0x1a0 [274762.344337] [<ffffffff810850e7>] ? kthread+0x97/0xa0 [274762.344341] [<ffffffff8100ce24>] ? kernel_thread_helper+0x4/0x10 [274762.344344] [<ffffffff81085050>] ? kthread+0x0/0xa0 [274762.344347] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10 [274762.344349] Mem-Info: [274762.344351] Node 0 DMA per-cpu: [274762.344354] CPU 0: hi: 0, btch: 1 usd: 0 [274762.344357] CPU 1: hi: 0, btch: 1 usd: 0 [274762.344358] Node 0 DMA32 per-cpu: [274762.344361] CPU 0: hi: 186, btch: 31 usd: 180 [274762.344363] CPU 1: hi: 186, btch: 31 usd: 181 [274762.344368] active_anon:334552 inactive_anon:83841 isolated_anon:20 [274762.344369] active_file:57732 inactive_file:58043 isolated_file:0 [274762.344370] unevictable:42 dirty:2414 writeback:0 unstable:0 [274762.344371] free:8641 slab_reclaimable:10621 slab_unreclaimable:9673 [274762.344372] mapped:170461 shmem:30731 pagetables:12009 bounce:0 [274762.344375] Node 0 DMA free:11556kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:768kB inactive_file:1976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15668kB mlocked:0kB dirty:0kB writeback:0kB mapped:1204kB shmem:0kB slab_reclaimable:68kB slab_unreclaimable:312kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [274762.344384] lowmem_reserve[]: 0 2883 2883 2883 [274762.344389] Node 0 DMA32 free:23008kB min:6848kB low:8560kB high:10272kB active_anon:1338208kB inactive_anon:335364kB active_file:230160kB inactive_file:230196kB unevictable:168kB isolated(anon):80kB isolated(file):0kB present:2952660kB mlocked:168kB dirty:9656kB writeback:0kB mapped:680640kB shmem:122924kB slab_reclaimable:42416kB slab_unreclaimable:38380kB kernel_stack:3480kB pagetables:48032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [274762.344399] lowmem_reserve[]: 0 0 0 0 [274762.344403] Node 0 DMA: 3*4kB 3*8kB 4*16kB 6*32kB 4*64kB 4*128kB 1*256kB 2*512kB 3*1024kB 3*2048kB 0*4096kB = 11556kB [274762.344413] Node 0 DMA32: 5438*4kB 8*8kB 14*16kB 0*32kB 1*64kB 1*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 23000kB [274762.344423] 165982 total pagecache pages [274762.344425] 19473 pages in swap cache [274762.344427] Swap cache stats: add 650825, delete 631352, find 1249740/1303578 [274762.344429] Free swap = 3180904kB [274762.344431] Total swap = 3574424kB [274762.350609] 752624 pages RAM [274762.350609] 153594 pages reserved [274762.350609] 275668 pages shared [274762.350609] 491506 pages non-shared [274762.350609] SLUB: Unable to allocate memory on node -1 (gfp=0x20) [274762.350609] cache: kmalloc-8192, object size: 8192, buffer size: 8192, default order: 3, min order: 1 [274762.350609] node 0: slabs: 173, objs: 677, free: 0 [274762.358458] skbuff alloc of size 3904 failed
Created attachment 47452 [details] lspci -vvnn
As long we do not have memory leak in ath9k, this is rather memory allocator problem not the driver. Can you reproduce that particular allocation failure? Is this regression?
Ok, I see you can reproduce, 28632 is duplicate of that bug. How frequently this happens? Any steps to reproduce?
(In reply to comment #3) > Ok, I see you can reproduce, 28632 is duplicate of that bug. How frequently > this happens? Any steps to reproduce? +Cristian in addtion please get the debug messages by sudo modprobe ath9k debug=0xe03(ATH_DBG_PS | ATH_DBG_FATAL | ATH_DBG_CONFIG | ATH_DBG_QUEUE | ATH_DBG_RESET)
#LANGUAGE=C modprobe ath9k debug=0xe03(ATH_DBG_PS | ATH_DBG_FATAL | ATH_DBG_CONFIG | ATH_DBG_QUEUE | ATH_DBG_RESET) bash: error sintáctico cerca del elemento inesperado `(' root@VGN-NS130FE:/home/caravena#
I think it is just "modprobe ath9k debug=0xe03"...
# modprobe ath9k debug=0xe03 # Is this OK?
You can try the patch from Felix for this. ath9k: use split rx buffers to get rid of order-1 skb allocations With this change, less CPU time is spent trying to look for consecutive pages for rx skbs. This also reduces the socket memory required for IP/UDP reassembly. Only two buffers per frame are supported. Frames spanning more buffers will be dropped, but the buffer size is enough to handle the required AMSDU size. Signed-off-by: Jouni Malinen <jouni.malinen@atheros.com> Signed-off-by: Felix Fietkau <nbd@openwrt.org> Signed-off-by: John W. Linville <linville@tuxdriver.com>
Created attachment 47772 [details] ath9k: use split rx buffers to get rid of order-1 skb allocations This may require a manual backport
(In reply to comment #6) > I think it is just "modprobe ath9k debug=0xe03"... Yes. Cristian this the right way.
(In reply to comment #9) > Created an attachment (id=47772) [details] > ath9k: use split rx buffers to get rid of order-1 skb allocations > > This may require a manual backport Luis thanks!
Created attachment 48232 [details] dmesg 2.6.38-rc5 Good call trace? I use 'sudo modprobe ath9k debug=0xe03' dmesg 2.6.38-rc5 [84876.326900] chromium-browse: page allocation failure. order:1, mode:0x4020 [84876.326906] Pid: 32696, comm: chromium-browse Not tainted 2.6.38-020638rc5-generic #201102160907 [84876.326909] Call Trace: [84876.326912] <IRQ> [<ffffffff8110f213>] ? __alloc_pages_slowpath+0x553/0x720 [84876.326928] [<ffffffff8110beff>] ? zone_watermark_ok+0x1f/0x30 [84876.326933] [<ffffffff8110f573>] ? __alloc_pages_nodemask+0x193/0x1d0 [84876.326940] [<ffffffff815acdb4>] ? _raw_spin_lock_irqsave+0x34/0x50 [84876.326947] [<ffffffff81144a03>] ? alloc_pages_current+0xa3/0x110 [84876.326953] [<ffffffff8114e1b5>] ? allocate_slab+0x175/0x180 [84876.326957] [<ffffffff8114e7b0>] ? new_slab+0x30/0x170 [84876.326961] [<ffffffff8114fd29>] ? __slab_alloc+0x1e9/0x250 [84876.326978] [<ffffffffa011c0c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [84876.326983] [<ffffffff81151398>] ? __kmalloc_node_track_caller+0x188/0x1e0 [84876.326987] [<ffffffffa011c0c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [84876.326992] [<ffffffff814af7cf>] ? __alloc_skb+0x7f/0x170 [84876.326996] [<ffffffffa011c0c9>] ? ath_rxbuf_alloc+0x39/0xc0 [ath] [84876.327008] [<ffffffffa0217874>] ? ath_rx_tasklet+0x294/0x600 [ath9k] [84876.327015] [<ffffffffa02142b8>] ? ath9k_tasklet+0xb8/0x180 [ath9k] [84876.327021] [<ffffffff8106b3ba>] ? tasklet_action+0x6a/0x110 [84876.327024] [<ffffffff8106ad40>] ? __do_softirq+0xb0/0x1c0 [84876.327031] [<ffffffff8102fdda>] ? ack_apic_level+0x7a/0xf0 [84876.327036] [<ffffffff8100cf1c>] ? call_softirq+0x1c/0x30 [84876.327040] [<ffffffff8100e885>] ? do_softirq+0x65/0xa0 [84876.327043] [<ffffffff8106aed5>] ? irq_exit+0x85/0x90 [84876.327048] [<ffffffff815b5239>] ? do_IRQ+0x69/0xe0 [84876.327052] [<ffffffff815ad453>] ? ret_from_intr+0x0/0x15 [84876.327054] <EOI> [<ffffffff8110e070>] ? get_page_from_freelist+0x1c0/0x440 [84876.327061] [<ffffffff8110cf42>] ? prep_new_page+0xe2/0x150 [84876.327065] [<ffffffff8110f4e3>] ? __alloc_pages_nodemask+0x103/0x1d0 [84876.327069] [<ffffffff81145834>] ? alloc_pages_vma+0x94/0x170 [84876.327076] [<ffffffff81128142>] ? do_anonymous_page+0x172/0x310 [84876.327079] [<ffffffff8112d3f1>] ? handle_pte_fault+0x191/0x1a0 [84876.327082] [<ffffffff81110c90>] ? __free_pages+0x20/0x30 [84876.327085] [<ffffffff8112d52f>] ? handle_mm_fault+0x12f/0x1a0 [84876.327089] [<ffffffff815b0c6c>] ? do_page_fault+0x1dc/0x520 [84876.327094] [<ffffffff8113255f>] ? do_mmap_pgoff+0x31f/0x350 [84876.327097] [<ffffffff811326f1>] ? sys_mmap_pgoff+0x161/0x220 [84876.327101] [<ffffffff815ad715>] ? page_fault+0x25/0x30 [84876.327103] Mem-Info: [84876.327106] Node 0 DMA per-cpu: [84876.327109] CPU 0: hi: 0, btch: 1 usd: 0 [84876.327111] CPU 1: hi: 0, btch: 1 usd: 0 [84876.327113] Node 0 DMA32 per-cpu: [84876.327117] CPU 0: hi: 186, btch: 31 usd: 38 [84876.327119] CPU 1: hi: 186, btch: 31 usd: 163 [84876.327125] active_anon:147613 inactive_anon:60500 isolated_anon:29 [84876.327126] active_file:185389 inactive_file:246775 isolated_file:0 [84876.327127] unevictable:30 dirty:1401 writeback:0 unstable:0 [84876.327128] free:39971 slab_reclaimable:17692 slab_unreclaimable:8745 [84876.327129] mapped:27424 shmem:19783 pagetables:9724 bounce:0 [84876.327131] Node 0 DMA free:11560kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:736kB inactive_file:3400kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15668kB mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:0kB slab_reclaimable:172kB slab_unreclaimable:24kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [84876.327141] lowmem_reserve[]: 0 2883 2883 2883 [84876.327146] Node 0 DMA32 free:148324kB min:6848kB low:8560kB high:10272kB active_anon:590452kB inactive_anon:242000kB active_file:740820kB inactive_file:983700kB unevictable:120kB isolated(anon):116kB isolated(file):0kB present:2952660kB mlocked:120kB dirty:5604kB writeback:0kB mapped:109616kB shmem:79132kB slab_reclaimable:70596kB slab_unreclaimable:34956kB kernel_stack:2984kB pagetables:38896kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [84876.327157] lowmem_reserve[]: 0 0 0 0 [84876.327160] Node 0 DMA: 10*4kB 4*8kB 4*16kB 3*32kB 3*64kB 1*128kB 5*256kB 1*512kB 3*1024kB 1*2048kB 1*4096kB = 11560kB [84876.327172] Node 0 DMA32: 36786*4kB 15*8kB 16*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 148352kB [84876.330017] 464819 total pagecache pages [84876.330017] 12848 pages in swap cache [84876.330017] Swap cache stats: add 134790, delete 121942, find 371960/378388 [84876.330017] Free swap = 3305632kB [84876.330017] Total swap = 3574424kB [84876.330017] 752624 pages RAM [84876.330017] 13833 pages reserved [84876.330017] 362506 pages shared [84876.330017] 481825 pages non-shared [84876.330017] SLUB: Unable to allocate memory on node -1 (gfp=0x20) [84876.330017] cache: kmalloc-8192, object size: 8192, buffer size: 8192, default order: 3, min order: 1 [84876.330017] node 0: slabs: 182, objs: 707, free: 0 [84876.355883] skbuff alloc of size 3904 failed
Created attachment 48242 [details] config kernel 2.6.38-rc5 I not full debug with 'modprobe ath9k debug=0xe03' ? CONFIG_ATH9K_HW=m CONFIG_ATH9K_COMMON=m CONFIG_ATH9K=m CONFIG_ATH9K_DEBUGFS=y CONFIG_ATH9K_RATE_CONTROL=y CONFIG_ATH9K_HTC=m CONFIG_ATH9K_HTC_DEBUGFS=y
(In reply to comment #13) > Created an attachment (id=48242) [details] > config kernel 2.6.38-rc5 > > I not full debug with 'modprobe ath9k debug=0xe03' ? > > CONFIG_ATH9K_HW=m > CONFIG_ATH9K_COMMON=m > CONFIG_ATH9K=m > CONFIG_ATH9K_DEBUGFS=y > CONFIG_ATH9K_RATE_CONTROL=y > CONFIG_ATH9K_HTC=m > CONFIG_ATH9K_HTC_DEBUGFS=y please make sure these are also there : CONFIG_ATH_COMMON=m CONFIG_ATH_DEBUG=y first of all please try with the patch Luis had suggested.
(In reply to comment #13) > Created an attachment (id=48242) [details] > config kernel 2.6.38-rc5 > > I not full debug with 'modprobe ath9k debug=0xe03' ? > > CONFIG_ATH9K_HW=m > CONFIG_ATH9K_COMMON=m > CONFIG_ATH9K=m > CONFIG_ATH9K_DEBUGFS=y > CONFIG_ATH9K_RATE_CONTROL=y > CONFIG_ATH9K_HTC=m > CONFIG_ATH9K_HTC_DEBUGFS=y from your kernel log its evident CONFIG_ATH_COMMON=m # CONFIG_ATH_DEBUG is not set please set CONFIG_ATH_DEBUG
On Fri, Feb 18, 2011 at 12:22 PM, <bugzilla-daemon@bugzilla.kernel.org> wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=28922 [...] > --- Comment #14 from shafi <shafi.wireless@gmail.com> 2011-02-18 15:21:57 > --- > (In reply to comment #13) >> Created an attachment (id=48242) > --> (https://bugzilla.kernel.org/attachment.cgi?id=48242) [details] >> config kernel 2.6.38-rc5 >> >> I not full debug with 'modprobe ath9k debug=0xe03' ? >> >> CONFIG_ATH9K_HW=m >> CONFIG_ATH9K_COMMON=m >> CONFIG_ATH9K=m >> CONFIG_ATH9K_DEBUGFS=y >> CONFIG_ATH9K_RATE_CONTROL=y >> CONFIG_ATH9K_HTC=m >> CONFIG_ATH9K_HTC_DEBUGFS=y > > please make sure these are also there : > > CONFIG_ATH_COMMON=m > CONFIG_ATH_DEBUG=y > > first of all please try with the patch Luis had suggested. Kernel source? 2.6.38-rc5? good? /usr/src#ln -s 2.6.38-rc5 linux /usr/src#patch -p0 < alloc.patch
(In reply to comment #16) > On Fri, Feb 18, 2011 at 12:22 PM, <bugzilla-daemon@bugzilla.kernel.org> > wrote: > > https://bugzilla.kernel.org/show_bug.cgi?id=28922 > > [...] > > > > --- Comment #14 from shafi <shafi.wireless@gmail.com> 2011-02-18 15:21:57 > --- > > (In reply to comment #13) > >> Created an attachment (id=48242) [details] > > --> (https://bugzilla.kernel.org/attachment.cgi?id=48242) [details] > >> config kernel 2.6.38-rc5 > >> > >> I not full debug with 'modprobe ath9k debug=0xe03' ? > >> > >> CONFIG_ATH9K_HW=m > >> CONFIG_ATH9K_COMMON=m > >> CONFIG_ATH9K=m > >> CONFIG_ATH9K_DEBUGFS=y > >> CONFIG_ATH9K_RATE_CONTROL=y > >> CONFIG_ATH9K_HTC=m > >> CONFIG_ATH9K_HTC_DEBUGFS=y > > > > please make sure these are also there : > > > > CONFIG_ATH_COMMON=m > > CONFIG_ATH_DEBUG=y > > > > first of all please try with the patch Luis had suggested. > > Kernel source? 2.6.38-rc5? > > good? > /usr/src#ln -s 2.6.38-rc5 linux > /usr/src#patch -p0 < alloc.patch The patch is already merged in wireless-testing on jan 26 and 2.6.38-rc5 does have this patch, are you still seeing this issue with latest wireless-testing?
Created attachment 49142 [details] dmesg 2.6.37.2
*** Bug 26652 has been marked as a duplicate of this bug. ***
*** Bug 28632 has been marked as a duplicate of this bug. ***
Created attachment 49162 [details] dmesg 2.6.38-rc6
Created attachment 49962 [details] dmesg 2.6.38-rc7
Patch from comment 9 is not 2.6.38, it's currently in wireless-testing tree scheduled for 2.6.39. You need to apply patch separately or compile wireless-testing. Do you have configured CONFIG_COMPACTION and/or CONFIG_TRANSPARENT_HUGEPAGE? I would try if disable these make any difference.
Created attachment 50042 [details] config-2.6.38-020638rc7-generic (In reply to comment #23) [...] > Do you have configured CONFIG_COMPACTION and/or CONFIG_TRANSPARENT_HUGEPAGE? > I > would try if disable these make any difference. Defualt: CONFIG_COMPACTION=y # CONFIG_TRANSPARENT_HUGEPAGE is not set
Created attachment 50342 [details] dmesg 2.6.38-rc8
Please try to recreate once 2.6.39-rc1 (or later) is available...
In kernel 2.6.39-rc1 -> problem with resolution of my monitor/laptop I hope version 2.6.39-rc2 for new testing
Have you been able to try a later 2.6.39-rc kernel?
Not problem in kernel 2.6.39-rc5; Thanks, close report please.