Bug 6688
Summary: | Memory allocation problem | ||
---|---|---|---|
Product: | Memory Management | Reporter: | Pawel Staszewski (pstaszewski) |
Component: | Other | Assignee: | Andrew Morton (akpm) |
Status: | CLOSED PATCH_ALREADY_AVAILABLE | ||
Severity: | normal | CC: | bunk, protasnb |
Priority: | P2 | ||
Hardware: | i386 | ||
OS: | Linux | ||
Kernel Version: | 2.6.16 | Subsystem: | |
Regression: | --- | Bisected commit-id: | |
Attachments: | slab-leak-detector.patch |
Description
Pawel Staszewski
2006-06-14 01:46:43 UTC
Looks like a slab leak. Can you please include the contents of /proc/slabinfo when the problem is occurring, so we can see which slab cache is causing the problem? cat /proc/slabinfo slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> ip_fib_alias 187280 188269 64 59 1 : tunables 120 60 8 : slabdata 3191 3191 3 UNIX 13 18 640 6 1 : tunables 54 27 8 : slabdata 3 3 0 ipt_hashlimit 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 scsi_cmd_cache 4 14 512 7 1 : tunables 54 27 8 : slabdata 2 2 0 msi_cache 1 1 5760 1 2 : tunables 8 4 0 : slabdata 1 1 0 cfq_ioc_pool 0 0 168 23 1 : tunables 120 60 8 : slabdata 0 0 0 cfq_pool 0 0 160 24 1 : tunables 120 60 8 : slabdata 0 0 0 crq_pool 0 0 88 44 1 : tunables 120 60 8 : slabdata 0 0 0 deadline_drq 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0 as_arq 4 34 112 34 1 : tunables 120 60 8 : slabdata 1 1 0 mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 : slabdata 1 1 0 isofs_inode_cache 0 0 616 6 1 : tunables 54 27 8 : slabdata 0 0 0 hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 27 8 : slabdata 1 1 0 ext2_inode_cache 0 0 680 6 1 : tunables 54 27 8 : slabdata 0 0 0 reiser_inode_cache 10077 10095 688 5 1 : tunables 54 27 8 : slabdata 2019 2019 0 dnotify_cache 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 eventpoll_pwq 0 0 72 53 1 : tunables 120 60 8 : slabdata 0 0 0 eventpoll_epi 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 inotify_event_cache 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 inotify_watch_cache 1 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0 kioctx 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 fasync_cache 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 shmem_inode_cache 5271 5275 760 5 1 : tunables 54 27 8 : slabdata 1055 1055 0 posix_timers_cache 0 0 136 28 1 : tunables 120 60 8 : slabdata 0 0 0 uid_cache 1 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0 tcp_bind_bucket 5 112 32 112 1 : tunables 120 60 8 : slabdata 1 1 0 inet_peer_cache 176 360 128 30 1 : tunables 120 60 8 : slabdata 12 12 0 ip_dst_cache 15680 15680 384 10 1 : tunables 54 27 8 : slabdata 1568 1568 0 arp_cache 3 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 RAW 3 5 768 5 1 : tunables 54 27 8 : slabdata 1 1 0 UDP 1 5 768 5 1 : tunables 54 27 8 : slabdata 1 1 0 tw_sock_TCP 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 TCP 7 10 1536 5 2 : tunables 24 12 8 : slabdata 2 2 0 sgpool-128 32 32 4096 1 1 : tunables 24 12 8 : slabdata 32 32 0 sgpool-64 32 32 2048 2 1 : tunables 24 12 8 : slabdata 16 16 0 sgpool-32 32 32 1024 4 1 : tunables 54 27 8 : slabdata 8 8 0 sgpool-16 32 32 512 8 1 : tunables 54 27 8 : slabdata 4 4 0 sgpool-8 32 45 256 15 1 : tunables 120 60 8 : slabdata 3 3 0 scsi_io_context 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 blkdev_ioc 13 67 56 67 1 : tunables 120 60 8 : slabdata 1 1 0 blkdev_queue 25 30 1624 5 2 : tunables 24 12 8 : slabdata 6 6 0 blkdev_requests 4 13 288 13 1 : tunables 54 27 8 : slabdata 1 1 0 biovec-(256) 7 7 4096 1 1 : tunables 24 12 8 : slabdata 7 7 0 biovec-128 7 8 2048 2 1 : tunables 24 12 8 : slabdata 4 4 0 biovec-64 7 8 1024 4 1 : tunables 54 27 8 : slabdata 2 2 0 biovec-16 7 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 biovec-4 7 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0 biovec-1 7 202 16 202 1 : tunables 120 60 8 : slabdata 1 1 0 bio 256 270 128 30 1 : tunables 120 60 8 : slabdata 9 9 0 sock_inode_cache 32 40 768 5 1 : tunables 54 27 8 : slabdata 8 8 0 skbuff_fclone_cache 63 63 512 7 1 : tunables 54 27 8 : slabdata 9 9 0 skbuff_head_cache 315 495 256 15 1 : tunables 120 60 8 : slabdata 33 33 0 file_lock_cache 2 22 176 22 1 : tunables 120 60 8 : slabdata 1 1 0 acpi_operand 1060 1113 72 53 1 : tunables 120 60 8 : slabdata 21 21 0 acpi_parse_ext 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 acpi_parse 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 acpi_state 0 0 88 44 1 : tunables 120 60 8 : slabdata 0 0 0 proc_inode_cache 280 288 600 6 1 : tunables 54 27 8 : slabdata 48 48 0 sigqueue 6 24 160 24 1 : tunables 120 60 8 : slabdata 1 1 0 radix_tree_node 9261 10451 536 7 1 : tunables 54 27 8 : slabdata 1493 1493 0 bdev_cache 5 5 768 5 1 : tunables 54 27 8 : slabdata 1 1 0 sysfs_dir_cache 3235 3312 80 48 1 : tunables 120 60 8 : slabdata 69 69 0 mnt_cache 19 45 256 15 1 : tunables 120 60 8 : slabdata 3 3 0 inode_cache 990 1001 568 7 1 : tunables 54 27 8 : slabdata 143 143 0 dentry_cache 18405 18620 208 19 1 : tunables 120 60 8 : slabdata 980 980 0 filp 258 630 256 15 1 : tunables 120 60 8 : slabdata 42 42 0 names_cache 1 1 4096 1 1 : tunables 24 12 8 : slabdata 1 1 0 idr_layer_cache 72 77 528 7 1 : tunables 54 27 8 : slabdata 11 11 0 buffer_head 41643 41680 96 40 1 : tunables 120 60 8 : slabdata 1042 1042 0 mm_struct 22 22 4096 1 1 : tunables 24 12 8 : slabdata 22 22 0 vm_area_struct 861 1122 176 22 1 : tunables 120 60 8 : slabdata 51 51 0 fs_cache 33 118 64 59 1 : tunables 120 60 8 : slabdata 2 2 0 files_cache 34 60 768 5 1 : tunables 54 27 8 : slabdata 12 12 0 signal_cache 63 84 640 6 1 : tunables 54 27 8 : slabdata 14 14 0 sighand_cache 54 54 2112 3 2 : tunables 24 12 8 : slabdata 18 18 0 task_struct 48 48 4096 1 1 : tunables 24 12 8 : slabdata 48 48 0 anon_vma 282 432 24 144 1 : tunables 120 60 8 : slabdata 3 3 0 pid 64 531 64 59 1 : tunables 120 60 8 : slabdata 9 9 0 size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0 size-65536 1 1 65536 1 16 : tunables 8 4 0 : slabdata 1 1 0 size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-32768 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0 size-16384 2 2 16384 1 4 : tunables 8 4 0 : slabdata 2 2 0 size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 size-8192 20 20 8192 1 2 : tunables 8 4 0 : slabdata 20 20 0 size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0 size-4096 462 468 4096 1 1 : tunables 24 12 8 : slabdata 462 468 0 size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0 size-2048 277 304 2048 2 1 : tunables 24 12 8 : slabdata 152 152 0 size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 size-1024 798 1016 1024 4 1 : tunables 54 27 8 : slabdata 254 254 0 size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0 size-512 1701 3240 512 8 1 : tunables 54 27 8 : slabdata 405 405 1 size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 size-256 4942 7575 256 15 1 : tunables 120 60 8 : slabdata 505 505 0 size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 size-64 393142 395064 64 59 1 : tunables 120 60 8 : slabdata 6696 6696 0 size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 size-128 16677 35400 128 30 1 : tunables 120 60 8 : slabdata 1180 1180 0 size-32 498 3472 32 112 1 : tunables 120 60 8 : slabdata 31 31 0 kmem_cache 106 110 384 10 1 : tunables 54 27 8 : slabdata 11 11 0 Call trace from another machine : Call Trace: <IRQ> <ffffffff8024abf7>{__alloc_pages+623} <ffffffff802145d4>{smp_local_timer_interrupt+35} <ffffffff80214639>{smp_apic_timer_interrupt+65} <ffffffff802612f2>{cache_alloc_refill+695} <ffffffff802616d7>{__kmalloc+114} <ffffffff8038d148>{__alloc_skb+85} <ffffffff8034123e>{e1000_clean_rx_irq+740} <ffffffff80340e94>{e1000_clean+239} <ffffffff80392be2>{net_rx_action+171} <ffffffff8022d553>{__do_softirq+103} <ffffffff8020aaa6>{call_softirq+30} <ffffffff8020c678>{do_softirq+61} <ffffffff8020c631>{do_IRQ+62} <ffffffff80207e94>{mwait_idle+0} <ffffffff80209e04>{ret_from_intr+0} <EOI> <ffffffff80207ede>{mwait_idle+74} <ffffffff80207e75>{cpu_idle+98} <ffffffff805da79b>{start_secondary+1127} Mem-info: DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:72 cpu 0 cold: high 62, batch 15 used:13 cpu 1 hot: high 186, batch 31 used:30 cpu 1 cold: high 62, batch 15 used:50 Normal per-cpu: empty HighMem per-cpu: empty Free pages: 4920kB (0kB HighMem) Active:48439 inactive:4651 dirty:0 writeback:0 unstable:0 free:1230 slab:68845 mapped:39985 pagetables:255 DMA free:1992kB min:172kB low:212kB high:256kB active:0kB inactive:0kB present:10604kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 481 481 481 DMA32 free:2928kB min:8016kB low:10020kB high:12024kB active:193756kB inactive:18604kB present:492776kB pages_scanned:126 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1992kB DMA32: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2928kB Normal: empty HighMem: empty Swap cache: add 0, delete 0, find 0/0, race 0+0 Free swap = 996020kB Total swap = 996020kB Free swap: 996020kB 128997 pages of RAM 3326 reserved pages 13499 pages shared 0 pages swap cached swapper: page allocation failure. order:0, mode:0x20 Call Trace: <IRQ> <ffffffff8024abf7>{__alloc_pages+623} <ffffffff802612f2>{cache_alloc_refill+695} <ffffffff802616d7>{__kmalloc+114} <ffffffff8038d148>{__alloc_skb+85} <ffffffff8034123e>{e1000_clean_rx_irq+740} <ffffffff80340e94>{e1000_clean+239} <ffffffff80392be2>{net_rx_action+171} <ffffffff8022d553>{__do_softirq+103} <ffffffff8020aaa6>{call_softirq+30} <ffffffff8020c678>{do_softirq+61} <ffffffff8020c631>{do_IRQ+62} <ffffffff80207e94>{mwait_idle+0} <ffffffff80209e04>{ret_from_intr+0} <EOI> <ffffffff80207ede>{mwait_idle+74} <ffffffff80207e75>{cpu_idle+98} <ffffffff805da79b>{start_secondary+1127} Mem-info: DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:72 cpu 0 cold: high 62, batch 15 used:13 cpu 1 hot: high 186, batch 31 used:30 cpu 1 cold: high 62, batch 15 used:50 Normal per-cpu: empty HighMem per-cpu: empty Free pages: 4920kB (0kB HighMem) Active:48439 inactive:4651 dirty:0 writeback:0 unstable:0 free:1230 slab:68845 mapped:39985 pagetables:255 DMA free:1992kB min:172kB low:212kB high:256kB active:0kB inactive:0kB present:10604kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 481 481 481 DMA32 free:2928kB min:8016kB low:10020kB high:12024kB active:193756kB inactive:18604kB present:492776kB pages_scanned:126 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1992kB DMA32: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2928kB Normal: empty HighMem: empty Swap cache: add 0, delete 0, find 0/0, race 0+0 Free swap = 996020kB Total swap = 996020kB Free swap: 996020kB 128997 pages of RAM 3326 reserved pages 13499 pages shared 0 pages swap cached printk: 21353 messages suppressed. swapper: page allocation failure. order:0, mode:0x20 Call Trace: <IRQ> <ffffffff8024abf7>{__alloc_pages+623} <ffffffff802612f2>{cache_alloc_refill+695} <ffffffff802616d7>{__kmalloc+114} <ffffffff8038d148>{__alloc_skb+85} <ffffffff80341c62>{e1000_alloc_rx_buffers+119} <ffffffff803414ef>{e1000_clean_rx_irq+1429} <ffffffff80340e94>{e1000_clean+239} <ffffffff80392be2>{net_rx_action+171} <ffffffff803166dd>{add_timer_randomness+284} <ffffffff8022d553>{__do_softirq+103} <ffffffff8020aaa6>{call_softirq+30} <ffffffff8020c678>{do_softirq+61} <ffffffff8020c631>{do_IRQ+62} <ffffffff80207e94>{mwait_idle+0} <ffffffff80209e04>{ret_from_intr+0} <EOI> <ffffffff80207ede>{mwait_idle+74} <ffffffff80207e75>{cpu_idle+98} <ffffffff805da79b>{start_secondary+1127} Mem-info: DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:129 cpu 0 cold: high 62, batch 15 used:57 cpu 1 hot: high 186, batch 31 used:30 cpu 1 cold: high 62, batch 15 used:60 Normal per-cpu: empty HighMem per-cpu: empty Free pages: 4976kB (0kB HighMem) Active:40234 inactive:1675 dirty:8 writeback:0 unstable:0 free:1244 slab:79910 mapped:39985 pagetables:255 DMA free:1992kB min:172kB low:212kB high:256kB active:0kB inactive:0kB present:10604kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 481 481 481 DMA32 free:2984kB min:8016kB low:10020kB high:12024kB active:160936kB inactive:6700kB present:492776kB pages_scanned:565 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1992kB DMA32: 0*4kB 1*8kB 2*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2984kB Normal: empty HighMem: empty Swap cache: add 38, delete 28, find 0/0, race 0+0 Free swap = 995868kB Total swap = 996020kB Free swap: 995868kB 128997 pages of RAM 3326 reserved pages 5576 pages shared 10 pages swap cached OK, so a third of that memory has disappeared into the ip_fib_alias cache. But two thirds is in size-64, so we don't know who is using that. Before we bug the net guys would you be able to apply slab-leak-detector.patch, set CONFIG_DEBUG_SLAB=y then, when the leak is happening, do echo "size-64 0 0 0" > /proc/slabinfo and then gather the dmesg output? Created attachment 8346 [details]
slab-leak-detector.patch
Hi Pawel, Have you been able to test with Andrew's path? Let me know if you need help applying the patch and setting up the test kernel on your systems. Thanks, --Natalie sorry that my late :) In 2.6.19 problem has gone :) |