With a single 2Gb/4-core VM on a 10Gb/8-core host, the _host_ ran out of 32kb page blocks. I can't be certain that it was kvm because the OOM condition triggered a GP fault in the SLUB allocator. (The other suspect is nouveau.) I have attached the kernel log and a SysRq Show Memory dump of the machine shortly after boot without running kvm.
Created attachment 88441 [details] kernel log showing OOM condition
Created attachment 88451 [details] sysrq/show memory without kvm shortly after boot
Created attachment 88461 [details] sysrq/show memory with kvm running a single 2gb vm
Not sure why I picked kvm/nouveau for this OOM ;) Could be any number of apps over a 3.5 period...
Dec 4 09:38:25 thor kernel: [322576.251464] firefox: page allocation failure: order:4, mode:0x80c0d0 This is an order-4 allocation failure. This is due to kernel failure to allocate physically contiguous region of memory. Please reassign to "Memory Management" component.
This is filesystem/block or memory manager-related. I just built -next-20121204 and by the end of the build a 10gb machine had < 100mb of zone DMA32 and < 70mb zone NORMAL.
Created attachment 88561 [details] /proc/slabinfo
Created attachment 88571 [details] lsmod > modlist.txt
md_raid is gobbling up memory. Looks like buffer_heads aren't being released. Check this out after cloning a single 8gb VM image straight after boot: # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> buffer_head 411176 411181 432 37 4 : tunables 0 0 0 : slabdata 11113 11113 0
Please reassign to IO/Storage, subcomponent LVM/DM
Sorry, please disregard last comment. This is probably related to the kswapd/flusher mess going on right now. https://lkml.org/lkml/2012/11/27/486