Bug 50181
| Summary: | Memory usage doubles after more then 20 hours of uptime. | ||
|---|---|---|---|
| Product: | Memory Management | Reporter: | Milos Jakovljevic (sukijaki) |
| Component: | Other | Assignee: | Andrew Morton (akpm) |
| Status: | CLOSED CODE_FIX | ||
| Severity: | normal | CC: | alan, dave, florian, kastner.karl, mike, sukijaki |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 3.10 | Subsystem: | |
| Regression: | Yes | Bisected commit-id: | |
| Attachments: | kernel config file | ||
It is a little better in -rc5. It manages to stay in normal memory counsumtion for more then a day now. But this morning I saw it was 3.5GB memory used, just like that. Only applications that ware used durring the night are deluge and tvtime. (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Tue, 6 Nov 2012 15:11:48 +0000 (UTC) bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=50181 > > Summary: Memory usage doubles after more then 20 hours of > uptime. > Product: Memory Management > Version: 2.5 > Kernel Version: 3.7-rc3 and 3.7-rc4 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Other > AssignedTo: akpm@linux-foundation.org > ReportedBy: sukijaki@gmail.com > Regression: Yes > > > Created an attachment (id=85721) > --> (https://bugzilla.kernel.org/attachment.cgi?id=85721) > kernel config file > > After 20 hours of uptime, memory usage starts going up. Normal usage for my > system was around 2.5GB max with all my apps and services up and running. But > with 3.7-rc3 and now -rc4 kernel, after more then 20 hours of uptime, it > starts > to going up. With kernel before 3.7-rc3, my machine could be up for 10 days > and > not go beyond 2.6GB memory usage. > > If I start some app that uses a lot of memory, when there is already 4 or > even > 6GB used already, insted of freeing the memory, it starts to swap it, and > everything slows down with a lot of iowait. > > Here is "free -m" output after 24 hours of uptime: > > free -m > total used free shared buffers cached > Mem: 7989 7563 426 0 146 2772 > -/+ buffers/cache: 4643 3345 > Swap: 1953 688 1264 > > > I know that it is ok for memory to be used this much for buffers and cache, > but > it is not normal not to relase it when it is needed. > > In attachment is my kernel config file. > Sounds like a memory leak. Please get the machine into this state and then send us - the contents of /proc/meminfo - the contents of /proc/slabinfo - the contents of /proc/vmstat - as root: dmesg -c echo m > /proc/sysrq-trigger dmesg thanks. On Tue, 2012-11-13 at 14:03 -0800, Andrew Morton wrote:
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Tue, 6 Nov 2012 15:11:48 +0000 (UTC)
> bugzilla-daemon@bugzilla.kernel.org wrote:
>
> > https://bugzilla.kernel.org/show_bug.cgi?id=50181
> >
> > Summary: Memory usage doubles after more then 20 hours of
> > uptime.
> > Product: Memory Management
> > Version: 2.5
> > Kernel Version: 3.7-rc3 and 3.7-rc4
> > Platform: All
> > OS/Version: Linux
> > Tree: Mainline
> > Status: NEW
> > Severity: normal
> > Priority: P1
> > Component: Other
> > AssignedTo: akpm@linux-foundation.org
> > ReportedBy: sukijaki@gmail.com
> > Regression: Yes
> >
> >
> > Created an attachment (id=85721)
> > --> (https://bugzilla.kernel.org/attachment.cgi?id=85721)
> > kernel config file
> >
> > After 20 hours of uptime, memory usage starts going up. Normal usage for my
> > system was around 2.5GB max with all my apps and services up and running.
> But
> > with 3.7-rc3 and now -rc4 kernel, after more then 20 hours of uptime, it
> starts
> > to going up. With kernel before 3.7-rc3, my machine could be up for 10 days
> and
> > not go beyond 2.6GB memory usage.
> >
> > If I start some app that uses a lot of memory, when there is already 4 or
> even
> > 6GB used already, insted of freeing the memory, it starts to swap it, and
> > everything slows down with a lot of iowait.
> >
> > Here is "free -m" output after 24 hours of uptime:
> >
> > free -m
> > total used free shared buffers cached
> > Mem: 7989 7563 426 0 146 2772
> > -/+ buffers/cache: 4643 3345
> > Swap: 1953 688 1264
> >
> >
> > I know that it is ok for memory to be used this much for buffers and cache,
> but
> > it is not normal not to relase it when it is needed.
> >
> > In attachment is my kernel config file.
> >
>
> Sounds like a memory leak.
>
> Please get the machine into this state and then send us
>
> - the contents of /proc/meminfo
>
> - the contents of /proc/slabinfo
>
> - the contents of /proc/vmstat
>
> - as root:
>
> dmesg -c
> echo m > /proc/sysrq-trigger
> dmesg
>
> thanks.
Will do. But it will take a day or two to get there, I rebooted today
because of this problem.
I'm seeing _some_ kind of leak on 3.7-rc5. The usual suspects (slab+anon+pagecache+free) are 3GB or so, so I'm missing ~5GB. Full stats are here: http://sr71.net/~dave/linux/leak-20121113/ MemTotal: 7973852 kB MemFree: 487944 kB Buffers: 42896 kB Cached: 599104 kB AnonPages: 1991104 kB Slab: 76708 kB I noticed something was wrong when I couldn't allocate a single hugetlbfs page. There's also some weirdness going on with my i915 video. Not sure if they're related, but the i915 code _is_ doing some order-9 allocs which are failing. On Tue, 2012-11-13 at 14:03 -0800, Andrew Morton wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Tue, 6 Nov 2012 15:11:48 +0000 (UTC) > bugzilla-daemon@bugzilla.kernel.org wrote: > > > https://bugzilla.kernel.org/show_bug.cgi?id=50181 > > > > Summary: Memory usage doubles after more then 20 hours of > > uptime. > > Product: Memory Management > > Version: 2.5 > > Kernel Version: 3.7-rc3 and 3.7-rc4 > > Platform: All > > OS/Version: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: Other > > AssignedTo: akpm@linux-foundation.org > > ReportedBy: sukijaki@gmail.com > > Regression: Yes > > > > > > Created an attachment (id=85721) > > --> (https://bugzilla.kernel.org/attachment.cgi?id=85721) > > kernel config file > > > > After 20 hours of uptime, memory usage starts going up. Normal usage for my > > system was around 2.5GB max with all my apps and services up and running. > But > > with 3.7-rc3 and now -rc4 kernel, after more then 20 hours of uptime, it > starts > > to going up. With kernel before 3.7-rc3, my machine could be up for 10 days > and > > not go beyond 2.6GB memory usage. > > > > If I start some app that uses a lot of memory, when there is already 4 or > even > > 6GB used already, insted of freeing the memory, it starts to swap it, and > > everything slows down with a lot of iowait. > > > > Here is "free -m" output after 24 hours of uptime: > > > > free -m > > total used free shared buffers cached > > Mem: 7989 7563 426 0 146 2772 > > -/+ buffers/cache: 4643 3345 > > Swap: 1953 688 1264 > > > > > > I know that it is ok for memory to be used this much for buffers and cache, > but > > it is not normal not to relase it when it is needed. > > > > In attachment is my kernel config file. > > > > Sounds like a memory leak. > > Please get the machine into this state and then send us > > - the contents of /proc/meminfo > > - the contents of /proc/slabinfo > > - the contents of /proc/vmstat > > - as root: > > dmesg -c > echo m > /proc/sysrq-trigger > dmesg > > thanks. Here is the requested content: free -m: http://pastebin.com/vb878a9Y cat /proc/meminfo : http://pastebin.com/zUDFcYEW cat /proc/slabinfo : http://pastebin.com/kswsJ7Hk cat /proc/vmstat : http://pastebin.com/wUebJqJe dmesg -c : http://pastebin.com/f7cTu8Wv echo m > /proc/sysrq-trigger && dmesg : http://pastebin.com/p68DcHUy And here are also files with that content: http://ubuntuone.com/5GUVahBTiZRP0QjQdP3gkQ On Thu, 15 Nov 2012 15:05:49 +0100 Milos Jakovljevic <sukijaki@gmail.com> wrote: > Here is the requested content: > > free -m: http://pastebin.com/vb878a9Y > cat /proc/meminfo : http://pastebin.com/zUDFcYEW > cat /proc/slabinfo : http://pastebin.com/kswsJ7Hk > cat /proc/vmstat : http://pastebin.com/wUebJqJe > > dmesg -c : http://pastebin.com/f7cTu8Wv > > echo m > /proc/sysrq-trigger && dmesg : http://pastebin.com/p68DcHUy > > And here are also files with that content: > http://ubuntuone.com/5GUVahBTiZRP0QjQdP3gkQ > You've lost 2-3GB of ZONE_NORMAL and I see no sign there to indicate where it went. /proc/slabinfo indicates that it isn't a slab leak, and kmemleak won't tell us about alloc_pages() leaks. I'm stumped. Dave, any progress at your end? (In reply to comment #6) > /proc/slabinfo indicates that it isn't a slab leak, and kmemleak won't > tell us about alloc_pages() leaks. I'm stumped. Dave, any progress at > your end? I turned on kmemleak and was able to reproduce this on a second reboot, but it went most of a workday before I noticed it had leaked a bunch. Unfortunately, kmemleak didn't help at all. It found a few small things that _may_ be leaks, but nothing to account for this _massive_ loss. I'm stumped so far. My next step is to add some logging to at least see if this is a gradual thing or it happens all at once, and maybe figure out what the heck I'm doing to trigger it. On Thu, 2012-11-15 at 14:12 -0800, Andrew Morton wrote:
> On Thu, 15 Nov 2012 15:05:49 +0100
> Milos Jakovljevic <sukijaki@gmail.com> wrote:
>
> > Here is the requested content:
> >
> > free -m: http://pastebin.com/vb878a9Y
> > cat /proc/meminfo : http://pastebin.com/zUDFcYEW
> > cat /proc/slabinfo : http://pastebin.com/kswsJ7Hk
> > cat /proc/vmstat : http://pastebin.com/wUebJqJe
> >
> > dmesg -c : http://pastebin.com/f7cTu8Wv
> >
> > echo m > /proc/sysrq-trigger && dmesg : http://pastebin.com/p68DcHUy
> >
> > And here are also files with that content:
> > http://ubuntuone.com/5GUVahBTiZRP0QjQdP3gkQ
> >
>
> You've lost 2-3GB of ZONE_NORMAL and I see no sign there to indicate
> where it went.
>
> /proc/slabinfo indicates that it isn't a slab leak, and kmemleak won't
> tell us about alloc_pages() leaks. I'm stumped. Dave, any progress at
> your end?
>
>
I didn't understood anything you sad, but never mind.
In -rc2 there was a problem with massive iowait when anything was done
(starting an app, loading a web page, etc ...), and there was a massive
read operation from my /home partition. In -rc3 that stopped, and this
started happening. Maybe it is related somehow?
Or maybe, it is just some problem with nvidia blob and 3.7 kernel
loosing VM_RELEASE (in a blob's mmap.c it was replaced with
VM_DONTEXPAND | VM_DONTDUMP ). - or maybe I'm just saying nonsense
here.
Reply-To: dave@linux.vnet.ibm.com resending to linux-mm@... On 11/15/2012 02:12 PM, Andrew Morton wrote: > /proc/slabinfo indicates that it isn't a slab leak, and kmemleak won't > tell us about alloc_pages() leaks. I'm stumped. Dave, any progress at > your end? I turned on kmemleak and was able to reproduce this on a second reboot, but it went most of a workday before I noticed it had leaked a bunch. Unfortunately, kmemleak didn't help at all. It found a few small things that _may_ be leaks, but nothing to account for this _massive_ loss. I'm stumped so far. My next step is to add some logging to at least see if this is a gradual thing or it happens all at once, and maybe figure out what the heck I'm doing to trigger it. Reply-To: dave@linux.vnet.ibm.com On 11/15/2012 03:11 PM, Milos Jakovljevic wrote: > Or maybe, it is just some problem with nvidia blob and 3.7 kernel > loosing VM_RELEASE (in a blob's mmap.c it was replaced with > VM_DONTEXPAND | VM_DONTDUMP ). - or maybe I'm just saying nonsense > here. I'm using Intel graphics, so it's not nvidia related for me, at least. I've been recording a bunch of gunk from /proc once a minute for the past 16 hours or so. I've grepped some of it in to a log file (but I've got a *LOT* more than this): http://sr71.net/~dave/linux/leak-20121113/log.1353087988.txt.gz From meminfo, it shows MemFree/Buffers/Cached/AnonPages/Slab/PageTables, and their sum. That should capture _most_ of the memory use on the system, and if we see that sum going down, it's probably a sign of the leak, especially when we see a trend over a long period. The file is in roughly this format, if anyone cares: <nr/date> <meminfo fields> sums: <sum fields> <delta> The system in question is my laptop. What I can tell is that it doesn't leak much when I'm not using it. But, it's leaking pretty steadily since I started using the system today (~6am in the logs). It _averages_ leaking about 400kB/minute when idle and almost 9MB/minute when in active use. I've tried to provoke the leak doing specific things like large downloads, kernel compiles, watching video, alloc'ing a bunch of transparent huge pages, then exiting... No smoking gun so far. Anybody have ideas what to try next or want to poke holes in my statistics? :) On Fri, 2012-11-16 at 10:34 -0800, Dave Hansen wrote:
> On 11/15/2012 03:11 PM, Milos Jakovljevic wrote:
> > Or maybe, it is just some problem with nvidia blob and 3.7 kernel
> > loosing VM_RELEASE (in a blob's mmap.c it was replaced with
> > VM_DONTEXPAND | VM_DONTDUMP ). - or maybe I'm just saying nonsense
> > here.
>
> I'm using Intel graphics, so it's not nvidia related for me, at least.
>
> I've been recording a bunch of gunk from /proc once a minute for the
> past 16 hours or so. I've grepped some of it in to a log file (but I've
> got a *LOT* more than this):
>
> http://sr71.net/~dave/linux/leak-20121113/log.1353087988.txt.gz
>
> From meminfo, it shows MemFree/Buffers/Cached/AnonPages/Slab/PageTables,
> and their sum. That should capture _most_ of the memory use on the
> system, and if we see that sum going down, it's probably a sign of the
> leak, especially when we see a trend over a long period. The file is in
> roughly this format, if anyone cares:
>
> <nr/date> <meminfo fields> sums: <sum fields> <delta>
>
> The system in question is my laptop. What I can tell is that it doesn't
> leak much when I'm not using it. But, it's leaking pretty steadily
> since I started using the system today (~6am in the logs). It
> _averages_ leaking about 400kB/minute when idle and almost 9MB/minute
> when in active use.
>
> I've tried to provoke the leak doing specific things like large
> downloads, kernel compiles, watching video, alloc'ing a bunch of
> transparent huge pages, then exiting... No smoking gun so far.
>
> Anybody have ideas what to try next or want to poke holes in my
> statistics? :)
>
For me, it mostly happens over night, when there is only tvtime and
deluge active (and rest of the programs are open but I don't use them -
firefox, evolution and pidgin).
Only ones it happened while I was on the PC doing something. It was
fresh after reboot, and I was restarting Firefox, 10 or more times (I
was experimenting with addons).
On Fri, 16 Nov 2012 10:34:00 -0800 Dave Hansen <dave@linux.vnet.ibm.com> wrote: > Anybody have ideas what to try next or want to poke holes in my > statistics? :) Maybe resurrect the below patch? It's probably six years old. It should allow us to find out who allocated those pages. Then perhaps we should merge the sucker this time. From: Alexander Nyberg <alexn@dsv.su.se> Introduces CONFIG_PAGE_OWNER that keeps track of the call chain under which a page was allocated. Includes a user-space helper in Documentation/page_owner.c to sort the enormous amount of output that this may give (thanks tridge). Information available through /proc/page_owner x86_64 introduces some stack noise in certain call chains so for exact output use of x86 && CONFIG_FRAME_POINTER is suggested. Tested on x86, x86 && CONFIG_FRAME_POINTER, x86_64 Output looks like: 4819 times: Page allocated via order 0, mask 0x50 [0xc012b7b9] find_lock_page+25 [0xc012b8c8] find_or_create_page+152 [0xc0147d74] grow_dev_page+36 [0xc0148164] __find_get_block+84 [0xc0147ebc] __getblk_slow+124 [0xc0148164] __find_get_block+84 [0xc01481e7] __getblk+55 [0xc0185d14] do_readahead+100 We use a custom stack unwinder because using __builtin_return_address([0-7]) causes gcc to generate code that might try to unwind the stack looking for function return addresses and "fall off" causing early panics if the call chain is not deep enough. So in that case we could have had a depth of around 3 functions in all traces (I experimented a bit with this). From: Dave Hansen <haveblue@us.ibm.com> make page_owner handle non-contiguous page ranges From: Alexander Nyberg <alexn@telia.com> I've cleaned up the __alloc_pages() part to a simple set_page_owner() call. Signed-off-by: Alexander Nyberg <alexn@dsv.su.se> Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Signed-Off-by: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitu.com> DESC Update page->order at an appropriate time when tracking PAGE_OWNER EDESC From: mel@skynet.ie (Mel Gorman) PAGE_OWNER tracks free pages by setting page->order to -1. However, it is set during __free_pages() which is not the only free path as __pagevec_free() and free_compound_page() do not go through __free_pages(). This leads to a situation where free pages are visible in /proc/page_owner which is confusing and might be interpreted as a memory leak. This patch sets page->owner when PageBuddy is set. It also prints a warning to the kernel log if a free page is found that does not appear free to PAGE_OWNER. This should be considered a fix to page-owner-tracking-leak-detector.patch. This only applies to -mm as PAGE_OWNER is not in mainline. Signed-off-by: Mel Gorman <mel@csn.ul.ie> Acked-by: Andy Whitcroft <apw@shadowen.org> DESC Print out PAGE_OWNER statistics in relation to fragmentation avoidance EDESC From: Mel Gorman <mel@csn.ul.ie> When PAGE_OWNER is set, more information is available of relevance to fragmentation avoidance. A second line is added to /proc/page_owner showing the PFN, the pageblock number, the mobility type of the page based on its allocation flags, whether the allocation is improperly placed and the flags. A sample entry looks like Page allocated via order 0, mask 0x1280d2 PFN 7355 Block 7 type 3 Fallback Flags LA [0xc01528c6] __handle_mm_fault+598 [0xc0320427] do_page_fault+279 [0xc031ed9a] error_code+114 This information can be used to identify pages that are improperly placed. As the format of PAGE_OWNER data is now different, the comment at the top of Documentation/page_owner.c is updated with new instructions. As PAGE_OWNER tracks the GFP flags used to allocate the pages, /proc/pagetypeinfo is enhanced to contain how many mixed blocks exist. The additional output looks like Number of mixed blocks Unmovable Reclaimable Movable Reserve Node 0, zone DMA 0 1 2 1 Node 0, zone Normal 2 11 33 0 Signed-off-by: Mel Gorman <mel@csn.ul.ie> Acked-by: Andy Whitcroft <apw@shadowen.org> Acked-by: Christoph Lameter <cl@linux-foundation.org> DESC Allow PAGE_OWNER to be set on any architecture EDESC From: Mel Gorman <mel@csn.ul.ie> Currently PAGE_OWNER depends on CONFIG_X86. This appears to be due to pfn_to_page() being called in an inappropriate for many memory models and the presense of memory holes. This patch ensures that pfn_valid() and pfn_valid_within() is called at the appropriate places and the offsets correctly updated so that PAGE_OWNER is safe on any architecture. In situations where CONFIG_HOLES_IN_ZONES is set (IA64 with VIRTUAL_MEM_MAP), there may be cases where pages allocated within a MAX_ORDER_NR_PAGES block of pages may not be displayed in /proc/page_owner if the hole is at the start of the block. Addressing this would be quite complex, perform slowly and is of no clear benefit. Once PAGE_OWNER is allowed on all architectures, the statistics for grouping pages by mobility that declare how many pageblocks contain mixed page types becomes optionally available on all arches. This patch was tested successfully on x86, x86_64, ppc64 and IA64 machines. Signed-off-by: Mel Gorman <mel@csn.ul.ie> Acked-by: Andy Whitcroft <apw@shadowen.org> DESC allow-page_owner-to-be-set-on-any-architecture-fix EDESC From: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Whitcroft <apw@shadowen.org> Cc: Mel Gorman <mel@csn.ul.ie> DESC allow-page_owner-to-be-set-on-any-architecture-fix fix EDESC From: mel@skynet.ie (Mel Gorman) Page-owner-tracking stores the a backtrace of an allocation in the struct page. How the stack trace is generated depends on whether CONFIG_FRAME_POINTER is set or not. If CONFIG_FRAME_POINTER is set, the frame pointer must be read using some inline assembler which is not available for all architectures. This patch uses the frame pointer where it is available but has a fallback where it is not. Signed-off-by: Mel Gorman <mel@csn.ul.ie> Cc: Andy Whitcroft <apw@shadowen.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> --- Documentation/page_owner.c | 141 ++++++++++++++++++++++++++++++++++ fs/proc/proc_misc.c | 145 +++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 5 + lib/Kconfig.debug | 10 ++ mm/page_alloc.c | 66 +++++++++++++++ mm/vmstat.c | 93 ++++++++++++++++++++++ 6 files changed, 460 insertions(+) diff -puN /dev/null Documentation/page_owner.c --- /dev/null +++ a/Documentation/page_owner.c @@ -0,0 +1,141 @@ +/* + * User-space helper to sort the output of /proc/page_owner + * + * Example use: + * cat /proc/page_owner > page_owner_full.txt + * grep -v ^PFN page_owner_full.txt > page_owner.txt + * ./sort page_owner.txt sorted_page_owner.txt +*/ + +#include <stdio.h> +#include <stdlib.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <unistd.h> +#include <string.h> + +struct block_list { + char *txt; + int len; + int num; +}; + + +static struct block_list *list; +static int list_size; +static int max_size; + +struct block_list *block_head; + +int read_block(char *buf, FILE *fin) +{ + int ret = 0; + int hit = 0; + char *curr = buf; + + for (;;) { + *curr = getc(fin); + if (*curr == EOF) return -1; + + ret++; + if (*curr == '\n' && hit == 1) + return ret - 1; + else if (*curr == '\n') + hit = 1; + else + hit = 0; + curr++; + } +} + +static int compare_txt(struct block_list *l1, struct block_list *l2) +{ + return strcmp(l1->txt, l2->txt); +} + +static int compare_num(struct block_list *l1, struct block_list *l2) +{ + return l2->num - l1->num; +} + +static void add_list(char *buf, int len) +{ + if (list_size != 0 && + len == list[list_size-1].len && + memcmp(buf, list[list_size-1].txt, len) == 0) { + list[list_size-1].num++; + return; + } + if (list_size == max_size) { + printf("max_size too small??\n"); + exit(1); + } + list[list_size].txt = malloc(len+1); + list[list_size].len = len; + list[list_size].num = 1; + memcpy(list[list_size].txt, buf, len); + list[list_size].txt[len] = 0; + list_size++; + if (list_size % 1000 == 0) { + printf("loaded %d\r", list_size); + fflush(stdout); + } +} + +int main(int argc, char **argv) +{ + FILE *fin, *fout; + char buf[1024]; + int ret, i, count; + struct block_list *list2; + struct stat st; + + fin = fopen(argv[1], "r"); + fout = fopen(argv[2], "w"); + if (!fin || !fout) { + printf("Usage: ./program <input> <output>\n"); + perror("open: "); + exit(2); + } + + fstat(fileno(fin), &st); + max_size = st.st_size / 100; /* hack ... */ + + list = malloc(max_size * sizeof(*list)); + + for(;;) { + ret = read_block(buf, fin); + if (ret < 0) + break; + + buf[ret] = '\0'; + add_list(buf, ret); + } + + printf("loaded %d\n", list_size); + + printf("sorting ....\n"); + + qsort(list, list_size, sizeof(list[0]), compare_txt); + + list2 = malloc(sizeof(*list) * list_size); + + printf("culling\n"); + + for (i=count=0;i<list_size;i++) { + if (count == 0 || + strcmp(list2[count-1].txt, list[i].txt) != 0) { + list2[count++] = list[i]; + } else { + list2[count-1].num += list[i].num; + } + } + + qsort(list2, count, sizeof(list[0]), compare_num); + + for (i=0;i<count;i++) { + fprintf(fout, "%d times:\n%s\n", list2[i].num, list2[i].txt); + } + return 0; +} diff -puN fs/proc/proc_misc.c~page-owner-tracking-leak-detector fs/proc/proc_misc.c --- a/fs/proc/proc_misc.c~page-owner-tracking-leak-detector +++ a/fs/proc/proc_misc.c @@ -855,6 +855,140 @@ static struct file_operations proc_kpage }; #endif /* CONFIG_PROC_PAGE_MONITOR */ +#ifdef CONFIG_PAGE_OWNER +#include <linux/bootmem.h> +#include <linux/kallsyms.h> +static ssize_t +read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) +{ + unsigned long pfn; + struct page *page; + char *kbuf, *modname; + const char *symname; + int ret = 0; + char namebuf[128]; + unsigned long offset = 0, symsize; + int i; + ssize_t num_written = 0; + int blocktype = 0, pagetype = 0; + + page = NULL; + pfn = min_low_pfn + *ppos; + + /* Find a valid PFN or the start of a MAX_ORDER_NR_PAGES area */ + while (!pfn_valid(pfn) && (pfn & (MAX_ORDER_NR_PAGES - 1)) != 0) + pfn++; + + /* Find an allocated page */ + for (; pfn < max_pfn; pfn++) { + /* + * If the new page is in a new MAX_ORDER_NR_PAGES area, + * validate the area as existing, skip it if not + */ + if ((pfn & (MAX_ORDER_NR_PAGES - 1)) == 0 && !pfn_valid(pfn)) { + pfn += MAX_ORDER_NR_PAGES - 1; + continue; + } + + /* Check for holes within a MAX_ORDER area */ + if (!pfn_valid_within(pfn)) + continue; + + page = pfn_to_page(pfn); + + /* Catch situations where free pages have a bad ->order */ + if (page->order >= 0 && PageBuddy(page)) + printk(KERN_WARNING + "PageOwner info inaccurate for PFN %lu\n", + pfn); + + /* Stop search if page is allocated and has trace info */ + if (page->order >= 0 && page->trace[0]) + break; + } + + if (!pfn_valid(pfn)) + return 0; + + /* Record the next PFN to read in the file offset */ + *ppos = (pfn - min_low_pfn) + 1; + + kbuf = kmalloc(count, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + ret = snprintf(kbuf, count, "Page allocated via order %d, mask 0x%x\n", + page->order, page->gfp_mask); + if (ret >= count) { + ret = -ENOMEM; + goto out; + } + + /* Print information relevant to grouping pages by mobility */ + blocktype = get_pageblock_migratetype(page); + pagetype = allocflags_to_migratetype(page->gfp_mask); + ret += snprintf(kbuf+ret, count-ret, + "PFN %lu Block %lu type %d %s " + "Flags %s%s%s%s%s%s%s%s%s%s%s%s\n", + pfn, + pfn >> pageblock_order, + blocktype, + blocktype != pagetype ? "Fallback" : " ", + PageLocked(page) ? "K" : " ", + PageError(page) ? "E" : " ", + PageReferenced(page) ? "R" : " ", + PageUptodate(page) ? "U" : " ", + PageDirty(page) ? "D" : " ", + PageLRU(page) ? "L" : " ", + PageActive(page) ? "A" : " ", + PageSlab(page) ? "S" : " ", + PageWriteback(page) ? "W" : " ", + PageCompound(page) ? "C" : " ", + PageSwapCache(page) ? "B" : " ", + PageMappedToDisk(page) ? "M" : " "); + if (ret >= count) { + ret = -ENOMEM; + goto out; + } + + num_written = ret; + + for (i = 0; i < 8; i++) { + if (!page->trace[i]) + break; + symname = kallsyms_lookup(page->trace[i], &symsize, &offset, + &modname, namebuf); + ret = snprintf(kbuf + num_written, count - num_written, + "[0x%lx] %s+%lu\n", + page->trace[i], namebuf, offset); + if (ret >= count - num_written) { + ret = -ENOMEM; + goto out; + } + num_written += ret; + } + + ret = snprintf(kbuf + num_written, count - num_written, "\n"); + if (ret >= count - num_written) { + ret = -ENOMEM; + goto out; + } + + num_written += ret; + ret = num_written; + + if (copy_to_user(buf, kbuf, ret)) + ret = -EFAULT; +out: + kfree(kbuf); + return ret; +} + +static struct file_operations proc_page_owner_operations = { + .read = read_page_owner, +}; +#endif + struct proc_dir_entry *proc_root_kcore; void __init proc_misc_init(void) @@ -932,4 +1066,15 @@ void __init proc_misc_init(void) #ifdef CONFIG_PROC_VMCORE proc_vmcore = proc_create("vmcore", S_IRUSR, NULL, &proc_vmcore_operations); #endif +#ifdef CONFIG_PAGE_OWNER + { + struct proc_dir_entry *entry; + entry = create_proc_entry("page_owner", + S_IWUSR | S_IRUGO, NULL); + if (entry) { + entry->proc_fops = &proc_page_owner_operations; + entry->size = 1024; + } + } +#endif } diff -puN include/linux/mm_types.h~page-owner-tracking-leak-detector include/linux/mm_types.h --- a/include/linux/mm_types.h~page-owner-tracking-leak-detector +++ a/include/linux/mm_types.h @@ -101,6 +101,11 @@ struct page { #ifdef CONFIG_KMEMCHECK void *shadow; #endif +#ifdef CONFIG_PAGE_OWNER + int order; + unsigned int gfp_mask; + unsigned long trace[8]; +#endif }; /* diff -puN lib/Kconfig.debug~page-owner-tracking-leak-detector lib/Kconfig.debug --- a/lib/Kconfig.debug~page-owner-tracking-leak-detector +++ a/lib/Kconfig.debug @@ -66,6 +66,16 @@ config UNUSED_SYMBOLS you really need it, and what the merge plan to the mainline kernel for your module is. +config PAGE_OWNER + bool "Track page owner" + depends on DEBUG_KERNEL + help + This keeps track of what call chain is the owner of a page, may + help to find bare alloc_page(s) leaks. Eats a fair amount of memory. + See Documentation/page_owner.c for user-space helper. + + If unsure, say N. + config DEBUG_FS bool "Debug Filesystem" depends on SYSFS diff -puN mm/page_alloc.c~page-owner-tracking-leak-detector mm/page_alloc.c --- a/mm/page_alloc.c~page-owner-tracking-leak-detector +++ a/mm/page_alloc.c @@ -316,6 +316,9 @@ static inline void set_page_order(struct { set_page_private(page, order); __SetPageBuddy(page); +#ifdef CONFIG_PAGE_OWNER + page->order = -1; +#endif } static inline void rmv_page_order(struct page *page) @@ -1434,6 +1437,62 @@ try_next_zone: return page; } +#ifdef CONFIG_PAGE_OWNER +static inline int valid_stack_ptr(struct thread_info *tinfo, void *p) +{ + return p > (void *)tinfo && + p < (void *)tinfo + THREAD_SIZE - 3; +} + +static inline void __stack_trace(struct page *page, unsigned long *stack, + unsigned long bp) +{ + int i = 0; + unsigned long addr; + struct thread_info *tinfo = (struct thread_info *) + ((unsigned long)stack & (~(THREAD_SIZE - 1))); + + memset(page->trace, 0, sizeof(long) * 8); + +#ifdef CONFIG_FRAME_POINTER + if (bp) { + while (valid_stack_ptr(tinfo, (void *)bp)) { + addr = *(unsigned long *)(bp + sizeof(long)); + page->trace[i] = addr; + if (++i >= 8) + break; + bp = *(unsigned long *)bp; + } + return; + } +#endif /* CONFIG_FRAME_POINTER */ + while (valid_stack_ptr(tinfo, stack)) { + addr = *stack++; + if (__kernel_text_address(addr)) { + page->trace[i] = addr; + if (++i >= 8) + break; + } + } +} + +static void set_page_owner(struct page *page, unsigned int order, + unsigned int gfp_mask) +{ + unsigned long address; + unsigned long bp = 0; +#ifdef CONFIG_X86_64 + asm ("movq %%rbp, %0" : "=r" (bp) : ); +#endif +#ifdef CONFIG_X86_32 + asm ("movl %%ebp, %0" : "=r" (bp) : ); +#endif + page->order = (int) order; + page->gfp_mask = gfp_mask; + __stack_trace(page, &address, bp); +} +#endif /* CONFIG_PAGE_OWNER */ + /* * This is the 'heart' of the zoned buddy allocator. */ @@ -1638,6 +1697,10 @@ nopage: show_mem(); } got_pg: +#ifdef CONFIG_PAGE_OWNER + if (page) + set_page_owner(page, order, gfp_mask); +#endif return page; } EXPORT_SYMBOL(__alloc_pages_internal); @@ -2635,6 +2698,9 @@ void __meminit memmap_init_zone(unsigned if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif +#ifdef CONFIG_PAGE_OWNER + page->order = -1; +#endif } } diff -puN mm/vmstat.c~page-owner-tracking-leak-detector mm/vmstat.c --- a/mm/vmstat.c~page-owner-tracking-leak-detector +++ a/mm/vmstat.c @@ -15,6 +15,7 @@ #include <linux/cpu.h> #include <linux/vmstat.h> #include <linux/sched.h> +#include "internal.h" #ifdef CONFIG_VM_EVENT_COUNTERS DEFINE_PER_CPU(struct vm_event_state, vm_event_states) = {{0}}; @@ -560,6 +561,97 @@ static int pagetypeinfo_showblockcount(s return 0; } +#ifdef CONFIG_PAGE_OWNER +static void pagetypeinfo_showmixedcount_print(struct seq_file *m, + pg_data_t *pgdat, + struct zone *zone) +{ + int mtype, pagetype; + unsigned long pfn; + unsigned long start_pfn = zone->zone_start_pfn; + unsigned long end_pfn = start_pfn + zone->spanned_pages; + unsigned long count[MIGRATE_TYPES] = { 0, }; + + /* Align PFNs to pageblock_nr_pages boundary */ + pfn = start_pfn & ~(pageblock_nr_pages-1); + + /* + * Walk the zone in pageblock_nr_pages steps. If a page block spans + * a zone boundary, it will be double counted between zones. This does + * not matter as the mixed block count will still be correct + */ + for (; pfn < end_pfn; pfn += pageblock_nr_pages) { + struct page *page; + unsigned long offset = 0; + + /* Do not read before the zone start, use a valid page */ + if (pfn < start_pfn) + offset = start_pfn - pfn; + + if (!pfn_valid(pfn + offset)) + continue; + + page = pfn_to_page(pfn + offset); + mtype = get_pageblock_migratetype(page); + + /* Check the block for bad migrate types */ + for (; offset < pageblock_nr_pages; offset++) { + /* Do not past the end of the zone */ + if (pfn + offset >= end_pfn) + break; + + if (!pfn_valid_within(pfn + offset)) + continue; + + page = pfn_to_page(pfn + offset); + + /* Skip free pages */ + if (PageBuddy(page)) { + offset += (1UL << page_order(page)) - 1UL; + continue; + } + if (page->order < 0) + continue; + + pagetype = allocflags_to_migratetype(page->gfp_mask); + if (pagetype != mtype) { + count[mtype]++; + break; + } + + /* Move to end of this allocation */ + offset += (1 << page->order) - 1; + } + } + + /* Print counts */ + seq_printf(m, "Node %d, zone %8s ", pgdat->node_id, zone->name); + for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) + seq_printf(m, "%12lu ", count[mtype]); + seq_putc(m, '\n'); +} +#endif /* CONFIG_PAGE_OWNER */ + +/* + * Print out the number of pageblocks for each migratetype that contain pages + * of other types. This gives an indication of how well fallbacks are being + * contained by rmqueue_fallback(). It requires information from PAGE_OWNER + * to determine what is going on + */ +static void pagetypeinfo_showmixedcount(struct seq_file *m, pg_data_t *pgdat) +{ +#ifdef CONFIG_PAGE_OWNER + int mtype; + + seq_printf(m, "\n%-23s", "Number of mixed blocks "); + for (mtype = 0; mtype < MIGRATE_TYPES; mtype++) + seq_printf(m, "%12s ", migratetype_names[mtype]); + seq_putc(m, '\n'); + + walk_zones_in_node(m, pgdat, pagetypeinfo_showmixedcount_print); +#endif /* CONFIG_PAGE_OWNER */ +} + /* * This prints out statistics in relation to grouping pages by mobility. * It is expensive to collect so do not constantly read the file. @@ -577,6 +669,7 @@ static int pagetypeinfo_show(struct seq_ seq_putc(m, '\n'); pagetypeinfo_showfree(m, pgdat); pagetypeinfo_showblockcount(m, pgdat); + pagetypeinfo_showmixedcount(m, pgdat); return 0; } _ Reply-To: dave@linux.vnet.ibm.com On 11/16/2012 11:15 AM, Andrew Morton wrote: > On Fri, 16 Nov 2012 10:34:00 -0800 > Dave Hansen <dave@linux.vnet.ibm.com> wrote: >> Anybody have ideas what to try next or want to poke holes in my >> statistics? :) > > Maybe resurrect the below patch? It's probably six years old. It > should allow us to find out who allocated those pages. > > Then perhaps we should merge the sucker this time. I at least got the sucker recompiling. Not my finest work, but here goes: http://sr71.net/~dave/linux/leak-20121113/pageowner_for_3.7-rc5.patch It's not pretty, and probably needs to (at least) get moved over to debugfs before getting merged, but it does appear to give some reasonable output. Figured I'd post it in case anyone else wants to give it a spin. Reply-To: dave@linux.vnet.ibm.com I managed to reproduce this on a second machine. The new system ran basically all weekend doing kernel compiles: no leak. But, I added some memory pressure, and made it start allocating a bunch of hugetlbfs pages. That made this bug kick in there too. It's somewhat hard to tell, but I _think_ the leaking is correlated with compaction activity. I'm trying a bisect now. A patch referencing this bug report has been merged in Linux v3.7-rc7: commit ef6c5be658f6a70c1256fbd18e18ee0dc24c3386 Author: Dave Hansen <dave@linux.vnet.ibm.com> Date: Wed Nov 21 14:21:51 2012 -0500 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) I think I'm seeing this bug too - however it's showing in top as cached memory - 6GB so far and it's always after compiles I ran echo 1 > /proc/sys/vm/drop_caches and it cleared it back up again I've been seeing this since the 3.7 rc's but didn't think it was a kernel issue I'd like to add I'm seeing this with 3.8-rc0 too This bug is bothering me for a month now, on a daily basis. The behaviour is always the similar. The ram is gradually filled up by the io cache until no free memory is left any more. At this point the cache usually occupies half of the total 8GB memory on my system. Instead of freeing space occupied by the cache when needed the system becomes irresponsive. Subsequently the oom-killer kills almost all processes in a sequence according to their memory consumption. I do not know what actually triggered this bug to occur first. At the moment it occurs with 3.10.0-031000rc7-generic, but strangely it seems to retroactively effect also older kernel versions up to 3.8.0-23-generic and probably even beyond that version. My system has no swap and an encrypted home partition, but this configuration was running for years without any problems. I assume that at least the following bugs are a related to this one: https://bugzilla.kernel.org/show_bug.cgi?id=13205 https://bugzilla.kernel.org/show_bug.cgi?id=40402 https://bugzilla.kernel.org/show_bug.cgi?id=59901 Folks, this was a pretty specific bug that Milos was hitting. I'm also fairly sure that we squashed it a year ago. If you are continuing to have problems on current kernels, please open a new bug. Andrew, can we close this please? |
Created attachment 85721 [details] kernel config file After 20 hours of uptime, memory usage starts going up. Normal usage for my system was around 2.5GB max with all my apps and services up and running. But with 3.7-rc3 and now -rc4 kernel, after more then 20 hours of uptime, it starts to going up. With kernel before 3.7-rc3, my machine could be up for 10 days and not go beyond 2.6GB memory usage. If I start some app that uses a lot of memory, when there is already 4 or even 6GB used already, insted of freeing the memory, it starts to swap it, and everything slows down with a lot of iowait. Here is "free -m" output after 24 hours of uptime: free -m total used free shared buffers cached Mem: 7989 7563 426 0 146 2772 -/+ buffers/cache: 4643 3345 Swap: 1953 688 1264 I know that it is ok for memory to be used this much for buffers and cache, but it is not normal not to relase it when it is needed. In attachment is my kernel config file.