Bug 67651

Thanks for the report - that is a gimp crash not a kernel one.

Actually it looks like you simply ran out of memory and gimp died in protest

GIMP developer Mike Henning say that it looks like a Linux kernel bug to him, as Gimp 2.8.8 crashes under Linux 3.12, but not Linux 3.11 and below. See also comment 10:
https://bugzilla.gnome.org/show_bug.cgi?id=719619#c10

I am sorry if this wasn't clear from my initial bug description.

The trace was absolutely clear - gimp ran out of memory and died (which is a bit rude of it)

(gimp:17857): GLib-ERROR **: gmem.c:110: failed to allocate 16384 bytes

So there are lots of things this could be but they depend upon knowing a fair bit more about your setup

Can you provide the output of "dmesg" after this occurs, and the contents of "free" off both boots.

The reason I said this was likely a kernel bug is because it disappears when I set the environment variable MALLOC_MMAP_MAX_ to 0 before running gimp (running under otherwise identical conditions).

I was under the impression that a normal out of memory condition would result in gimp being killed by the kernel, not malloc returning null. (That message only means that malloc returned null.)

This is the output of free before running gimp on my machine:
mike@mike-newbox-arch:~$  free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       2.5G       5.3G       2.8M       2.8M       1.3G
-/+ buffers/cache:       1.1G       6.6G
Swap:           0B         0B         0B

As you can see, I have over 6 GB of memory free. Running gimp and creating a 16000x16000 image (about 2.5 GB) then crashes. Only this is appended to dmesg:
[  754.663771] traps: gimp[844] trap int3 ip:7fdb67e07289 sp:7fffcd30c7d0 error:0

(I will attach my full kernel log. Other than this line, you're only missing the boot process.)

As I stated earlier, I can then successfully create an image with the same dimensions by running gimp like this: MALLOC_MMAP_MAX_=0 gimp

Free reports this with that image open:
mike@mike-newbox-arch:~$  free -h
             total       used       free     shared    buffers     cached
Mem:          7.7G       4.9G       2.8G        10M       2.8M       1.4G
-/+ buffers/cache:       3.5G       4.2G
Swap:           0B         0B         0B

Because of this, I believe it is caused by a change in the kernel mmap implementation. (But then, you can probably diagnose it better than I can.)

I have kernel 3.12.5, Arch Linux x86_64, glibc 2.18

Created attachment 119741 [details]
Output of dmesg after reproducing this bug.

The kernel (and userspace set policy) try and avoid getting into a situation where there is too much overcommitting of resources. It's quite likely you will see a malloc NULL return from the kernel refusing an mmap especially if fedora uses memcfg and cgroups to stop one set of tasks (eg the desktop) from taking down the entire box.

The kernel kill case is an extreme 'worst case scenario' recovery which the rest of the policy tries to avoid allowing to happen in the first place.

The fact MALLOC_MMAP_MAX=0 makes it go away does indeed seem suspicious.

One thing related to similar failures has been fixed:  2afc745f3e3079ab16c826be4860da2529054dd2


Hopefully Andrew has some idea

I also see this change between 3.11 and 3.12. I have seen that it is due to a change in the way maps are handled. Trying a particular sequence of gimp operations with 3.11, I see 1177 entries in /proc/<pid>/maps. Doing the same sequence with the 3.12 kernel, I see 70054 maps entries. The default vm.max_map_count (on my Fedora 19 system) is 65530, so gimp's allocations fail before reaching that 70054 mark.

Increasing vm.max_map_count allows gimp to continue working, but something has definitely changed in the way the maps are managed by the kernel.

I haven't used gimp in years.  Please precisely describe the sequence of steps you're using to create this image.  Also, the kernel .config would be useful, thanks.

Exactly what I was doing would be somewhat hard for you to reproduce, but here is a simple set of similar steps that demonstrates it. Firstly, I am using 64 bit Fedora 19, updated to the latest package updates. I have the following versions of pertinent packages:

kernel-3.12.5-200.fc19.x86_64
gimp-2.8.10-4.fc19.x86_64
glib2-2.36.3-4.fc19.x86_64

My machine has 16 GB of RAM and 16 GB of swap; it has an Intel Core i7 CPU with 4 cores, 2 threads per core.

Some steps. Numbers in parentheses are the result of "wc -l maps" in /proc/<pid> for the gimp process.

1. run gimp. (1074 maps)
2. File -> New. Create an image of 9000x4000 pixels. (29122)
3. Scribble a bit in the window with the paintbrush tool. (31123)
4. In the layers dialog, duplicate the layer. (29766 -- lower)
5. In the layers dialog, right click on the top layer and select "add layer mask"; choose white (full opacity). (37060)
6. Scribble a bit in the layer mask. (37551)
7. Select Image -> Flatten Image. (46543)

More of the same sorts of actions quickly drive the maps count over the maximum and gimp dies an ungraceful death.

Doing the same sequence with Kernel 3.11, the maps count never goes about about 1200.

I am using the stock kernel from Fedora 19 updates. I didn't build it myself. I'll attach its .config shortly.

Created attachment 120301 [details]
.config file for Fedora 19 3.12 kernel.

If you just want the crash, an easier way to reproduce it is:
1. Open GIMP
2. Go to File->New
3. Type in 16000 in for both the width and height.
4. Click OK. If another dialog appears, click OK on that too.
5. Gimp should now crash on recent kernels.

My kernel is the Arch Linux stock kernel. The config can be found here:
https://projects.archlinux.org/svntogit/packages.git/tree/linux/trunk/config.x86_64?id=17d596f51b07023cbbae5dac4c7aeb166913131a

Cyrill,

Gimp is broken due to a kernel bug included in 3.12. It cannot open
large files without failing memory allocations due to exceeding
vm.max_map_count. The relevant bugzilla entries are

https://bugzilla.kernel.org/show_bug.cgi?id=67651
https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0

They include details on how to reproduce the issue. In my case, a
failure shows messages like this

	(gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes

	(file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load: gimp_wire_read(): error
	xinit: connection to X server lost

	waiting for X server to shut down
	/usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
	/usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
	/usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup

X-related junk is there was because I was using a headless server and
xinit directly to launch gimp to reproduce the bug.

Automated bisection using mmtests (https://github.com/gormanm/mmtests)
and the configuration file configs/config-global-dhp__gimp-simple (needs
local web server with a copy of the image file) identified the following
commit. Test case was simple -- try and open the large file described in
the bug. I did not investigate the patch itself as I'm just reporting
the results of the bisection. If I had to guess, I'd say that VMA
merging has been affected.

d9104d1ca9662498339c0de975b4666c30485f4e is the first bad commit
commit d9104d1ca9662498339c0de975b4666c30485f4e
Author: Cyrill Gorcunov <gorcunov@gmail.com>
Date:   Wed Sep 11 14:22:24 2013 -0700

    mm: track vma changes with VM_SOFTDIRTY bit
    
    Pavel reported that in case if vma area get unmapped and then mapped (or
    expanded) in-place, the soft dirty tracker won't be able to recognize this
    situation since it works on pte level and ptes are get zapped on unmap,
    loosing soft dirty bit of course.
    
    So to resolve this situation we need to track actions on vma level, there
    VM_SOFTDIRTY flag comes in.  When new vma area created (or old expanded)
    we set this bit, and keep it here until application calls for clearing
    soft dirty bit.
    
    Thus when user space application track memory changes now it can detect if
    vma area is renewed.
    
    Reported-by: Pavel Emelyanov <xemul@parallels.com>
    Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
    Cc: Andy Lutomirski <luto@amacapital.net>
    Cc: Matt Mackall <mpm@selenic.com>
    Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
    Cc: Marcelo Tosatti <mtosatti@redhat.com>
    Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
    Cc: Stephen Rothwell <sfr@canb.auug.org.au>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
    Cc: Rob Landley <rob@landley.net>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

On Wed, Jan 22, 2014 at 07:08:16PM +0000, Mel Gorman wrote:
> Cyrill,
> 
> Gimp is broken due to a kernel bug included in 3.12. It cannot open
> large files without failing memory allocations due to exceeding
> vm.max_map_count. The relevant bugzilla entries are
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=67651
> https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0
> 
> They include details on how to reproduce the issue. In my case, a
> failure shows messages like this
> 
>       (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
> 
>       (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load:
> gimp_wire_read(): error
>       xinit: connection to X server lost
> 
>       waiting for X server to shut down
>       /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
>       /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
>       /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
> 
> X-related junk is there was because I was using a headless server and
> xinit directly to launch gimp to reproduce the bug.
> 
> Automated bisection using mmtests (https://github.com/gormanm/mmtests)
> and the configuration file configs/config-global-dhp__gimp-simple (needs
> local web server with a copy of the image file) identified the following
> commit. Test case was simple -- try and open the large file described in
> the bug. I did not investigate the patch itself as I'm just reporting
> the results of the bisection. If I had to guess, I'd say that VMA
> merging has been affected.

Thanks a lot for report, Mel! I'm investigating...

On Wed, 22 Jan 2014 19:08:16 +0000 Mel Gorman <mgorman@suse.de> wrote:

> X-related junk is there was because I was using a headless server and
> xinit directly to launch gimp to reproduce the bug.

I've never done this.  Can you share the magic recipe for running an X
app in this way?

Thanks.

On Wed, Jan 22, 2014 at 11:19:28PM +0400, Cyrill Gorcunov wrote:
> > commit. Test case was simple -- try and open the large file described in
> > the bug. I did not investigate the patch itself as I'm just reporting
> > the results of the bisection. If I had to guess, I'd say that VMA
> > merging has been affected.
> 
> Thanks a lot for report, Mel! I'm investigating...

Mel, here is a quick fix for bring merging back (just in case if you
have a minute to test it and confirm the merging were affected). It
seems I've lost setting up vma-softdirty bit somewhere and procedure
which tests vma flags mathcing fails, will continue investigating/testing
tomorrow.
---
 mm/mmap.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

Index: linux-2.6.git/mm/mmap.c
===================================================================
--- linux-2.6.git.orig/mm/mmap.c
+++ linux-2.6.git/mm/mmap.c
@@ -893,8 +893,18 @@ again:			remove_next = 1 + (end > next->
 static inline int is_mergeable_vma(struct vm_area_struct *vma,
 			struct file *file, unsigned long vm_flags)
 {
+	/*
+	 * VM_SOFTDIRTY should not prevent from VMA merging, if we
+	 * match the flags but dirty bit -- just mark merged one as
+	 * a dirty then.
+	 */
+#ifdef CONFIG_MEM_SOFT_DIRTY
+	if ((vma->vm_flags ^ vm_flags) & ~VM_SOFTDIRTY)
+		return 0;
+#else
 	if (vma->vm_flags ^ vm_flags)
 		return 0;
+#endif
 	if (vma->vm_file != file)
 		return 0;
 	if (vma->vm_ops && vma->vm_ops->close)
@@ -1082,7 +1092,11 @@ static int anon_vma_compatible(struct vm
 	return a->vm_end == b->vm_start &&
 		mpol_equal(vma_policy(a), vma_policy(b)) &&
 		a->vm_file == b->vm_file &&
+#ifdef CONFIG_MEM_SOFT_DIRTY
+		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC|VM_SOFTDIRTY)) &&
+#else
 		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC)) &&
+#endif
 		b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT);
 }

On 01/22/2014 11:08 AM, Mel Gorman wrote:
> Cyrill,
> 
> Gimp is broken due to a kernel bug included in 3.12. It cannot open
> large files without failing memory allocations due to exceeding
> vm.max_map_count. The relevant bugzilla entries are
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=67651
> https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0
> 
> They include details on how to reproduce the issue. In my case, a
> failure shows messages like this
> 
>       (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
> 
>       (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load:
> gimp_wire_read(): error
>       xinit: connection to X server lost
> 
>       waiting for X server to shut down
>       /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
>       /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
>       /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
> 
> X-related junk is there was because I was using a headless server and
> xinit directly to launch gimp to reproduce the bug.
> 
> Automated bisection using mmtests (https://github.com/gormanm/mmtests)
> and the configuration file configs/config-global-dhp__gimp-simple (needs
> local web server with a copy of the image file) identified the following
> commit. Test case was simple -- try and open the large file described in
> the bug. I did not investigate the patch itself as I'm just reporting
> the results of the bisection. If I had to guess, I'd say that VMA
> merging has been affected.
> 
> d9104d1ca9662498339c0de975b4666c30485f4e is the first bad commit
> commit d9104d1ca9662498339c0de975b4666c30485f4e
> Author: Cyrill Gorcunov <gorcunov@gmail.com>
> Date:   Wed Sep 11 14:22:24 2013 -0700
> 
>     mm: track vma changes with VM_SOFTDIRTY bit
>     
>     Pavel reported that in case if vma area get unmapped and then mapped (or
>     expanded) in-place, the soft dirty tracker won't be able to recognize
>     this
>     situation since it works on pte level and ptes are get zapped on unmap,
>     loosing soft dirty bit of course.
>     
>     So to resolve this situation we need to track actions on vma level, there
>     VM_SOFTDIRTY flag comes in.  When new vma area created (or old expanded)
>     we set this bit, and keep it here until application calls for clearing
>     soft dirty bit.
>     
>     Thus when user space application track memory changes now it can detect
>     if
>     vma area is renewed.

Presumably some path is failing to set VM_SOFTDIRTY, thus preventing mms
from being merged.

That being said, this could cause vma blowups for programs that are
actually using this thing.

--Andy

On Wed, Jan 22, 2014 at 02:45:53PM -0800, Andy Lutomirski wrote:
> >     
> >     Thus when user space application track memory changes now it can detect
> if
> >     vma area is renewed.
> 
> Presumably some path is failing to set VM_SOFTDIRTY, thus preventing mms
> from being merged.
> 
> That being said, this could cause vma blowups for programs that are
> actually using this thing.

Hi Andy, indeed, this could happen. The easiest way is to ignore softdirty bit
when we're trying to merge vmas and set it one new merged. I think this should
be correct. Once I finish I'll send the patch.

	Cyrill

On Thu, 23 Jan 2014 09:59:06 +0400 Cyrill Gorcunov <gorcunov@gmail.com> wrote:

> On Wed, Jan 22, 2014 at 02:45:53PM -0800, Andy Lutomirski wrote:
> > >     
> > >     Thus when user space application track memory changes now it can
> detect if
> > >     vma area is renewed.
> > 
> > Presumably some path is failing to set VM_SOFTDIRTY, thus preventing mms
> > from being merged.
> > 
> > That being said, this could cause vma blowups for programs that are
> > actually using this thing.
> 
> Hi Andy, indeed, this could happen. The easiest way is to ignore softdirty
> bit
> when we're trying to merge vmas and set it one new merged. I think this
> should
> be correct. Once I finish I'll send the patch.

Hang on.  We think the problem is that gimp is generating vmas which
*should* be merged, but for unknown reasons they differ in
VM_SOFTDIRTY, yes?

Shouldn't we work out where we're forgetting to set VM_SOFTDIRTY? 
Putting bandaids over this error when we come to trying to merge the
vmas sounds very wrong?

On Wed, Jan 22, 2014 at 10:09:10PM -0800, Andrew Morton wrote:
> > > 
> > > That being said, this could cause vma blowups for programs that are
> > > actually using this thing.
> > 
> > Hi Andy, indeed, this could happen. The easiest way is to ignore softdirty
> bit
> > when we're trying to merge vmas and set it one new merged. I think this
> should
> > be correct. Once I finish I'll send the patch.
> 
> Hang on.  We think the problem is that gimp is generating vmas which
> *should* be merged, but for unknown reasons they differ in
> VM_SOFTDIRTY, yes?

Yes. One place where I forgot to set softdirty bit is setup_arg_pages. But
it called once on elf load, so it can't cause such effect (but should be
fixed too). Also there is do_brk where vmasoftdirty is missed too :/

Another problem is the potential scenario when we have a bunch of vmas
and clear vma-softdirty bit on them, then we try to map new one, flags
won't match and instead of extending old vma the new one will be created.
I think (if only I'm not missing something) that vma-softdirty should
be ignored in such case (ie inside is_mergeable_vma) and once vma extended
it should be marked as dirty one. Again, I need to think and test more.

> Shouldn't we work out where we're forgetting to set VM_SOFTDIRTY? 
> Putting bandaids over this error when we come to trying to merge the
> vmas sounds very wrong?

I'm looking into this as well.

	Cyrill

On Wed, Jan 22, 2014 at 11:52:15AM -0800, Andrew Morton wrote:
> On Wed, 22 Jan 2014 19:08:16 +0000 Mel Gorman <mgorman@suse.de> wrote:
> 
> > X-related junk is there was because I was using a headless server and
> > xinit directly to launch gimp to reproduce the bug.
> 
> I've never done this.  Can you share the magic recipe for running an X
> app in this way?
> 

The relevant part of the test script is

# Build a wrapper script to launch gimp
cat > gimp-launch.sh << EOF
/usr/bin/gimp -i -b "(mmtests-open-image \"$FILENAME\")" -b "(gimp-quit 0)" > $LOGDIR_RESULTS/gimp-out.1 2>&1
echo \$? > gimp-exit-code
EOF
chmod u+x gimp-launch.sh

$TIME_CMD xinit ./gimp-launch.sh 2> $LOGDIR_RESULTS/time.1
RETVAL=`cat gimp-exit-code`

It's clumsy because the application would start with no window manager
and looking at it again, it probably was not even necessary because of
the -i switch in gimp.

Previously when I needed to automate an X app I configured the machine to
login automatically, exported the DISPLAY variable in the test script and
used wmctrl to detect if an application had a window displayed yet.

On Thu, Jan 23, 2014 at 02:33:25AM +0400, Cyrill Gorcunov wrote:
> On Wed, Jan 22, 2014 at 11:19:28PM +0400, Cyrill Gorcunov wrote:
> > > commit. Test case was simple -- try and open the large file described in
> > > the bug. I did not investigate the patch itself as I'm just reporting
> > > the results of the bisection. If I had to guess, I'd say that VMA
> > > merging has been affected.
> > 
> > Thanks a lot for report, Mel! I'm investigating...
> 
> Mel, here is a quick fix for bring merging back (just in case if you
> have a minute to test it and confirm the merging were affected). It
> seems I've lost setting up vma-softdirty bit somewhere and procedure
> which tests vma flags mathcing fails, will continue investigating/testing
> tomorrow.

The test case passes with this patch applied to 3.13 so that appears
to confirm that this is related to VM_SOFTDIRTY preventing merges.
Unfortunately I did not have slabinfo tracking enabled to double check
the number of vm_area_structs in teh system.

On Thu, Jan 23, 2014 at 02:33:25AM +0400, Cyrill Gorcunov wrote:
> On Wed, Jan 22, 2014 at 11:19:28PM +0400, Cyrill Gorcunov wrote:
> > > commit. Test case was simple -- try and open the large file described in
> > > the bug. I did not investigate the patch itself as I'm just reporting
> > > the results of the bisection. If I had to guess, I'd say that VMA
> > > merging has been affected.
> > 
> > Thanks a lot for report, Mel! I'm investigating...
> 
> Mel, here is a quick fix for bring merging back (just in case if you
> have a minute to test it and confirm the merging were affected). It
> seems I've lost setting up vma-softdirty bit somewhere and procedure
> which tests vma flags mathcing fails, will continue investigating/testing
> tomorrow.

The test case passes with this patch applied to 3.13 so that appears
to confirm that this is related to VM_SOFTDIRTY preventing merges.
Unfortunately I did not have slabinfo tracking enabled to double check
the number of vm_area_structs in teh system.

On Thu, Jan 23, 2014 at 09:55:41AM +0000, Mel Gorman wrote:
> On Thu, Jan 23, 2014 at 02:33:25AM +0400, Cyrill Gorcunov wrote:
> > On Wed, Jan 22, 2014 at 11:19:28PM +0400, Cyrill Gorcunov wrote:
> > > > commit. Test case was simple -- try and open the large file described
> in
> > > > the bug. I did not investigate the patch itself as I'm just reporting
> > > > the results of the bisection. If I had to guess, I'd say that VMA
> > > > merging has been affected.
> > > 
> > > Thanks a lot for report, Mel! I'm investigating...
> > 
> > Mel, here is a quick fix for bring merging back (just in case if you
> > have a minute to test it and confirm the merging were affected). It
> > seems I've lost setting up vma-softdirty bit somewhere and procedure
> > which tests vma flags mathcing fails, will continue investigating/testing
> > tomorrow.
> 
> The test case passes with this patch applied to 3.13 so that appears
> to confirm that this is related to VM_SOFTDIRTY preventing merges.
> Unfortunately I did not have slabinfo tracking enabled to double check
> the number of vm_area_structs in teh system.

Thanks a lot, Mel! I'm testing the patch as well (manually though :).
I'll send the final fix today.

On Thu, Jan 23, 2014 at 02:36:06PM +0400, Cyrill Gorcunov wrote:
> > The test case passes with this patch applied to 3.13 so that appears
> > to confirm that this is related to VM_SOFTDIRTY preventing merges.
> > Unfortunately I did not have slabinfo tracking enabled to double check
> > the number of vm_area_structs in teh system.
> 
> Thanks a lot, Mel! I'm testing the patch as well (manually though :).
> I'll send the final fix today.

The patch below should fix the problem. I would really appreaciate
some additional testing.
---
From: Cyrill Gorcunov <gorcunov@gmail.com>
Subject: [PATCH] mm: Ignore VM_SOFTDIRTY on VMA merging

VM_SOFTDIRTY bit affects vma merge routine: if two VMAs has all
bits in vm_flags matched except dirty bit the kernel can't longer
merge them and this forces the kernel to generate new VMAs instead.

It finally may lead to the situation when userspace application
reaches vm.max_map_count limit and get crashed in worse case

 | (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
 |
 | (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load: gimp_wire_read(): error
 | xinit: connection to X server lost
 |
 | waiting for X server to shut down
 | /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup

https://bugzilla.kernel.org/show_bug.cgi?id=67651
https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0

Initial problem came from missed VM_SOFTDIRTY in do_brk() routine
but even if we would set up VM_SOFTDIRTY here, there is still a way to
prevent VMAs from merging: one can call

 | echo 4 > /proc/$PID/clear_refs

and clear all VM_SOFTDIRTY over all VMAs presented in memory map,
then new do_brk() will try to extend old VMA and finds that dirty
bit doesn't match thus new VMA will be generated.

As discussed to Pavel, the right approach should be to ignore
VM_SOFTDIRTY bit when we're trying to merge VMAs and if merged
successed we mark extended VMA with dirty bit.

Reported-by: Mel Gorman <mgorman@suse.de>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
---
 mm/mmap.c |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

Index: linux-2.6.git/mm/mmap.c
===================================================================
--- linux-2.6.git.orig/mm/mmap.c
+++ linux-2.6.git/mm/mmap.c
@@ -893,7 +893,15 @@ again:			remove_next = 1 + (end > next->
 static inline int is_mergeable_vma(struct vm_area_struct *vma,
 			struct file *file, unsigned long vm_flags)
 {
-	if (vma->vm_flags ^ vm_flags)
+	/*
+	 * VM_SOFTDIRTY should not prevent from VMA merging, if we
+	 * match the flags but dirty bit -- the caller should mark
+	 * merged VMA as dirty. If dirty bit won't be excluded from
+	 * comparison, we increase pressue on the memory system forcing
+	 * the kernel to generate new VMAs when old one could be extended
+	 * instead.
+	 */
+	if ((vma->vm_flags ^ vm_flags) & VM_SOFTDIRTY)
 		return 0;
 	if (vma->vm_file != file)
 		return 0;
@@ -1038,6 +1046,8 @@ struct vm_area_struct *vma_merge(struct
 				end, prev->vm_pgoff, NULL);
 		if (err)
 			return NULL;
+		else
+			prev->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(prev);
 		return prev;
 	}
@@ -1057,6 +1067,8 @@ struct vm_area_struct *vma_merge(struct
 				next->vm_pgoff - pglen, NULL);
 		if (err)
 			return NULL;
+		else
+			area->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(area);
 		return area;
 	}
@@ -1082,7 +1094,7 @@ static int anon_vma_compatible(struct vm
 	return a->vm_end == b->vm_start &&
 		mpol_equal(vma_policy(a), vma_policy(b)) &&
 		a->vm_file == b->vm_file &&
-		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC)) &&
+		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC|VM_SOFTDIRTY)) &&
 		b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT);
 }

On Thu, Jan 23, 2014 at 04:15:55PM +0400, Cyrill Gorcunov wrote:
> On Thu, Jan 23, 2014 at 02:36:06PM +0400, Cyrill Gorcunov wrote:
> > > The test case passes with this patch applied to 3.13 so that appears
> > > to confirm that this is related to VM_SOFTDIRTY preventing merges.
> > > Unfortunately I did not have slabinfo tracking enabled to double check
> > > the number of vm_area_structs in teh system.
> > 
> > Thanks a lot, Mel! I'm testing the patch as well (manually though :).
> > I'll send the final fix today.
> 
> The patch below should fix the problem. I would really appreaciate
> some additional testing.

Forgot to refresh the patch, sorry.
---
From: Cyrill Gorcunov <gorcunov@gmail.com>
Subject: [PATCH] mm: Ignore VM_SOFTDIRTY on VMA merging

VM_SOFTDIRTY bit affects vma merge routine: if two VMAs has all
bits in vm_flags matched except dirty bit the kernel can't longer
merge them and this forces the kernel to generate new VMAs instead.

It finally may lead to the situation when userspace application
reaches vm.max_map_count limit and get crashed in worse case

 | (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
 |
 | (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load: gimp_wire_read(): error
 | xinit: connection to X server lost
 |
 | waiting for X server to shut down
 | /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup

https://bugzilla.kernel.org/show_bug.cgi?id=67651
https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0

Initial problem came from missed VM_SOFTDIRTY in do_brk() routine
but even if we would set up VM_SOFTDIRTY here, there is still a way to
prevent VMAs from merging: one can call

 | echo 4 > /proc/$PID/clear_refs

and clear all VM_SOFTDIRTY over all VMAs presented in memory map,
then new do_brk() will try to extend old VMA and finds that dirty
bit doesn't match thus new VMA will be generated.

As discussed to Pavel, the right approach should be to ignore
VM_SOFTDIRTY bit when we're trying to merge VMAs and if merged
successed we mark extended VMA with dirty bit.

Reported-by: Mel Gorman <mgorman@suse.de>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
CC: Pavel Emelyanov <xemul@parallels.com>
CC: Andrew Morton <akpm@linux-foundation.org>
---
 mm/mmap.c |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

Index: linux-2.6.git/mm/mmap.c
===================================================================
--- linux-2.6.git.orig/mm/mmap.c
+++ linux-2.6.git/mm/mmap.c
@@ -893,7 +893,15 @@ again:			remove_next = 1 + (end > next->
 static inline int is_mergeable_vma(struct vm_area_struct *vma,
 			struct file *file, unsigned long vm_flags)
 {
-	if (vma->vm_flags ^ vm_flags)
+	/*
+	 * VM_SOFTDIRTY should not prevent from VMA merging, if we
+	 * match the flags but dirty bit -- the caller should mark
+	 * merged VMA as dirty. If dirty bit won't be excluded from
+	 * comparison, we increase pressue on the memory system forcing
+	 * the kernel to generate new VMAs when old one could be extended
+	 * instead.
+	 */
+	if ((vma->vm_flags ^ vm_flags) & ~VM_SOFTDIRTY)
 		return 0;
 	if (vma->vm_file != file)
 		return 0;
@@ -1038,6 +1046,8 @@ struct vm_area_struct *vma_merge(struct
 				end, prev->vm_pgoff, NULL);
 		if (err)
 			return NULL;
+		else
+			prev->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(prev);
 		return prev;
 	}
@@ -1057,6 +1067,8 @@ struct vm_area_struct *vma_merge(struct
 				next->vm_pgoff - pglen, NULL);
 		if (err)
 			return NULL;
+		else
+			area->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(area);
 		return area;
 	}
@@ -1082,7 +1094,7 @@ static int anon_vma_compatible(struct vm
 	return a->vm_end == b->vm_start &&
 		mpol_equal(vma_policy(a), vma_policy(b)) &&
 		a->vm_file == b->vm_file &&
-		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC)) &&
+		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC|VM_SOFTDIRTY)) &&
 		b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT);
 }

FYI: This bug hits us in AddressSanitizer, where the allocator creates lots
of adjacent mappings and expects the kernel to merge them.

FYI: This bug hits us in AddressSanitizer, where the allocator creates lots
of adjacent mappings and expects the kernel to merge them.
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59733

FTR, here is a small test case that would fail if the bug is present: 

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <assert.h>
int main() {
  char *p = (char*)0x600000000000;
  size_t i;
  for (i = 0; i < 100000; i++) {
    void *addr = p + i * 4096;
    void *ret = mmap(addr, 4096, PROT_WRITE | PROT_READ,
                     MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
    if (addr != ret) {
      fprintf(stderr, "failed at iteration %zd\n", i);
      char command[100];
      snprintf(command, sizeof(command), "cat /proc/%d/maps | head -30", getpid());
      system(command);
      return 1;
    }
  }
}


On a broken kernel it will print something like this:

failed at iteration 65514
00400000-00401000 r-xp 00000000 fc:00 20991316  /tmp/a.out
00600000-00601000 r--p 00000000 fc:00 20991316  /tmp/a.out
00601000-00602000 rw-p 00001000 fc:00 20991316  /tmp/a.out
600000000000-600000001000 rw-p 00000000 00:00 0 
600000001000-600000002000 rw-p 00000000 00:00 0 
600000002000-600000003000 rw-p 00000000 00:00 0 
600000003000-600000004000 rw-p 00000000 00:00 0 
600000004000-600000005000 rw-p 00000000 00:00 0 
...