Created attachment 186421 [details]
testcase and the bug information

Hi all,

I think I find a linux bug, I have the test cases is constructed. I can stable recurring problems in fedora22(4.0.4) kernel version, arch for x86_64.
I construct transparent huge page, when the parent and child process with MAP_SHARE, MAP_PRIVATE way to access the same huge page area, it has the opportunity to lead to huge page copy on write failure, and then it will munmap the child corresponding mmap area, but then the child mmap area with VM_MAYSHARE attributes, child process munmap this area can trigger VM_BUG_ON in set_vma_resv_flags functions (vma - > vm_flags & VM_MAYSHARE).

Testcase and the information can refer to the attachment.

I think we can through the following two methods to modify the problem. The two methods are validated. 

patch 1:
diff --git a/mm/mmap.c b/mm/mmap.c
index bb50cac..19c43ca 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1347,6 +1347,7 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
                                return -ENODEV;
                        if (vm_flags & (VM_GROWSDOWN|VM_GROWSUP))
                                return -EINVAL;
+                       vm_flags &= ~(VM_SHARED | VM_MAYSHARE);
                        break;
 
                default:
@@ -1368,6 +1369,7 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
                         * Set pgoff according to addr for anon_vma.
                         */
                        pgoff = addr >> PAGE_SHIFT;
+                       vm_flags &= ~(VM_SHARED | VM_MAYSHARE);
                        break;
                default:
                        return -EINVAL;

patch 2: 
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 650ee57..153ceff 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -423,7 +423,6 @@ static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map)
 static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags)
 {
        VM_BUG_ON(!is_vm_hugetlb_page(vma));
-       VM_BUG_ON(vma->vm_flags & VM_MAYSHARE);
 
        set_vma_private_data(vma, get_vma_private_data(vma) | flags);
 }

Notice:
If you used low level kernel version, you may fix other bugs and you will find this problem.

Reference：
commit 4998a6c0edce7fae9c0a5463f6ec3fa585258ee7
commit 66aebce747eaf9bc456bf1f1b217d8db843031d0
commit f12d5bfceb7e1f9051563381ec047f7f13956c3c