Bug 206569 - BUG at mm/filemap.c:2529: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)
Summary: BUG at mm/filemap.c:2529: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)
Status: NEW
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-02-17 13:16 UTC by Denis Lisov
Modified: 2020-10-11 20:21 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.5.3
Tree: Mainline
Regression: No


Attachments

Description Denis Lisov 2020-02-17 13:16:10 UTC
Gentoo Linux, amd64. Vanilla 5.5.3 kernel, custom config (might be weird - it has been updated with `make oldconfig` since 3.18, usually jumping a few mainline versions at a time).

After the last big kernel update (from kernel 5.0.17 to 5.5.3) I've had multiple BUGs with messages in dmesg like the following set (including some additional context):


page:ffffc9b318af02c0 refcount:0 mapcount:3 mapping:dead000000000400 index:0x60b compound_mapcount: 3                                                                                                                                                                         
ext4_da_aops name:"cc1plus"                                                                                                                                                                                                                                                   
flags: 0x8000000000000000()                                                                                                                                                                                                                                                   
raw: 8000000000000000 ffffc9b318af0001 ffffc9b318af02c8 dead000000000400                                                                                                                                                                                                      
raw: 0000000000000000 0000000000000000 0000000000000002 0000000000000000                                                                                                                                                                                                      
page dumped because: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)                                                                                                                                                                                                            
------------[ cut here ]------------                                                                                                                                                                                                                                          
kernel BUG at mm/filemap.c:2529!                                                                                                                                                                                                                                              
invalid opcode: 0000 [#1] PREEMPT SMP PTI                                                                                                                                                                                                                                     
CPU: 2 PID: 62637 Comm: cc1plus Not tainted 5.5.4 #1                                                                                                                                                                                                                          
Hardware name: Gigabyte Technology Co., Ltd. Z97M-D3H/Z97M-D3H, BIOS F5 05/30/2014
RIP: 0010:filemap_fault+0x82e/0x9f0
Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3 e0 e9 c5 f8 ff ff 48 c7 c6 e0 2a 35 a2 4c 89 ff e8 e2 13 03 00 <0f> 0b 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
RSP: 0000:ffffa42d8a3dfcf8 EFLAGS: 00010246
RAX: 0000000000000042 RBX: ffffa42d8a3dfe28 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa23566f2
RBP: 000000000000020b R08: 00000000000003b0 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: ffff9fd323c9c1a0
R13: ffff9fd268395700 R14: ffff9fd323c9c030 R15: ffffc9b318af02c0
FS:  00007fbc1415aac0(0000) GS:ffff9fd40fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9f1c870000 CR3: 000000066821c003 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? preempt_count_add+0x63/0x90
 ? _raw_spin_lock+0xe/0x30
 ? alloc_set_pte+0x4c6/0x7b0
 ? filemap_map_pages+0x2ab/0x3d0
 ext4_filemap_fault+0x28/0x3a
 __do_fault+0x33/0x110
 __handle_mm_fault+0xda9/0x13e0
 handle_mm_fault+0xb3/0x1e0
 do_user_addr_fault+0x1de/0x440
 page_fault+0x2f/0x40
RIP: 0033:0x60b0c0
Code: Bad RIP value.
RSP: 002b:00007ffc6566f338 EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00007ffc6566f2f4 RDI: 00007ffc6566f340
RBP: 000000000000000d R08: 00007ffc6566f340 R09: 0000000000000007
R10: 0000000000000001 R11: 00000000034711d8 R12: 0000000000000000
R13: 00000000006bfee0 R14: 0000000000000000 R15: 0000000000705770
Modules linked in: x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass iTCO_wdt ppdev iTCO_vendor_support crct10dif_pclmul crc32c_intel snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi ghash_clmulni_intel snd_hda_intel pcspkr snd_intel_d
spcfg i2c_i801 radeon snd_hda_codec snd_hda_core ttm snd_pcm snd_timer r8169 mei_me realtek snd mei soundcore lpc_ich mfd_core parport_pc parport acpi_pad sch_fq_codel efivarfs
---[ end trace aa2fe94307da24fc ]---
RIP: 0010:filemap_fault+0x82e/0x9f0
Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3 e0 e9 c5 f8 ff ff 48 c7 c6 e0 2a 35 a2 4c 89 ff e8 e2 13 03 00 <0f> 0b 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
RSP: 0000:ffffa42d8a3dfcf8 EFLAGS: 00010246
RAX: 0000000000000042 RBX: ffffa42d8a3dfe28 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa23566f2
RBP: 000000000000020b R08: 00000000000003b0 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: ffff9fd323c9c1a0
R13: ffff9fd268395700 R14: ffff9fd323c9c030 R15: ffffc9b318af02c0
FS:  00007fbc1415aac0(0000) GS:ffff9fd40fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000060b096 CR3: 000000066821c003 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

These happened, in all cases, during compilation and resulted in some processes hanging, seemingly forever.

I'm not sure whether MM or ext4 is more likely to be the problem, so my apologies if I've chosen the wrong component.
Comment 1 Denis Lisov 2020-02-19 21:33:32 UTC
Upd. Double-checking my kernel settings I found that I've accidentally enabled experimental CONFIG_READ_ONLY_THP_FOR_FS - may be related to that.

In my new kernel build the problem failed to reproduce (I've disabled CONFIG_READ_ONLY_THP_FOR_FS, CONFIG_KSM and switched from CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS to CONFIG_TRANSPARENT_HUGEPAGE_MADVISE - sorry for multiple changes).
Comment 2 Andrew Morton 2020-02-19 22:46:38 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 19 Feb 2020 21:33:32 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=206569
> 
> --- Comment #1 from Denis Lisov (dennis.lissov@gmail.com) ---
> Upd. Double-checking my kernel settings I found that I've accidentally
> enabled
> experimental CONFIG_READ_ONLY_THP_FOR_FS - may be related to that.
> 
> In my new kernel build the problem failed to reproduce (I've disabled
> CONFIG_READ_ONLY_THP_FOR_FS, CONFIG_KSM and switched from
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS to CONFIG_TRANSPARENT_HUGEPAGE_MADVISE -
> sorry for multiple changes).

Thanks.

Song Liu, could you please take a look?
Comment 3 willy 2020-02-20 00:25:24 UTC
On Wed, Feb 19, 2020 at 02:46:35PM -0800, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 19 Feb 2020 21:33:32 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=206569
> > 
> > --- Comment #1 from Denis Lisov (dennis.lissov@gmail.com) ---
> > Upd. Double-checking my kernel settings I found that I've accidentally
> enabled
> > experimental CONFIG_READ_ONLY_THP_FOR_FS - may be related to that.
> > 
> > In my new kernel build the problem failed to reproduce (I've disabled
> > CONFIG_READ_ONLY_THP_FOR_FS, CONFIG_KSM and switched from
> > CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS to CONFIG_TRANSPARENT_HUGEPAGE_MADVISE -
> > sorry for multiple changes).
> 
> Thanks.
> 
> Song Liu, could you please take a look?

This is a compound page ... Dennis, are you willing to apply this patch
https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
and retest?  It will give us more information about what's going on.
Comment 4 Yang Shi 2020-02-20 00:37:31 UTC
On Wed, Feb 19, 2020 at 4:25 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Wed, Feb 19, 2020 at 02:46:35PM -0800, Andrew Morton wrote:
> >
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> >
> > On Wed, 19 Feb 2020 21:33:32 +0000 bugzilla-daemon@bugzilla.kernel.org
> wrote:
> >
> > > https://bugzilla.kernel.org/show_bug.cgi?id=206569
> > >
> > > --- Comment #1 from Denis Lisov (dennis.lissov@gmail.com) ---
> > > Upd. Double-checking my kernel settings I found that I've accidentally
> enabled
> > > experimental CONFIG_READ_ONLY_THP_FOR_FS - may be related to that.
> > >
> > > In my new kernel build the problem failed to reproduce (I've disabled
> > > CONFIG_READ_ONLY_THP_FOR_FS, CONFIG_KSM and switched from
> > > CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS to CONFIG_TRANSPARENT_HUGEPAGE_MADVISE
> -
> > > sorry for multiple changes).
> >
> > Thanks.
> >
> > Song Liu, could you please take a look?
>
> This is a compound page ... Dennis, are you willing to apply this patch
>
> https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
> and retest?  It will give us more information about what's going on.

page:ffffc9b318af02c0 refcount:0 mapcount:3 mapping:dead000000000400
index:0x60b compound_mapcount: 3

It looks like a tail page.


>
>
Comment 5 willy 2020-02-20 01:21:55 UTC
On Wed, Feb 19, 2020 at 04:37:11PM -0800, Yang Shi wrote:
> On Wed, Feb 19, 2020 at 4:25 PM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Wed, Feb 19, 2020 at 02:46:35PM -0800, Andrew Morton wrote:
> > >
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > >
> > > On Wed, 19 Feb 2020 21:33:32 +0000 bugzilla-daemon@bugzilla.kernel.org
> wrote:
> > >
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=206569
> > > >
> > > > --- Comment #1 from Denis Lisov (dennis.lissov@gmail.com) ---
> > > > Upd. Double-checking my kernel settings I found that I've accidentally
> enabled
> > > > experimental CONFIG_READ_ONLY_THP_FOR_FS - may be related to that.
> > > >
> > > > In my new kernel build the problem failed to reproduce (I've disabled
> > > > CONFIG_READ_ONLY_THP_FOR_FS, CONFIG_KSM and switched from
> > > > CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS to
> CONFIG_TRANSPARENT_HUGEPAGE_MADVISE -
> > > > sorry for multiple changes).
> > >
> > > Thanks.
> > >
> > > Song Liu, could you please take a look?
> >
> > This is a compound page ... Dennis, are you willing to apply this patch
> >
> https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
> > and retest?  It will give us more information about what's going on.
> 
> page:ffffc9b318af02c0 refcount:0 mapcount:3 mapping:dead000000000400
> index:0x60b compound_mapcount: 3
> 
> It looks like a tail page.

... yes, that's the point; that patch prints the information from the head
page when the page is a tail page.
Comment 6 Denis Lisov 2020-02-20 23:56:57 UTC
Hello,

First, I've tried to limit changes between a working build and a
broken one. The current findings are that:
CONFIG_KSM does not matter (the bug reproduces even without it)
Both CONFIG_READ_ONLY_THP_FOR_FS and
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS are needed to reproduce the bug (or
at least I haven't managed to reproduce it with any of them disabled
so far).

On Thu, Feb 20, 2020 at 3:25 AM Matthew Wilcox <willy@infradead.org> wrote:
> This is a compound page ... Dennis, are you willing to apply this patch
>
> https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
> and retest?  It will give us more information about what's going on.

Here's the new log. The patch was applied to 5.5.4 manually as it did
not apply automatically.

page:ffffd52adae60180 refcount:522 mapcount:0 mapping:00000000b5351a5d
index:0x406 head:ffffd52adae60000 order:9 compound_mapcount:0
ext4_da_aops name:"cc1plus"
flags: 0x8000000000000000()
raw: 8000000000000000 ffffd52adae60001 dead000000000122
dead000000000400
raw: 0000000000000001 0000000000000000 00000000ffffffff
0000000000000000
head: 8000000000010095 ffffd52ade75e848 ffff93b3c624f020
ffff93b387d7daf0
head: 0000000000000400 0000000000000000 0000020affffffff
ffff93b3bff56000
page dumped because: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)
------------[ cut here ]------------
kernel BUG at mm/filemap.c:2529!
invalid opcode: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 33773 Comm: cc1plus Not tainted 5.5.4-r1-bughunt #5
Hardware name: Gigabyte Technology Co., Ltd. Z97M-D3H/Z97M-D3H, BIOS
F5 05/30/2014
RIP: 0010:filemap_fault+0x82e/0x9f0
Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000809350 CR3: 00000007ac748003 CR4: 00000000001606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? preempt_count_add+0x63/0x90
 ? _raw_spin_lock+0xe/0x30
 ? alloc_set_pte+0x4c6/0x7b0
 ? call_function_single_interrupt+0xa/0x20
 ? filemap_map_pages+0x298/0x3b0
 ext4_filemap_fault+0x28/0x3a
 __do_fault+0x33/0xb0
 __handle_mm_fault+0xda9/0x13e0
 ? _raw_spin_unlock_irq+0x9/0x20
 handle_mm_fault+0xb3/0x1e0
 do_user_addr_fault+0x1de/0x440
 page_fault+0x2f/0x40
RIP: 0033:0x606c20
Code: Bad RIP value.
RSP: 002b:00007fff7a679d98 EFLAGS: 00010283
RAX: 000000000000000a RBX: 00007feb8e462540 RCX: 0000000000000001
RDX: 00007feb8e462540 RSI: 0000000006864625 RDI: 0000000000000000
RBP: 00007feb90cc4da8 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000ffffef05 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000006864625
Modules linked in: x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support ppdev
crct10dif_pclmul crc32c_intel ghash_clmulni_intel
snd_hda_codec_realtek snd_hda_codec_generic pcspkr snd_hda_codec_hdmi
i2c_i801 radeon snd_hda_intel snd_intel_dspcfg snd_hda_codec
snd_hda_core r8169 snd_pcm ttm realtek snd_timer mei_me snd mei
soundcore lpc_ich mfd_core parport_pc parport acpi_pad sch_fq_codel
efivarfs
---[ end trace 33b86223ecc3d3ab ]---
RIP: 0010:filemap_fault+0x82e/0x9f0
Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000606bf6 CR3: 00000007ac748003 CR4: 00000000001606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Comment 7 songliubraving 2020-02-21 01:26:07 UTC
> On Feb 20, 2020, at 3:56 PM, Denis Lisov <dennis.lissov@gmail.com> wrote:
> 
> Hello,
> 
> First, I've tried to limit changes between a working build and a
> broken one. The current findings are that:
> CONFIG_KSM does not matter (the bug reproduces even without it)
> Both CONFIG_READ_ONLY_THP_FOR_FS and
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS are needed to reproduce the bug (or
> at least I haven't managed to reproduce it with any of them disabled
> so far).

Sorry for the delayed reply. I am on vacation this week. 

CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is disabled in our production kernel, 
so I guess the bug is triggered by the combination of the two. 

Denis, how do you repro this issue? I haven't got luck on my vm. 

Thanks,
Song

> 
> On Thu, Feb 20, 2020 at 3:25 AM Matthew Wilcox <willy@infradead.org> wrote:
>> This is a compound page ... Dennis, are you willing to apply this patch
>>
>> https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
>> and retest?  It will give us more information about what's going on.
> 
> Here's the new log. The patch was applied to 5.5.4 manually as it did
> not apply automatically.
> 
> page:ffffd52adae60180 refcount:522 mapcount:0 mapping:00000000b5351a5d
> index:0x406 head:ffffd52adae60000 order:9 compound_mapcount:0
> ext4_da_aops name:"cc1plus"
> flags: 0x8000000000000000()
> raw: 8000000000000000 ffffd52adae60001 dead000000000122
> dead000000000400
> raw: 0000000000000001 0000000000000000 00000000ffffffff
> 0000000000000000
> head: 8000000000010095 ffffd52ade75e848 ffff93b3c624f020
> ffff93b387d7daf0
> head: 0000000000000400 0000000000000000 0000020affffffff
> ffff93b3bff56000
> page dumped because: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)
> ------------[ cut here ]------------
> kernel BUG at mm/filemap.c:2529!
> invalid opcode: 0000 [#1] PREEMPT SMP PTI
> CPU: 0 PID: 33773 Comm: cc1plus Not tainted 5.5.4-r1-bughunt #5
> Hardware name: Gigabyte Technology Co., Ltd. Z97M-D3H/Z97M-D3H, BIOS
> F5 05/30/2014
> RIP: 0010:filemap_fault+0x82e/0x9f0
> Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
> e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
> 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
> RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
> RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
> RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
> R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
> FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000809350 CR3: 00000007ac748003 CR4: 00000000001606f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> ? preempt_count_add+0x63/0x90
> ? _raw_spin_lock+0xe/0x30
> ? alloc_set_pte+0x4c6/0x7b0
> ? call_function_single_interrupt+0xa/0x20
> ? filemap_map_pages+0x298/0x3b0
> ext4_filemap_fault+0x28/0x3a
> __do_fault+0x33/0xb0
> __handle_mm_fault+0xda9/0x13e0
> ? _raw_spin_unlock_irq+0x9/0x20
> handle_mm_fault+0xb3/0x1e0
> do_user_addr_fault+0x1de/0x440
> page_fault+0x2f/0x40
> RIP: 0033:0x606c20
> Code: Bad RIP value.
> RSP: 002b:00007fff7a679d98 EFLAGS: 00010283
> RAX: 000000000000000a RBX: 00007feb8e462540 RCX: 0000000000000001
> RDX: 00007feb8e462540 RSI: 0000000006864625 RDI: 0000000000000000
> RBP: 00007feb90cc4da8 R08: 0000000000000000 R09: 0000000000000000
> R10: 00000000ffffef05 R11: 0000000000000000 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000006864625
> Modules linked in: x86_pkg_temp_thermal intel_powerclamp coretemp
> kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support ppdev
> crct10dif_pclmul crc32c_intel ghash_clmulni_intel
> snd_hda_codec_realtek snd_hda_codec_generic pcspkr snd_hda_codec_hdmi
> i2c_i801 radeon snd_hda_intel snd_intel_dspcfg snd_hda_codec
> snd_hda_core r8169 snd_pcm ttm realtek snd_timer mei_me snd mei
> soundcore lpc_ich mfd_core parport_pc parport acpi_pad sch_fq_codel
> efivarfs
> ---[ end trace 33b86223ecc3d3ab ]---
> RIP: 0010:filemap_fault+0x82e/0x9f0
> Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
> e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
> 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
> RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
> RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
> RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
> R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
> FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000606bf6 CR3: 00000007ac748003 CR4: 00000000001606f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Comment 8 Denis Lisov 2020-02-21 08:58:11 UTC
The reproducer I've been using was to start mass parallel rebuild of
binary packages with

/usr/bin/emerge -1 --buildpkgonly --ask=n kde-plasma/bluedevil
kde-plasma/breeze kde-plasma/breeze-gtk kde-plasma/drkonqi
kde-plasma/kactivitymanagerd kde-plasma/kde-cli-tools
kde-plasma/kde-gtk-config kde-plasma/kdecoration
kde-plasma/kdeplasma-addons kde-plasma/kgamma kde-plasma/khotkeys
kde-plasma/kinfocenter kde-plasma/kmenuedit kde-plasma/kscreen
kde-plasma/kscreenlocker kde-plasma/ksshaskpass kde-plasma/ksysguard
kde-plasma/kwallet-pam kde-plasma/kwayland-integration kde-plasma/kwin
kde-plasma/kwrited kde-plasma/libkscreen kde-plasma/libksysguard
kde-plasma/libkworkspace kde-plasma/milou kde-plasma/oxygen
kde-plasma/plasma-browser-integration kde-plasma/plasma-desktop
kde-plasma/plasma-integration kde-plasma/plasma-meta
kde-plasma/plasma-vault kde-plasma/plasma-workspace
kde-plasma/plasma-workspace-wallpapers kde-plasma/polkit-kde-agent
kde-plasma/powerdevil kde-plasma/sddm-kcm kde-plasma/systemsettings
kde-plasma/user-manager kde-plasma/xdg-desktop-portal-kde
kde-plasma/xembed-sni-proxy

and the bug usually reproduces in 5-10 min after starting this. I've
also seen it reproduce during a plain kernel build with `make clean &&
make -j8`, but less reliably (the build could accidentally pass).

I'll see if I can make a self-contained reproducer in some way.

Denis Lisov.

On Fri, Feb 21, 2020 at 4:26 AM Song Liu <songliubraving@fb.com> wrote:
>
>
>
> > On Feb 20, 2020, at 3:56 PM, Denis Lisov <dennis.lissov@gmail.com> wrote:
> >
> > Hello,
> >
> > First, I've tried to limit changes between a working build and a
> > broken one. The current findings are that:
> > CONFIG_KSM does not matter (the bug reproduces even without it)
> > Both CONFIG_READ_ONLY_THP_FOR_FS and
> > CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS are needed to reproduce the bug (or
> > at least I haven't managed to reproduce it with any of them disabled
> > so far).
>
> Sorry for the delayed reply. I am on vacation this week.
>
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is disabled in our production kernel,
> so I guess the bug is triggered by the combination of the two.
>
> Denis, how do you repro this issue? I haven't got luck on my vm.
>
> Thanks,
> Song
>
> >
> > On Thu, Feb 20, 2020 at 3:25 AM Matthew Wilcox <willy@infradead.org> wrote:
> >> This is a compound page ... Dennis, are you willing to apply this patch
> >>
> https://lore.kernel.org/linux-mm/20200211001536.1027652-12-jhubbard@nvidia.com/
> >> and retest?  It will give us more information about what's going on.
> >
> > Here's the new log. The patch was applied to 5.5.4 manually as it did
> > not apply automatically.
> >
> > page:ffffd52adae60180 refcount:522 mapcount:0 mapping:00000000b5351a5d
> > index:0x406 head:ffffd52adae60000 order:9 compound_mapcount:0
> > ext4_da_aops name:"cc1plus"
> > flags: 0x8000000000000000()
> > raw: 8000000000000000 ffffd52adae60001 dead000000000122
> > dead000000000400
> > raw: 0000000000000001 0000000000000000 00000000ffffffff
> > 0000000000000000
> > head: 8000000000010095 ffffd52ade75e848 ffff93b3c624f020
> > ffff93b387d7daf0
> > head: 0000000000000400 0000000000000000 0000020affffffff
> > ffff93b3bff56000
> > page dumped because: VM_BUG_ON_PAGE(page_to_pgoff(page) != offset)
> > ------------[ cut here ]------------
> > kernel BUG at mm/filemap.c:2529!
> > invalid opcode: 0000 [#1] PREEMPT SMP PTI
> > CPU: 0 PID: 33773 Comm: cc1plus Not tainted 5.5.4-r1-bughunt #5
> > Hardware name: Gigabyte Technology Co., Ltd. Z97M-D3H/Z97M-D3H, BIOS
> > F5 05/30/2014
> > RIP: 0010:filemap_fault+0x82e/0x9f0
> > Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
> > e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
> > 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
> > RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
> > RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
> > RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
> > R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
> > R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
> > FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000)
> knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000809350 CR3: 00000007ac748003 CR4: 00000000001606f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > Call Trace:
> > ? preempt_count_add+0x63/0x90
> > ? _raw_spin_lock+0xe/0x30
> > ? alloc_set_pte+0x4c6/0x7b0
> > ? call_function_single_interrupt+0xa/0x20
> > ? filemap_map_pages+0x298/0x3b0
> > ext4_filemap_fault+0x28/0x3a
> > __do_fault+0x33/0xb0
> > __handle_mm_fault+0xda9/0x13e0
> > ? _raw_spin_unlock_irq+0x9/0x20
> > handle_mm_fault+0xb3/0x1e0
> > do_user_addr_fault+0x1de/0x440
> > page_fault+0x2f/0x40
> > RIP: 0033:0x606c20
> > Code: Bad RIP value.
> > RSP: 002b:00007fff7a679d98 EFLAGS: 00010283
> > RAX: 000000000000000a RBX: 00007feb8e462540 RCX: 0000000000000001
> > RDX: 00007feb8e462540 RSI: 0000000006864625 RDI: 0000000000000000
> > RBP: 00007feb90cc4da8 R08: 0000000000000000 R09: 0000000000000000
> > R10: 00000000ffffef05 R11: 0000000000000000 R12: 0000000000000000
> > R13: 0000000000000000 R14: 0000000000000001 R15: 0000000006864625
> > Modules linked in: x86_pkg_temp_thermal intel_powerclamp coretemp
> > kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support ppdev
> > crct10dif_pclmul crc32c_intel ghash_clmulni_intel
> > snd_hda_codec_realtek snd_hda_codec_generic pcspkr snd_hda_codec_hdmi
> > i2c_i801 radeon snd_hda_intel snd_intel_dspcfg snd_hda_codec
> > snd_hda_core r8169 snd_pcm ttm realtek snd_timer mei_me snd mei
> > soundcore lpc_ich mfd_core parport_pc parport acpi_pad sch_fq_codel
> > efivarfs
> > ---[ end trace 33b86223ecc3d3ab ]---
> > RIP: 0010:filemap_fault+0x82e/0x9f0
> > Code: 47 20 f7 c2 00 00 01 00 0f 84 d2 f8 ff ff 41 0f b6 4f 51 48 d3
> > e0 e9 c5 f8 ff ff 48 c7 c6 a0 15 35 98 4c 89 ff e8 52 14 03 00 <0f> 0b
> > 45 31 ed e9 58 f8 ff ff 49 8b 57 08 48 8d 42 ff 83 e2 01 49
> > RSP: 0000:ffff9d3b8c163cf8 EFLAGS: 00010286
> > RAX: 0000000000000042 RBX: ffff9d3b8c163e28 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000ffffffff
> > RBP: 0000000000000206 R08: 00000000000003b1 R09: 0000000000000004
> > R10: 0000000000000000 R11: 0000000000000001 R12: ffff93b387d7daf0
> > R13: ffff93b36fcaf200 R14: ffff93b387d7d980 R15: ffffd52adae60180
> > FS:  00007feb915e4ac0(0000) GS:ffff93b3cfe00000(0000)
> knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000606bf6 CR3: 00000007ac748003 CR4: 00000000001606f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>
Comment 9 songliubraving 2020-02-26 22:45:03 UTC
On Feb 21, 2020, at 12:57 AM, Denis Lisov <dennis.lissov@gmail.com> wrote:
> 
> The reproducer I've been using was to start mass parallel rebuild of
> binary packages with
> 
> /usr/bin/emerge -1 --buildpkgonly --ask=n kde-plasma/bluedevil
> kde-plasma/breeze kde-plasma/breeze-gtk kde-plasma/drkonqi
> kde-plasma/kactivitymanagerd kde-plasma/kde-cli-tools
> kde-plasma/kde-gtk-config kde-plasma/kdecoration
> kde-plasma/kdeplasma-addons kde-plasma/kgamma kde-plasma/khotkeys
> kde-plasma/kinfocenter kde-plasma/kmenuedit kde-plasma/kscreen
> kde-plasma/kscreenlocker kde-plasma/ksshaskpass kde-plasma/ksysguard
> kde-plasma/kwallet-pam kde-plasma/kwayland-integration kde-plasma/kwin
> kde-plasma/kwrited kde-plasma/libkscreen kde-plasma/libksysguard
> kde-plasma/libkworkspace kde-plasma/milou kde-plasma/oxygen
> kde-plasma/plasma-browser-integration kde-plasma/plasma-desktop
> kde-plasma/plasma-integration kde-plasma/plasma-meta
> kde-plasma/plasma-vault kde-plasma/plasma-workspace
> kde-plasma/plasma-workspace-wallpapers kde-plasma/polkit-kde-agent
> kde-plasma/powerdevil kde-plasma/sddm-kcm kde-plasma/systemsettings
> kde-plasma/user-manager kde-plasma/xdg-desktop-portal-kde
> kde-plasma/xembed-sni-proxy
> 
> and the bug usually reproduces in 5-10 min after starting this. I've
> also seen it reproduce during a plain kernel build with `make clean &&
> make -j8`, but less reliably (the build could accidentally pass).
> 
> I'll see if I can make a self-contained reproducer in some way.
> 
> Denis Lisov.

Some updates:

On my VM, I saw khugepaged put .text of python binary in 2MB pages. 
This is caused by khugepaged_always() == true in hugepage_vma_check(). 
I guess this means the bug exists w/o CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS, 
but CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS makes it easier to trigger. 

However, I haven't got luck to repro this issue. 

Denis, what is the version of gcc are you using to repro this issue?
Also, could you please try the repro with he following patch:

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b679908743cb..5b78590d4d6d 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1718,6 +1718,10 @@ static void collapse_file(struct mm_struct *mm,
        else {
                __inc_node_page_state(new_page, NR_FILE_THPS);
                filemap_nr_thps_inc(mapping);
+               pr_info("%s ino %lu, d_iname %s comm %s\n", __func__,
+                       mapping->host->i_ino,
+                       file->f_path.dentry->d_iname,
+                       mm->owner->comm);
        }

        if (nr_none) {

Thanks,
Song
Comment 10 willy 2020-05-01 18:05:20 UTC
On Wed, Feb 26, 2020 at 10:44:29PM +0000, Song Liu wrote:
> On my VM, I saw khugepaged put .text of python binary in 2MB pages. 
> This is caused by khugepaged_always() == true in hugepage_vma_check(). 
> I guess this means the bug exists w/o CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS, 
> but CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS makes it easier to trigger. 
> 
> However, I haven't got luck to repro this issue. 

I fixed something as part of the large pages work which might have also
fixed this problem.  However, you have CONFIG_DEBUG_VM enabled, so it
would have actually triggered the bug.

http://git.infradead.org/users/willy/linux-dax.git/commitdiff/2f864b6c27caa737ccfe65a86972f6b73a66a095

I don't think anything else I've changed will fix this bug, whatever it is ;-(
Comment 11 songliubraving 2020-05-19 00:52:52 UTC
> On May 1, 2020, at 11:05 AM, Matthew Wilcox <willy@infradead.org> wrote:
> 
> On Wed, Feb 26, 2020 at 10:44:29PM +0000, Song Liu wrote:
>> On my VM, I saw khugepaged put .text of python binary in 2MB pages. 
>> This is caused by khugepaged_always() == true in hugepage_vma_check(). 
>> I guess this means the bug exists w/o CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS, 
>> but CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS makes it easier to trigger. 
>> 
>> However, I haven't got luck to repro this issue. 
> 
> I fixed something as part of the large pages work which might have also
> fixed this problem.  However, you have CONFIG_DEBUG_VM enabled, so it
> would have actually triggered the bug.
> 
>
> http://git.infradead.org/users/willy/linux-dax.git/commitdiff/2f864b6c27caa737ccfe65a86972f6b73a66a095
> 
> I don't think anything else I've changed will fix this bug, whatever it is
> ;-(

I am testing with 5.5.3. However, I still got no luck here. :(

I haven't figured out how to build KDE in Gentoo. I tried to build kernel, 
many many times, without repro of the issue. 

Denis, could you please share some information about your test system:

    1. # of CPUs;
    2. size of RAM;
    3. size of swap, if any; 
    4. .config file of the kernel that repros the issue. 

Thanks,
Song
Comment 12 Matthew Wilcox 2020-08-24 02:02:12 UTC
I think I have a fix for this problem.  Denis, are you willing to apply these three patches and re-enable the config option?

http://git.infradead.org/users/willy/pagecache.git/shortlog/refs/heads/for-5.9
Comment 13 Hugh Dickins 2020-10-11 20:21:28 UTC
Adding a note to help update bugzilla 206569: the v5.9 tree now has
033b5d775511 ("mm/khugepaged: fix filemap page_to_pgoff(page) != offset")
which fixes this VM_BUG_ON_PAGE (or data corruption) when CONFIG_NUMA
is not set: we are hoping that may have have been the problem here.

Note You need to log in before you can comment on or make changes to this bug.