Bug 204165 - 100% CPU usage in compact_zone
Summary: 100% CPU usage in compact_zone
Status: NEW
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Page Allocator (show other bugs)
Hardware: x86-64 Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-13 19:20 UTC by GYt2bW
Modified: 2019-07-19 22:13 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.2.0-g0ecfebd2b524
Tree: Mainline
Regression: No


Attachments
zcat-ed kernel .config from /proc (127.67 KB, text/plain)
2019-07-13 20:05 UTC, GYt2bW
Details
publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc (3.15 KB, application/pgp-keys)
2019-07-16 01:08 UTC, howaboutsynergy
Details
signature.asc (855 bytes, application/pgp-signature)
2019-07-16 01:08 UTC, howaboutsynergy
Details
publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc (3.15 KB, application/pgp-keys)
2019-07-16 19:15 UTC, howaboutsynergy
Details
signature.asc (855 bytes, application/pgp-signature)
2019-07-16 19:15 UTC, howaboutsynergy
Details

Description GYt2bW 2019-07-13 19:20:21 UTC
```
[ 1319.078876] CPU: 8 PID: 25800 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078880] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078882] RIP: 0010:compact_zone+0x587/0xa70
[ 1319.078887] Code: ff 4c 8b 34 24 49 89 ef 8b 4c 24 50 4c 89 e2 4c 89 f6 48 89 ef e8 c9 e8 ff ff 49 89 c4 48 85 c0 0f 84 c3 02 00 00 4d 89 67 30 <41> 8b 77 24 85 f6 0f 85 9f 00 00 00 80 7c 24 56 00 0f 84 c3 02 00
[ 1319.078896] RSP: 0018:ffffb2d88bd23a18 EFLAGS: 00000202
[ 1319.078905] RAX: 000000000016b420 RBX: 000000000016b800 RCX: 0000000000000000
[ 1319.078908] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 1319.078915] RBP: ffffb2d88bd23ad0 R08: 0000000000000001 R09: ffff99f62da21450
[ 1319.078923] R10: 0000000000000000 R11: ffff99f62da20a80 R12: 000000000016b420
[ 1319.078930] R13: ffff99f64dfded00 R14: 000000000016b420 R15: ffffb2d88bd23ad0
[ 1319.078936] FS:  0000720b85407740(0000) GS:ffff99f62da00000(0000) knlGS:0000000000000000
[ 1319.078940] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.078944] CR2: 00007ecf94de9578 CR3: 00000007ec28a006 CR4: 00000000003606e0
[ 1319.078952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.078961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.078964] Call Trace:
[ 1319.078969]  compact_zone_order+0xde/0x120
[ 1319.078974]  try_to_compact_pages+0x187/0x240
[ 1319.078978]  __alloc_pages_direct_compact+0x87/0x170
[ 1319.078985]  __alloc_pages_slowpath+0x454/0xc20
[ 1319.078993]  ? release_pages+0x348/0x3b0
[ 1319.078997]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1319.079003]  __alloc_pages_nodemask+0x268/0x2b0
[ 1319.079007]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1319.079012]  __handle_mm_fault+0xc11/0x1890
[ 1319.079017]  handle_mm_fault+0xa9/0x1d0
[ 1319.079030]  __do_page_fault+0x237/0x480
[ 1319.079034]  do_page_fault+0x1d/0x67
[ 1319.079041]  ? page_fault+0x8/0x30
[ 1319.079047]  page_fault+0x1e/0x30
[ 1319.079052] RIP: 0033:0x5c2e1a191c10
[ 1319.079055] Code: Bad RIP value.
[ 1319.079060] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1319.079074] RAX: 00000000183a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1319.079079] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1319.079084] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1319.079088] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1319.079093] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
```

This process is using 100% CPU and you can't gdb attach to it and can't kill -9 it (even from root).

I've actually 5 or 6 of these processes stuck like this. This only happens sometimes!
And I'm not sure if I've only seen it do this with the `stress` process.
Comment 1 GYt2bW 2019-07-13 20:03:47 UTC
```
  PID  %CPU COMMAND                                                                            PR  NI    VIRT    RES S USER     
25677 100.0 stress                                                                             20   0  980352 116676 R user     
25752 100.0 stress                                                                             20   0  980352 122820 R user     
25800 100.0 stress                                                                             20   0  980352      0 R user     
25727  98.1 stress                                                                             20   0  980352  96196 R user     
25777  98.1 stress                                                                             20   0  980352 124868 R user     
 
```

```
[ 1044.371191] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(V) show-blocked-tasks(w) 
[ 1073.867794] sysrq: Show Blocked State
[ 1073.867806]   task                        PC stack   pid father
[ 1094.149119] sysrq: Show backtrace of all active CPUs
[ 1094.149129] NMI backtrace for cpu 8
[ 1094.149138] CPU: 8 PID: 14722 Comm: bash Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.149142] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.149144] Call Trace:
[ 1094.149157]  dump_stack+0x46/0x60
[ 1094.149163]  nmi_cpu_backtrace.cold+0x14/0x53
[ 1094.149170]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 1094.149174]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 1094.149181]  __handle_sysrq.cold+0x48/0x102
[ 1094.149187]  write_sysrq_trigger+0x1f/0x30
[ 1094.149193]  proc_reg_write+0x37/0x60
[ 1094.149201]  vfs_write+0xb1/0x190
[ 1094.149207]  ksys_write+0x5d/0xe0
[ 1094.149213]  do_syscall_64+0x50/0x170
[ 1094.149220]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1094.149225] RIP: 0033:0x79787f47a4ba
[ 1094.149230] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20
[ 1094.149234] RSP: 002b:00007ffc61be1698 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 1094.149240] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000079787f47a4ba
[ 1094.149242] RDX: 0000000000000002 RSI: 000055f82dde0ee0 RDI: 0000000000000001
[ 1094.149245] RBP: 00007ffc61be16c0 R08: 000000000000000a R09: 0000000000000001
[ 1094.149248] R10: 000055f82d3c305e R11: 0000000000000246 R12: 0000000000000002
[ 1094.149251] R13: 000055f82dde0ee0 R14: 000079787f54f500 R15: 000079787f550300
[ 1094.149256] Sending NMI from CPU 8 to CPUs 0-7,9-11:
[ 1094.149387] NMI backtrace for cpu 0
[ 1094.149389] CPU: 0 PID: 25727 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.149391] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.149392] RIP: 0010:isolate_migratepages_block+0x3b/0xa50
[ 1094.149394] Code: 49 89 fe 48 89 f3 48 89 54 24 08 89 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 c0 48 8b 47 40 48 8b 68 50 <48> 89 6c 24 20 48 c7 44 24 78 00 00 00 00 c6 44 24 77 00 65 48 8b
[ 1094.149395] RSP: 0018:ffffb2d88badb958 EFLAGS: 00000246
[ 1094.149397] RAX: ffff99f64dfded00 RBX: 000000000058fcc0 RCX: 000000000000000c
[ 1094.149398] RDX: 000000000058fe00 RSI: 000000000058fcc0 RDI: ffffb2d88badbad0
[ 1094.149399] RBP: ffff99f64dfde000 R08: ffff99f64dfd9b10 R09: ffff99f62d821450
[ 1094.149400] R10: 0000000000000001 R11: ffff99f62d820a80 R12: 000000000058fe00
[ 1094.149401] R13: ffff99f64dfded00 R14: ffffb2d88badbad0 R15: ffffb2d88badbad0
[ 1094.149402] FS:  0000720b85407740(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000
[ 1094.149403] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.149404] CR2: 0000631d9a784fe0 CR3: 00000007c23a4003 CR4: 00000000003606f0
[ 1094.149405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.149406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.149406] Call Trace:
[ 1094.149407]  compact_zone+0x577/0xa70
[ 1094.149408]  compact_zone_order+0xde/0x120
[ 1094.149409]  try_to_compact_pages+0x187/0x240
[ 1094.149409]  __alloc_pages_direct_compact+0x87/0x170
[ 1094.149410]  __alloc_pages_slowpath+0x20e/0xc20
[ 1094.149411]  ? release_pages+0x348/0x3b0
[ 1094.149412]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1094.149413]  __alloc_pages_nodemask+0x268/0x2b0
[ 1094.149413]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1094.149414]  __handle_mm_fault+0xc11/0x1890
[ 1094.149415]  handle_mm_fault+0xa9/0x1d0
[ 1094.149416]  __do_page_fault+0x237/0x480
[ 1094.149416]  do_page_fault+0x1d/0x67
[ 1094.149417]  ? page_fault+0x8/0x30
[ 1094.149418]  page_fault+0x1e/0x30
[ 1094.149419] RIP: 0033:0x5c2e1a191c10
[ 1094.149419] Code: Bad RIP value.
[ 1094.149420] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1094.149422] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1094.149423] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1094.149424] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1094.149425] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1094.149426] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1094.149478] NMI backtrace for cpu 2 skipped: idling at intel_idle+0x7d/0x120
[ 1094.149483] NMI backtrace for cpu 1
[ 1094.149487] CPU: 1 PID: 25677 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.149490] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.149493] RIP: 0010:compact_zone+0x499/0xa70
[ 1094.149497] Code: f6 48 2b 35 29 7e d1 00 48 c1 fe 06 b9 01 00 00 00 ba 03 00 00 00 4c 89 f7 e8 93 10 02 00 48 85 c0 0f 85 93 00 00 00 4c 89 f7 <e8> 62 d8 ff ff 84 c0 75 56 44 8b 45 70 45 85 c0 0f 85 af 00 00 00
[ 1094.149501] RSP: 0018:ffffb2d88b94ba18 EFLAGS: 00000202
[ 1094.149504] RAX: 0000000000718420 RBX: 0000000000718800 RCX: 00000000000038c2
[ 1094.149507] RDX: 0000000000100000 RSI: 000000000000000c RDI: ffffde0b9c610000
[ 1094.149511] RBP: ffffb2d88b94bad0 R08: 0000000000000001 R09: ffff99f62d861450
[ 1094.149525] R10: 0000000000000001 R11: ffff99f62d860a80 R12: 0000000000718600
[ 1094.149538] R13: ffff99f64dfded00 R14: ffffde0b9c610000 R15: 0000000000000000
[ 1094.149541] FS:  0000720b85407740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000
[ 1094.149544] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.149547] CR2: 0000631d9a784fe0 CR3: 00000007c8b04001 CR4: 00000000003606e0
[ 1094.149550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.149552] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.149555] Call Trace:
[ 1094.149558]  compact_zone_order+0xde/0x120
[ 1094.149560]  try_to_compact_pages+0x187/0x240
[ 1094.149563]  __alloc_pages_direct_compact+0x87/0x170
[ 1094.149569]  __alloc_pages_slowpath+0x454/0xc20
[ 1094.149575]  ? release_pages+0x348/0x3b0
[ 1094.149577]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1094.149589]  __alloc_pages_nodemask+0x268/0x2b0
[ 1094.149600]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1094.149606]  __handle_mm_fault+0xc11/0x1890
[ 1094.149609]  handle_mm_fault+0xa9/0x1d0
[ 1094.149612]  __do_page_fault+0x237/0x480
[ 1094.149615]  do_page_fault+0x1d/0x67
[ 1094.149620]  ? page_fault+0x8/0x30
[ 1094.149622]  page_fault+0x1e/0x30
[ 1094.149625] RIP: 0033:0x5c2e1a191c10
[ 1094.149630] Code: Bad RIP value.
[ 1094.149636] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1094.149645] RAX: 00000000071a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1094.149650] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1094.149654] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1094.149657] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1094.149663] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1094.149704] NMI backtrace for cpu 4
[ 1094.149708] CPU: 4 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.149710] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.149712] RIP: 0033:0x571ede0e4420
[ 1094.149716] Code: 44 24 2c 48 89 44 24 10 48 85 f6 0f 84 97 00 00 00 49 89 cc 31 db 4c 8d 6c 24 30 4c 8d 35 59 6c 00 00 0f 1f 84 00 00 00 00 00 <48> 8b 44 24 08 48 8d 2c 18 48 8b 14 24 48 29 da 4c 89 e9 48 89 ee
[ 1094.149720] RSP: 002b:00007ffe25d18460 EFLAGS: 00000202
[ 1094.149726] RAX: 0000000000000001 RBX: 0000000000000046 RCX: 0000000000000000
[ 1094.149728] RDX: 0000000000000001 RSI: 0000571ede0f049a RDI: 00007c7ea567b320
[ 1094.149729] RBP: 0000571ede0f049a R08: 0000000000000001 R09: 0000000000000000
[ 1094.149731] R10: 00007ffe25d1848c R11: 0000571ede0f049a R12: 00007c7ea5679500
[ 1094.149735] R13: 00007ffe25d18490 R14: 0000571ede0eb071 R15: 0000000000000001
[ 1094.149741] FS:  00007c7ea548a740 GS:  0000000000000000
[ 1094.149752] NMI backtrace for cpu 3
[ 1094.149758] CPU: 3 PID: 25800 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.149767] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.149772] RIP: 0010:isolate_migratepages_block+0x1c/0xa50
[ 1094.149777] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 <89> 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31
[ 1094.149783] RSP: 0018:ffffb2d88bd23958 EFLAGS: 00000292
[ 1094.149792] RAX: 0000000000000000 RBX: 000000000016b420 RCX: 000000000000000c
[ 1094.149800] RDX: 000000000016b600 RSI: 000000000016b420 RDI: ffffb2d88bd23ad0
[ 1094.149803] RBP: ffffb2d88bd23ad0 R08: 0000000000000001 R09: ffff99f62d8e1450
[ 1094.149808] R10: 0000000000000001 R11: ffff99f62d8e0a80 R12: 000000000016b600
[ 1094.149816] R13: ffff99f64dfded00 R14: ffffb2d88bd23ad0 R15: ffffb2d88bd23ad0
[ 1094.149822] FS:  0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000
[ 1094.149825] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.149831] CR2: 00007210ba3fc578 CR3: 00000007ec28a005 CR4: 00000000003606e0
[ 1094.149838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.149842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.149848] Call Trace:
[ 1094.149853]  compact_zone+0x577/0xa70
[ 1094.149859]  compact_zone_order+0xde/0x120
[ 1094.149863]  try_to_compact_pages+0x187/0x240
[ 1094.149868]  __alloc_pages_direct_compact+0x87/0x170
[ 1094.149871]  __alloc_pages_slowpath+0x454/0xc20
[ 1094.149875]  ? release_pages+0x348/0x3b0
[ 1094.149878]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1094.149883]  __alloc_pages_nodemask+0x268/0x2b0
[ 1094.149889]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1094.149894]  __handle_mm_fault+0xc11/0x1890
[ 1094.149897]  handle_mm_fault+0xa9/0x1d0
[ 1094.149900]  __do_page_fault+0x237/0x480
[ 1094.149905]  do_page_fault+0x1d/0x67
[ 1094.149907]  ? page_fault+0x8/0x30
[ 1094.149910]  page_fault+0x1e/0x30
[ 1094.149915] RIP: 0033:0x5c2e1a191c10
[ 1094.149920] Code: Bad RIP value.
[ 1094.149923] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1094.149932] RAX: 00000000183a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1094.149935] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1094.149943] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1094.149946] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1094.149949] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1094.149954] NMI backtrace for cpu 5
[ 1094.149960] CPU: 5 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150093] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150099] RIP: 0033:0x77d47d3fd585
[ 1094.150104] Code: 8b 04 24 25 00 0a 00 00 3d 00 0a 00 00 74 66 49 8b 44 24 30 48 39 f8 76 7c 48 29 f8 48 89 c2 31 c9 48 85 d2 74 6f 48 89 4d c8 <48> 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24
[ 1094.150107] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000246
[ 1094.150114] RAX: 00005ffb5747045f RBX: 0000000000000001 RCX: 0000000000000000
[ 1094.150117] RDX: 0000000000000ff6 RSI: 00005ffb5747045f RDI: 00005ffb591e43ca
[ 1094.150121] RBP: 00007ffe06db5e30 R08: 00005ffb57470460 R09: 0000000000000000
[ 1094.150124] R10: 00007ffe06db5ebc R11: 00005ffb5747045f R12: 000077d47d554500
[ 1094.150127] R13: 00005ffb5747045f R14: 0000000000000000 R15: 00005ffb5747045f
[ 1094.150131] FS:  000077d47d365740 GS:  0000000000000000
[ 1094.150134] NMI backtrace for cpu 6
[ 1094.150139] CPU: 6 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150143] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150151] RIP: 0010:ahci_scr_read+0x41/0x60
[ 1094.150156] Code: 07 48 05 00 01 00 00 41 b8 ea ff ff ff 83 fe 04 77 1d 74 1f 89 f6 48 63 0c b5 f0 84 cc a2 85 c9 74 0d 48 01 c8 48 01 f8 8b 00 <89> 02 45 31 c0 44 89 c0 c3 f6 41 1b 20 74 f6 b9 3c 00 00 00 eb e2
[ 1094.150159] RSP: 0000:ffffb2d883340e60 EFLAGS: 00000082
[ 1094.150171] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 000000000000003c
[ 1094.150174] RDX: ffffb2d883340e6c RSI: 0000000000000004 RDI: ffffb2d883189000
[ 1094.150180] RBP: ffff99f628f28000 R08: 00000000ffffffea R09: 0000000000000000
[ 1094.150184] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f628f2a040
[ 1094.150190] R13: ffffb2d883189180 R14: ffff99f628f2a040 R15: ffff99f628f22018
[ 1094.150196] FS:  00007c32c37f6740(0000) GS:ffff99f62d980000(0000) knlGS:0000000000000000
[ 1094.150200] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.150204] CR2: 00007da5da76cb50 CR3: 000000077cd3c003 CR4: 00000000003606e0
[ 1094.150209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.150216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.150219] Call Trace:
[ 1094.150222]  <IRQ>
[ 1094.150226]  sata_async_notification+0x59/0x130
[ 1094.150231]  ahci_handle_port_interrupt+0xf2/0x590
[ 1094.150236]  ahci_handle_port_intr+0x52/0xa0
[ 1094.150240]  ahci_single_level_irq_intr+0x34/0x50
[ 1094.150243]  __handle_irq_event_percpu+0x2f/0xc0
[ 1094.150248]  handle_irq_event_percpu+0x2c/0x80
[ 1094.150255]  handle_irq_event+0x23/0x43
[ 1094.150258]  handle_edge_irq+0x78/0x190
[ 1094.150261]  handle_irq+0x17/0x20
[ 1094.150264]  do_IRQ+0x3e/0xd0
[ 1094.150269]  common_interrupt+0xf/0xf
[ 1094.150272]  </IRQ>
[ 1094.150275] RIP: 0033:0x58517a058690
[ 1094.150280] Code: 68 63 00 00 00 e9 b0 f9 ff ff ff 25 c2 dc 00 00 68 64 00 00 00 e9 a0 f9 ff ff ff 25 ba dc 00 00 68 65 00 00 00 e9 90 f9 ff ff <ff> 25 b2 dc 00 00 68 66 00 00 00 e9 80 f9 ff ff ff 25 aa dc 00 00
[ 1094.150283] RSP: 002b:00007ffe934c7338 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffde
[ 1094.150291] RAX: 0000000000000001 RBX: 0000000000000012 RCX: 00007c32c39e5500
[ 1094.150295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 000058517a066466
[ 1094.150300] RBP: 000058517a066466 R08: 000058517a066467 R09: 0000000000000000
[ 1094.150303] R10: 00007ffe934c736c R11: 000058517a066466 R12: 00007c32c39e5500
[ 1094.150306] R13: 00007ffe934c7370 R14: 000058517a061071 R15: 0000000000000001
[ 1094.150309] NMI backtrace for cpu 7
[ 1094.150313] CPU: 7 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150319] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150322] RIP: 0010:format_decode+0x91/0x3f0
[ 1094.150329] Code: 01 00 00 80 4d 04 40 48 89 c8 48 8d 48 01 48 89 0c 24 0f b6 50 01 80 fa 2b 0f 84 af 01 00 00 7e d0 80 fa 2d 0f 84 e9 01 00 00 <80> fa 30 0f 85 a4 01 00 00 80 4d 04 10 eb ce 29 d8 48 83 c4 08 5b
[ 1094.150331] RSP: 0018:ffffb2d88413fd40 EFLAGS: 00000016
[ 1094.150337] RAX: ffffffffa2e31775 RBX: ffffffffa2e31775 RCX: ffffffffa2e31776
[ 1094.150340] RDX: 000000000000006c RSI: ffffb2d88413fd70 RDI: ffffffffa2e31775
[ 1094.150343] RBP: ffffb2d88413fd70 R08: ffff99f5c714405f R09: 0000000000000004
[ 1094.150349] R10: 0000000000000004 R11: ffff99f5c714405f R12: ffffffffa2e31775
[ 1094.150355] R13: 0000000000002000 R14: ffffffffa2e31775 R15: ffff99f5c7144060
[ 1094.150358] FS:  000074e35af43740(0000) GS:ffff99f62d9c0000(0000) knlGS:0000000000000000
[ 1094.150365] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.150371] CR2: 00007b610f6c0020 CR3: 000000077cedc006 CR4: 00000000003606e0
[ 1094.150376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.150383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.150386] Call Trace:
[ 1094.150389]  vsnprintf+0x6e/0x5b0
[ 1094.150391]  scnprintf+0x4d/0x90
[ 1094.150394]  msg_print_ext_header.constprop.0+0x7f/0xa0
[ 1094.150397]  ? _cond_resched+0x10/0x20
[ 1094.150403]  ? mutex_lock_interruptible+0x9/0x30
[ 1094.150405]  devkmsg_read+0x17f/0x270
[ 1094.150408]  vfs_read+0x98/0x150
[ 1094.150410]  ksys_read+0x5d/0xe0
[ 1094.150413]  do_syscall_64+0x50/0x170
[ 1094.150418]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1094.150420] RIP: 0033:0x74e35b05d415
[ 1094.150427] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1094.150432] RSP: 002b:00007ffda01e7bc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1094.150438] RAX: ffffffffffffffda RBX: 00005d76c75ae440 RCX: 000074e35b05d415
[ 1094.150441] RDX: 0000000000001fff RSI: 00005d76c75ae440 RDI: 0000000000000003
[ 1094.150446] RBP: 00005d76c75aa7fa R08: 000000000000000a R09: 0000000000000000
[ 1094.150454] R10: 00007ffda01e79fc R11: 0000000000000246 R12: 0000000000000029
[ 1094.150457] R13: 00005d76c75a9377 R14: 00005d76c75ae440 R15: 00005d76c75ae455
[ 1094.150462] NMI backtrace for cpu 9
[ 1094.150466] CPU: 9 PID: 1254 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150469] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150474] RIP: 0010:do_syscall_64+0x1c/0x170
[ 1094.150484] Code: da fa ff ff eb 94 0f 1f 84 00 00 00 00 00 55 53 48 89 fb 48 89 f5 0f 1f 44 00 00 0f 1f 44 00 00 fb 65 48 8b 04 25 00 5d 01 00 <48> 8b 00 a9 c1 01 08 10 0f 85 f4 00 00 00 48 81 fb b1 01 00 00 77
[ 1094.150492] RSP: 0018:ffffb2d883f1ff40 EFLAGS: 00000246
[ 1094.150500] RAX: ffff99f57cc38000 RBX: 0000000000000000 RCX: 0000000000000000
[ 1094.150504] RDX: 0000000000000000 RSI: ffffb2d883f1ff58 RDI: 0000000000000000
[ 1094.150507] RBP: ffffb2d883f1ff58 R08: 0000000000000000 R09: 0000000000000000
[ 1094.150510] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1094.150513] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1094.150516] FS:  0000705c7f390740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000
[ 1094.150519] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.150524] CR2: 00007d2b869ac010 CR3: 00000007c71a2004 CR4: 00000000003606e0
[ 1094.150532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.150535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.150537] Call Trace:
[ 1094.150541]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1094.150544] RIP: 0033:0x705c7f6af415
[ 1094.150552] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1094.150557] RSP: 002b:00007ffc9e431508 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1094.150564] RAX: ffffffffffffffda RBX: 0000705c7f7837e0 RCX: 0000705c7f6af415
[ 1094.150570] RDX: 0000000000000001 RSI: 0000705c7f783863 RDI: 0000000000000000
[ 1094.150577] RBP: 00007ffc9e431550 R08: 0000000000000000 R09: 0000000000000000
[ 1094.150583] R10: 00005c354d1911f7 R11: 0000000000000246 R12: 0000705c7f785300
[ 1094.150586] R13: 0000705c7f784500 R14: 0000705c7f784700 R15: 0000705c7f785468
[ 1094.150589] NMI backtrace for cpu 10
[ 1094.150595] CPU: 10 PID: 25752 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150597] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150602] RIP: 0010:compact_zone+0x3a3/0xa70
[ 1094.150609] Code: 48 31 c2 48 f7 c2 00 fe ff ff 41 0f 94 47 7a 49 8b 6f 40 31 f6 44 8b 15 fb 6a ef 00 45 85 d2 40 0f 95 c6 c1 e6 03 89 74 24 50 <41> 83 7f 70 02 74 07 83 ce 04 89 74 24 50 45 0f b6 6f 74 48 8b 55
[ 1094.150614] RSP: 0018:ffffb2d88bba3a18 EFLAGS: 00000202
[ 1094.150620] RAX: 00000000004f3c60 RBX: 00000000004f4000 RCX: 000000000000279e
[ 1094.150622] RDX: 00000000004f3c60 RSI: 0000000000000008 RDI: ffff99f64dfffb00
[ 1094.150625] RBP: ffff99f64dfded00 R08: 0000000000000001 R09: ffff99f62daa1450
[ 1094.150627] R10: 0000000000000001 R11: ffff99f62daa0a80 R12: 0000000000000009
[ 1094.150630] R13: ffff99f64dfded00 R14: 00000000004f3c60 R15: ffffb2d88bba3ad0
[ 1094.150642] FS:  0000720b85407740(0000) GS:ffff99f62da80000(0000) knlGS:0000000000000000
[ 1094.150644] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.150647] CR2: 00007f962a63c020 CR3: 000000081439e005 CR4: 00000000003606e0
[ 1094.150649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.150653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.150658] Call Trace:
[ 1094.150662]  compact_zone_order+0xde/0x120
[ 1094.150664]  try_to_compact_pages+0x187/0x240
[ 1094.150666]  __alloc_pages_direct_compact+0x87/0x170
[ 1094.150671]  __alloc_pages_slowpath+0x454/0xc20
[ 1094.150680]  ? release_pages+0x348/0x3b0
[ 1094.150684]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1094.150686]  __alloc_pages_nodemask+0x268/0x2b0
[ 1094.150688]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1094.150692]  __handle_mm_fault+0xc11/0x1890
[ 1094.150696]  handle_mm_fault+0xa9/0x1d0
[ 1094.150698]  __do_page_fault+0x237/0x480
[ 1094.150701]  do_page_fault+0x1d/0x67
[ 1094.150702]  ? page_fault+0x8/0x30
[ 1094.150704]  page_fault+0x1e/0x30
[ 1094.150706] RIP: 0033:0x5c2e1a191c10
[ 1094.150717] Code: Bad RIP value.
[ 1094.150720] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1094.150724] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1094.150728] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1094.150730] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1094.150733] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1094.150738] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1094.150746] NMI backtrace for cpu 11
[ 1094.150752] CPU: 11 PID: 25777 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1094.150758] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1094.150763] RIP: 0010:compact_zone+0x5db/0xa70
[ 1094.150770] Code: 8b 47 60 85 c0 7e 23 48 8b 4c 24 08 48 85 c9 74 19 48 c7 c2 ff ff ff ff c4 e2 f9 f7 c2 49 23 47 30 48 39 c1 0f 82 63 03 00 00 <48> 8b 44 24 28 48 85 c0 0f 84 14 fb ff ff 48 83 78 08 00 0f 84 09
[ 1094.150777] RSP: 0018:ffffb2d88bc6ba18 EFLAGS: 00000246
[ 1094.150786] RAX: 0000000000000009 RBX: 0000000000718800 RCX: 0000000000000000
[ 1094.150789] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 1094.150792] RBP: ffffb2d88bc6bad0 R08: 0000000000000001 R09: ffff99f62dae1450
[ 1094.150800] R10: 0000000000000000 R11: ffff99f62dae0a80 R12: 0000000000718440
[ 1094.150805] R13: ffff99f64dfded00 R14: 0000000000718440 R15: ffffb2d88bc6bad0
[ 1094.150809] FS:  0000720b85407740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000
[ 1094.150814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.150818] CR2: 0000700c4abe4000 CR3: 00000007ccf6a003 CR4: 00000000003606e0
[ 1094.150823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1094.150830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1094.150834] Call Trace:
[ 1094.150837]  compact_zone_order+0xde/0x120
[ 1094.150847]  try_to_compact_pages+0x187/0x240
[ 1094.150853]  __alloc_pages_direct_compact+0x87/0x170
[ 1094.150856]  __alloc_pages_slowpath+0x454/0xc20
[ 1094.150862]  ? release_pages+0x348/0x3b0
[ 1094.150869]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1094.150874]  __alloc_pages_nodemask+0x268/0x2b0
[ 1094.150879]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1094.150882]  __handle_mm_fault+0xc11/0x1890
[ 1094.150885]  handle_mm_fault+0xa9/0x1d0
[ 1094.150892]  __do_page_fault+0x237/0x480
[ 1094.150900]  do_page_fault+0x1d/0x67
[ 1094.150904]  ? page_fault+0x8/0x30
[ 1094.150909]  page_fault+0x1e/0x30
[ 1094.150912] RIP: 0033:0x5c2e1a191c10
[ 1094.150915] Code: Bad RIP value.
[ 1094.150918] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1094.150933] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1094.150938] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1094.150941] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1094.150947] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1094.150952] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1096.504728] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started
[ 1096.645895] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated

```
Comment 2 GYt2bW 2019-07-13 20:04:07 UTC
```
[ 1319.077352] sysrq: Show backtrace of all active CPUs
[ 1319.077365] NMI backtrace for cpu 7
[ 1319.077376] CPU: 7 PID: 14722 Comm: bash Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.077381] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.077386] Call Trace:
[ 1319.077415]  dump_stack+0x46/0x60
[ 1319.077426]  nmi_cpu_backtrace.cold+0x14/0x53
[ 1319.077440]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 1319.077447]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 1319.077458]  __handle_sysrq.cold+0x48/0x102
[ 1319.077467]  write_sysrq_trigger+0x1f/0x30
[ 1319.077481]  proc_reg_write+0x37/0x60
[ 1319.077497]  vfs_write+0xb1/0x190
[ 1319.077506]  ksys_write+0x5d/0xe0
[ 1319.077516]  do_syscall_64+0x50/0x170
[ 1319.077529]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1319.077537] RIP: 0033:0x79787f47a4ba
[ 1319.077546] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20
[ 1319.077553] RSP: 002b:00007ffc61be1698 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 1319.077561] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000079787f47a4ba
[ 1319.077567] RDX: 0000000000000002 RSI: 000055f82dde0ee0 RDI: 0000000000000001
[ 1319.077572] RBP: 00007ffc61be16c0 R08: 000000000000000a R09: 0000000000000001
[ 1319.077576] R10: 000055f82d3c305e R11: 0000000000000246 R12: 0000000000000002
[ 1319.077582] R13: 000055f82dde0ee0 R14: 000079787f54f500 R15: 000079787f550300
[ 1319.077680] Sending NMI from CPU 7 to CPUs 0-6,8-11:
[ 1319.077882] NMI backtrace for cpu 0
[ 1319.077890] CPU: 0 PID: 25727 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.077892] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.077894] RIP: 0010:get_pfnblock_flags_mask+0x1e/0x70
[ 1319.077897] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 90 49 89 f0 49 c1 e8 0f 48 8b 3d ea 5b 5f 01 48 85 ff 74 1d 48 89 f0 48 c1 e8 17 48 8b 3c c7 <45> 0f b6 c0 49 c1 e0 04 49 01 f8 48 85 ff 49 0f 45 f8 48 c1 ee 09
[ 1319.077899] RSP: 0018:ffffb2d88badba10 EFLAGS: 00000247
[ 1319.077910] RAX: 0000000000000000 RBX: 0000000000590000 RCX: 0000000000000007
[ 1319.077912] RDX: 0000000000000002 RSI: 000000000058fc00 RDI: ffff99f64dfd9000
[ 1319.077914] RBP: ffffb2d88badbad0 R08: 00000000000000b1 R09: ffff99f62d821450
[ 1319.077917] R10: 0000000000000001 R11: ffff99f57cfb5c00 R12: 000000000058fe00
[ 1319.077920] R13: ffff99f64dfded00 R14: ffffde0b963f0000 R15: 0000000000000000
[ 1319.077924] FS:  0000720b85407740(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000
[ 1319.077926] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.077931] CR2: 0000631d9a780ac0 CR3: 00000007c23a4006 CR4: 00000000003606f0
[ 1319.077933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.077935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.077939] Call Trace:
[ 1319.077945]  compact_zone+0x4d9/0xa70
[ 1319.077948]  compact_zone_order+0xde/0x120
[ 1319.077950]  try_to_compact_pages+0x187/0x240
[ 1319.077952]  __alloc_pages_direct_compact+0x87/0x170
[ 1319.077953]  __alloc_pages_slowpath+0x20e/0xc20
[ 1319.077957]  ? release_pages+0x348/0x3b0
[ 1319.077960]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1319.077965]  __alloc_pages_nodemask+0x268/0x2b0
[ 1319.077967]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1319.077971]  __handle_mm_fault+0xc11/0x1890
[ 1319.077977]  handle_mm_fault+0xa9/0x1d0
[ 1319.077978]  __do_page_fault+0x237/0x480
[ 1319.077980]  do_page_fault+0x1d/0x67
[ 1319.077981]  ? page_fault+0x8/0x30
[ 1319.077983]  page_fault+0x1e/0x30
[ 1319.077985] RIP: 0033:0x5c2e1a191c10
[ 1319.077988] Code: Bad RIP value.
[ 1319.077991] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1319.077999] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1319.078000] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1319.078003] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1319.078005] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1319.078008] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1319.078016] NMI backtrace for cpu 1
[ 1319.078023] CPU: 1 PID: 25752 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078033] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078039] RIP: 0010:node_page_state+0x10/0x20
[ 1319.078044] Code: 49 39 ee 75 b3 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 80 00 00 00 00 89 f6 48 8d 84 f7 c8 1b 02 00 48 8b 00 48 85 c0 <ba> 00 00 00 00 48 0f 48 c2 c3 66 0f 1f 44 00 00 48 83 ec 20 41 89
[ 1319.078047] RSP: 0018:ffffb2d88bba3950 EFLAGS: 00000246
[ 1319.078056] RAX: 0000000000000000 RBX: 00000000004f3c60 RCX: 000000000000000c
[ 1319.078059] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfde000
[ 1319.078061] RBP: ffff99f64dfde000 R08: 0000000000000001 R09: ffff99f62d861450
[ 1319.078064] R10: 0000000000000001 R11: ffff99f57cfb6200 R12: 0000000000000000
[ 1319.078068] R13: 0000000000048a59 R14: ffffb2d88bba3ad0 R15: 0000000000007c63
[ 1319.078071] FS:  0000720b85407740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000
[ 1319.078076] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.078079] CR2: 00007210ba498ff8 CR3: 000000081439e006 CR4: 00000000003606e0
[ 1319.078084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.078087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.078090] Call Trace:
[ 1319.078092]  isolate_migratepages_block+0xba/0xa50
[ 1319.078098]  compact_zone+0x577/0xa70
[ 1319.078101]  compact_zone_order+0xde/0x120
[ 1319.078104]  try_to_compact_pages+0x187/0x240
[ 1319.078108]  __alloc_pages_direct_compact+0x87/0x170
[ 1319.078111]  __alloc_pages_slowpath+0x454/0xc20
[ 1319.078114]  ? release_pages+0x348/0x3b0
[ 1319.078120]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1319.078123]  __alloc_pages_nodemask+0x268/0x2b0
[ 1319.078126]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1319.078131]  __handle_mm_fault+0xc11/0x1890
[ 1319.078133]  handle_mm_fault+0xa9/0x1d0
[ 1319.078136]  __do_page_fault+0x237/0x480
[ 1319.078139]  do_page_fault+0x1d/0x67
[ 1319.078142]  ? page_fault+0x8/0x30
[ 1319.078148]  page_fault+0x1e/0x30
[ 1319.078151] RIP: 0033:0x5c2e1a191c10
[ 1319.078153] Code: Bad RIP value.
[ 1319.078159] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1319.078172] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1319.078176] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1319.078181] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1319.078190] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1319.078199] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1319.078206] NMI backtrace for cpu 2
[ 1319.078212] CPU: 2 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078217] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078221] RIP: 0010:new_sync_write+0x57/0x1b0
[ 1319.078229] Code: f6 c6 04 0f 95 c0 01 c0 89 c1 83 c9 04 f6 c6 40 0f 45 c1 f6 c6 10 75 18 48 8b 8f f0 00 00 00 48 8b 09 48 8b 71 28 f6 46 50 10 <0f> 84 f9 00 00 00 83 c8 10 89 c1 83 c9 20 81 e2 00 00 10 00 0f 45
[ 1319.078233] RSP: 0018:ffffb2d884117e40 EFLAGS: 00000246
[ 1319.078246] RAX: 0000000000000000 RBX: ffff99f5c67e9e00 RCX: ffff99f5c7d50de0
[ 1319.078250] RDX: 0000000000000001 RSI: ffff99f629132800 RDI: ffff99f5c67e9e00
[ 1319.078253] RBP: ffffb2d884117f08 R08: 0000000000000020 R09: 0000000000000000
[ 1319.078256] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020
[ 1319.078260] R13: ffffb2d884117f08 R14: 000058517b6f83c0 R15: 0000000000000000
[ 1319.078266] FS:  00007c32c37f6740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000
[ 1319.078271] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.078283] CR2: 0000631d9a77fcc0 CR3: 000000077cd3c002 CR4: 00000000003606e0
[ 1319.078288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.078292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.078296] Call Trace:
[ 1319.078298]  vfs_write+0xb1/0x190
[ 1319.078302]  ksys_write+0x5d/0xe0
[ 1319.078305]  do_syscall_64+0x50/0x170
[ 1319.078309]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1319.078313] RIP: 0033:0x7c32c39104ba
[ 1319.078319] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20
[ 1319.078324] RSP: 002b:00007ffe934c7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 1319.078332] RAX: ffffffffffffffda RBX: 0000000000000020 RCX: 00007c32c39104ba
[ 1319.078336] RDX: 0000000000000020 RSI: 000058517b6f83c0 RDI: 0000000000000001
[ 1319.078340] RBP: 00007ffe934c7340 R08: 000000000000000a R09: 0000000000000000
[ 1319.078344] R10: 00007ffe934c736c R11: 0000000000000246 R12: 0000000000000020
[ 1319.078348] R13: 000058517b6f83c0 R14: 00007c32c39e5500 R15: 00007c32c39e6300
[ 1319.078355] NMI backtrace for cpu 4
[ 1319.078363] CPU: 4 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078369] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078372] RIP: 0033:0x70f212b62cff
[ 1319.078377] Code: 64 8b 04 25 18 00 00 00 85 c0 0f 85 0b 01 00 00 ba 01 00 00 00 0f b1 17 48 8b bb 88 00 00 00 48 8b 45 c8 48 89 47 08 ff 47 04 <8b> 83 c0 00 00 00 85 c0 0f 85 a3 00 00 00 c7 83 c0 00 00 00 ff ff
[ 1319.078379] RSP: 002b:00007fffa4d872b0 EFLAGS: 00000202
[ 1319.078385] RAX: 000070f212ad7740 RBX: 000070f212cc6500 RCX: 000070f212cc6500
[ 1319.078388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 000070f212cc8320
[ 1319.078396] RBP: 00007fffa4d872f0 R08: 00005acd2cb39459 R09: 0000000000000000
[ 1319.078402] R10: 00007fffa4d8732c R11: 00005acd2cb39458 R12: 0000000000000001
[ 1319.078405] R13: 0000000000000001 R14: 0000000000000001 R15: 00005acd2cb39458
[ 1319.078410] FS:  000070f212ad7740 GS:  0000000000000000
[ 1319.078416] NMI backtrace for cpu 3
[ 1319.078421] CPU: 3 PID: 25777 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078425] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078428] RIP: 0010:compact_unlock_should_abort.isra.0+0x3b/0x50
[ 1319.078440] Code: 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 74 e0 <c6> 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00 00 41
[ 1319.078442] RSP: 0018:ffffb2d88bc6b940 EFLAGS: 00000202
[ 1319.078448] RAX: ffff99f370429e40 RBX: 0000000000718440 RCX: ffffb2d88bc6bb49
[ 1319.078453] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 1319.078455] RBP: ffffb2d88bc6bb49 R08: 0000000000000001 R09: 0000000000000000
[ 1319.078457] R10: 0000000000000000 R11: ffff99f57cfb4e00 R12: ffffb2d88bc6bad0
[ 1319.078460] R13: 0000000000000000 R14: ffffb2d88bc6bad0 R15: 0000000000000000
[ 1319.078465] FS:  0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000
[ 1319.078471] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.078473] CR2: 000076e9fabf2020 CR3: 00000007ccf6a001 CR4: 00000000003606e0
[ 1319.078476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.078479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.078486] Call Trace:
[ 1319.078490]  isolate_migratepages_block+0x2ed/0xa50
[ 1319.078494]  compact_zone+0x577/0xa70
[ 1319.078628]  compact_zone_order+0xde/0x120
[ 1319.078637]  try_to_compact_pages+0x187/0x240
[ 1319.078641]  __alloc_pages_direct_compact+0x87/0x170
[ 1319.078644]  __alloc_pages_slowpath+0x454/0xc20
[ 1319.078649]  ? release_pages+0x348/0x3b0
[ 1319.078653]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1319.078655]  __alloc_pages_nodemask+0x268/0x2b0
[ 1319.078659]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1319.078665]  __handle_mm_fault+0xc11/0x1890
[ 1319.078670]  handle_mm_fault+0xa9/0x1d0
[ 1319.078673]  __do_page_fault+0x237/0x480
[ 1319.078678]  do_page_fault+0x1d/0x67
[ 1319.078680]  ? page_fault+0x8/0x30
[ 1319.078683]  page_fault+0x1e/0x30
[ 1319.078688] RIP: 0033:0x5c2e1a191c10
[ 1319.078697] Code: Bad RIP value.
[ 1319.078700] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1319.078707] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1319.078714] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1319.078720] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1319.078727] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1319.078729] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1319.078735] NMI backtrace for cpu 5
[ 1319.078740] CPU: 5 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078743] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078747] RIP: 0033:0x7c7ea55225a9
[ 1319.078760] Code: 74 6f 48 89 4d c8 48 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24 28 4d 01 f5 49 89 df 4d 29 f7 48 8b 4d c8 <4c> 01 f9 75 45 4c 29 fb 49 89 de 4c 89 f0 48 83 c4 18 5b 41 5c 41
[ 1319.078766] RSP: 002b:00007ffe25d183c0 EFLAGS: 00000246
[ 1319.078772] RAX: 0000571ede4bd476 RBX: 0000000000000001 RCX: 0000000000000000
[ 1319.078775] RDX: 0000000000000001 RSI: 0000571ede0f050a RDI: 0000571ede4bd475
[ 1319.078784] RBP: 00007ffe25d18400 R08: 0000571ede0f050b R09: 0000000000000000
[ 1319.078792] R10: 00007ffe25d1848c R11: 0000571ede0f050a R12: 00007c7ea5679500
[ 1319.078796] R13: 0000571ede0f050b R14: 0000000000000001 R15: 0000000000000000
[ 1319.078801] FS:  00007c7ea548a740 GS:  0000000000000000
[ 1319.078806] NMI backtrace for cpu 6
[ 1319.078809] CPU: 6 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078812] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078821] RIP: 0033:0x77d47d3fd5a2
[ 1319.078826] Code: 89 c2 31 c9 48 85 d2 74 6f 48 89 4d c8 48 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24 28 4d 01 f5 49 89 df <4d> 29 f7 48 8b 4d c8 4c 01 f9 75 45 4c 29 fb 49 89 de 4c 89 f0 48
[ 1319.078829] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000206
[ 1319.078834] RAX: 00005ffb591e43c1 RBX: 0000000000000001 RCX: 0000000000000052
[ 1319.078841] RDX: 0000000000000001 RSI: 00005ffb57470455 RDI: 00005ffb591e43c0
[ 1319.078847] RBP: 00007ffe06db5e30 R08: 00005ffb57470456 R09: 0000000000000000
[ 1319.078852] R10: 00007ffe06db5ebc R11: 00005ffb57470455 R12: 000077d47d554500
[ 1319.078856] R13: 00005ffb57470456 R14: 0000000000000001 R15: 0000000000000001
[ 1319.078860] FS:  000077d47d365740 GS:  0000000000000000
[ 1319.078867] NMI backtrace for cpu 8
[ 1319.078876] CPU: 8 PID: 25800 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.078880] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.078882] RIP: 0010:compact_zone+0x587/0xa70
[ 1319.078887] Code: ff 4c 8b 34 24 49 89 ef 8b 4c 24 50 4c 89 e2 4c 89 f6 48 89 ef e8 c9 e8 ff ff 49 89 c4 48 85 c0 0f 84 c3 02 00 00 4d 89 67 30 <41> 8b 77 24 85 f6 0f 85 9f 00 00 00 80 7c 24 56 00 0f 84 c3 02 00
[ 1319.078896] RSP: 0018:ffffb2d88bd23a18 EFLAGS: 00000202
[ 1319.078905] RAX: 000000000016b420 RBX: 000000000016b800 RCX: 0000000000000000
[ 1319.078908] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 1319.078915] RBP: ffffb2d88bd23ad0 R08: 0000000000000001 R09: ffff99f62da21450
[ 1319.078923] R10: 0000000000000000 R11: ffff99f62da20a80 R12: 000000000016b420
[ 1319.078930] R13: ffff99f64dfded00 R14: 000000000016b420 R15: ffffb2d88bd23ad0
[ 1319.078936] FS:  0000720b85407740(0000) GS:ffff99f62da00000(0000) knlGS:0000000000000000
[ 1319.078940] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.078944] CR2: 00007ecf94de9578 CR3: 00000007ec28a006 CR4: 00000000003606e0
[ 1319.078952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.078961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.078964] Call Trace:
[ 1319.078969]  compact_zone_order+0xde/0x120
[ 1319.078974]  try_to_compact_pages+0x187/0x240
[ 1319.078978]  __alloc_pages_direct_compact+0x87/0x170
[ 1319.078985]  __alloc_pages_slowpath+0x454/0xc20
[ 1319.078993]  ? release_pages+0x348/0x3b0
[ 1319.078997]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1319.079003]  __alloc_pages_nodemask+0x268/0x2b0
[ 1319.079007]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1319.079012]  __handle_mm_fault+0xc11/0x1890
[ 1319.079017]  handle_mm_fault+0xa9/0x1d0
[ 1319.079030]  __do_page_fault+0x237/0x480
[ 1319.079034]  do_page_fault+0x1d/0x67
[ 1319.079041]  ? page_fault+0x8/0x30
[ 1319.079047]  page_fault+0x1e/0x30
[ 1319.079052] RIP: 0033:0x5c2e1a191c10
[ 1319.079055] Code: Bad RIP value.
[ 1319.079060] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 1319.079074] RAX: 00000000183a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 1319.079079] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 1319.079084] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 1319.079088] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 1319.079093] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 1319.079103] NMI backtrace for cpu 9
[ 1319.079109] CPU: 9 PID: 1238 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.079113] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.079121] RIP: 0010:touch_atime+0x26/0xe0
[ 1319.079133] Code: 1f 44 00 00 55 48 89 e5 41 54 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 4c 8b 27 48 8b 47 08 48 8b 58 30 <48> 89 de e8 02 ff ff ff 84 c0 75 1d 48 8b 44 24 10 65 48 33 04 25
[ 1319.079141] RSP: 0018:ffffb2d884137db0 EFLAGS: 00000246
[ 1319.079151] RAX: ffff99f594bc4840 RBX: ffff99f5c7ec6750 RCX: 0000000000000000
[ 1319.079158] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f5c8398810
[ 1319.079164] RBP: ffffb2d884137dd8 R08: 0000000000000001 R09: 0000000000000000
[ 1319.079170] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f629139220
[ 1319.079186] R13: ffff99f5c92c4d68 R14: 0000000000000001 R15: ffff99f5c75b0180
[ 1319.079197] FS:  00007d2186c2e740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000
[ 1319.079202] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.079209] CR2: 00005892d7c57e02 CR3: 0000000788c3c002 CR4: 00000000003606e0
[ 1319.079224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.079234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.079241] Call Trace:
[ 1319.079250]  pipe_read+0x2aa/0x2c0
[ 1319.079258]  new_sync_read+0x10f/0x1a0
[ 1319.079264]  vfs_read+0x98/0x150
[ 1319.079270]  ksys_read+0x5d/0xe0
[ 1319.079277]  do_syscall_64+0x50/0x170
[ 1319.079293]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1319.079302] RIP: 0033:0x7d2186f4d415
[ 1319.079308] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1319.079317] RSP: 002b:00007ffd33329a38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1319.079337] RAX: ffffffffffffffda RBX: 00007d21870217e0 RCX: 00007d2186f4d415
[ 1319.079346] RDX: 0000000000000001 RSI: 00007d2187021863 RDI: 0000000000000000
[ 1319.079351] RBP: 00007ffd33329a80 R08: 0000000000000000 R09: 0000000000000000
[ 1319.079360] R10: 0000647dee70b1f7 R11: 0000000000000246 R12: 00007d2187023300
[ 1319.079367] R13: 00007d2187022500 R14: 00007d2187022700 R15: 00007d2187023468
[ 1319.079372] NMI backtrace for cpu 10
[ 1319.079384] CPU: 10 PID: 1053 Comm: xfwm4 Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.079392] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.079395] RIP: 0033:0x624fe3bc1622
[ 1319.079403] Code: 8b 85 24 04 00 00 ff c0 41 39 45 00 0f 85 1c fc ff ff 49 8b 75 20 48 89 ef e8 2a dc ff ff 48 89 c7 48 85 c0 0f 84 fe fb ff ff <49> 8b 75 28 e8 d5 5d 02 00 e9 f0 fb ff ff 48 8b 73 08 ba 06 00 00
[ 1319.079412] RSP: 002b:00007ffdd444b6f0 EFLAGS: 00000206
[ 1319.079428] RAX: 0000624fe5f3ce00 RBX: 0000624fe5a93830 RCX: 0000000000000001
[ 1319.079433] RDX: 0000000000000000 RSI: 00000000008004bd RDI: 0000624fe5f3ce00
[ 1319.079441] RBP: 0000624fe597a100 R08: 0000624fe5f3ce00 R09: 0000000000000020
[ 1319.079449] R10: 00007388623d19e0 R11: 0000000000000058 R12: 0000000000000000
[ 1319.079454] R13: 00007ffdd444b810 R14: 00007ffdd444b810 R15: 0000624fe5ea6da0
[ 1319.079465] FS:  0000738860b0c980 GS:  0000000000000000
[ 1319.079473] NMI backtrace for cpu 11
[ 1319.079484] CPU: 11 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 1319.079491] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1319.079495] RIP: 0010:queued_spin_lock_slowpath+0x110/0x1a0
[ 1319.079505] Code: 85 c0 74 f2 eb f6 41 8d 40 01 c1 e0 12 c1 e1 10 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 8b 07 <66> 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 63 c6 07 01 48 85
[ 1319.079508] RSP: 0018:ffffb2d88413fe50 EFLAGS: 00000002
[ 1319.079517] RAX: 0000000000100000 RBX: ffff99f5c7144000 RCX: 0000000000300000
[ 1319.079520] RDX: ffff99f62dae16c0 RSI: 0000000000000000 RDI: ffffffffa355d2ac
[ 1319.079527] RBP: ffff99f57cd9b400 R08: 000000000000000b R09: 0000000000000000
[ 1319.079531] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001fff
[ 1319.079535] R13: 00005d76c75ae440 R14: ffff99f5c7144038 R15: ffff99f57cd9b400
[ 1319.079545] FS:  000074e35af43740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000
[ 1319.079552] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1319.079563] CR2: 00007ecf94e85ff8 CR3: 000000077cedc004 CR4: 00000000003606e0
[ 1319.079568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1319.079573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1319.079580] Call Trace:
[ 1319.079589]  devkmsg_read+0x7f/0x270
[ 1319.079594]  vfs_read+0x98/0x150
[ 1319.079597]  ksys_read+0x5d/0xe0
[ 1319.079610]  do_syscall_64+0x50/0x170
[ 1319.079616]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1319.079622] RIP: 0033:0x74e35b05d415
[ 1319.079630] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1319.079637] RSP: 002b:00007ffda01e7bc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1319.079648] RAX: ffffffffffffffda RBX: 00005d76c75ae440 RCX: 000074e35b05d415
[ 1319.079656] RDX: 0000000000001fff RSI: 00005d76c75ae440 RDI: 0000000000000003
[ 1319.079661] RBP: 00005d76c75aa7fa R08: 000000000000000a R09: 0000000000000000
[ 1319.079664] R10: 00007ffda01e79fc R11: 0000000000000246 R12: 0000000000000041
[ 1319.079676] R13: 00005d76c75a9377 R14: 00005d76c75ae440 R15: 00005d76c75ae455
[ 1336.744759] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started
[ 1336.846541] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated
[ 1343.485167] dbus-daemon[653]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.111' (uid=1000 pid=523 comm="mousepad /tmp/dmesg.log ")
[ 1343.505410] systemd[1]: Starting Hostname Service...
[ 1343.507309] device: 'lo': device_add
[ 1343.507448] PM: Adding info for No Bus:lo
[ 1344.616004] systemd[1]: Started Hostname Service.
```
Comment 3 GYt2bW 2019-07-13 20:04:47 UTC
some of them finished/disappeared , only 3 are still running now:
```
  PID  %CPU COMMAND                                                                            PR  NI    VIRT    RES S USER     
25777 100.0 stress                                                                             20   0  980352 124868 R user     
25727 100.0 stress                                                                             20   0  980352  96196 R user     
25752 100.0 stress                                                                             20   0  980352 122820 R user     
```


```
[ 4245.323214] snd_hda_intel 0000:00:1f.3: PME# enabled
[ 4245.334733] snd_hda_intel 0000:00:1f.3: power state changed by ACPI to D3hot
[ 4279.678617] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started
[ 4279.765483] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated
[ 4324.861160] i2c i2c-2: NAK from device addr 0x50 msg #0
[ 4324.864949] i2c i2c-1: NAK from device addr 0x50 msg #0
[ 4324.868589] i2c i2c-3: NAK from device addr 0x50 msg #0
[ 4339.738573] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started
[ 4339.811766] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated
[ 4399.798582] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started
[ 4399.886406] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated
[ 4459.859214] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started
[ 4459.952334] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated
[ 4519.918645] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started
[ 4520.000915] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated
[ 4537.428195] sysrq: Show backtrace of all active CPUs
[ 4537.428204] NMI backtrace for cpu 8
[ 4537.428214] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.428219] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.428223] Call Trace:
[ 4537.428230]  <IRQ>
[ 4537.428243]  dump_stack+0x46/0x60
[ 4537.428251]  nmi_cpu_backtrace.cold+0x14/0x53
[ 4537.428260]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 4537.428267]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 4537.428275]  __handle_sysrq.cold+0x48/0x102
[ 4537.428283]  sysrq_filter+0x2ea/0x3b0
[ 4537.428291]  input_to_handler+0x4d/0xf0
[ 4537.428300]  input_pass_values.part.0+0x109/0x130
[ 4537.428308]  input_handle_event+0x171/0x5a0
[ 4537.428316]  input_event+0x4d/0x70
[ 4537.428325]  hidinput_report_event+0x2e/0x40
[ 4537.428333]  hid_report_raw_event+0x260/0x430
[ 4537.428341]  hid_input_report+0xfb/0x150
[ 4537.428350]  hid_irq_in+0x168/0x190
[ 4537.428360]  __usb_hcd_giveback_urb+0x77/0xe0
[ 4537.428383]  xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd]
[ 4537.428406]  xhci_td_cleanup+0xf7/0x140 [xhci_hcd]
[ 4537.428428]  xhci_irq+0x7e8/0x1be0 [xhci_hcd]
[ 4537.428438]  __handle_irq_event_percpu+0x2f/0xc0
[ 4537.428445]  handle_irq_event_percpu+0x2c/0x80
[ 4537.428453]  handle_irq_event+0x23/0x43
[ 4537.428462]  handle_edge_irq+0x78/0x190
[ 4537.428470]  handle_irq+0x17/0x20
[ 4537.428476]  do_IRQ+0x3e/0xd0
[ 4537.428485]  common_interrupt+0xf/0xf
[ 4537.428490]  </IRQ>
[ 4537.428500] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0
[ 4537.428508] Code: e8 d6 f9 a1 ff 49 89 c6 31 ff e8 fc 09 a2 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 f5 f3 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c
[ 4537.428514] RSP: 0018:ffffb2d883217e80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde
[ 4537.428521] RAX: ffff99f62da20a00 RBX: ffffffffa3061940 RCX: 000000000000001f
[ 4537.428526] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000
[ 4537.428531] RBP: 0000042073c1a3e8 R08: 0000042073cb420a R09: 0000000000000f70
[ 4537.428535] R10: ffff99f62da1fb44 R11: ffff99f62da1fb24 R12: ffff99f62da29a00
[ 4537.428540] R13: 0000000000000004 R14: 0000042073cb420a R15: 0000000000000000
[ 4537.428551]  cpuidle_enter+0x24/0x40
[ 4537.428559]  do_idle+0x1c1/0x230
[ 4537.428567]  cpu_startup_entry+0x14/0x20
[ 4537.428575]  start_secondary+0x168/0x1b0
[ 4537.428585]  secondary_startup_64+0xa4/0xb0
[ 4537.428594] Sending NMI from CPU 8 to CPUs 0-7,9-11:
[ 4537.428681] NMI backtrace for cpu 0
[ 4537.428683] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.428685] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.428686] RIP: 0010:menu_select+0x283/0x600
[ 4537.428688] Code: 00 85 c9 0f 8e 26 01 00 00 48 8d 45 4c 48 8b 3c 24 48 8d 57 18 41 bf ff ff ff ff 45 31 f6 80 78 0c 00 75 25 48 83 3a 00 75 1f <41> 83 ff ff 45 0f 44 fe 8b 70 08 44 39 ee 0f 87 70 02 00 00 39 18
[ 4537.428689] RSP: 0018:ffffffffa3003e60 EFLAGS: 00000046
[ 4537.428691] RAX: ffffffffa3061aac RBX: 0000000000000940 RCX: 0000000000000009
[ 4537.428692] RDX: ffff99f62d829ac0 RSI: 0000000000000014 RDI: ffff99f62d829a00
[ 4537.428693] RBP: ffffffffa3061940 R08: 0000000000001b9a R09: 0000000000001f67
[ 4537.428694] R10: ffff99f62d81fb44 R11: ffff99f62d81fb24 R12: ffff99f62d81fae0
[ 4537.428695] R13: 0000000000000940 R14: 0000000000000003 R15: 0000000000000002
[ 4537.428696] FS:  0000000000000000(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000
[ 4537.428697] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4537.428698] CR2: 0000787391901145 CR3: 000000080600a005 CR4: 00000000003606f0
[ 4537.428699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4537.428700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4537.428700] Call Trace:
[ 4537.428701]  do_idle+0x19d/0x230
[ 4537.428702]  cpu_startup_entry+0x14/0x20
[ 4537.428702]  start_kernel+0x453/0x470
[ 4537.428703]  secondary_startup_64+0xa4/0xb0
[ 4537.428761] NMI backtrace for cpu 1 skipped: idling at intel_idle+0x7d/0x120
[ 4537.428927] NMI backtrace for cpu 2
[ 4537.428930] CPU: 2 PID: 25777 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.428936] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.428940] RIP: 0010:compact_zone+0x411/0xa70
[ 4537.428942] Code: 48 0f 42 fa 45 31 ed 49 8d 9c 24 00 02 00 00 48 81 e3 00 fe ff ff 49 3b 5f 28 0f 87 7c 01 00 00 48 81 c3 00 02 00 00 49 89 fe <4c> 89 f8 45 89 ef 49 89 ed 48 89 c5 4c 89 e0 4c 89 24 24 4c 8d a3
[ 4537.428946] RSP: 0018:ffffb2d88bc6ba18 EFLAGS: 00000206
[ 4537.428954] RAX: 0000000000718440 RBX: 0000000000718800 RCX: 00000000000038c2
[ 4537.428957] RDX: 0000000000100000 RSI: 000000000000000c RDI: 0000000000718400
[ 4537.428959] RBP: ffff99f64dfded00 R08: 0000000000000001 R09: ffff99f62d8a1450
[ 4537.428962] R10: 0000000000000001 R11: ffff99f62d8a0a80 R12: 0000000000718440
[ 4537.428966] R13: 0000000000000000 R14: 0000000000718400 R15: ffffb2d88bc6bad0
[ 4537.428969] FS:  0000720b85407740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000
[ 4537.428973] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4537.428974] CR2: 000057450363c268 CR3: 00000007ccf6a003 CR4: 00000000003606e0
[ 4537.428976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4537.428977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4537.428980] Call Trace:
[ 4537.428984]  compact_zone_order+0xde/0x120
[ 4537.428990]  try_to_compact_pages+0x187/0x240
[ 4537.428991]  __alloc_pages_direct_compact+0x87/0x170
[ 4537.428992]  __alloc_pages_slowpath+0x454/0xc20
[ 4537.428993]  ? release_pages+0x348/0x3b0
[ 4537.428995]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 4537.428998]  __alloc_pages_nodemask+0x268/0x2b0
[ 4537.429001]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 4537.429005]  __handle_mm_fault+0xc11/0x1890
[ 4537.429008]  handle_mm_fault+0xa9/0x1d0
[ 4537.429009]  __do_page_fault+0x237/0x480
[ 4537.429013]  do_page_fault+0x1d/0x67
[ 4537.429014]  ? page_fault+0x8/0x30
[ 4537.429017]  page_fault+0x1e/0x30
[ 4537.429018] RIP: 0033:0x5c2e1a191c10
[ 4537.429019] Code: Bad RIP value.
[ 4537.429023] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 4537.429029] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 4537.429033] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 4537.429037] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 4537.429038] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 4537.429039] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 4537.429048] NMI backtrace for cpu 3
[ 4537.429056] CPU: 3 PID: 1252 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429060] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429064] RIP: 0010:mutex_lock+0x1a/0x30
[ 4537.429073] Code: be 02 00 00 00 e9 46 fb ff ff 66 0f 1f 44 00 00 55 48 89 fd e8 97 df ff ff 65 48 8b 14 25 00 5d 01 00 31 c0 f0 48 0f b1 55 00 <74> 06 48 89 ef 5d eb ce 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 90
[ 4537.429078] RSP: 0018:ffffb2d883effdd8 EFLAGS: 00000246
[ 4537.429091] RAX: 0000000000000000 RBX: ffff99f588eb2d00 RCX: 0000000000000001
[ 4537.429096] RDX: ffff99f57cc3dac0 RSI: ffffb2d883effe50 RDI: ffff99f5c8094c00
[ 4537.429103] RBP: ffff99f5c8094c00 R08: 0000000000000001 R09: 0000000000000000
[ 4537.429110] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 4537.429115] R13: ffffb2d883efff08 R14: 00007c3fc3975863 R15: ffff99f5c8094c00
[ 4537.429121] FS:  00007c3fc3582740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000
[ 4537.429127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4537.429130] CR2: 000078739192f578 CR3: 000000077ce7c006 CR4: 00000000003606e0
[ 4537.429135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4537.429141] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4537.429147] Call Trace:
[ 4537.429152]  pipe_read+0x37/0x2c0
[ 4537.429157]  new_sync_read+0x10f/0x1a0
[ 4537.429164]  vfs_read+0x98/0x150
[ 4537.429168]  ksys_read+0x5d/0xe0
[ 4537.429173]  do_syscall_64+0x50/0x170
[ 4537.429180]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 4537.429184] RIP: 0033:0x7c3fc38a1415
[ 4537.429192] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 4537.429199] RSP: 002b:00007ffdfcb6a818 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 4537.429207] RAX: ffffffffffffffda RBX: 00007c3fc39757e0 RCX: 00007c3fc38a1415
[ 4537.429210] RDX: 0000000000000001 RSI: 00007c3fc3975863 RDI: 0000000000000000
[ 4537.429215] RBP: 00007ffdfcb6a860 R08: 0000000000000000 R09: 0000000000000000
[ 4537.429222] R10: 000063939c42a58c R11: 0000000000000246 R12: 00007c3fc3977300
[ 4537.429226] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468
[ 4537.429229] NMI backtrace for cpu 4
[ 4537.429235] CPU: 4 PID: 25727 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429245] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429249] RIP: 0010:rcu_all_qs+0x7/0x50
[ 4537.429256] Code: 43 30 48 83 7b 20 00 74 d2 0f 0b 48 85 c0 74 d0 c6 03 00 5b 5d c3 0f 0b c3 66 2e 0f 1f 84 00 00 00 00 00 65 8a 05 16 27 f2 5d <84> c0 74 2e 48 c7 c0 bd 17 02 00 65 48 03 05 7e 10 f1 5d 0f b6 00
[ 4537.429261] RSP: 0018:ffffb2d88badb948 EFLAGS: 00000286
[ 4537.429276] RAX: 0000000080000000 RBX: 000000000058fcc0 RCX: 000000000000000c
[ 4537.429279] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfde000
[ 4537.429282] RBP: ffff99f64dfde000 R08: ffff99f64dfd9b10 R09: ffff99f62d921450
[ 4537.429289] R10: 0000000000000001 R11: ffff99f62d920a80 R12: 0000000000000000
[ 4537.429295] R13: 000000000003389a R14: ffffb2d88badbad0 R15: 000000000001f5a8
[ 4537.429302] FS:  0000720b85407740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000
[ 4537.429307] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4537.429309] CR2: 000072d5e050ac40 CR3: 00000007c23a4002 CR4: 00000000003606e0
[ 4537.429312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4537.429319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4537.429324] Call Trace:
[ 4537.429331]  _cond_resched+0x10/0x20
[ 4537.429334]  isolate_migratepages_block+0xd5/0xa50
[ 4537.429337]  compact_zone+0x577/0xa70
[ 4537.429343]  compact_zone_order+0xde/0x120
[ 4537.429348]  try_to_compact_pages+0x187/0x240
[ 4537.429353]  __alloc_pages_direct_compact+0x87/0x170
[ 4537.429357]  __alloc_pages_slowpath+0x20e/0xc20
[ 4537.429361]  ? release_pages+0x348/0x3b0
[ 4537.429364]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 4537.429367]  __alloc_pages_nodemask+0x268/0x2b0
[ 4537.429373]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 4537.429379]  __handle_mm_fault+0xc11/0x1890
[ 4537.429384]  handle_mm_fault+0xa9/0x1d0
[ 4537.429386]  __do_page_fault+0x237/0x480
[ 4537.429389]  do_page_fault+0x1d/0x67
[ 4537.429391]  ? page_fault+0x8/0x30
[ 4537.429398]  page_fault+0x1e/0x30
[ 4537.429401] RIP: 0033:0x5c2e1a191c10
[ 4537.429404] Code: Bad RIP value.
[ 4537.429406] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 4537.429414] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 4537.429417] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 4537.429419] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 4537.429426] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 4537.429429] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 4537.429437] NMI backtrace for cpu 5
[ 4537.429440] CPU: 5 PID: 25752 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429443] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429450] RIP: 0010:isolate_migratepages_block+0x2de/0xa50
[ 4537.429455] Code: 00 00 48 81 c4 88 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 4c 89 54 24 18 48 8b 44 24 20 48 8d b8 00 1b 02 00 48 8b 4c 24 28 <48> 8d 54 24 77 48 8b 74 24 78 e8 33 f0 ff ff 84 c0 4c 8b 54 24 18
[ 4537.429458] RSP: 0018:ffffb2d88bba3958 EFLAGS: 00000246
[ 4537.429468] RAX: ffff99f64dfde000 RBX: 00000000004f3c60 RCX: ffffb2d88bba3b49
[ 4537.429472] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfffb00
[ 4537.429475] RBP: ffff99f64dfde000 R08: 0000000000000001 R09: ffff99f62d961450
[ 4537.429478] R10: 0000000000000000 R11: ffff99f57cfb7400 R12: ffffb2d88bba3ad0
[ 4537.429480] R13: 0000000000000000 R14: ffffb2d88bba3ad0 R15: 0000000000000000
[ 4537.429485] FS:  0000720b85407740(0000) GS:ffff99f62d940000(0000) knlGS:0000000000000000
[ 4537.429490] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4537.429493] CR2: 00007ac70ee0a000 CR3: 000000081439e001 CR4: 00000000003606e0
[ 4537.429496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4537.429500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 4537.429503] Call Trace:
[ 4537.429506]  compact_zone+0x577/0xa70
[ 4537.429510]  compact_zone_order+0xde/0x120
[ 4537.429515]  try_to_compact_pages+0x187/0x240
[ 4537.429518]  __alloc_pages_direct_compact+0x87/0x170
[ 4537.429520]  __alloc_pages_slowpath+0x454/0xc20
[ 4537.429523]  ? release_pages+0x348/0x3b0
[ 4537.429526]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 4537.429530]  __alloc_pages_nodemask+0x268/0x2b0
[ 4537.429533]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 4537.429536]  __handle_mm_fault+0xc11/0x1890
[ 4537.429538]  handle_mm_fault+0xa9/0x1d0
[ 4537.429543]  __do_page_fault+0x237/0x480
[ 4537.429546]  do_page_fault+0x1d/0x67
[ 4537.429548]  ? page_fault+0x8/0x30
[ 4537.429551]  page_fault+0x1e/0x30
[ 4537.429554] RIP: 0033:0x5c2e1a191c10
[ 4537.429561] Code: Bad RIP value.
[ 4537.429564] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 4537.429574] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 4537.429579] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 4537.429582] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 4537.429585] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 4537.429590] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 4537.429595] NMI backtrace for cpu 6
[ 4537.429601] CPU: 6 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429604] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429609] RIP: 0033:0x7c32c38bcbd3
[ 4537.429614] Code: 00 31 c0 eb 09 66 90 48 c7 c0 fe ff ff ff 48 8b 7d c8 64 48 33 3c 25 28 00 00 00 0f 85 7c 00 00 00 48 8d 65 d8 5b 41 5c 41 5d <41> 5e 41 5f 5d c3 0f 1f 80 00 00 00 00 48 8b 45 80 4c 29 e0 eb ce
[ 4537.429617] RSP: 002b:00007ffe934c7320 EFLAGS: 00000246
[ 4537.429624] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000000
[ 4537.429627] RDX: 0000000000000068 RSI: 0000000000000001 RDI: 0000000000000000
[ 4537.429630] RBP: 00007ffe934c7330 R08: 000058517a066458 R09: 0000000000000000
[ 4537.429633] R10: 00007ffe934c736c R11: 000058517a066457 R12: 00007c32c39e5500
[ 4537.429636] R13: 00007ffe934c7370 R14: 000058517a06648a R15: 000058517b6d4b10
[ 4537.429640] FS:  00007c32c37f6740 GS:  0000000000000000
[ 4537.429644] NMI backtrace for cpu 7
[ 4537.429647] CPU: 7 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429653] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429656] RIP: 0033:0x70f212b62d36
[ 4537.429661] Code: 00 c7 83 c0 00 00 00 ff ff ff ff 48 8b 8b d8 00 00 00 48 89 4d c8 48 8d 15 d7 39 16 00 48 8d 05 38 47 16 00 48 29 d0 48 29 d1 <48> 39 c8 0f 86 d1 00 00 00 4c 89 ea 4c 89 fe 48 89 df 48 8b 45 c8
[ 4537.429664] RSP: 002b:00007fffa4d872b0 EFLAGS: 00000206
[ 4537.429670] RAX: 0000000000000d68 RBX: 000070f212cc6500 RCX: 0000000000000c00
[ 4537.429675] RDX: 000070f212cc6700 RSI: 0000000000000001 RDI: 000070f212cc8320
[ 4537.429678] RBP: 00007fffa4d872f0 R08: 00005acd2cb3946a R09: 0000000000000000
[ 4537.429681] R10: 00007fffa4d8732c R11: 00005acd2cb39469 R12: 0000000000000001
[ 4537.429684] R13: 0000000000000001 R14: 0000000000000001 R15: 00005acd2cb39469
[ 4537.429690] FS:  000070f212ad7740 GS:  0000000000000000
[ 4537.429696] NMI backtrace for cpu 9
[ 4537.429699] CPU: 9 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429702] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429706] RIP: 0033:0x74e35afcec9c
[ 4537.429712] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 18 49 89 f5 4c 0f af ea <4d> 85 ed 0f 84 f0 00 00 00 49 89 ff 49 89 f6 49 89 d4 48 89 cb 8b
[ 4537.429714] RSP: 002b:00007ffda01e7980 EFLAGS: 00000202
[ 4537.429729] RAX: 0000000000000001 RBX: 000000000000000a RCX: 000074e35b132500
[ 4537.429732] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00005d76c75ae45e
[ 4537.429737] RBP: 00007ffda01e79c0 R08: 00005d76c75ae45f R09: 0000000000000000
[ 4537.429743] R10: 00007ffda01e79fc R11: 00005d76c75ae45e R12: 000074e35b132500
[ 4537.429748] R13: 0000000000000001 R14: 00005d76c75a9071 R15: 0000000000000001
[ 4537.429753] FS:  000074e35af43740 GS:  0000000000000000
[ 4537.429757] NMI backtrace for cpu 10 skipped: idling at intel_idle+0x7d/0x120
[ 4537.429764] NMI backtrace for cpu 11
[ 4537.429770] CPU: 11 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 4537.429774] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 4537.429779] RIP: 0033:0x77d47d42bb16
[ 4537.429790] Code: 12 00 64 48 8b 00 4c 8b 28 4d 8b 7d 28 4d 85 ff 0f 84 06 01 00 00 4c 89 65 80 4d 01 e6 0f 82 21 01 00 00 49 8b 07 4c 8b 68 28 <48> 83 38 00 74 0d 49 c1 cd 11 64 4c 33 2c 25 30 00 00 00 4c 89 ef
[ 4537.429797] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000216
[ 4537.429811] RAX: 00005ffb591c31d0 RBX: 00007ffe06db5ebc RCX: 00007ffe06db5ec0
[ 4537.429814] RDX: 000000000000000d RSI: 000077d47d556b60 RDI: 00007ffe06db5ebc
[ 4537.429820] RBP: 00007ffe06db5e80 R08: 0000000000000001 R09: 0000000000000000
[ 4537.429827] R10: 00007ffe06db5ebc R11: 00005ffb57470458 R12: 00005ffb57470459
[ 4537.429833] R13: 000077d47d3962d0 R14: 00005ffb57470466 R15: 00005ffb591c0b10
[ 4537.429838] FS:  000077d47d365740 GS:  0000000000000000
```
Comment 4 GYt2bW 2019-07-13 20:05:52 UTC
Created attachment 283669 [details]
zcat-ed kernel .config from /proc
Comment 5 GYt2bW 2019-07-13 20:18:42 UTC
the initial command was this:
```
$ time stress -m 220 --vm-bytes 1000000000 --timeout 10
stress: info: [25580] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: FAIL: [25580] (415) <-- worker 25793 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25795 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25799 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25792 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25797 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25798 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25791 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
stress: FAIL: [25580] (415) <-- worker 25800 got signal 9
stress: WARN: [25580] (417) now reaping child worker processes
```

(and it's still running 3 `stress` processes at the moment)

```
$ cat /proc/uptime 
5490.76 43065.45
```
Comment 6 GYt2bW 2019-07-13 20:19:58 UTC
omg that bugzilla bug again :(((((((

that's not what I pasted in the previous comment!!!!!!!!!!11
Comment 7 GYt2bW 2019-07-13 20:20:40 UTC
oh great, it got partially fixed as I posted my previous comment!
Comment 8 GYt2bW 2019-07-13 20:25:52 UTC
^C

real	80m19.315s
user	0m0.464s
sys	50m33.525s

That stress command has been running for 80mins, but it was supposed to exit after 10 seconds!
There are still 3 stress processes running.

```
[ 5905.542236] sysrq: Show backtrace of all active CPUs
[ 5905.542244] NMI backtrace for cpu 8
[ 5905.542250] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.542253] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.542256] Call Trace:
[ 5905.542260]  <IRQ>
[ 5905.542271]  dump_stack+0x46/0x60
[ 5905.542276]  nmi_cpu_backtrace.cold+0x14/0x53
[ 5905.542283]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 5905.542287]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 5905.542294]  __handle_sysrq.cold+0x48/0x102
[ 5905.542299]  sysrq_filter+0x2ea/0x3b0
[ 5905.542306]  input_to_handler+0x4d/0xf0
[ 5905.542312]  input_pass_values.part.0+0x109/0x130
[ 5905.542319]  input_handle_event+0x171/0x5a0
[ 5905.542324]  input_event+0x4d/0x70
[ 5905.542331]  hidinput_report_event+0x2e/0x40
[ 5905.542337]  hid_report_raw_event+0x260/0x430
[ 5905.542343]  hid_input_report+0xfb/0x150
[ 5905.542350]  hid_irq_in+0x168/0x190
[ 5905.542358]  __usb_hcd_giveback_urb+0x77/0xe0
[ 5905.542376]  xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd]
[ 5905.542391]  xhci_td_cleanup+0xf7/0x140 [xhci_hcd]
[ 5905.542406]  xhci_irq+0x7e8/0x1be0 [xhci_hcd]
[ 5905.542413]  __handle_irq_event_percpu+0x2f/0xc0
[ 5905.542419]  handle_irq_event_percpu+0x2c/0x80
[ 5905.542424]  handle_irq_event+0x23/0x43
[ 5905.542431]  handle_edge_irq+0x78/0x190
[ 5905.542437]  handle_irq+0x17/0x20
[ 5905.542442]  do_IRQ+0x3e/0xd0
[ 5905.542450]  common_interrupt+0xf/0xf
[ 5905.542453]  </IRQ>
[ 5905.542460] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0
[ 5905.542466] Code: e8 d6 f9 a1 ff 49 89 c6 31 ff e8 fc 09 a2 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 f5 f3 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c
[ 5905.542470] RSP: 0018:ffffb2d883217e80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde
[ 5905.542475] RAX: ffff99f62da20a00 RBX: ffffffffa3061940 RCX: 000000000000001f
[ 5905.542478] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000
[ 5905.542482] RBP: 0000055efdb5caf8 R08: 0000055efdbed96b R09: 0000000000000384
[ 5905.542484] R10: ffff99f62da1fb44 R11: ffff99f62da1fb24 R12: ffff99f62da29a00
[ 5905.542487] R13: 0000000000000004 R14: 0000055efdbed96b R15: 0000000000000000
[ 5905.542495]  cpuidle_enter+0x24/0x40
[ 5905.542501]  do_idle+0x1c1/0x230
[ 5905.542507]  cpu_startup_entry+0x14/0x20
[ 5905.542514]  start_secondary+0x168/0x1b0
[ 5905.542522]  secondary_startup_64+0xa4/0xb0
[ 5905.542528] Sending NMI from CPU 8 to CPUs 0-7,9-11:
[ 5905.542583] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x7d/0x120
[ 5905.542835] NMI backtrace for cpu 1
[ 5905.542840] CPU: 1 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.542843] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.542844] RIP: 0010:queued_spin_lock_slowpath+0x3d/0x1a0
[ 5905.542848] Code: 41 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 1b 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 f6 c4 01 75 04 c6 47
[ 5905.542850] RSP: 0018:ffffb2d883fc7e50 EFLAGS: 00000002
[ 5905.542858] RAX: 00000000000c0101 RBX: ffff99f57ce54000 RCX: 0000000000080000
[ 5905.542860] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa355d2ac
[ 5905.542862] RBP: ffff99f57cd9e300 R08: ffffb2d883fa7e78 R09: ffffb2d883f8fe78
[ 5905.542864] R10: 0000000000000004 R11: ffff99f588c48600 R12: 0000000000001fff
[ 5905.542866] R13: 00005acd2cb39440 R14: ffff99f57ce54038 R15: ffff99f57cd9e300
[ 5905.542868] FS:  000070f212ad7740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000
[ 5905.542872] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.542874] CR2: 000078739192f578 CR3: 000000077cdac005 CR4: 00000000003606e0
[ 5905.542876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.542878] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.542879] Call Trace:
[ 5905.542880]  devkmsg_read+0xa2/0x270
[ 5905.542881]  ? finish_wait+0x80/0x80
[ 5905.542883]  vfs_read+0x98/0x150
[ 5905.542884]  ksys_read+0x5d/0xe0
[ 5905.542887]  do_syscall_64+0x50/0x170
[ 5905.542895]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5905.542896] RIP: 0033:0x70f212bf1415
[ 5905.542900] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 5905.542901] RSP: 002b:00007fffa4d874f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 5905.542905] RAX: ffffffffffffffda RBX: 00005acd2cb39440 RCX: 000070f212bf1415
[ 5905.542907] RDX: 0000000000001fff RSI: 00005acd2cb39440 RDI: 0000000000000003
[ 5905.542908] RBP: 00005acd2cb357fa R08: 000000000000000a R09: 0000000000000000
[ 5905.542913] R10: 00007fffa4d8732c R11: 0000000000000246 R12: 000000000000003c
[ 5905.542916] R13: 00005acd2cb34377 R14: 00005acd2cb39440 R15: 00005acd2cb39456
[ 5905.542919] NMI backtrace for cpu 2
[ 5905.542924] CPU: 2 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.542932] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.542936] RIP: 0010:queued_spin_lock_slowpath+0x16d/0x1a0
[ 5905.542944] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 16 02 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 e5 ec a2 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb
[ 5905.542948] RSP: 0018:ffffb2d884117e50 EFLAGS: 00000046
[ 5905.542963] RAX: 0000000000000000 RBX: ffff99f57ce60000 RCX: 00000000000c0000
[ 5905.542969] RDX: ffff99f62d8a16c0 RSI: 000000000000000b RDI: ffffffffa355d2ac
[ 5905.542975] RBP: ffff99f588e03a00 R08: 0000000000000002 R09: ffffb2d883fa7e78
[ 5905.542979] R10: 0000000000000002 R11: ffff99f588c49800 R12: 0000000000001fff
[ 5905.542982] R13: 000058517a066440 R14: ffff99f57ce60038 R15: ffff99f588e03a00
[ 5905.542988] FS:  00007c32c37f6740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000
[ 5905.542993] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.542997] CR2: 00007ecf94de9578 CR3: 000000077cd3c003 CR4: 00000000003606e0
[ 5905.543002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.543008] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.543013] Call Trace:
[ 5905.543018]  devkmsg_read+0xa2/0x270
[ 5905.543021]  ? finish_wait+0x80/0x80
[ 5905.543023]  vfs_read+0x98/0x150
[ 5905.543029]  ksys_read+0x5d/0xe0
[ 5905.543036]  do_syscall_64+0x50/0x170
[ 5905.543039]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5905.543042] RIP: 0033:0x7c32c3910415
[ 5905.543048] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 5905.543054] RSP: 002b:00007ffe934c7538 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 5905.543066] RAX: ffffffffffffffda RBX: 000058517a066440 RCX: 00007c32c3910415
[ 5905.543072] RDX: 0000000000001fff RSI: 000058517a066440 RDI: 0000000000000003
[ 5905.543076] RBP: 000058517a0627fa R08: 000000000000000a R09: 0000000000000000
[ 5905.543084] R10: 00007ffe934c736c R11: 0000000000000246 R12: 000000000000003c
[ 5905.543087] R13: 000058517a061377 R14: 000058517a066440 R15: 000058517a066456
[ 5905.543094] NMI backtrace for cpu 3
[ 5905.543100] CPU: 3 PID: 25752 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543106] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543112] RIP: 0010:isolate_migratepages_block+0x6/0xa50
[ 5905.543119] Code: ff 0f b7 43 60 ff c8 66 89 43 5a e9 8b fa ff ff e8 1f 82 ee ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 <41> 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08
[ 5905.543122] RSP: 0018:ffffb2d88bba39f8 EFLAGS: 00000202
[ 5905.543136] RAX: 0000000000000000 RBX: 00000000004f4000 RCX: 000000000000000c
[ 5905.543142] RDX: 00000000004f3e00 RSI: 00000000004f3c60 RDI: ffffb2d88bba3ad0
[ 5905.543149] RBP: ffffb2d88bba3ad0 R08: 0000000000000001 R09: ffff99f62d8e1450
[ 5905.543158] R10: 0000000000000001 R11: ffff99f62d8e0a80 R12: 00000000004f3e00
[ 5905.543163] R13: ffff99f64dfded00 R14: 00000000004f3c60 R15: ffffb2d88bba3ad0
[ 5905.543169] FS:  0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000
[ 5905.543174] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.543182] CR2: 00007ecf94e85ff8 CR3: 000000081439e002 CR4: 00000000003606e0
[ 5905.543190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.543195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.543200] Call Trace:
[ 5905.543205]  compact_zone+0x577/0xa70
[ 5905.543210]  compact_zone_order+0xde/0x120
[ 5905.543217]  try_to_compact_pages+0x187/0x240
[ 5905.543224]  __alloc_pages_direct_compact+0x87/0x170
[ 5905.543230]  __alloc_pages_slowpath+0x454/0xc20
[ 5905.543236]  ? release_pages+0x348/0x3b0
[ 5905.543243]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 5905.543250]  __alloc_pages_nodemask+0x268/0x2b0
[ 5905.543255]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 5905.543260]  __handle_mm_fault+0xc11/0x1890
[ 5905.543267]  handle_mm_fault+0xa9/0x1d0
[ 5905.543275]  __do_page_fault+0x237/0x480
[ 5905.543281]  do_page_fault+0x1d/0x67
[ 5905.543286]  ? page_fault+0x8/0x30
[ 5905.543294]  page_fault+0x1e/0x30
[ 5905.543299] RIP: 0033:0x5c2e1a191c10
[ 5905.543304] Code: Bad RIP value.
[ 5905.543310] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 5905.543324] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 5905.543331] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 5905.543337] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 5905.543343] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 5905.543348] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 5905.543360] NMI backtrace for cpu 4
[ 5905.543367] CPU: 4 PID: 25727 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543374] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543381] RIP: 0010:isolate_migratepages_block+0x20/0xa50
[ 5905.543388] Code: 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 89 4c 24 3c <65> 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 c0 48 8b 47
[ 5905.543393] RSP: 0018:ffffb2d88badb958 EFLAGS: 00000292
[ 5905.543403] RAX: 0000000000000001 RBX: 000000000058fcc0 RCX: 000000000000000c
[ 5905.543406] RDX: 000000000058fe00 RSI: 000000000058fcc0 RDI: ffffb2d88badbad0
[ 5905.543411] RBP: ffffb2d88badbad0 R08: ffff99f64dfd9b10 R09: ffff99f62d921450
[ 5905.543414] R10: 0000000000000001 R11: ffff99f62d920a80 R12: 000000000058fe00
[ 5905.543419] R13: ffff99f64dfded00 R14: ffffb2d88badbad0 R15: ffffb2d88badbad0
[ 5905.543423] FS:  0000720b85407740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000
[ 5905.543426] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.543431] CR2: 000079ad89768970 CR3: 00000007c23a4003 CR4: 00000000003606e0
[ 5905.543440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.543443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.543445] Call Trace:
[ 5905.543451]  compact_zone+0x577/0xa70
[ 5905.543454]  compact_zone_order+0xde/0x120
[ 5905.543459]  try_to_compact_pages+0x187/0x240
[ 5905.543463]  __alloc_pages_direct_compact+0x87/0x170
[ 5905.543466]  __alloc_pages_slowpath+0x20e/0xc20
[ 5905.543471]  ? release_pages+0x348/0x3b0
[ 5905.543475]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 5905.543480]  __alloc_pages_nodemask+0x268/0x2b0
[ 5905.543485]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 5905.543487]  __handle_mm_fault+0xc11/0x1890
[ 5905.543492]  handle_mm_fault+0xa9/0x1d0
[ 5905.543496]  __do_page_fault+0x237/0x480
[ 5905.543503]  do_page_fault+0x1d/0x67
[ 5905.543506]  ? page_fault+0x8/0x30
[ 5905.543510]  page_fault+0x1e/0x30
[ 5905.543513] RIP: 0033:0x5c2e1a191c10
[ 5905.543515] Code: Bad RIP value.
[ 5905.543520] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 5905.543529] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 5905.543532] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 5905.543636] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 5905.543642] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 5905.543645] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 5905.543648] NMI backtrace for cpu 6
[ 5905.543652] CPU: 6 PID: 1252 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543655] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543658] RIP: 0033:0x7c3fc3821982
[ 5905.543664] Code: 57 58 48 8b 57 18 48 8b 4f 48 48 89 4f 18 48 89 57 48 48 89 47 08 c3 90 f3 0f 1e fa 55 48 89 e5 41 54 53 48 89 fb 48 8b 47 28 <48> 3b 47 20 76 39 4c 8b a7 d8 00 00 00 48 8d 15 6a 4d 15 00 48 8d
[ 5905.543667] RSP: 002b:00007ffdfcb6a800 EFLAGS: 00000246
[ 5905.543677] RAX: 00007c3fc3975863 RBX: 00007c3fc39757e0 RCX: 0000000000000000
[ 5905.543680] RDX: 0000000000000000 RSI: 00007c3fc3975864 RDI: 00007c3fc39757e0
[ 5905.543685] RBP: 00007ffdfcb6a810 R08: 00007c3fc3976500 R09: 0000000000000000
[ 5905.543688] R10: 000063939c42a5af R11: 0000000000000246 R12: 00007c3fc3976700
[ 5905.543693] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468
[ 5905.543695] FS:  00007c3fc3582740 GS:  0000000000000000
[ 5905.543701] NMI backtrace for cpu 5
[ 5905.543707] CPU: 5 PID: 396 Comm: systemd-journal Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543711] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543713] RIP: 0010:find_get_pages_range_tag+0xbf/0x2a0
[ 5905.543718] Code: 49 c7 c5 ff ff ff ff eb 75 48 8b 42 08 48 8d 78 ff a8 01 48 0f 44 fa 8b 47 34 85 c0 74 3c 8d 48 01 4c 8d 47 34 f0 0f b1 4f 34 <75> ee 48 8b 4a 08 48 8d 41 ff 83 e1 01 48 0f 44 c2 48 39 c7 0f 84
[ 5905.543720] RSP: 0018:ffffb2d883503b68 EFLAGS: 00000246
[ 5905.543726] RAX: 0000000000000003 RBX: 0000000000000002 RCX: 0000000000000004
[ 5905.543731] RDX: ffffde0b9dcd6180 RSI: ffffb2d883503c38 RDI: ffffde0b9dcd6180
[ 5905.543734] RBP: 0000000000000238 R08: ffffde0b9dcd61b4 R09: ffffffffffffffff
[ 5905.543737] R10: ffffffffffffffc0 R11: 0000000000000000 R12: 0000000000000009
[ 5905.543740] R13: ffffffffffffffff R14: 0007ffffffffffff R15: ffffb2d883503c28
[ 5905.543743] FS:  00007d2b87dae840(0000) GS:ffff99f62d940000(0000) knlGS:0000000000000000
[ 5905.543746] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.543748] CR2: 00007d2b896b2010 CR3: 0000000826cce003 CR4: 00000000003606e0
[ 5905.543752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.543757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.543759] Call Trace:
[ 5905.543765]  pagevec_lookup_range_tag+0x1f/0x30
[ 5905.543767]  extent_write_cache_pages+0xe1/0x3a0
[ 5905.543770]  ? clear_state_bit+0xde/0x190
[ 5905.543773]  extent_writepages+0x4c/0xa0
[ 5905.543778]  do_writepages+0x3c/0xd0
[ 5905.543780]  ? btrfs_page_mkwrite+0x38e/0x480
[ 5905.543783]  __filemap_fdatawrite_range+0xec/0x130
[ 5905.543786]  btrfs_fdatawrite_range+0x16/0x40
[ 5905.543789]  start_ordered_ops+0x47/0x80
[ 5905.543792]  btrfs_sync_file+0xa1/0x3b0
[ 5905.543799]  do_fsync+0x33/0x70
[ 5905.543802]  __x64_sys_fsync+0xb/0x10
[ 5905.543805]  do_syscall_64+0x50/0x170
[ 5905.543807]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5905.543810] RIP: 0033:0x7d2b895b9c38
[ 5905.543813] Code: ff ff 77 3e c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 89 7d fc e8 58 bd f7 ff 41 89 c0 8b 7d fc b8 4a 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 22 89 45 fc 44 89 c7 e8 95 bd f7 ff 8b 45 fc
[ 5905.543818] RSP: 002b:00007fff5ff3eba0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[ 5905.543824] RAX: ffffffffffffffda RBX: 00005dc473d022d0 RCX: 00007d2b895b9c38
[ 5905.543829] RDX: 00007d2b896b2000 RSI: 00005dc473d2f550 RDI: 0000000000000024
[ 5905.543832] RBP: 00007fff5ff3ebb0 R08: 0000000000000000 R09: 0000000000000014
[ 5905.543835] R10: 00007fff5ff3eb80 R11: 0000000000000293 R12: 00007fff5ff3ecf0
[ 5905.543839] R13: 00005dc473d022d0 R14: 00007fff5ff3f250 R15: 0000000000000180
[ 5905.543843] NMI backtrace for cpu 7
[ 5905.543848] CPU: 7 PID: 25777 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543851] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543853] RIP: 0010:compact_unlock_should_abort.isra.0+0x39/0x50
[ 5905.543858] Code: 00 5d 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 <74> e0 c6 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00
[ 5905.543860] RSP: 0018:ffffb2d88bc6b940 EFLAGS: 00000202
[ 5905.543868] RAX: ffff99f370429e40 RBX: 0000000000718440 RCX: ffffb2d88bc6bb49
[ 5905.543870] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 5905.543872] RBP: ffffb2d88bc6bb49 R08: 0000000000000001 R09: ffff99f62d9e1450
[ 5905.543876] R10: 0000000000000000 R11: ffff99f62d9e0a80 R12: ffffb2d88bc6bad0
[ 5905.543878] R13: 0000000000000000 R14: ffffb2d88bc6bad0 R15: 0000000000000000
[ 5905.543881] FS:  0000720b85407740(0000) GS:ffff99f62d9c0000(0000) knlGS:0000000000000000
[ 5905.543886] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.543891] CR2: 00007873919cbff8 CR3: 00000007ccf6a003 CR4: 00000000003606e0
[ 5905.543893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.543895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.543898] Call Trace:
[ 5905.543903]  isolate_migratepages_block+0x2ed/0xa50
[ 5905.543907]  compact_zone+0x577/0xa70
[ 5905.543909]  compact_zone_order+0xde/0x120
[ 5905.543913]  try_to_compact_pages+0x187/0x240
[ 5905.543915]  __alloc_pages_direct_compact+0x87/0x170
[ 5905.543918]  __alloc_pages_slowpath+0x454/0xc20
[ 5905.543920]  ? release_pages+0x348/0x3b0
[ 5905.543925]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 5905.543930]  __alloc_pages_nodemask+0x268/0x2b0
[ 5905.543932]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 5905.543934]  __handle_mm_fault+0xc11/0x1890
[ 5905.543938]  handle_mm_fault+0xa9/0x1d0
[ 5905.543940]  __do_page_fault+0x237/0x480
[ 5905.543942]  do_page_fault+0x1d/0x67
[ 5905.543945]  ? page_fault+0x8/0x30
[ 5905.543947]  page_fault+0x1e/0x30
[ 5905.543949] RIP: 0033:0x5c2e1a191c10
[ 5905.543953] Code: Bad RIP value.
[ 5905.543958] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 5905.543962] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 5905.543964] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 5905.543967] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 5905.543969] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 5905.543973] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 5905.543976] NMI backtrace for cpu 9
[ 5905.543980] CPU: 9 PID: 1254 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.543985] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.543991] RIP: 0010:ktime_get_coarse_real_ts64+0x0/0x40
[ 5905.543996] Code: ef e8 c4 fb ff ff 85 c0 75 0f 48 8b 74 24 28 48 8b 44 24 30 e9 5d ff ff ff 41 89 c6 e9 6a ff ff ff e8 74 fa f8 ff 0f 1f 40 00 <8b> 15 7a 85 65 01 f6 c2 01 75 26 8b 05 93 85 65 01 c4 e2 fb f7 05
[ 5905.543999] RSP: 0018:ffffb2d883f1fd58 EFLAGS: 00000246
[ 5905.544005] RAX: 0000000000000000 RBX: ffff99f5c7d50250 RCX: 0000000000000000
[ 5905.544011] RDX: 0000000000000000 RSI: ffff99f5c7d50250 RDI: ffffb2d883f1fd60
[ 5905.544016] RBP: ffffb2d883f1fd80 R08: 0000000000000001 R09: 0000000000000000
[ 5905.544019] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f629139220
[ 5905.544022] R13: ffff99f5c8648878 R14: 0000000000000001 R15: ffff99f5c86a8cc0
[ 5905.544026] FS:  0000705c7f390740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000
[ 5905.544029] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.544032] CR2: 00007e392404fb50 CR3: 00000007c71a2004 CR4: 00000000003606e0
[ 5905.544035] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.544040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.544044] Call Trace:
[ 5905.544047]  current_time+0x24/0x80
[ 5905.544053]  atime_needs_update+0x6e/0xd0
[ 5905.544059]  touch_atime+0x2e/0xe0
[ 5905.544062]  pipe_read+0x2aa/0x2c0
[ 5905.544065]  new_sync_read+0x10f/0x1a0
[ 5905.544069]  vfs_read+0x98/0x150
[ 5905.544072]  ksys_read+0x5d/0xe0
[ 5905.544075]  do_syscall_64+0x50/0x170
[ 5905.544078]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5905.544081] RIP: 0033:0x705c7f6af415
[ 5905.544085] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 5905.544091] RSP: 002b:00007ffc9e431508 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 5905.544099] RAX: ffffffffffffffda RBX: 0000705c7f7837e0 RCX: 0000705c7f6af415
[ 5905.544104] RDX: 0000000000000001 RSI: 0000705c7f783863 RDI: 0000000000000000
[ 5905.544108] RBP: 00007ffc9e431550 R08: 0000000000000000 R09: 0000000000000000
[ 5905.544111] R10: 00005c354d18d56f R11: 0000000000000246 R12: 0000705c7f785300
[ 5905.544114] R13: 0000705c7f784500 R14: 0000705c7f784700 R15: 0000705c7f785468
[ 5905.544123] NMI backtrace for cpu 10
[ 5905.544127] CPU: 10 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.544130] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.544133] RIP: 0033:0x77d47d3fd535
[ 5905.544139] Code: 02 44 89 c7 75 0f e9 ea 1e 08 00 66 2e 0f 1f 84 00 00 00 00 00 e9 1b 7a 08 00 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 18 45 31 f6 48 85 d2
[ 5905.544144] RSP: 002b:00007ffe06db5e30 EFLAGS: 00000202
[ 5905.544151] RAX: 000077d47d555300 RBX: 000077d47d554500 RCX: 0000000000000c00
[ 5905.544154] RDX: 0000000000000001 RSI: 00005ffb57470462 RDI: 000077d47d554500
[ 5905.544158] RBP: 00007ffe06db5e80 R08: 00005ffb57470463 R09: 0000000000000000
[ 5905.544161] R10: 00007ffe06db5ebc R11: 00005ffb57470462 R12: 0000000000000001
[ 5905.544167] R13: 0000000000000001 R14: 0000000000000001 R15: 00005ffb57470462
[ 5905.544170] FS:  000077d47d365740 GS:  0000000000000000
[ 5905.544174] NMI backtrace for cpu 11
[ 5905.544178] CPU: 11 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 5905.544182] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 5905.544187] RIP: 0010:format_decode+0x3df/0x3f0
[ 5905.544194] Code: e9 58 fe ff ff 31 c0 e9 e1 fc ff ff c6 45 00 02 48 8d 41 02 29 d8 e9 d2 fc ff ff 83 c2 0b 88 55 00 e9 37 fe ff ff 48 8d 41 02 <48> 89 04 24 be 4c 00 00 00 e9 36 fd ff ff 0f 1f 00 41 54 55 53 49
[ 5905.544198] RSP: 0018:ffffb2d883fa7d40 EFLAGS: 00000046
[ 5905.544208] RAX: ffffffffa2e31773 RBX: ffffffffa2e31770 RCX: ffffffffa2e31771
[ 5905.544213] RDX: 0000000000000004 RSI: 000000000000006c RDI: ffffffffa2e31770
[ 5905.544219] RBP: ffffb2d883fa7d70 R08: ffff99f61c780059 R09: 0000000000000000
[ 5905.544223] R10: 0000000000000004 R11: ffff99f61c780059 R12: ffffffffa2e31770
[ 5905.544226] R13: 0000000000002000 R14: ffffffffa2e31770 R15: ffff99f61c78005a
[ 5905.544231] FS:  00007c7ea548a740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000
[ 5905.544238] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5905.544241] CR2: 000056eb222ca148 CR3: 00000007c97bc001 CR4: 00000000003606e0
[ 5905.544244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5905.544247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 5905.544253] Call Trace:
[ 5905.544258]  vsnprintf+0x6e/0x5b0
[ 5905.544262]  scnprintf+0x4d/0x90
[ 5905.544268]  msg_print_ext_header.constprop.0+0x7f/0xa0
[ 5905.544273]  ? _cond_resched+0x10/0x20
[ 5905.544277]  ? mutex_lock_interruptible+0x9/0x30
[ 5905.544280]  devkmsg_read+0x17f/0x270
[ 5905.544286]  vfs_read+0x98/0x150
[ 5905.544289]  ksys_read+0x5d/0xe0
[ 5905.544297]  do_syscall_64+0x50/0x170
[ 5905.544300]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5905.544303] RIP: 0033:0x7c7ea55a4415
[ 5905.544307] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 5905.544310] RSP: 002b:00007ffe25d18658 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 5905.544321] RAX: ffffffffffffffda RBX: 0000571ede0f0440 RCX: 00007c7ea55a4415
[ 5905.544327] RDX: 0000000000001fff RSI: 0000571ede0f0440 RDI: 0000000000000003
[ 5905.544330] RBP: 0000571ede0ec7fa R08: 000000000000000a R09: 0000000000000000
[ 5905.544333] R10: 00007ffe25d1848c R11: 0000000000000246 R12: 0000000000000041
[ 5905.544338] R13: 0000571ede0eb377 R14: 0000571ede0f0440 R15: 0000571ede0f0455


```
Comment 9 GYt2bW 2019-07-13 20:49:43 UTC
Now only two stress processes left:
(I actually tried to gdb attach to one this time, then tried to kill -9 another, and tried to kill -11 another, no effect - but eventually, on its own(!), one process/thread went away, so two left now)

```
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  27127.4 avail Mem 

  PID  %CPU COMMAND                                                                            PR  NI    VIRT    RES S USER     
25727  98.1 stress                                                                             20   0  980352  96196 R user     
25752  98.1 stress                                                                             20   0  980352 122820 R user     
```


```
[ 7163.859183] sysrq: Show backtrace of all active CPUs
[ 7163.859189] NMI backtrace for cpu 8
[ 7163.859196] CPU: 8 PID: 25727 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.859199] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.859202] Call Trace:
[ 7163.859206]  <IRQ>
[ 7163.859216]  dump_stack+0x46/0x60
[ 7163.859222]  nmi_cpu_backtrace.cold+0x14/0x53
[ 7163.859228]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 7163.859233]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 7163.859239]  __handle_sysrq.cold+0x48/0x102
[ 7163.859244]  sysrq_filter+0x2ea/0x3b0
[ 7163.859250]  input_to_handler+0x4d/0xf0
[ 7163.859257]  input_pass_values.part.0+0x109/0x130
[ 7163.859263]  input_handle_event+0x171/0x5a0
[ 7163.859269]  input_event+0x4d/0x70
[ 7163.859275]  hidinput_report_event+0x2e/0x40
[ 7163.859280]  hid_report_raw_event+0x260/0x430
[ 7163.859286]  hid_input_report+0xfb/0x150
[ 7163.859293]  hid_irq_in+0x168/0x190
[ 7163.859300]  __usb_hcd_giveback_urb+0x77/0xe0
[ 7163.859316]  xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd]
[ 7163.859331]  xhci_td_cleanup+0xf7/0x140 [xhci_hcd]
[ 7163.859346]  xhci_irq+0x7e8/0x1be0 [xhci_hcd]
[ 7163.859353]  __handle_irq_event_percpu+0x2f/0xc0
[ 7163.859359]  handle_irq_event_percpu+0x2c/0x80
[ 7163.859364]  handle_irq_event+0x23/0x43
[ 7163.859370]  handle_edge_irq+0x78/0x190
[ 7163.859376]  handle_irq+0x17/0x20
[ 7163.859381]  do_IRQ+0x3e/0xd0
[ 7163.859388]  common_interrupt+0xf/0xf
[ 7163.859391]  </IRQ>
[ 7163.859397] RIP: 0010:compact_zone+0x101/0xa70
[ 7163.859403] Code: 98 90 05 00 00 0f 94 44 24 56 e8 ca 0e 04 00 48 c7 44 24 08 00 00 00 00 65 48 8b 04 25 00 5d 01 00 48 89 44 24 58 49 8b 57 30 <b8> 09 00 00 00 c4 c2 fb f7 47 28 48 89 d1 48 c1 e9 09 48 39 c8 0f
[ 7163.859407] RSP: 0018:ffffb2d88badba18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffde
[ 7163.859412] RAX: ffffb2d88badbac0 RBX: 0000000000590000 RCX: 000000000058fe00
[ 7163.859415] RDX: 000000000058fcc0 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 7163.859418] RBP: ffffb2d88badbad0 R08: ffff99f64dfd9b10 R09: ffff99f62da21450
[ 7163.859420] R10: 0000000000000000 R11: ffff99f62da20a80 R12: 000000000058fcc0
[ 7163.859423] R13: ffff99f64dfded00 R14: 000000000058fcc0 R15: ffffb2d88badbad0
[ 7163.859429]  ? compact_zone+0x577/0xa70
[ 7163.859434]  compact_zone_order+0xde/0x120
[ 7163.859440]  try_to_compact_pages+0x187/0x240
[ 7163.859445]  __alloc_pages_direct_compact+0x87/0x170
[ 7163.859450]  __alloc_pages_slowpath+0x20e/0xc20
[ 7163.859456]  ? release_pages+0x348/0x3b0
[ 7163.859461]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 7163.859465]  __alloc_pages_nodemask+0x268/0x2b0
[ 7163.859470]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 7163.859476]  __handle_mm_fault+0xc11/0x1890
[ 7163.859483]  handle_mm_fault+0xa9/0x1d0
[ 7163.859489]  __do_page_fault+0x237/0x480
[ 7163.859495]  do_page_fault+0x1d/0x67
[ 7163.859499]  ? page_fault+0x8/0x30
[ 7163.859502]  page_fault+0x1e/0x30
[ 7163.859506] RIP: 0033:0x5c2e1a191c10
[ 7163.859514] Code: Bad RIP value.
[ 7163.859518] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 7163.859521] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 7163.859524] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 7163.859527] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 7163.859530] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 7163.859533] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 7163.859539] Sending NMI from CPU 8 to CPUs 0-7,9-11:
[ 7163.859585] NMI backtrace for cpu 0
[ 7163.859588] CPU: 0 PID: 6555 Comm: xz Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.859591] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.859592] RIP: 0033:0x719ad7a360f0
[ 7163.859595] Code: 85 1e 02 00 00 bf fe ff ff ff 8d 42 02 29 d7 01 f7 48 01 c7 eb 14 0f 1f 40 00 41 0f b6 34 04 48 83 c0 01 40 3a 74 01 ff 75 08 <48> 39 c7 41 89 c0 75 e8 41 29 d0 45 8d 68 ff 44 89 44 24 30 41 83
[ 7163.859597] RSP: 002b:0000719ab1d1bb50 EFLAGS: 00000246
[ 7163.859600] RAX: 0000000000000010 RBX: 0000000000000acc RCX: 0000719ab1430869
[ 7163.859602] RDX: 000000000000000b RSI: 0000000000000000 RDI: 000000000000004c
[ 7163.859604] RBP: 000000000000000c R08: 000000000000000f R09: 000000000000003a
[ 7163.859606] R10: 0000719ab1471163 R11: 00000000000408f9 R12: 0000719ab1471163
[ 7163.859607] R13: 00000000000009a5 R14: 0000719ab14df010 R15: 0000000000000032
[ 7163.859608] FS:  0000719ab1d1c700 GS:  0000000000000000
[ 7163.859684] NMI backtrace for cpu 1
[ 7163.859687] CPU: 1 PID: 6565 Comm: xz Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.859689] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.859691] RIP: 0033:0x719ad7a36320
[ 7163.859694] Code: 0f 84 2c f6 ff ff 89 c1 89 ea e9 c3 fc ff ff 0f 1f 00 44 89 de 48 c1 e0 07 48 8d 84 06 a4 41 00 00 41 8b 04 86 e9 0e fd ff ff <45> 85 db 78 1b 4c 8d 05 34 aa 00 00 c1 ee 12 41 0f b6 34 30 83 c6
[ 7163.859695] RSP: 002b:0000719ab0859b50 EFLAGS: 00000216
[ 7163.859698] RAX: 0000000000000002 RBX: 0000000000000591 RCX: 0000000000000001
[ 7163.859705] RDX: 0000000000000004 RSI: 0000000000570d43 RDI: 0000719ab001d018
[ 7163.859706] RBP: 0000000000000004 R08: 0000000000000000 R09: 00000000000003f3
[ 7163.859708] R10: 0000719aa56925b4 R11: 0000000000570d43 R12: 0000719aa56925b4
[ 7163.859709] R13: 00000000000003ed R14: 0000719ab001d010 R15: 0000000000000021
[ 7163.859711] FS:  0000719ab085a700 GS:  0000000000000000
[ 7163.859719] NMI backtrace for cpu 2
[ 7163.859726] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.859734] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.859745] RIP: 0010:tick_nohz_idle_exit+0x26/0xa0
[ 7163.859754] Code: 00 0f 1f 00 41 55 41 54 55 53 48 c7 c5 80 cf 01 00 65 48 03 2d e3 65 ef 5d fa f6 45 4c 01 74 7a 48 83 bd b0 00 00 00 00 75 6c <0f> b6 5d 4c 83 e3 fe 88 5d 4c 41 89 dc 41 c0 ec 02 d0 eb 83 e3 01
[ 7163.859757] RSP: 0018:ffffb2d8831e7eb0 EFLAGS: 00000046
[ 7163.859765] RAX: 0000000080204008 RBX: 0000000000000002 RCX: ffff99f62d8a1700
[ 7163.859771] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff99f62d8a9a00
[ 7163.859773] RBP: ffff99f62d89cf80 R08: 0000000000000000 R09: 0000000000000daf
[ 7163.859776] R10: ffff99f62d89fb44 R11: ffff99f62d89fb24 R12: ffff99f62a20bc80
[ 7163.859782] R13: ffff99f62a20bc80 R14: ffff99f62d8a9a00 R15: ffffffffa3061940
[ 7163.859785] FS:  0000000000000000(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000
[ 7163.859787] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7163.859795] CR2: 00007112d0bcfb50 CR3: 000000080600a003 CR4: 00000000003606e0
[ 7163.859798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7163.859805] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7163.859809] Call Trace:
[ 7163.859812]  do_idle+0x137/0x230
[ 7163.859817]  cpu_startup_entry+0x14/0x20
[ 7163.859821]  start_secondary+0x168/0x1b0
[ 7163.859824]  secondary_startup_64+0xa4/0xb0
[ 7163.859967] NMI backtrace for cpu 3
[ 7163.859972] CPU: 3 PID: 25752 Comm: stress Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.859978] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.859983] RIP: 0010:compact_unlock_should_abort.isra.0+0x39/0x50
[ 7163.859988] Code: 00 5d 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 <74> e0 c6 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00
[ 7163.859993] RSP: 0018:ffffb2d88bba3940 EFLAGS: 00000202
[ 7163.860000] RAX: ffff99f517c53c80 RBX: 00000000004f3c60 RCX: ffffb2d88bba3b49
[ 7163.860004] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00
[ 7163.860009] RBP: ffffb2d88bba3b49 R08: 0000000000000001 R09: ffff99f62d8e1450
[ 7163.860013] R10: 0000000000000000 R11: ffff99f57cfb4e00 R12: ffffb2d88bba3ad0
[ 7163.860017] R13: 0000000000000000 R14: ffffb2d88bba3ad0 R15: 0000000000000000
[ 7163.860022] FS:  0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000
[ 7163.860027] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7163.860034] CR2: 000078739192f578 CR3: 000000081439e002 CR4: 00000000003606e0
[ 7163.860039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7163.860043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7163.860046] Call Trace:
[ 7163.860051]  isolate_migratepages_block+0x2ed/0xa50
[ 7163.860055]  compact_zone+0x577/0xa70
[ 7163.860059]  compact_zone_order+0xde/0x120
[ 7163.860064]  try_to_compact_pages+0x187/0x240
[ 7163.860068]  __alloc_pages_direct_compact+0x87/0x170
[ 7163.860073]  __alloc_pages_slowpath+0x454/0xc20
[ 7163.860077]  ? release_pages+0x348/0x3b0
[ 7163.860081]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 7163.860086]  __alloc_pages_nodemask+0x268/0x2b0
[ 7163.860090]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 7163.860094]  __handle_mm_fault+0xc11/0x1890
[ 7163.860101]  handle_mm_fault+0xa9/0x1d0
[ 7163.860103]  __do_page_fault+0x237/0x480
[ 7163.860108]  do_page_fault+0x1d/0x67
[ 7163.860110]  ? page_fault+0x8/0x30
[ 7163.860115]  page_fault+0x1e/0x30
[ 7163.860119] RIP: 0033:0x5c2e1a191c10
[ 7163.860124] Code: Bad RIP value.
[ 7163.860128] RSP: 002b:00007fff325961b0 EFLAGS: 00010206
[ 7163.860136] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db
[ 7163.860140] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000
[ 7163.860145] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000
[ 7163.860147] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff
[ 7163.860152] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00
[ 7163.860159] NMI backtrace for cpu 4
[ 7163.860164] CPU: 4 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860167] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860172] RIP: 0010:queued_spin_lock_slowpath+0x16d/0x1a0
[ 7163.860176] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 16 02 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 e5 ec a2 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb
[ 7163.860180] RSP: 0018:ffffb2d884117e50 EFLAGS: 00000046
[ 7163.860188] RAX: 0000000000000000 RBX: ffff99f57ce60000 RCX: 0000000000140000
[ 7163.860191] RDX: ffff99f62d9216c0 RSI: 0000000000000001 RDI: ffffffffa355d2ac
[ 7163.860196] RBP: ffff99f588e03a00 R08: 0000000000000004 R09: 0000000000000000
[ 7163.860198] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001fff
[ 7163.860203] R13: 000058517a066440 R14: ffff99f57ce60038 R15: ffff99f588e03a00
[ 7163.860207] FS:  00007c32c37f6740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000
[ 7163.860210] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7163.860213] CR2: 000078739192f578 CR3: 000000077cd3c006 CR4: 00000000003606e0
[ 7163.860215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7163.860221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7163.860224] Call Trace:
[ 7163.860227]  devkmsg_read+0x7f/0x270
[ 7163.860229]  vfs_read+0x98/0x150
[ 7163.860233]  ksys_read+0x5d/0xe0
[ 7163.860237]  do_syscall_64+0x50/0x170
[ 7163.860240]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 7163.860244] RIP: 0033:0x7c32c3910415
[ 7163.860248] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 7163.860254] RSP: 002b:00007ffe934c7538 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 7163.860260] RAX: ffffffffffffffda RBX: 000058517a066440 RCX: 00007c32c3910415
[ 7163.860265] RDX: 0000000000001fff RSI: 000058517a066440 RDI: 0000000000000003
[ 7163.860269] RBP: 000058517a0627fa R08: 000000000000000a R09: 0000000000000000
[ 7163.860274] R10: 00007ffe934c736c R11: 0000000000000246 R12: 000000000000000b
[ 7163.860276] R13: 000058517a061377 R14: 000058517a066440 R15: 000058517a066455
[ 7163.860280] NMI backtrace for cpu 5
[ 7163.860287] CPU: 5 PID: 1098 Comm: panel-24-cpugra Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860290] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860297] RIP: 0033:0x7b5fc1db0606
[ 7163.860304] Code: 28 f2 0f 11 47 28 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f2 0f 10 06 f2 0f 10 0a f2 0f 10 57 08 f2 0f 10 5f 18 f2 0f 59 d0 <f2> 0f 59 d9 f2 0f 59 07 f2 0f 59 4f 10 f2 0f 58 d3 f2 0f 58 c1 f2
[ 7163.860309] RSP: 002b:00007ffd6a2c6e88 EFLAGS: 00000206
[ 7163.860317] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000000076
[ 7163.860320] RDX: 00007ffd6a2c6ed8 RSI: 00007ffd6a2c6ed0 RDI: 00007ffd6a2c7f80
[ 7163.860323] RBP: 00007ffd6a2c7f80 R08: 00007ffd6a2c76c0 R09: 0000000000000000
[ 7163.860328] R10: 0000000000000083 R11: 000055caec6febb8 R12: 00007ffd6a2c6fd8
[ 7163.860331] R13: 00007ffd6a2c7058 R14: 00007ffd6a2c6eb0 R15: 00007ffd6a2c6eb8
[ 7163.860335] FS:  00007b5fbfea8980 GS:  0000000000000000
[ 7163.860339] NMI backtrace for cpu 6
[ 7163.860342] CPU: 6 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860347] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860350] RIP: 0033:0x77d47d3f0d6c
[ 7163.860356] Code: 4c 89 fe 48 89 df 48 8b 45 c8 ff 50 38 49 89 c0 48 83 f8 ff 41 0f 94 c1 f7 03 00 80 00 00 75 26 48 8b bb 88 00 00 00 ff 4f 04 <75> 1a 48 c7 47 08 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 0f 85
[ 7163.860360] RSP: 002b:00007ffe06db5e40 EFLAGS: 00000246
[ 7163.860368] RAX: 0000000000000001 RBX: 000077d47d554500 RCX: 0000000000000000
[ 7163.860372] RDX: 0000000000000001 RSI: 00005ffb57470470 RDI: 000077d47d556320
[ 7163.860378] RBP: 00007ffe06db5e80 R08: 0000000000000001 R09: 0000000000000000
[ 7163.860383] R10: 00007ffe06db5ebc R11: 00005ffb57470470 R12: 0000000000000001
[ 7163.860386] R13: 0000000000000001 R14: 0000000000000001 R15: 00005ffb57470470
[ 7163.860390] FS:  000077d47d365740 GS:  0000000000000000
[ 7163.860395] NMI backtrace for cpu 7
[ 7163.860398] CPU: 7 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860400] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860410] RIP: 0033:0x70f212b08368
[ 7163.860415] Code: 45 a0 48 83 7f 68 00 74 11 48 c1 c8 11 64 48 33 04 25 30 00 00 00 48 89 45 a0 45 85 ed 0f 85 df 07 00 00 48 8b 45 a8 4c 8b 18 <48> 8b 7d 98 48 85 ff 48 89 f8 49 0f 44 c4 4c 8b 10 4d 8b 6c 24 08
[ 7163.860417] RSP: 002b:00007fffa4d87190 EFLAGS: 00000246
[ 7163.860426] RAX: 00007fffa4d87270 RBX: 00005acd2cb3947b RCX: 00005acd2cb3947b
[ 7163.860430] RDX: 00007fffa4d87270 RSI: 00007fffa4d87280 RDI: 00005acd2cc741d0
[ 7163.860434] RBP: 00007fffa4d87240 R08: 0000000000000000 R09: 0000000000000001
[ 7163.860439] R10: 00007fffa4d8732c R11: 00005acd2cb39456 R12: 00007fffa4d87280
[ 7163.860445] R13: 0000000000000000 R14: 00005acd2cb3947b R15: 00005acd2cc71b10
[ 7163.860449] FS:  000070f212ad7740 GS:  0000000000000000
[ 7163.860452] NMI backtrace for cpu 9
[ 7163.860460] CPU: 9 PID: 1238 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860465] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860472] RIP: 0010:_raw_spin_lock+0xb/0x20
[ 7163.860478] Code: 81 05 fd 94 82 5d 00 02 00 00 31 c0 ba ff 00 00 00 f0 0f b1 17 75 01 c3 e9 22 f1 8f ff 66 90 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 01 c3 89 c6 e9 db e0 8f ff 66 66 2e 0f 1f 84 00 00 00 00 00 65
[ 7163.860486] RSP: 0018:ffffb2d884137cb8 EFLAGS: 00000046
[ 7163.860491] RAX: 0000000000000000 RBX: ffff99f57ce09e40 RCX: 0000000000000009
[ 7163.860499] RDX: 0000000000000001 RSI: ffff99f62da40000 RDI: ffff99f62da60a00
[ 7163.860504] RBP: ffff99f62da60a00 R08: 0000000000000009 R09: ffffb2d884137d90
[ 7163.860507] R10: 0000000000000000 R11: 0000000000000009 R12: 0000000000000001
[ 7163.860511] R13: 0000000000000046 R14: ffff99f57ce0a584 R15: 0000000000020a00
[ 7163.860519] FS:  00007d2186c2e740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000
[ 7163.860524] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7163.860531] CR2: 00007743d494ec40 CR3: 0000000788c3c005 CR4: 00000000003606e0
[ 7163.860537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7163.860540] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7163.860546] Call Trace:
[ 7163.860554]  try_to_wake_up+0x18d/0x500
[ 7163.860556]  autoremove_wake_function+0xc/0x50
[ 7163.860559]  __wake_up_common+0x7a/0x190
[ 7163.860562]  __wake_up_common_lock+0x79/0xc0
[ 7163.860565]  pipe_write+0x1e3/0x410
[ 7163.860568]  new_sync_write+0x116/0x1b0
[ 7163.860571]  vfs_write+0xb1/0x190
[ 7163.860573]  ksys_write+0x5d/0xe0
[ 7163.860580]  do_syscall_64+0x50/0x170
[ 7163.860585]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 7163.860591] RIP: 0033:0x7d2186f4d4ba
[ 7163.860596] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20
[ 7163.860602] RSP: 002b:00007ffd33328988 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 7163.860612] RAX: ffffffffffffffda RBX: 0000000000000022 RCX: 00007d2186f4d4ba
[ 7163.860618] RDX: 0000000000000022 RSI: 0000647dee7025a0 RDI: 0000000000000001
[ 7163.860622] RBP: 00007ffd333289b0 R08: 0000000000000000 R09: 0000000000000000
[ 7163.860628] R10: 0000647dee707571 R11: 0000000000000246 R12: 0000000000000022
[ 7163.860632] R13: 0000647dee7025a0 R14: 00007d2187022500 R15: 00007d2187023300
[ 7163.860640] NMI backtrace for cpu 10
[ 7163.860645] CPU: 10 PID: 1252 Comm: sed Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860648] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860651] RIP: 0033:0x7c3fc3820b9e
[ 7163.860657] Code: 00 4c 89 f8 4c 29 e0 48 89 45 c8 4d 89 e6 81 e2 88 02 00 00 81 fa 80 02 00 00 0f 84 5c 01 00 00 41 f7 45 00 00 80 00 00 75 40 <49> 8b bd 88 00 00 00 8b 47 04 ff c8 89 47 04 75 2f 48 c7 47 08 00
[ 7163.860663] RSP: 002b:00007ffdfcb6a820 EFLAGS: 00000246
[ 7163.860673] RAX: 0000000000000d68 RBX: 00007c3fc39757e0 RCX: 0000000000000000
[ 7163.860678] RDX: 0000000000000080 RSI: 00007c3fc3975864 RDI: 00007c3fc3978320
[ 7163.860685] RBP: 00007ffdfcb6a860 R08: 00007c3fc3976500 R09: 0000000000000000
[ 7163.860690] R10: 000063939c42a5b0 R11: 0000000000000246 R12: 00007c3fc3976700
[ 7163.860696] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468
[ 7163.860703] FS:  00007c3fc3582740 GS:  0000000000000000
[ 7163.860707] NMI backtrace for cpu 11
[ 7163.860710] CPU: 11 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #22
[ 7163.860715] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 7163.860718] RIP: 0033:0x7c7ea5550a75
[ 7163.860726] Code: 8d 05 07 b1 12 00 48 0f 44 c8 48 89 f2 48 89 fe 31 ff e9 0e 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 68 48 89 fb 49 89 f4
[ 7163.860730] RSP: 002b:00007ffe25d18450 EFLAGS: 00000202
[ 7163.860737] RAX: 0000571ede0f0455 RBX: 0000000000000010 RCX: 00007ffe25d18490
[ 7163.860742] RDX: 0000000000000031 RSI: 0000571ede0f0465 RDI: 00007ffe25d1848c
[ 7163.860747] RBP: 0000571ede0f0465 R08: 0000000000000001 R09: 0000000000000000
[ 7163.860751] R10: 00007ffe25d1848c R11: 0000571ede0f0464 R12: 00007c7ea5679500
[ 7163.860754] R13: 00007ffe25d18490 R14: 0000571ede0eb071 R15: 0000000000000001
[ 7163.860764] FS:  00007c7ea548a740 GS:  0000000000000000
```
Comment 10 GYt2bW 2019-07-13 20:54:07 UTC
As you can see Comment 8 got hillariously botched! (must be my usage of markdown)
https://bugzilla.mozilla.org/show_bug.cgi?id=1557932
Comment 11 GYt2bW 2019-07-14 14:49:45 UTC
(I mean, if you look at what you've received as notifications in email, for each of my comments, you see how they differ with what is in here!)
Comment 12 howaboutsynergy 2019-07-15 23:45:18 UTC
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 15, 2019 11:25 PM, Andrew Morton <akpm@linux-foundation.org> wrote:

> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
Roger that.

> 

> On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 

> > https://bugzilla.kernel.org/show_bug.cgi?id=204165
> > 

> >             Bug ID: 204165
> >            Summary: 100% CPU usage in compact_zone_order
> >     

> 

> Looks like we have a lockup in compact_zone()
> 

> >            Product: Memory Management
> >            Version: 2.5
> >     Kernel Version: 5.2.0-g0ecfebd2b524
> >           Hardware: x86-64
> >                 OS: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Page Allocator
> >           Assignee: akpm@linux-foundation.org
> >           Reporter: howaboutsynergy@pm.me
> >         Regression: No
> >     

> 

> I assume this should be "yes". Did previous kernels exhibit this
> behavior or is it new in 5.2?

Regression: yes? 

I'm not sure... 

tl;dr: seen with kernel linux-stable 5.1.8.r0.g937cc0cc22a2-1
And really not sure if I've seen it before.

long read:
At least one previous kernel did because I've seen this before in https://bugzilla.kernel.org/show_bug.cgi?id=203833#c1 where I thought it was due to 'teo' governor but it wasn't (I'm using 'menu' gov. now)

I didn't mention kernel version there, but according to irc logs where I seek'd help at the time, the date was June 10th 2019 3am, checking which kernel I was running at the time (thanks to my q1q repo. git logs) it was 5.1.8 (stable)kernel to which I updated on Sun Jun 9 10:50:09 2019 +0200 and have not changed until Tue Jun 11 05:16:59 2019 +0200
local/linux-stable 5.1.8.r0.g937cc0cc22a2-1 (builtbydaddy)

Side note:
I've encountered something similar here https://github.com/constantoverride/qubes-linux-kernel/issues/2
 where kworker would randomly start using 100% CPU and couldn't be killed. The kernel there was 4.18.7, the VM had very little RAM in Qubes there. Since it's probably a different bug due to that stacktrace being so different/unrelated, please ignore.
Comment 13 howaboutsynergy 2019-07-16 01:08:35 UTC
Created attachment 283715 [details]
publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, July 15, 2019 11:25 PM, Andrew Morton <akpm@linux-foundation.org> wrote:

> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 

> On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 

> > https://bugzilla.kernel.org/show_bug.cgi?id=204165
> > 

> >             Bug ID: 204165
> >            Summary: 100% CPU usage in compact_zone_order
> >     

> 

> Looks like we have a lockup in compact_zone()
> 

> >            Product: Memory Management
> >            Version: 2.5
> >     Kernel Version: 5.2.0-g0ecfebd2b524
> >           Hardware: x86-64
> >                 OS: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Page Allocator
> >           Assignee: akpm@linux-foundation.org
> >           Reporter: howaboutsynergy@pm.me
> >         Regression: No
> >     

> 

> I assume this should be "yes". Did previous kernels exhibit this
> behavior or is it new in 5.2?

Is the reason that you switched to email somehow related to bugzilla not even posting this very reply of yours(seen above) that you sent via email even though you included bugzilla in the CC ?! I noticed that only mine got posted and mine was after yours in the emails. Yet there's no trace of yours when looking at the issue through the bugzilla interface.

Your (emailed)comment should be right after this https://bugzilla.kernel.org/show_bug.cgi?id=204165#c11
but it's not.

Anyway, I'm ready to test stuff, please let me know how I can help.
Meanwhile I'll try to find a better way to replicate the issue.
Comment 14 howaboutsynergy 2019-07-16 01:08:36 UTC
Created attachment 283717 [details]
signature.asc
Comment 15 howaboutsynergy 2019-07-16 02:01:40 UTC
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, July 16, 2019 1:28 AM, <howaboutsynergy@protonmail.com> wrote:

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, July 15, 2019 11:25 PM, Andrew Morton akpm@linux-foundation.org
> wrote:
>
> > (switched to email. Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
>
> Roger that.
>
> > On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org
> wrote:
> >
> > > https://bugzilla.kernel.org/show_bug.cgi?id=204165
> > >
> > >             Bug ID: 204165
> > >            Summary: 100% CPU usage in compact_zone_order
> > >
> >
> > Looks like we have a lockup in compact_zone()
> >
> > >            Product: Memory Management
> > >            Version: 2.5
> > >     Kernel Version: 5.2.0-g0ecfebd2b524
> > >           Hardware: x86-64
> > >                 OS: Linux
> > >               Tree: Mainline
> > >             Status: NEW
> > >           Severity: normal
> > >           Priority: P1
> > >          Component: Page Allocator
> > >           Assignee: akpm@linux-foundation.org
> > >           Reporter: howaboutsynergy@pm.me
> > >         Regression: No
> > >
> >
> > I assume this should be "yes". Did previous kernels exhibit this
> > behavior or is it new in 5.2?
>
> Regression: yes?
> I'm not sure...
> tl;dr: seen with kernel linux-stable 5.1.8.r0.g937cc0cc22a2-1
> And really not sure if I've seen it before.
>
> long read:
> At least one previous kernel did because I've seen this before in
> https://bugzilla.kernel.org/show_bug.cgi?id=203833#c1 where I thought it was
> due to 'teo' governor but it wasn't (I'm using 'menu' gov. now)
>
> I didn't mention kernel version there, but according to irc logs where I
> seek'd help at the time, the date was June 10th 2019 3am, checking which
> kernel I was running at the time (thanks to my q1q repo. git logs) it was
> 5.1.8 (stable)kernel to which I updated on Sun Jun 9 10:50:09 2019 +0200 and
> have not changed until Tue Jun 11 05:16:59 2019 +0200
> local/linux-stable 5.1.8.r0.g937cc0cc22a2-1 (builtbydaddy)
>
> Side note:
> I've encountered something similar here
> https://github.com/constantoverride/qubes-linux-kernel/issues/2
> where kworker would randomly start using 100% CPU and couldn't be killed. The
> kernel there was 4.18.7, the VM had very little RAM in Qubes there. Since
> it's probably a different bug due to that stacktrace being so
> different/unrelated, please ignore.

Well after a maybe 50+ tries, got it triggered on kernel 5.2.1-g527a3db363a3

```
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [19317] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: FAIL: [19317] (415) <-- worker 19536 got signal 9
stress: WARN: [19317] (417) now reaping child worker processes
stress: FAIL: [19317] (415) <-- worker 19530 got signal 9
stress: WARN: [19317] (417) now reaping child worker processes
```


MiB Mem :  31745.3 total,  28945.2 free,   2055.4 used,    744.7 buff/cache
MiB Swap:  65536.0 total,  63410.3 free,   2125.7 used.  28774.5 avail Mem

  PID  %CPU COMMAND                                                                            PR  NI    VIRT    RES S USER
19335 100.0 stress                                                                             20   0 9769416     12 R user
19376 100.0 stress                                                                             20   0 9769416      0 R user
19418 100.0 stress                                                                             20   0 9769416     12 R user
19490 100.0 stress                                                                             20   0 9769416      8 R user
21020 100.0 dmesg                                                                              20   0    6112   2636 R user


```
[ 1572.246553] sysrq: Show backtrace of all active CPUs
[ 1572.246563] NMI backtrace for cpu 8
[ 1572.246572] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.246579] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.246582] Call Trace:
[ 1572.246589]  <IRQ>
[ 1572.246603]  dump_stack+0x46/0x60
[ 1572.246615]  nmi_cpu_backtrace.cold+0x14/0x53
[ 1572.246623]  ? lapic_can_unplug_cpu.cold+0x42/0x42
[ 1572.246634]  nmi_trigger_cpumask_backtrace+0x8e/0x90
[ 1572.246641]  __handle_sysrq.cold+0x48/0x102
[ 1572.246650]  sysrq_filter+0x2ea/0x3b0
[ 1572.246658]  input_to_handler+0x4d/0xf0
[ 1572.246668]  input_pass_values.part.0+0x109/0x130
[ 1572.246675]  input_handle_event+0x171/0x5a0
[ 1572.246684]  input_event+0x4d/0x70
[ 1572.246694]  hidinput_report_event+0x2e/0x40
[ 1572.246703]  hid_report_raw_event+0x260/0x430
[ 1572.246710]  hid_input_report+0xfb/0x150
[ 1572.246721]  hid_irq_in+0x168/0x190
[ 1572.246730]  __usb_hcd_giveback_urb+0x77/0xe0
[ 1572.246754]  xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd]
[ 1572.246777]  xhci_td_cleanup+0xf7/0x140 [xhci_hcd]
[ 1572.246799]  xhci_irq+0x7e8/0x1be0 [xhci_hcd]
[ 1572.246811]  __handle_irq_event_percpu+0x2f/0xc0
[ 1572.246820]  handle_irq_event_percpu+0x2c/0x80
[ 1572.246827]  handle_irq_event+0x23/0x43
[ 1572.246838]  handle_edge_irq+0x78/0x190
[ 1572.246846]  handle_irq+0x17/0x20
[ 1572.246857]  do_IRQ+0x3e/0xd0
[ 1572.246865]  common_interrupt+0xf/0xf
[ 1572.246872]  </IRQ>
[ 1572.246880] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0
[ 1572.246890] Code: e8 d6 1c a1 ff 49 89 c6 31 ff e8 fc 2c a1 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 d5 18 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c
[ 1572.246895] RSP: 0018:ffffbd38c3217e80 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffde
[ 1572.246904] RAX: ffff9699ada5cb00 RBX: ffffffff93062340 RCX: 000000000000001f
[ 1572.246909] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000
[ 1572.246916] RBP: 0000016e1126287d R08: 0000016e11320f1b R09: 000000007fffffff
[ 1572.246921] R10: ffff9699ada5bc44 R11: ffff9699ada5bc24 R12: ffff9699ada65b00
[ 1572.246925] R13: 0000000000000006 R14: 0000016e11320f1b R15: 0000000000000000
[ 1572.246936]  cpuidle_enter+0x24/0x40
[ 1572.246945]  do_idle+0x1c1/0x230
[ 1572.246955]  cpu_startup_entry+0x14/0x20
[ 1572.246963]  start_secondary+0x168/0x1b0
[ 1572.246972]  secondary_startup_64+0xa4/0xb0
[ 1572.246981] Sending NMI from CPU 8 to CPUs 0-7,9-11:
[ 1572.247108] NMI backtrace for cpu 0
[ 1572.247110] CPU: 0 PID: 19376 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247111] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247112] RIP: 0010:compact_zone+0x3cf/0xa70
[ 1572.247115] Code: 7f 70 02 74 07 83 ce 04 89 74 24 50 45 0f b6 6f 74 48 8b 55 60 45 84 ed 0f 85 5d 02 00 00 48 39 d0 0f 84 3d 02 00 00 48 89 c7 <48> 81 e7 00 fe ff ff a9 ff 01 00 00 0f 84 28 02 00 00 49 89 c4 48
[ 1572.247116] RSP: 0018:ffffbd38cfca7a18 EFLAGS: 00000202
[ 1572.247118] RAX: 000000000022c720 RBX: 000000000022ca00 RCX: 0000000000001163
[ 1572.247119] RDX: 0000000000100000 RSI: 000000000000000c RDI: 000000000022c720
[ 1572.247122] RBP: ffff9699cdfded00 R08: 0000000000000001 R09: ffff9699ad65d550
[ 1572.247123] R10: 0000000000000001 R11: ffff9699ad65cb80 R12: 0000000000000009
[ 1572.247124] R13: 0000000000000000 R14: 000000000022c720 R15: ffffbd38cfca7ad0
[ 1572.247125] FS:  0000710f1082f740(0000) GS:ffff9699ad600000(0000) knlGS:0000000000000000
[ 1572.247126] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.247127] CR2: 000056d077f9cea8 CR3: 000000027ed88005 CR4: 00000000003606f0
[ 1572.247128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.247129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.247130] Call Trace:
[ 1572.247130]  compact_zone_order+0xde/0x120
[ 1572.247131]  try_to_compact_pages+0x187/0x240
[ 1572.247132]  __alloc_pages_direct_compact+0x87/0x170
[ 1572.247133]  __alloc_pages_slowpath+0x454/0xc20
[ 1572.247134]  ? release_pages+0x348/0x3b0
[ 1572.247134]  __alloc_pages_nodemask+0x268/0x2b0
[ 1572.247137]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1572.247138]  __handle_mm_fault+0xc0c/0x1310
[ 1572.247139]  handle_mm_fault+0xa9/0x1d0
[ 1572.247139]  __do_page_fault+0x237/0x480
[ 1572.247140]  do_page_fault+0x1d/0x67
[ 1572.247141]  ? page_fault+0x8/0x30
[ 1572.247142]  page_fault+0x1e/0x30
[ 1572.247143] RIP: 0033:0x5f7c81c80c10
[ 1572.247145] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1572.247145] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206
[ 1572.247147] RAX: 0000000016e90000 RBX: 0000710cbc770010 RCX: 0000710f109543db
[ 1572.247148] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000
[ 1572.247149] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000
[ 1572.247152] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1572.247153] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1572.247212] NMI backtrace for cpu 1
[ 1572.247214] CPU: 1 PID: 19335 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247216] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247217] RIP: 0010:isolate_migratepages_block+0x267/0xa50
[ 1572.247219] Code: 00 00 00 c4 e2 f9 f7 c6 48 8d 5c 03 ff 45 84 ed 0f 85 17 ff ff ff 48 ff c3 48 39 5c 24 08 0f 87 46 ff ff ff 4d 89 d7 4d 89 e6 <48> 89 d8 0f b6 54 24 77 48 3b 5c 24 08 0f 87 25 02 00 00 84 d2 0f
[ 1572.247220] RSP: 0018:ffffbd38cfb5f958 EFLAGS: 00000202
[ 1572.247223] RAX: 0000000000000001 RBX: 000000000018c060 RCX: ffffbd38cfb5fb49
[ 1572.247224] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff9699cdfffb00
[ 1572.247225] RBP: ffff9699cdfde000 R08: 0000000000000001 R09: ffff9699ad6dd550
[ 1572.247226] R10: 0000000000000000 R11: ffff969946236600 R12: ffffbd38cfb5fad0
[ 1572.247230] R13: 0000000000000000 R14: ffffbd38cfb5fad0 R15: 0000000000000000
[ 1572.247232] FS:  0000710f1082f740(0000) GS:ffff9699ad680000(0000) knlGS:0000000000000000
[ 1572.247233] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.247234] CR2: 00007f6eb6d40faa CR3: 000000080b004002 CR4: 00000000003606e0
[ 1572.247235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.247237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.247237] Call Trace:
[ 1572.247238]  compact_zone+0x577/0xa70
[ 1572.247239]  compact_zone_order+0xde/0x120
[ 1572.247241]  try_to_compact_pages+0x187/0x240
[ 1572.247242]  __alloc_pages_direct_compact+0x87/0x170
[ 1572.247248]  __alloc_pages_slowpath+0x454/0xc20
[ 1572.247259]  ? release_pages+0x348/0x3b0
[ 1572.247265]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1572.247266]  __alloc_pages_nodemask+0x268/0x2b0
[ 1572.247267]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1572.247268]  __handle_mm_fault+0xc0c/0x1310
[ 1572.247269]  handle_mm_fault+0xa9/0x1d0
[ 1572.247270]  __do_page_fault+0x237/0x480
[ 1572.247272]  do_page_fault+0x1d/0x67
[ 1572.247273]  ? page_fault+0x8/0x30
[ 1572.247275]  page_fault+0x1e/0x30
[ 1572.247276] RIP: 0033:0x5f7c81c80c10
[ 1572.247280] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1572.247281] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206
[ 1572.247287] RAX: 0000000015c90000 RBX: 0000710cbc770010 RCX: 0000710f109543db
[ 1572.247289] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000
[ 1572.247290] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000
[ 1572.247292] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1572.247293] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1572.247321] NMI backtrace for cpu 2
[ 1572.247328] CPU: 2 PID: 19490 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247331] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247335] RIP: 0010:isolate_migratepages_block+0x1c/0xa50
[ 1572.247339] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 <89> 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31
[ 1572.247346] RSP: 0018:ffffbd3929043958 EFLAGS: 00000292
[ 1572.247358] RAX: 0000000000000000 RBX: 000000000062b760 RCX: 000000000000000c
[ 1572.247361] RDX: 000000000062b800 RSI: 000000000062b760 RDI: ffffbd3929043ad0
[ 1572.247364] RBP: ffffbd3929043ad0 R08: 0000000000000001 R09: ffff9699ad75d550
[ 1572.247368] R10: 0000000000000001 R11: ffff9699ad75cb80 R12: 000000000062b800
[ 1572.247371] R13: ffff9699cdfded00 R14: ffffbd3929043ad0 R15: ffffbd3929043ad0
[ 1572.247377] FS:  0000710f1082f740(0000) GS:ffff9699ad700000(0000) knlGS:0000000000000000
[ 1572.247380] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.247390] CR2: 000077de6c7b3ec4 CR3: 000000080bde2005 CR4: 00000000003606e0
[ 1572.247393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.247397] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.247402] Call Trace:
[ 1572.247409]  compact_zone+0x577/0xa70
[ 1572.247415]  compact_zone_order+0xde/0x120
[ 1572.247418]  try_to_compact_pages+0x187/0x240
[ 1572.247423]  __alloc_pages_direct_compact+0x87/0x170
[ 1572.247426]  __alloc_pages_slowpath+0x454/0xc20
[ 1572.247432]  ? release_pages+0x348/0x3b0
[ 1572.247439]  ? __pagevec_lru_add_fn+0x189/0x2a0
[ 1572.247442]  __alloc_pages_nodemask+0x268/0x2b0
[ 1572.247448]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1572.247450]  __handle_mm_fault+0xc0c/0x1310
[ 1572.247455]  handle_mm_fault+0xa9/0x1d0
[ 1572.247460]  __do_page_fault+0x237/0x480
[ 1572.247464]  do_page_fault+0x1d/0x67
[ 1572.247468]  ? page_fault+0x8/0x30
[ 1572.247473]  page_fault+0x1e/0x30
[ 1572.247476] RIP: 0033:0x5f7c81c80c10
[ 1572.247482] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1572.247488] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206
[ 1572.247494] RAX: 0000000019c90000 RBX: 0000710cbc770010 RCX: 0000710f109543db
[ 1572.247500] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000
[ 1572.247504] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000
[ 1572.247507] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1572.247511] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1572.247516] NMI backtrace for cpu 5
[ 1572.247520] CPU: 5 PID: 1273 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247525] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247530] RIP: 0033:0x7de7c0c81388
[ 1572.247534] Code: 85 e4 fa ff ff 00 00 00 00 c7 85 e0 fa ff ff 00 00 00 00 e9 38 f4 ff ff 66 2e 0f 1f 84 00 00 00 00 00 48 83 bd e8 fa ff ff 00 <0f> 84 5c 0c 00 00 0f b6 85 dc fa ff ff 83 e0 01 83 bd f0 fa ff ff
[ 1572.247537] RSP: 002b:00007ffeedcb2aa0 EFLAGS: 00000206
[ 1572.247546] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000004
[ 1572.247551] RDX: 0000000000000000 RSI: 00007de7c0dc95c0 RDI: 0000000000000000
[ 1572.247557] RBP: 00007ffeedcb3010 R08: 00007ffeedcb2fd4 R09: 0000000000000001
[ 1572.247560] R10: 0000560441280093 R11: 00007ffeedcb2fd4 R12: 00007ffeedcb3030
[ 1572.247563] R13: 000056044128008f R14: 00007ffeedcb31b0 R15: 0000000000000064
[ 1572.247568] FS:  00007de7c0c01740 GS:  0000000000000000
[ 1572.247575] NMI backtrace for cpu 4
[ 1572.247580] CPU: 4 PID: 1245 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247584] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247587] RIP: 0010:_raw_spin_lock+0xb/0x20
[ 1572.247690] Code: 81 05 6d b7 81 6d 00 02 00 00 31 c0 ba ff 00 00 00 f0 0f b1 17 75 01 c3 e9 92 14 8f ff 66 90 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 01 c3 89 c6 e9 1b 04 8f ff 66 66 2e 0f 1f 84 00 00 00 00 00 65
[ 1572.247801] RSP: 0018:ffffbd38c546bcb8 EFLAGS: 00000097
[ 1572.247812] RAX: 0000000000000001 RBX: ffff96994b4e5ac0 RCX: 000000000000000a
[ 1572.247815] RDX: 0000000000000001 RSI: ffff9699ad800000 RDI: ffff9699adb5cb00
[ 1572.247818] RBP: ffff9699adb5cb00 R08: 000000000000000a R09: ffffbd38c546bd90
[ 1572.247821] R10: 0000000000000000 R11: 0000000000000004 R12: 0000000000000001
[ 1572.247827] R13: 0000000000000046 R14: ffff96994b4e6204 R15: 000000000005cb00
[ 1572.247830] FS:  000077a36f17e740(0000) GS:ffff9699ad800000(0000) knlGS:0000000000000000
[ 1572.247833] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.247837] CR2: 000072d7ae4d9578 CR3: 00000007ca59c006 CR4: 00000000003606e0
[ 1572.247840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.247847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.247850] Call Trace:
[ 1572.247853]  try_to_wake_up+0x18d/0x500
[ 1572.247855]  autoremove_wake_function+0xc/0x50
[ 1572.247858]  __wake_up_common+0x7a/0x190
[ 1572.247861]  __wake_up_common_lock+0x79/0xc0
[ 1572.247864]  pipe_write+0x1e3/0x410
[ 1572.247869]  new_sync_write+0x116/0x1b0
[ 1572.247875]  vfs_write+0xb1/0x190
[ 1572.247878]  ksys_write+0x5d/0xe0
[ 1572.247881]  do_syscall_64+0x50/0x170
[ 1572.247884]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1572.247886] RIP: 0033:0x77a36f2984ba
[ 1572.247894] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20
[ 1572.247897] RSP: 002b:00007fffd15cf728 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 1572.247903] RAX: ffffffffffffffda RBX: 0000000000000029 RCX: 000077a36f2984ba
[ 1572.247906] RDX: 0000000000000029 RSI: 00006261bcaad3c0 RDI: 0000000000000001
[ 1572.247909] RBP: 00007fffd15cf750 R08: 000000000000000a R09: 0000000000000000
[ 1572.247912] R10: 00007fffd15cf77c R11: 0000000000000246 R12: 0000000000000029
[ 1572.247918] R13: 00006261bcaad3c0 R14: 000077a36f36d500 R15: 000077a36f36e300
[ 1572.247922] NMI backtrace for cpu 3
[ 1572.247926] CPU: 3 PID: 19418 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.247929] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.247932] RIP: 0010:isolate_migratepages_block+0xad/0xa50
[ 1572.247938] Code: 89 ef e8 06 91 ff ff 49 89 c7 be 01 00 00 00 48 89 ef e8 f6 90 ff ff 49 89 c5 be 08 00 00 00 48 89 ef e8 e6 90 ff ff 49 89 c4 <be> 07 00 00 00 48 89 ef e8 d6 90 ff ff 49 01 c4 4c 03 3c 24 4d 01
[ 1572.247944] RSP: 0018:ffffbd38cfdf7958 EFLAGS: 00000246
[ 1572.247952] RAX: 0000000000000000 RBX: 00000000006f1ba0 RCX: 000000000000000c
[ 1572.247955] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9699cdfde000
[ 1572.247958] RBP: ffff9699cdfde000 R08: 0000000000000001 R09: ffff9699ad7dd550
[ 1572.247964] R10: 0000000000000001 R11: ffff9699ad7dcb80 R12: 0000000000000000
[ 1572.247969] R13: 00000000000012b2 R14: ffffbd38cfdf7ad0 R15: 00000000000018a8
[ 1572.247972] FS:  0000710f1082f740(0000) GS:ffff9699ad780000(0000) knlGS:0000000000000000
[ 1572.247974] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.247977] CR2: 000072d7ae4d9578 CR3: 00000007c59c4001 CR4: 00000000003606e0
[ 1572.247980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.247983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.247987] Call Trace:
[ 1572.247990]  compact_zone+0x577/0xa70
[ 1572.247992]  compact_zone_order+0xde/0x120
[ 1572.247995]  try_to_compact_pages+0x187/0x240
[ 1572.248000]  __alloc_pages_direct_compact+0x87/0x170
[ 1572.248004]  __alloc_pages_slowpath+0x454/0xc20
[ 1572.248007]  __alloc_pages_nodemask+0x268/0x2b0
[ 1572.248010]  do_huge_pmd_anonymous_page+0x131/0x5c0
[ 1572.248013]  ? mem_cgroup_throttle_swaprate+0x20/0x113
[ 1572.248018]  __handle_mm_fault+0xc0c/0x1310
[ 1572.248023]  handle_mm_fault+0xa9/0x1d0
[ 1572.248026]  __do_page_fault+0x237/0x480
[ 1572.248030]  do_page_fault+0x1d/0x67
[ 1572.248035]  ? page_fault+0x8/0x30
[ 1572.248039]  page_fault+0x1e/0x30
[ 1572.248042] RIP: 0033:0x5f7c81c80c10
[ 1572.248046] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1572.248050] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206
[ 1572.248058] RAX: 0000000026a90000 RBX: 0000710cbc770010 RCX: 0000710f109543db
[ 1572.248060] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000
[ 1572.248065] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000
[ 1572.248070] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1572.248073] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1572.248078] NMI backtrace for cpu 6
[ 1572.248082] CPU: 6 PID: 1258 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.248087] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.248092] RIP: 0033:0x7cfec2fbca97
[ 1572.248097] Code: 00 00 00 83 e1 1f 48 83 e7 e0 c5 7e 6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1 d3 f8 85 c0 74 1d f3 0f bc c0 <48> 01 c8 31 d2 48 8d 04 07 40 3a 30 48 0f 45 c2 c5 f8 77 c3 0f 1f
[ 1572.248102] RSP: 002b:00007fffc45dab98 EFLAGS: 00000202
[ 1572.248110] RAX: 0000000000000001 RBX: 0000582b21b12475 RCX: 000000000000001a
[ 1572.248113] RDX: 0000000000000000 RSI: 000000000000005f RDI: 0000582b21b0e7e0
[ 1572.248118] RBP: 0000582b21b0e7fa R08: 000000000000000a R09: 0000000000000000
[ 1572.248123] R10: 00007fffc45da9ac R11: 0000000000000246 R12: 0000582b21b12459
[ 1572.248126] R13: 0000582b21b0d377 R14: 0000582b21b12440 R15: 0000582b21b12456
[ 1572.248131] FS:  00007cfec2e27740 GS:  0000000000000000
[ 1572.248134] NMI backtrace for cpu 7
[ 1572.248137] CPU: 7 PID: 1249 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.248143] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.248145] RIP: 0010:number+0x234/0x360
[ 1572.248149] Code: c6 00 20 48 ff c0 4c 39 c0 75 f0 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 31 01 00 00 4c 89 c0 48 83 c4 38 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 83 e5 ef 44 88 7c 24 16 e9 2e fe ff ff 48 f7
[ 1572.248152] RSP: 0018:ffffbd38c541bd40 EFLAGS: 00000086
[ 1572.248163] RAX: ffff96994b6f0059 RBX: ffffbd38c541bdb8 RCX: 00000000fffffffb
[ 1572.248168] RDX: ffffbd38c541bd07 RSI: 0000000000000034 RDI: ffff969a4b6f0056
[ 1572.248170] RBP: ffff96994b6f2058 R08: ffff96994b6f0059 R09: 0000000000000000
[ 1572.248172] R10: 0000000000000004 R11: ffff96994b6f0059 R12: ffffffff92e33a07
[ 1572.248179] R13: 0000000000000000 R14: 0000000000ffff0a R15: 0000000000000000
[ 1572.248183] FS:  00007ef911609740(0000) GS:ffff9699ad980000(0000) knlGS:0000000000000000
[ 1572.248186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.248188] CR2: 00007d5d665cfae0 CR3: 00000007cb6ec002 CR4: 00000000003606e0
[ 1572.248192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.248199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.248201] Call Trace:
[ 1572.248203]  vsnprintf+0x3e1/0x5b0
[ 1572.248207]  scnprintf+0x4d/0x90
[ 1572.248212]  msg_print_ext_header.constprop.0+0x7f/0xa0
[ 1572.248217]  ? _cond_resched+0x10/0x20
[ 1572.248219]  ? mutex_lock_interruptible+0x9/0x30
[ 1572.248225]  devkmsg_read+0x17f/0x270
[ 1572.248227]  vfs_read+0x98/0x150
[ 1572.248232]  ksys_read+0x5d/0xe0
[ 1572.248234]  do_syscall_64+0x50/0x170
[ 1572.248241]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1572.248245] RIP: 0033:0x7ef911723415
[ 1572.248251] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1572.248255] RSP: 002b:00007ffcf81cb498 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1572.248262] RAX: ffffffffffffffda RBX: 00005571c56a2440 RCX: 00007ef911723415
[ 1572.248267] RDX: 0000000000001fff RSI: 00005571c56a2440 RDI: 0000000000000003
[ 1572.248271] RBP: 00005571c569e7fa R08: 000000000000000a R09: 0000000000000000
[ 1572.248273] R10: 00007ffcf81cb2cc R11: 0000000000000246 R12: 000000000000001c
[ 1572.248278] R13: 00005571c569d377 R14: 00005571c56a2440 R15: 00005571c56a2456
[ 1572.248285] NMI backtrace for cpu 9
[ 1572.248288] CPU: 9 PID: 400 Comm: systemd-journal Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.248292] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.248300] RIP: 0010:queued_spin_lock_slowpath+0x17b/0x1d0
[ 1572.248307] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 d7 05 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 15 ed 92 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb
[ 1572.248310] RSP: 0018:ffffbd38c34cbd50 EFLAGS: 00000046
[ 1572.248322] RAX: 0000000000000000 RBX: ffff9699a86f4370 RCX: 0000000000280000
[ 1572.248325] RDX: ffff9699adadd7c0 RSI: 0000000000000005 RDI: ffffffff935a02ac
[ 1572.248333] RBP: ffffbd38c34cbda8 R08: 0000000000000009 R09: 0000000000000000
[ 1572.248336] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9699a4980000
[ 1572.248342] R13: 00007ffddcf63680 R14: ffffbd38c34cbe00 R15: ffff9699a86f4318
[ 1572.248346] FS:  00007c88db99c840(0000) GS:ffff9699ada80000(0000) knlGS:0000000000000000
[ 1572.248351] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.248359] CR2: 00007c88cf9be3c0 CR3: 00000008286d4001 CR4: 00000000003606e0
[ 1572.248364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.248369] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.248375] Call Trace:
[ 1572.248382]  devkmsg_poll+0x3c/0x80
[ 1572.248388]  ep_item_poll.isra.0+0x3a/0xb0
[ 1572.248391]  ep_send_events_proc+0xec/0x230
[ 1572.248394]  ? ep_read_events_proc+0xe0/0xe0
[ 1572.248403]  ep_scan_ready_list.constprop.0+0x9b/0x1f0
[ 1572.248408]  ep_poll+0x87/0x450
[ 1572.248412]  do_epoll_wait+0xab/0xd0
[ 1572.248419]  __x64_sys_epoll_wait+0x15/0x20
[ 1572.248424]  do_syscall_64+0x50/0x170
[ 1572.248428]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1572.248435] RIP: 0033:0x7c88dd1b20b5
[ 1572.248439] Code: 89 55 f8 48 89 75 f0 89 7d fc e8 e6 18 f7 ff 41 89 c0 44 8b 55 ec 8b 55 f8 48 8b 75 f0 8b 7d fc b8 e8 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 25 89 45 fc 44 89 c7 e8 18 19 f7 ff 8b 45 fc
[ 1572.248444] RSP: 002b:00007ffddcf63650 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[ 1572.248458] RAX: ffffffffffffffda RBX: 00006265e2ccc3d0 RCX: 00007c88dd1b20b5
[ 1572.248461] RDX: 0000000000000020 RSI: 00007ffddcf63680 RDI: 0000000000000007
[ 1572.248465] RBP: 00007ffddcf63670 R08: 0000000000000000 R09: 0000000000000020
[ 1572.248471] R10: 00000000dd84bf82 R11: 0000000000000293 R12: 00007ffddcf63680
[ 1572.248478] R13: 000007494e8c1088 R14: 0000000000000000 R15: 0000000000000001
[ 1572.248486] NMI backtrace for cpu 11
[ 1572.248490] CPU: 11 PID: 20040 Comm: dmesg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.248493] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.248502] RIP: 0033:0x7717bcea7cbb
[ 1572.248509] Code: 48 83 ec 18 49 89 f5 4c 0f af ea 4d 85 ed 0f 84 f0 00 00 00 49 89 ff 49 89 f6 49 89 d4 48 89 cb 8b 11 81 e2 00 80 00 00 75 44 <64> 48 8b 04 25 10 00 00 00 48 89 45 c8 48 8b b9 88 00 00 00 48 39
[ 1572.248518] RSP: 002b:00007fff79d5f6a0 EFLAGS: 00000246
[ 1572.248531] RAX: 0000000000000001 RBX: 00007717bd00b500 RCX: 00007717bd00b500
[ 1572.248534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000062592cea0487
[ 1572.248538] RBP: 00007fff79d5f6e0 R08: 000062592cea0488 R09: 0000000000000000
[ 1572.248544] R10: 00007fff79d5f71c R11: 000062592cea0487 R12: 0000000000000001
[ 1572.248552] R13: 0000000000000001 R14: 0000000000000001 R15: 000062592cea0487
[ 1572.248557] FS:  00007717bce1c740 GS:  0000000000000000
[ 1572.248563] NMI backtrace for cpu 10
[ 1572.248567] CPU: 10 PID: 1260 Comm: grep Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #68
[ 1572.248570] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
[ 1572.248577] RIP: 0010:ZSTD_decompressSequences+0x6be/0xb80
[ 1572.248586] Code: 8b 39 48 89 7c 24 68 84 c0 74 13 f3 0f bd c0 8d 48 e9 89 ce 49 83 fc ea 0f 86 2c fb ff ff 49 c7 c5 f2 ff ff ff e9 21 fa ff ff <89> d0 44 8b 34 85 00 f0 c7 92 44 8b 54 24 70 c4 e2 a9 f7 44 24 68
[ 1572.248589] RSP: 0018:ffffbd38c5a37838 EFLAGS: 00000202
[ 1572.248599] RAX: 0000000000000008 RBX: 0000000000000000 RCX: ffffbd38c6022048
[ 1572.248606] RDX: 0000000000000008 RSI: ffffbd38c6022d30 RDI: ffffbd38c60228a4
[ 1572.248611] RBP: 0000000000000000 R08: 0000000000000099 R09: 0000000000000001
[ 1572.248617] R10: 0000000000000007 R11: 0000000000000006 R12: 0000000000000001
[ 1572.248622] R13: 0000000000000006 R14: fffffffffffffffe R15: ffff969928146263
[ 1572.248628] FS:  000077c96a004740(0000) GS:ffff9699adb00000(0000) knlGS:0000000000000000
[ 1572.248635] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1572.248642] CR2: 00005a61235ce000 CR3: 00000007cba90004 CR4: 00000000003606e0
[ 1572.248648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1572.248659] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1572.248666] Call Trace:
[ 1572.248672]  ZSTD_decompressMultiFrame+0x329/0x370
[ 1572.248677]  ZSTD_decompressDCtx+0xc/0x10
[ 1572.248680]  zstd_decompress+0x1f/0x40
[ 1572.248685]  zcomp_decompress+0x2e/0x50
[ 1572.248690]  zram_bvec_rw.isra.0+0x5e2/0x6a0
[ 1572.248699]  zram_rw_page+0x7f/0xe0
[ 1572.248705]  bdev_read_page+0x6e/0xa0
[ 1572.248710]  swap_readpage+0xaf/0x1f0
[ 1572.248713]  do_swap_page+0x72e/0x7e0
[ 1572.248718]  __handle_mm_fault+0xa77/0x1310
[ 1572.248722]  handle_mm_fault+0xa9/0x1d0
[ 1572.248725]  __do_page_fault+0x237/0x480
[ 1572.248734]  do_page_fault+0x1d/0x67
[ 1572.248741]  ? reweight_entity+0x15a/0x1a0
[ 1572.248744]  page_fault+0x1e/0x30
[ 1572.248749] RIP: 0010:copy_user_generic_unrolled+0x89/0xc0
[ 1572.248758] Code: 38 4c 89 47 20 4c 89 4f 28 4c 89 57 30 4c 89 5f 38 48 8d 76 40 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 4c 8b 06 <4c> 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10 89 d1 8a
[ 1572.248766] RSP: 0018:ffffbd38c5a37d90 EFLAGS: 00050203
[ 1572.248780] RAX: 0000000000000016 RBX: ffff96972af7c000 RCX: 0000000000000001
[ 1572.248783] RDX: 0000000000000006 RSI: ffff96972af7c008 RDI: 00005a61235cdffe
[ 1572.248789] RBP: 0000000000000016 R08: 363478302b6b6361 R09: 0000000000000000
[ 1572.248794] R10: 0000000000000003 R11: ffff9699a2261e00 R12: 0000000000000016
[ 1572.248800] R13: ffffbd38c5a37e50 R14: 0000000000000000 R15: 0000000000000000
[ 1572.248803]  copyout+0x28/0x30
[ 1572.248806]  copy_page_to_iter+0xbc/0x2f0
[ 1572.248809]  pipe_read+0xad/0x2c0
[ 1572.248814]  new_sync_read+0x10f/0x1a0
[ 1572.248821]  vfs_read+0x98/0x150
[ 1572.248826]  ksys_read+0x5d/0xe0
[ 1572.248832]  do_syscall_64+0x50/0x170
[ 1572.248837]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1572.248840] RIP: 0033:0x77c96a140415
[ 1572.248849] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55
[ 1572.248855] RSP: 002b:00007fff87565c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1572.248869] RAX: ffffffffffffffda RBX: 000000000000e000 RCX: 000077c96a140415
[ 1572.248877] RDX: 000000000000e000 RSI: 00005a61235cdff6 RDI: 0000000000000000
[ 1572.248886] RBP: 000000000000e000 R08: 0000000000000000 R09: 0000000000019008
[ 1572.248890] R10: 00005a61235b85a0 R11: 0000000000000246 R12: 00005a61235cdff6
[ 1572.248893] R13: 0000000000000000 R14: 00005a61235c3100 R15: 0000000000000000

```

Ignore the dmesg process, I've had two programs that keep querrying it and I've put process to 800Mhz max, to avoid high power usage during this.

the swap is in zram

/etc/fstab shows:
/dev/zram0 none swap defaults,discard 0 0

$ zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram2 zstd           64G   4.7M 136.3K   248K      12 /var/tmp
/dev/zram1 zstd           64G 320.3M  84.1M    88M      12 /tmp
/dev/zram0 zstd           64G   1.6G  75.6M 230.6M      12 [SWAP]

I can't find a reliable way to reproduce it, it seems to happen sort of random...
Comment 16 GYt2bW 2019-07-16 03:29:19 UTC
```
crash> bt -Tsx 21199
PID: 21199  TASK: ffff8fd7ef4e8000  CPU: 11  COMMAND: "stress"
  [ffff9cb9aa1cf160] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf170] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf200] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf210] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf220] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf318] wake_all_kswapds+0x4f at ffffffff981d5e5f
  [ffff9cb9aa1cf358] __alloc_pages_slowpath+0x334 at ffffffff981d9b14
  [ffff9cb9aa1cf388] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf438] update_group_capacity+0x20 at ffffffff980d2530
  [ffff9cb9aa1cf468] cpumask_next_and+0x18 at ffffffff987df928
  [ffff9cb9aa1cf470] update_sd_lb_stats+0x159 at ffffffff980d2839
  [ffff9cb9aa1cf560] update_load_avg+0x4b1 at ffffffff980cdae1
  [ffff9cb9aa1cf568] account_entity_enqueue+0xbf at ffffffff980cc13f
  [ffff9cb9aa1cf5b8] reweight_entity+0x15a at ffffffff980cd17a
  [ffff9cb9aa1cf5e0] enqueue_task_fair+0x11f at ffffffff980cff0f
  [ffff9cb9aa1cf608] check_preempt_wakeup+0xfd at ffffffff980cd39d
  [ffff9cb9aa1cf650] check_preempt_curr+0x70 at ffffffff980c2ce0
  [ffff9cb9aa1cf668] __update_load_avg_se+0x200 at ffffffff980e05d0
  [ffff9cb9aa1cf6b8] update_load_avg+0x42b at ffffffff980cda5b
  [ffff9cb9aa1cf6c0] account_entity_enqueue+0xbf at ffffffff980cc13f
  [ffff9cb9aa1cf6f8] enqueue_entity+0xe4 at ffffffff980cea64
  [ffff9cb9aa1cf700] record_times+0x10 at ffffffff980e8140
  [ffff9cb9aa1cf750] check_preempt_wakeup+0x182 at ffffffff980cd422
  [ffff9cb9aa1cf798] check_preempt_curr+0x70 at ffffffff980c2ce0
  [ffff9cb9aa1cf7a8] ttwu_do_wakeup+0x12 at ffffffff980c2d02
  [ffff9cb9aa1cf7c8] __update_load_avg_se+0x200 at ffffffff980e05d0
  [ffff9cb9aa1cf810] update_load_avg+0x76 at ffffffff980cd6a6
  [ffff9cb9aa1cf850] set_next_entity+0x89 at ffffffff980cdc29
  [ffff9cb9aa1cf870] pick_next_task_fair+0x590 at ffffffff980d4a50
  [ffff9cb9aa1cf890] apic_timer_interrupt+0xa at ffffffff9880146a
  [ffff9cb9aa1cf8a0] apic_timer_interrupt+0xa at ffffffff9880146a
  [ffff9cb9aa1cf928] node_page_state+0x15 at ffffffff981ac675
  [ffff9cb9aa1cf950] isolate_migratepages_block+0x69 at ffffffff981b3539
  [ffff9cb9aa1cfa10] compact_zone+0x577 at ffffffff981b4c07
  [ffff9cb9aa1cfab8] compact_zone_order+0xde at ffffffff981b51de
  [ffff9cb9aa1cfb78] try_to_compact_pages+0x187 at ffffffff981b5a17
  [ffff9cb9aa1cfbd8] __alloc_pages_direct_compact+0x87 at ffffffff981d95a7
  [ffff9cb9aa1cfc30] __alloc_pages_slowpath+0x20e at ffffffff981d99ee
  [ffff9cb9aa1cfc48] try_charge+0x51a at ffffffff9821044a
  [ffff9cb9aa1cfcd0] release_pages+0x348 at ffffffff9819aa08
  [ffff9cb9aa1cfd00] __pagevec_lru_add_fn+0x189 at ffffffff9819ada9
  [ffff9cb9aa1cfd40] __alloc_pages_nodemask+0x268 at ffffffff981da668
  [ffff9cb9aa1cfda0] do_huge_pmd_anonymous_page+0x131 at ffffffff98202c01
  [ffff9cb9aa1cfdf0] __handle_mm_fault+0xc0c at ffffffff981bfa4c
  [ffff9cb9aa1cfea0] handle_mm_fault+0xa9 at ffffffff981c01f9
  [ffff9cb9aa1cfec8] __do_page_fault+0x237 at ffffffff9803d5c7
  [ffff9cb9aa1cff28] do_page_fault+0x1d at ffffffff9803d85d
  [ffff9cb9aa1cff48] page_fault+0x8 at ffffffff98800de8
  [ffff9cb9aa1cff50] page_fault+0x1e at ffffffff98800dfe
    RIP: 00005879fc8c4c10  RSP: 00007ffd393aa0d0  RFLAGS: 00010206
    RAX: 000000002438a000  RBX: 0000714104876010  RCX: 0000714358a5a3db
    RDX: 0000000000000000  RSI: 00000002540bf000  RDI: 0000714104876000
    RBP: 00005879fc8c5a54   R8: 0000714104876010   R9: 0000000000000000
    R10: 0000000000000022  R11: 00000002540be400  R12: ffffffffffffffff
    R13: 0000000000000002  R14: 0000000000001000  R15: 00000002540be400
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b

crash> bt -Tsx 21199
PID: 21199  TASK: ffff8fd7ef4e8000  CPU: 1   COMMAND: "stress"
  [ffff9cb9aa1cf160] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf170] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf200] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf210] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf220] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf318] wake_all_kswapds+0x4f at ffffffff981d5e5f
  [ffff9cb9aa1cf358] __alloc_pages_slowpath+0x334 at ffffffff981d9b14
  [ffff9cb9aa1cf388] get_page_from_freelist+0xe88 at ffffffff981d9078
  [ffff9cb9aa1cf438] update_group_capacity+0x20 at ffffffff980d2530
  [ffff9cb9aa1cf468] cpumask_next_and+0x18 at ffffffff987df928
  [ffff9cb9aa1cf470] update_sd_lb_stats+0x159 at ffffffff980d2839
  [ffff9cb9aa1cf560] update_load_avg+0x4b1 at ffffffff980cdae1
  [ffff9cb9aa1cf568] account_entity_enqueue+0xbf at ffffffff980cc13f
  [ffff9cb9aa1cf5b8] reweight_entity+0x15a at ffffffff980cd17a
  [ffff9cb9aa1cf5e0] enqueue_task_fair+0x11f at ffffffff980cff0f
  [ffff9cb9aa1cf608] check_preempt_wakeup+0xfd at ffffffff980cd39d
  [ffff9cb9aa1cf650] check_preempt_curr+0x70 at ffffffff980c2ce0
  [ffff9cb9aa1cf668] __update_load_avg_se+0x200 at ffffffff980e05d0
  [ffff9cb9aa1cf6b8] update_load_avg+0x42b at ffffffff980cda5b
  [ffff9cb9aa1cf6c0] account_entity_enqueue+0xbf at ffffffff980cc13f
  [ffff9cb9aa1cf6f8] enqueue_entity+0xe4 at ffffffff980cea64
  [ffff9cb9aa1cf700] record_times+0x10 at ffffffff980e8140
  [ffff9cb9aa1cf750] check_preempt_wakeup+0x182 at ffffffff980cd422
  [ffff9cb9aa1cf798] check_preempt_curr+0x70 at ffffffff980c2ce0
  [ffff9cb9aa1cf7a8] ttwu_do_wakeup+0x12 at ffffffff980c2d02
  [ffff9cb9aa1cf7c8] __update_load_avg_se+0x200 at ffffffff980e05d0
  [ffff9cb9aa1cf808] __update_load_avg_cfs_rq+0x1df at ffffffff980e085f
  [ffff9cb9aa1cf810] update_load_avg+0x76 at ffffffff980cd6a6
  [ffff9cb9aa1cf850] set_next_entity+0x89 at ffffffff980cdc29
  [ffff9cb9aa1cf870] pick_next_task_fair+0x590 at ffffffff980d4a50
  [ffff9cb9aa1cf890] apic_timer_interrupt+0xa at ffffffff9880146a
  [ffff9cb9aa1cf918] compact_unlock_should_abort+0x44 at ffffffff981b2834
  [ffff9cb9aa1cf950] isolate_migratepages_block+0x2ed at ffffffff981b37bd
  [ffff9cb9aa1cfa10] compact_zone+0x577 at ffffffff981b4c07
  [ffff9cb9aa1cfab8] compact_zone_order+0xde at ffffffff981b51de
  [ffff9cb9aa1cfb78] try_to_compact_pages+0x187 at ffffffff981b5a17
  [ffff9cb9aa1cfbd8] __alloc_pages_direct_compact+0x87 at ffffffff981d95a7
  [ffff9cb9aa1cfc30] __alloc_pages_slowpath+0x20e at ffffffff981d99ee
  [ffff9cb9aa1cfc48] try_charge+0x51a at ffffffff9821044a
  [ffff9cb9aa1cfcd0] release_pages+0x348 at ffffffff9819aa08
  [ffff9cb9aa1cfd00] __pagevec_lru_add_fn+0x189 at ffffffff9819ada9
  [ffff9cb9aa1cfd40] __alloc_pages_nodemask+0x268 at ffffffff981da668
  [ffff9cb9aa1cfda0] do_huge_pmd_anonymous_page+0x131 at ffffffff98202c01
  [ffff9cb9aa1cfdf0] __handle_mm_fault+0xc0c at ffffffff981bfa4c
  [ffff9cb9aa1cfea0] handle_mm_fault+0xa9 at ffffffff981c01f9
  [ffff9cb9aa1cfec8] __do_page_fault+0x237 at ffffffff9803d5c7
  [ffff9cb9aa1cff28] do_page_fault+0x1d at ffffffff9803d85d
  [ffff9cb9aa1cff48] page_fault+0x8 at ffffffff98800de8
  [ffff9cb9aa1cff50] page_fault+0x1e at ffffffff98800dfe
    RIP: 00005879fc8c4c10  RSP: 00007ffd393aa0d0  RFLAGS: 00010206
    RAX: 000000002438a000  RBX: 0000714104876010  RCX: 0000714358a5a3db
    RDX: 0000000000000000  RSI: 00000002540bf000  RDI: 0000714104876000
    RBP: 00005879fc8c5a54   R8: 0000714104876010   R9: 0000000000000000
    R10: 0000000000000022  R11: 00000002540be400  R12: ffffffffffffffff
    R13: 0000000000000002  R14: 0000000000001000  R15: 00000002540be400
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
```
Comment 17 GYt2bW 2019-07-16 03:38:49 UTC
ok I just realized that the comments look broken only when cookies+scripts are allowed, because otherwise the markdown doesn't get interpreted so you just see the comments in their raw form!

So the above comment has completely different contents when views via uMatrix extension as css+images allowed only, vs when viewed with everything allowed(cookies+scripts+css+images+everything else)
Comment 18 mgorman 2019-07-16 07:17:57 UTC
On Tue, Jul 16, 2019 at 03:57:30AM +0000, howaboutsynergy@protonmail.com wrote:
> > PID: 21120 TASK: ffff8fd7f10ddac0 CPU: 9 COMMAND: "stress"
> > START: __schedule at ffffffff987f5a0c
> > [ffff9cb9a9f57928] isolate_migratepages_block at ffffffff981b35c6
> > [ffff9cb9a9f57950] isolate_migratepages_block at ffffffff981b37bd
> > [ffff9cb9a9f57a10] compact_zone at ffffffff981b4c07
> > [ffff9cb9a9f57ab8] compact_zone_order at ffffffff981b51de
> > [ffff9cb9a9f57b78] try_to_compact_pages at ffffffff981b5a17
> > [ffff9cb9a9f57bd8] __alloc_pages_direct_compact at ffffffff981d95a7
> > [ffff9cb9a9f57c30] __alloc_pages_slowpath at ffffffff981d99ee
> > [ffff9cb9a9f57cd0] release_pages at ffffffff9819aa08
> > [ffff9cb9a9f57d00] __pagevec_lru_add_fn at ffffffff9819ada9
> > [ffff9cb9a9f57d40] __alloc_pages_nodemask at ffffffff981da668
> > [ffff9cb9a9f57da0] do_huge_pmd_anonymous_page at ffffffff98202c01
> > [ffff9cb9a9f57df0] __handle_mm_fault at ffffffff981bfa4c
> > [ffff9cb9a9f57ea0] handle_mm_fault at ffffffff981c01f9
> > [ffff9cb9a9f57ec8] __do_page_fault at ffffffff9803d5c7
> > [ffff9cb9a9f57f28] do_page_fault at ffffffff9803d85d
> > [ffff9cb9a9f57f48] page_fault at ffffffff98800de8
> > [ffff9cb9a9f57f50] page_fault at ffffffff98800dfe
> > RIP: 00005879fc8c4c10 RSP: 00007ffd393aa0d0 RFLAGS: 00010206
> > RAX: 0000000010b8a000 RBX: 0000714104876010 RCX: 0000714358a5a3db
> > RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000714104876000
> > RBP: 00005879fc8c5a54 R8: 0000714104876010 R9: 0000000000000000
> > R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
> > R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
> > ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
> > crash>

High CPU usage in this path is not something I've observed recently.
When it happens and CPU usage is high, can you run the following commands
please?

trace-cmd record -e compaction:* sleep 10
trace-cmd report > trace.log

and send me the resulting trace.log please?
Comment 19 mgorman 2019-07-16 10:03:52 UTC
I tried reproducing this but after 300 attempts with various parameters
and adding other workloads in the background, I was unable to reproduce
the problem.
Comment 20 howaboutsynergy 2019-07-16 19:15:29 UTC
Created attachment 283749 [details]
publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc

On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net> wrote:
> I tried reproducing this but after 300 attempts with various parameters
> and adding other workloads in the background, I was unable to reproduce
> the problem.
> 


The third time I ran this command `$ time stress -m 220 --vm-bytes 10000000000 --timeout 10`, got 10+ hung:

  PID  %CPU COMMAND                                                                            PR  NI    VIRT    RES S USER     

 3785  94.5 stress                                                                             20   0 9769416      4 R user     

 3777  87.3 stress                                                                             20   0 9769416      4 R user     

 3923  85.5 stress                                                                             20   0 9769416      4 R user     

 3937  85.5 stress                                                                             20   0 9769416      4 R user     

 3943  81.8 stress                                                                             20   0 9769416      4 R user     

 3885  80.0 stress                                                                             20   0 9769416      4 R user     

 3970  80.0 stress                                                                             20   0 9769416      4 R user     

 3902  76.4 stress                                                                             20   0 9769416      4 R user     

 3954  72.7 stress                                                                             20   0 9769416      4 R user     

 3868  70.9 stress                                                                             20   0 9769416      4 R user     

 3893  69.1 stress                                                                             20   0 9769416      4 R user     

 3786  65.5 stress                                                                             20   0 9769416      4 R user     

 3783  60.0 stress                                                                             20   0 9769416      4 R user     

 3848  58.2 stress                                                                             20   0 9769416      4 R user     

 3863  58.2 stress                                                                             20   0 9769416      4 R user     


looked like this:
```
TERM='xterm-256color'
53.48 573.49
-----------
user@i87k 2019/07/16 20:36:47 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut53
!41866 1 0  5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019
/home/user 

$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [1744] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [1744] successful run completed in 19s

real	0m19.036s
user	0m0.794s
sys	2m59.583s
-----------
user@i87k 2019/07/16 20:37:12 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut79
!41867 2 0  5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019
/home/user 

$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [3520] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [3520] successful run completed in 18s

real	0m18.657s
user	0m0.901s
sys	2m59.700s
-----------
user@i87k 2019/07/16 20:42:28 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut394
!41868 3 0  5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019
/home/user 

$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [3771] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd


```
(sure I waited a few minutes until I ran the 3rd one, I don't remember what trivial things I've been doing during the time - like making sure I got the right trace command(s))

I'm pretty sure you need swap in (ext4)zram for this to trigger (faster?), judging by the fuller stacktrace that I showed in prev. email(s) gotten via 'crash'-s `bt -Tsx pidhere`

On Tuesday, July 16, 2019 9:11 AM, Mel Gorman <mgorman@techsingularity.net> wrote:
> High CPU usage in this path is not something I've observed recently.
> When it happens and CPU usage is high, can you run the following commands
> please?
> 

> trace-cmd record -e compaction:* sleep 10
> trace-cmd report > trace.log
> 

> and send me the resulting trace.log please?

Ok, getting trace.log as requested:

```
$ sudo trace-cmd record -e compaction:* sleep 10
[sudo] password for user: 

CPU 0: 12430 events lost
CPU 1: 83959 events lost
CPU 4: 13447 events lost
CPU 6: 2825 events lost
CPU 8: 791 events lost
CPU 11: 8475 events lost
CPU0 data recorded at offset=0x5bc000
    114487296 bytes in size
CPU1 data recorded at offset=0x72eb000
    106885120 bytes in size
CPU2 data recorded at offset=0xd8da000
    125046784 bytes in size
CPU3 data recorded at offset=0x1501b000
    111022080 bytes in size
CPU4 data recorded at offset=0x1b9fc000
    120532992 bytes in size
CPU5 data recorded at offset=0x22cef000
    115990528 bytes in size
CPU6 data recorded at offset=0x29b8d000
    116109312 bytes in size
CPU7 data recorded at offset=0x30a48000
    73822208 bytes in size
CPU8 data recorded at offset=0x350af000
    98643968 bytes in size
CPU9 data recorded at offset=0x3aec2000
    96514048 bytes in size
CPU10 data recorded at offset=0x40acd000
    113967104 bytes in size
CPU11 data recorded at offset=0x4777d000
    127184896 bytes in size

trace.dat is 1.3G
-rw-r--r--  1 root root 1326219264 16.07.2019 20:45 trace.dat

$ LD_PRELOAD=/usr/lib/trace-cmd/python/ctracecmd.so trace-cmd report > trace.log
trace-cmd: symbol lookup error: /usr/lib/trace-cmd/python/ctracecmd.so: undefined symbol: PyExc_SystemError

$ trace-cmd report > trace.log
  could not load plugin '/usr/lib/trace-cmd/plugins/plugin_python.so'
/usr/lib/trace-cmd/plugins/plugin_python.so: undefined symbol: PyString_FromString

I guess we could ignore that?


trace.log is like 4.3G
-rw-r--r--  1 user user 4370520245 16.07.2019 20:50 trace.log


On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net> wrote:
> I tried reproducing this but after 300 attempts with various parameters
> and adding other workloads in the background, I was unable to reproduce
> the problem.
> 



As a reminder, here's how sysrq+l stacktraces look like (for two of the stress pids):
```
[ 1294.913508] NMI backtrace for cpu 5
[ 1294.913517] CPU: 5 PID: 3848 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #71
[ 1294.913522] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019
[ 1294.913526] RIP: 0010:ftrace_likely_update+0x1a/0x200
[ 1294.913533] Code: 0b eb bb 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 fb 41 89 d4 9c 41 5f 0f 01 ca <85> c9 0f 84 8b 00 00 00 48 ff 47 28 8b 15 34 72 47 01 85 d2 75 16
[ 1294.913537] RSP: 0000:ffffa3cd4c7ef848 EFLAGS: 00000286
[ 1294.913552] RAX: 0000000000000000 RBX: ffffffff97546180 RCX: 0000000000000000
[ 1294.913557] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff97546180
[ 1294.913562] RBP: 000000000017cc6f R08: 0000000000000000 R09: 0000000000000000
[ 1294.913567] R10: 0000000000000001 R11: 0000000000000004 R12: 0000000000000000
[ 1294.913571] R13: ffff93560dfdc000 R14: 000000000012f44c R15: 0000000000000286
[ 1294.913577] FS:  000072d6afdff740(0000) GS:ffff9355ed880000(0000) knlGS:0000000000000000
[ 1294.913582] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1294.913590] CR2: 000070fbe8abdc88 CR3: 0000000827a46004 CR4: 00000000003606e0
[ 1294.913595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1294.913600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1294.913605] Call Trace:
[ 1294.913609]  _cond_resched+0x2d/0x50
[ 1294.913613]  isolate_migratepages_block+0xf8/0xe40
[ 1294.913618]  compact_zone+0x4f9/0xe10
[ 1294.913626]  compact_zone_order+0xe3/0x120
[ 1294.913630]  try_to_compact_pages+0xde/0x3b0
[ 1294.913635]  __alloc_pages_direct_compact+0x8c/0x170
[ 1294.913639]  __alloc_pages_slowpath+0x65c/0x1290
[ 1294.913644]  __alloc_pages_nodemask+0x4cf/0x530
[ 1294.913648]  do_huge_pmd_anonymous_page+0x17c/0x780
[ 1294.913653]  __handle_mm_fault+0xeee/0x17d0
[ 1294.913661]  handle_mm_fault+0x17b/0x330
[ 1294.913666]  __do_page_fault+0x34e/0x800
[ 1294.913671]  do_page_fault+0x57/0x1f9
[ 1294.913675]  ? page_fault+0x8/0x30
[ 1294.913680]  page_fault+0x1e/0x30
[ 1294.913684] RIP: 0033:0x5a46e611ac10
[ 1294.913690] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1294.913695] RSP: 002b:00007fffa51f0f60 EFLAGS: 00010206
[ 1294.913705] RAX: 00000000098c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db
[ 1294.913710] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000
[ 1294.913715] RBP: 00005a46e611ba54 R08: 000072d45bd40010 R09: 0000000000000000
[ 1294.913720] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1294.913724] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1294.913730] NMI backtrace for cpu 11
[ 1294.913739] CPU: 11 PID: 3902 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #71
[ 1294.913744] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019
[ 1294.913748] RIP: 0010:isolate_migratepages_block+0x97b/0xe40
[ 1294.913754] Code: c6 43 79 01 4c 8b 74 24 08 4d 39 fe b9 00 00 00 00 ba 00 00 00 00 40 0f 92 c6 40 0f b6 f6 48 c7 c7 90 ac 56 97 e8 05 fe f5 ff <4d> 39 fe 0f 82 7a f9 ff ff 4c 39 7c 24 08 0f 84 6f f9 ff ff 0f 1f
[ 1294.913758] RSP: 0000:ffffa3cd4c99f8b0 EFLAGS: 00000286
[ 1294.913768] RAX: 0000000000000000 RBX: ffffa3cd4c99fa50 RCX: 0000000000000000
[ 1294.913777] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9756ac90
[ 1294.913781] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
[ 1294.913786] R10: 0000000000000001 R11: 0000000000000004 R12: 0000000000000000
[ 1294.913790] R13: 0000000000000000 R14: 000000000070ee00 R15: 000000000070ece0
[ 1294.913795] FS:  000072d6afdff740(0000) GS:ffff9355edb80000(0000) knlGS:0000000000000000
[ 1294.913799] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1294.913804] CR2: 00007d99c6200000 CR3: 00000007ecb1a003 CR4: 00000000003606e0
[ 1294.913813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1294.913817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1294.913822] Call Trace:
[ 1294.913826]  compact_zone+0x4f9/0xe10
[ 1294.913830]  compact_zone_order+0xe3/0x120
[ 1294.913835]  try_to_compact_pages+0xde/0x3b0
[ 1294.913839]  __alloc_pages_direct_compact+0x8c/0x170
[ 1294.913848]  __alloc_pages_slowpath+0x65c/0x1290
[ 1294.913852]  __alloc_pages_nodemask+0x4cf/0x530
[ 1294.913856]  do_huge_pmd_anonymous_page+0x17c/0x780
[ 1294.913861]  __handle_mm_fault+0xeee/0x17d0
[ 1294.913865]  handle_mm_fault+0x17b/0x330
[ 1294.913869]  __do_page_fault+0x34e/0x800
[ 1294.913874]  do_page_fault+0x57/0x1f9
[ 1294.913882]  ? page_fault+0x8/0x30
[ 1294.913886]  page_fault+0x1e/0x30
[ 1294.913891] RIP: 0033:0x5a46e611ac10
[ 1294.913896] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1294.913901] RSP: 002b:00007fffa51f0f60 EFLAGS: 00010206
[ 1294.913907] RAX: 000000000d0c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db
[ 1294.913916] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000
[ 1294.913921] RBP: 00005a46e611ba54 R08: 000072d45bd40010 R09: 0000000000000000
[ 1294.913925] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1294.913929] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
```

and here's how they look via crash:
```
$ sudo crash "/usr/lib/modules/$(uname -r)/build/vmlinux"
[sudo] password for user: 


crash 7.2.6
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [336MB]: patching 121800 gdb minimal_symbol values

      KERNEL: /usr/lib/modules/5.2.1-g527a3db363a3/build/vmlinux       

    DUMPFILE: /proc/kcore
        CPUS: 12
        DATE: Tue Jul 16 20:59:48 2019
      UPTIME: 00:23:54
LOAD AVERAGE: 26.83, 21.74, 16.35
       TASKS: 508
    NODENAME: i87k
     RELEASE: 5.2.1-g527a3db363a3
     VERSION: #71 SMP Tue Jul 16 19:41:12 CEST 2019
     MACHINE: x86_64  (3700 Mhz)
      MEMORY: 31.9 GB
         PID: 6417
     COMMAND: "crash"
        TASK: ffff934f9c9a8000  [THREAD_INFO: ffff934f9c9a8000]
         CPU: 2
       STATE: TASK_RUNNING (ACTIVE)

crash> bt -Tsx 3848
PID: 3848   TASK: ffff934f83373d80  CPU: 5   COMMAND: "stress"
  [ffffa3cd4c7eefb8] get_page_from_freelist+0xaa9 at ffffffff96291799
  [ffffa3cd4c7ef0e8] get_page_from_freelist+0xaa9 at ffffffff96291799
  [ffffa3cd4c7ef108] __alloc_pages_slowpath+0x216 at ffffffff96292cb6
  [ffffa3cd4c7ef130] get_page_from_freelist+0xb76 at ffffffff96291866
  [ffffa3cd4c7ef178] get_page_from_freelist+0xaa9 at ffffffff96291799
  [ffffa3cd4c7ef1a0] trace_hardirqs_on_caller+0x32 at ffffffff961bc4b2
  [ffffa3cd4c7ef2e0] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b
  [ffffa3cd4c7ef388] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b
  [ffffa3cd4c7ef4b0] get_zspage_mapping+0x40 at ffffffff962e2220
  [ffffa3cd4c7ef538] decay_load+0x6a at ffffffff96111d1a
  [ffffa3cd4c7ef5a0] __list_add_valid+0x90 at ffffffff965e3ff0
  [ffffa3cd4c7ef650] check_preempt_wakeup+0x267 at ffffffff960f7097
  [ffffa3cd4c7ef6b8] decay_load+0x6a at ffffffff96111d1a
  [ffffa3cd4c7ef700] trace_hardirqs_on_thunk+0x1a at ffffffff96001b02
  [ffffa3cd4c7ef738] update_load_avg+0xca at ffffffff960f73aa
  [ffffa3cd4c7ef740] update_load_avg+0xca at ffffffff960f73aa
  [ffffa3cd4c7ef7a0] finish_task_switch+0xc9 at ffffffff960e6879
  [ffffa3cd4c7ef7e8] finish_task_switch+0x17f at ffffffff960e692f
  [ffffa3cd4c7ef820] __schedule+0x552 at ffffffff969f4752
  [ffffa3cd4c7ef8a8] isolate_migratepages_block+0x97b at ffffffff9626165b
  [ffffa3cd4c7ef918] isolate_migratepages_block+0xf at ffffffff96260cef
  [ffffa3cd4c7ef978] compact_zone+0x4f9 at ffffffff96263eb9
  [ffffa3cd4c7efa38] compact_zone_order+0xe3 at ffffffff962648b3
  [ffffa3cd4c7efaf0] try_to_compact_pages+0xde at ffffffff9626523e
  [ffffa3cd4c7efb60] __alloc_pages_direct_compact+0x8c at ffffffff9629285c
  [ffffa3cd4c7efbb8] __alloc_pages_slowpath+0x65c at ffffffff962930fc
  [ffffa3cd4c7efce8] __alloc_pages_nodemask+0x4cf at ffffffff962941ff
  [ffffa3cd4c7efd60] do_huge_pmd_anonymous_page+0x17c at ffffffff962c6fcc
  [ffffa3cd4c7efdb0] __handle_mm_fault+0xeee at ffffffff96272c3e
  [ffffa3cd4c7efe78] handle_mm_fault+0x17b at ffffffff9627369b
  [ffffa3cd4c7efeb0] __do_page_fault+0x34e at ffffffff9604f8ee
  [ffffa3cd4c7eff20] do_page_fault+0x57 at ffffffff9604fe27
  [ffffa3cd4c7eff38] page_fault+0x8 at ffffffff96a00e78
  [ffffa3cd4c7eff50] page_fault+0x1e at ffffffff96a00e8e
    RIP: 00005a46e611ac10  RSP: 00007fffa51f0f60  RFLAGS: 00010206
    RAX: 00000000098c0000  RBX: 000072d45bd40010  RCX: 000072d6aff243db
    RDX: 0000000000000000  RSI: 00000002540bf000  RDI: 000072d45bd40000
    RBP: 00005a46e611ba54   R8: 000072d45bd40010   R9: 0000000000000000
    R10: 0000000000000022  R11: 00000002540be400  R12: ffffffffffffffff
    R13: 0000000000000002  R14: 0000000000001000  R15: 00000002540be400
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
crash> bt -l 3848
PID: 3848   TASK: ffff934f83373d80  CPU: 11  COMMAND: "stress"
 #0 [ffffa3cd4c7ef7f0] __schedule at ffffffff969f471a
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/kernel/sched/core.c: 2818
 #1 [ffffa3cd4c7ef830] trace_hardirqs_on_thunk at ffffffff96001b02
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/entry/thunk_64.S: 42
 #2 [ffffa3cd4c7ef8a8] isolate_migratepages_block at ffffffff9626165b
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 1039
 #3 [ffffa3cd4c7ef978] compact_zone at ffffffff96263eb9
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 1817
 #4 [ffffa3cd4c7efa38] compact_zone_order at ffffffff962648b3
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 2313
 #5 [ffffa3cd4c7efaf0] try_to_compact_pages at ffffffff9626523e
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 2362
 #6 [ffffa3cd4c7efb60] __alloc_pages_direct_compact at ffffffff9629285c
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 3831
 #7 [ffffa3cd4c7efbb8] __alloc_pages_slowpath at ffffffff962930fc
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 4470
 #8 [ffffa3cd4c7efce8] __alloc_pages_nodemask at ffffffff962941ff
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 4678
 #9 [ffffa3cd4c7efd60] do_huge_pmd_anonymous_page at ffffffff962c6fcc
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/./include/linux/topology.h: 73
#10 [ffffa3cd4c7efdb0] __handle_mm_fault at ffffffff96272c3e
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/memory.c: 3788
#11 [ffffa3cd4c7efe78] handle_mm_fault at ffffffff9627369b
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/memory.c: 4058
#12 [ffffa3cd4c7efeb0] __do_page_fault at ffffffff9604f8ee
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/mm/fault.c: 1457
#13 [ffffa3cd4c7eff20] do_page_fault at ffffffff9604fe27
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/./arch/x86/include/asm/jump_label.h: 23
#14 [ffffa3cd4c7eff50] page_fault at ffffffff96a00e8e
    /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/entry/entry_64.S: 1156
    RIP: 00005a46e611ac10  RSP: 00007fffa51f0f60  RFLAGS: 00010206
    RAX: 00000000098c0000  RBX: 000072d45bd40010  RCX: 000072d6aff243db
    RDX: 0000000000000000  RSI: 00000002540bf000  RDI: 000072d45bd40000
    RBP: 00005a46e611ba54   R8: 000072d45bd40010   R9: 0000000000000000
    R10: 0000000000000022  R11: 00000002540be400  R12: ffffffffffffffff
    R13: 0000000000000002  R14: 0000000000001000  R15: 00000002540be400
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
crash> bt -Tsx 3902
PID: 3902   TASK: ffff934f5264bd80  CPU: 1   COMMAND: "stress"
  [ffffa3cd4c99efb8] get_page_from_freelist+0xaa9 at ffffffff96291799
  [ffffa3cd4c99f108] __alloc_pages_slowpath+0x216 at ffffffff96292cb6
  [ffffa3cd4c99f130] get_page_from_freelist+0xb76 at ffffffff96291866
  [ffffa3cd4c99f178] get_page_from_freelist+0xaa9 at ffffffff96291799
  [ffffa3cd4c99f2e0] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b
  [ffffa3cd4c99f480] test_clear_page_writeback+0x12a at ffffffff9623ad8a
  [ffffa3cd4c99f4a8] trace_hardirqs_on+0x2c at ffffffff961bc30c
  [ffffa3cd4c99f4c8] test_clear_page_writeback+0x18c at ffffffff9623adec
  [ffffa3cd4c99f538] decay_load+0x6a at ffffffff96111d1a
  [ffffa3cd4c99f5a0] __list_add_valid+0x90 at ffffffff965e3ff0
  [ffffa3cd4c99f650] check_preempt_wakeup+0x267 at ffffffff960f7097
  [ffffa3cd4c99f6b8] decay_load+0x6a at ffffffff96111d1a
  [ffffa3cd4c99f6e0] __accumulate_pelt_segments+0x29 at ffffffff96111d89
  [ffffa3cd4c99f700] __update_load_avg_se+0x1cb at ffffffff961120ab
  [ffffa3cd4c99f738] update_load_avg+0xca at ffffffff960f73aa
  [ffffa3cd4c99f740] update_load_avg+0xca at ffffffff960f73aa
  [ffffa3cd4c99f7e0] switch_mm_irqs_off+0x270 at ffffffff96057920
  [ffffa3cd4c99f820] __schedule+0x51a at ffffffff969f471a
  [ffffa3cd4c99f880] preempt_schedule_common+0x15 at ffffffff969f4ed5
  [ffffa3cd4c99f898] _cond_resched+0x3f at ffffffff969f4f3f
  [ffffa3cd4c99f8a8] isolate_migratepages_block+0xf8 at ffffffff96260dd8
  [ffffa3cd4c99f978] compact_zone+0x4f9 at ffffffff96263eb9
  [ffffa3cd4c99fa38] compact_zone_order+0xe3 at ffffffff962648b3
  [ffffa3cd4c99faf0] try_to_compact_pages+0xde at ffffffff9626523e
  [ffffa3cd4c99fb60] __alloc_pages_direct_compact+0x8c at ffffffff9629285c
  [ffffa3cd4c99fbb8] __alloc_pages_slowpath+0x65c at ffffffff962930fc
  [ffffa3cd4c99fce8] __alloc_pages_nodemask+0x4cf at ffffffff962941ff
  [ffffa3cd4c99fd60] do_huge_pmd_anonymous_page+0x17c at ffffffff962c6fcc
  [ffffa3cd4c99fdb0] __handle_mm_fault+0xeee at ffffffff96272c3e
  [ffffa3cd4c99fe78] handle_mm_fault+0x17b at ffffffff9627369b
  [ffffa3cd4c99feb0] __do_page_fault+0x34e at ffffffff9604f8ee
  [ffffa3cd4c99ff20] do_page_fault+0x57 at ffffffff9604fe27
  [ffffa3cd4c99ff38] page_fault+0x8 at ffffffff96a00e78
  [ffffa3cd4c99ff50] page_fault+0x1e at ffffffff96a00e8e
    RIP: 00005a46e611ac10  RSP: 00007fffa51f0f60  RFLAGS: 00010206
    RAX: 000000000d0c0000  RBX: 000072d45bd40010  RCX: 000072d6aff243db
    RDX: 0000000000000000  RSI: 00000002540bf000  RDI: 000072d45bd40000
    RBP: 00005a46e611ba54   R8: 000072d45bd40010   R9: 0000000000000000
    R10: 0000000000000022  R11: 00000002540be400  R12: ffffffffffffffff
    R13: 0000000000000002  R14: 0000000000001000  R15: 00000002540be400
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
crash> bt -l 3902
PID: 3902   TASK: ffff934f5264bd80  CPU: 9   COMMAND: "stress"
(active)
crash> 

```
As can be seen, it's doing stuff with(in) zstd, maybe that's why swap in zstd is needed to reproduce this (more easily? or at all?)


trace.log is still being compressed(via xz) but those running 'stress' processes using 100% cpu are only making this slower :)
I'll send it when it's ready?
Comment 21 howaboutsynergy 2019-07-16 19:15:30 UTC
Created attachment 283751 [details]
signature.asc
Comment 22 mgorman 2019-07-17 17:53:38 UTC
On Tue, Jul 16, 2019 at 07:15:08PM +0000, howaboutsynergy@protonmail.com wrote:
> On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net>
> wrote:
> > I tried reproducing this but after 300 attempts with various parameters
> > and adding other workloads in the background, I was unable to reproduce
> > the problem.
> > 
> 
> 
> The third time I ran this command `$ time stress -m 220 --vm-bytes
> 10000000000 --timeout 10`, got 10+ hung:
> 
>   PID  %CPU COMMAND                                                          
>                    PR  NI    VIRT    RES S USER     
>  3785  94.5 stress                                                           
>                   20   0 9769416      4 R user     
>  3777  87.3 stress                                                           
>                   20   0 9769416      4 R user     
>  3923  85.5 stress                                                           
>                   20   0 9769416      4 R user     
>  3937  85.5 stress                                                           
>                   20   0 9769416      4 R user     
>  3943  81.8 stress                                                           
>                   20   0 9769416      4 R user     
>  3885  80.0 stress                                                           
>                   20   0 9769416      4 R user     
>  3970  80.0 stress                                                           
>                   20   0 9769416      4 R user     
>
> <SNIP>
>
> trace.dat is 1.3G
> -rw-r--r--  1 root root 1326219264 16.07.2019 20:45 trace.dat
> 

Ok, great. From the trace, it was obvious that the scanner is making no
progress. I don't think zswap is involved as such but it *may* be making
it easier to trigger due to altering timing. At least, I see no reason
why zswap would materially affect the termination conditions.

From the path and your trace, I think what *might* be happening is that
a fatal signal is pending which does not advance the scanner or look like
a proper abort. I think it ends up looping in compaction instead of dying
without either aborting or progressing the scanner.  It might explain why
stress-ng is hitting is as it is probably sending fatal signals on timeout
(I didn't check the source).

Can you try this (compile tested only) patch please? Note that the stress
test might still take time to exit normally if it's stuck in a swap
storm of some sort but I'm hoping the 100% compaction CPU usage goes away
at least.

diff --git a/mm/compaction.c b/mm/compaction.c
index 9e1b9acb116b..952dc2fb24e5 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -842,13 +842,15 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 
 		/*
 		 * Periodically drop the lock (if held) regardless of its
-		 * contention, to give chance to IRQs. Abort async compaction
-		 * if contended.
+		 * contention, to give chance to IRQs. Abort completely if
+		 * a fatal signal is pending.
 		 */
 		if (!(low_pfn % SWAP_CLUSTER_MAX)
 		    && compact_unlock_should_abort(&pgdat->lru_lock,
-					    flags, &locked, cc))
-			break;
+					    flags, &locked, cc)) {
+			low_pfn = 0;
+			goto fatal_pending;
+		}
 
 		if (!pfn_valid_within(low_pfn))
 			goto isolate_fail;
@@ -1060,6 +1062,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 	trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn,
 						nr_scanned, nr_isolated);
 
+fatal_pending:
 	cc->total_migrate_scanned += nr_scanned;
 	if (nr_isolated)
 		count_compact_events(COMPACTISOLATED, nr_isolated);
Comment 23 howaboutsynergy 2019-07-17 22:00:37 UTC
tl;dr: patch seems to work, thank you very much!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, July 17, 2019 7:53 PM, Mel Gorman <mgorman@techsingularity.net> wrote:

> Ok, great. From the trace, it was obvious that the scanner is making no
> progress. I don't think zswap is involved as such but it may be making
> it easier to trigger due to altering timing. At least, I see no reason
> why zswap would materially affect the termination conditions.
I don't know if it matters in this context, but I've been using the term `zswap`(somewhere else I think) to (wrongly)refer to swap in zram (and even sometimes called it ext4(in this bug report too) without realizing at the time that ext4 is only for /tmp and /var/tmp instead! they are ext4 in zram) but in fact this isn't zswap that I have been using (even though I have CONFIG_ZSWAP=y in .config) but it's just CONFIG_ZRAM=y with CONFIG_SWAP=y (and probably a bunch of others being needed too).

>
> From the path and your trace, I think what might be happening is that
> a fatal signal is pending which does not advance the scanner or look like
> a proper abort. I think it ends up looping in compaction instead of dying
> without either aborting or progressing the scanner. It might explain why
> stress-ng is hitting is as it is probably sending fatal signals on timeout
> (I didn't check the source).
Ah I didn't know there are multiple `stress` versions, here's what I used:

/usr/bin/stress is owned by stress 1.0.4-5

$ pacman -Qs stress
local/stress 1.0.4-5
    A tool that stress tests your system (CPU, memory, I/O, disks)

$ pacman -Qi stress
Name            : stress
Version         : 1.0.4-5
Description     : A tool that stress tests your system (CPU, memory, I/O, disks)
Architecture    : x86_64
URL             : http://people.seas.harvard.edu/~apw/stress/
Licenses        : GPL
Groups          : None
Provides        : None
Depends On      : None
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 25.00 KiB
Packager        : Florian Pritz <bluewind@xinu.at>
Build Date      : Thu 31 May 2018 01:46:16 PM CEST
Install Date    : Mon 10 Jun 2019 02:44:14 AM CEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Note: not:
$ yaourt -Ss stress-ng
aur/stress-ng 0.09.59-1 (11) (0.10)
    stress-ng will stress test a computer system in various selectable ways

(it probably doesn't matter anyway :) as I didn't look at the source either)


>
> Can you try this (compile tested only) patch please? Note that the stress
> test might still take time to exit normally if it's stuck in a swap
> storm of some sort but I'm hoping the 100% compaction CPU usage goes away
> at least.
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 9e1b9acb116b..952dc2fb24e5 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -842,13 +842,15 @@ isolate_migratepages_block(struct compact_control cc,
> unsigned long low_pfn,
> /* Periodically drop the lock (if held) regardless of its
>
> -          * contention, to give chance to IRQs. Abort async compaction
>
>
> -          * if contended.
>
>
>
> -          * contention, to give chance to IRQs. Abort completely if
>
>
> -          * a fatal signal is pending.
>            */
>           if (!(low_pfn % SWAP_CLUSTER_MAX)
>               && compact_unlock_should_abort(&pgdat->lru_lock,
>
>
>
> -                                 flags, &locked, cc))
>
>
> -             break;
>
>
>
> -                                 flags, &locked, cc)) {
>
>
> -             low_pfn = 0;
>
>
> -             goto fatal_pending;
>
>
> -         }
>
>
>
> if (!pfn_valid_within(low_pfn))
> goto isolate_fail;
> @@ -1060,6 +1062,7 @@ isolate_migratepages_block(struct compact_control *cc,
> unsigned long low_pfn,
> trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn,
> nr_scanned, nr_isolated);
>
> +fatal_pending:
> cc->total_migrate_scanned += nr_scanned;
>
>     if (nr_isolated)
>       count_compact_events(COMPACTISOLATED, nr_isolated);
>
>
> --
>
> Mel Gorman
> SUSE Labs



Now the "problem" is I can't tell if it would get stuck :D but it usually ends in no more than 17 sec:
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [7981] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: FAIL: [7981] (415) <-- worker 8202 got signal 9
stress: WARN: [7981] (417) now reaping child worker processes
stress: FAIL: [7981] (415) <-- worker 8199 got signal 9
stress: WARN: [7981] (417) now reaping child worker processes
stress: FAIL: [7981] (451) failed run completed in 18s

real	0m17.397s
user	0m1.069s
sys	2m42.774s

sometimes it's 14,15 or 16, but usually it's between 17 and 18(exclusive) seconds.



Interestingly sometimes (rarely) it succeeds:
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [12629] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [12629] successful run completed in 15s

real	0m14.571s
user	0m1.122s
sys	2m18.623s

But it succeedes 100% of the time if I limit CPU max to 800Mhz (lowest):
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [8434] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [8434] successful run completed in 23s

real	0m22.639s
user	0m1.653s
sys	3m18.693s
sometimes it's in 7 sec less that the usual 22sec:
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [8689] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [8689] successful run completed in 15s

real	0m15.585s
user	0m1.576s
sys	2m26.495s


(probably irrelevant)Sometimes Xorg says it can't allocate any more memory but stacktrace looks like it's inside some zram i915 kernel stuff:

[ 1416.842931] [drm] Atomic update on pipe (A) took 188 us, max time under evasion is 100 us
[ 1425.416979] Xorg: page allocation failure: order:0, mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null),cpuset=/,mems_allowed=0
[ 1425.416984] CPU: 1 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #74
[ 1425.416985] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019
[ 1425.416986] Call Trace:
[ 1425.416991]  dump_stack+0x46/0x60
[ 1425.416993]  warn_alloc.cold+0x7b/0xfb
[ 1425.416996]  __alloc_pages_slowpath+0xbe2/0xc20
[ 1425.416997]  __alloc_pages_nodemask+0x268/0x2b0
[ 1425.416999]  new_slab+0x3ff/0x610
[ 1425.417001]  ? xas_alloc+0x9c/0xc0
[ 1425.417002]  ___slab_alloc+0x33b/0x540
[ 1425.417003]  ? xas_nomem+0x49/0x70
[ 1425.417005]  ? xas_alloc+0x9c/0xc0
[ 1425.417006]  ? xas_nomem+0x49/0x70
[ 1425.417007]  __slab_alloc+0x9/0x10
[ 1425.417008]  kmem_cache_alloc+0x118/0x150
[ 1425.417010]  xas_nomem+0x49/0x70
[ 1425.417011]  add_to_swap_cache+0x25c/0x310
[ 1425.417013]  __read_swap_cache_async+0xef/0x1e0
[ 1425.417014]  swap_cluster_readahead+0x1ac/0x2a0
[ 1425.417015]  shmem_swapin+0x55/0xa0
[ 1425.417018]  ? call_function_single_interrupt+0xa/0x20
[ 1425.417019]  ? xas_init_marks+0x19/0x40
[ 1425.417020]  ? xas_store+0x3e4/0x640
[ 1425.417022]  shmem_swapin_page+0x418/0x590
[ 1425.417023]  ? xas_load+0x5/0x70
[ 1425.417025]  ? find_get_entry+0x55/0x120
[ 1425.417026]  shmem_getpage_gfp.isra.0+0x37f/0x820
[ 1425.417028]  shmem_read_mapping_page_gfp+0x3b/0x70
[ 1425.417051]  i915_gem_object_get_pages_gtt+0x1f9/0x550 [i915]
[ 1425.417070]  ? gen8_ppgtt_alloc_4lvl+0x5a/0x150 [i915]
[ 1425.417086]  __i915_gem_object_get_pages+0x4e/0x60 [i915]
[ 1425.417104]  __i915_vma_do_pin+0x4bf/0x540 [i915]
[ 1425.417120]  eb_lookup_vmas+0x68b/0xb90 [i915]
[ 1425.417136]  ? i915_gem_do_execbuffer+0x351/0x1040 [i915]
[ 1425.417152]  ? i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915]
[ 1425.417154]  ? krealloc+0x21/0xa0
[ 1425.417169]  i915_gem_do_execbuffer+0x502/0x1040 [i915]
[ 1425.417172]  ? ZSTD_decompressMultiFrame+0x329/0x370
[ 1425.417174]  ? ZSTD_decompressDCtx+0xc/0x10
[ 1425.417175]  ? ktime_get_with_offset+0x5c/0x130
[ 1425.417177]  ? zram_bvec_rw.isra.0+0x154/0x6a0
[ 1425.417179]  ? _cond_resched+0x10/0x20
[ 1425.417194]  i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915]
[ 1425.417196]  ? zram_rw_page+0xba/0xe0
[ 1425.417211]  ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915]
[ 1425.417220]  drm_ioctl_kernel+0xa5/0xf0 [drm]
[ 1425.417228]  drm_ioctl+0x201/0x380 [drm]
[ 1425.417243]  ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915]
[ 1425.417245]  do_vfs_ioctl+0x3f4/0x660
[ 1425.417246]  ? handle_mm_fault+0xa9/0x1d0
[ 1425.417248]  ksys_ioctl+0x59/0x90
[ 1425.417249]  __x64_sys_ioctl+0x11/0x20
[ 1425.417251]  do_syscall_64+0x50/0x170
[ 1425.417253]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1425.417255] RIP: 0033:0x78521f1b1c2b
[ 1425.417256] Code: 0f 1e fa 48 8b 05 65 d2 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 35 d2 0c 00 f7 d8 64 89 01 48
[ 1425.417258] RSP: 002b:00007ffcf8fd0d08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 1425.417259] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 000078521f1b1c2b
[ 1425.417260] RDX: 00007ffcf8fd0d40 RSI: 0000000040406469 RDI: 000000000000000a
[ 1425.417261] RBP: 00005a6a541acad0 R08: 0000000000000000 R09: 00007852196c2a60
[ 1425.417262] R10: 0000785213f3e000 R11: 0000000000000246 R12: 0000000000002000
[ 1425.417263] R13: 000078521af98000 R14: 00007ffcf8fd0d40 R15: 00007ffcf8fd0d40
[ 1425.417264] Mem-Info:
[ 1425.417267] active_anon:6998108 inactive_anon:442511 isolated_anon:9465
                active_file:2251 inactive_file:2329 isolated_file:259
                unevictable:16735 dirty:20 writeback:0 unstable:0
                slab_reclaimable:28873 slab_unreclaimable:37272
                mapped:3501 shmem:39527 pagetables:32186 bounce:0
                free:87238 free_pcp:2904 free_cma:0
[ 1425.417269] Node 0 active_anon:27992432kB inactive_anon:1770044kB active_file:9004kB inactive_file:9316kB unevictable:66940kB isolated(anon):37860kB isolated(file):1036kB mapped:14004kB dirty:80kB writeback:0kB shmem:158108kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 22118400kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1425.417271] Node 0 DMA free:15892kB min:100kB low:124kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15892kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1425.417273] lowmem_reserve[]: 0 2346 31714 31714
[ 1425.417275] Node 0 DMA32 free:132920kB min:15608kB low:19508kB high:23408kB active_anon:2259000kB inactive_anon:1412kB active_file:0kB inactive_file:0kB unevictable:1296kB writepending:0kB present:2744176kB managed:2416496kB mlocked:0kB kernel_stack:16kB pagetables:3420kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB
[ 1425.417277] lowmem_reserve[]: 0 0 29368 29368
[ 1425.417279] Node 0 Normal free:200140kB min:198384kB low:246956kB high:295528kB active_anon:25733140kB inactive_anon:1769340kB active_file:10288kB inactive_file:10456kB unevictable:65560kB writepending:80kB present:30654464kB managed:30077908kB mlocked:32kB kernel_stack:9408kB pagetables:125324kB bounce:0kB free_pcp:11740kB local_pcp:1420kB free_cma:0kB
[ 1425.417281] lowmem_reserve[]: 0 0 0 0
[ 1425.417282] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[ 1425.417287] Node 0 DMA32: 248*4kB (UME) 256*8kB (UME) 172*16kB (UME) 138*32kB (UME) 87*64kB (UME) 68*128kB (UME) 38*256kB (UE) 22*512kB (UME) 11*1024kB (E) 0*2048kB 19*4096kB (M) = 134560kB
[ 1425.417292] Node 0 Normal: 17*4kB (UM) 20*8kB (UM) 16*16kB (UM) 1721*32kB (UM) 733*64kB (UME) 140*128kB (UM) 1*256kB (E) 1*512kB (M) 2*1024kB (ME) 37*2048kB (M) 0*4096kB = 198980kB
[ 1425.417297] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1425.417298] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1425.417299] 50365 total pagecache pages
[ 1425.417306] 6377 pages in swap cache
[ 1425.417307] Swap cache stats: add 728656996, delete 728204250, find 6351630/8862186
[ 1425.417308] Free swap  = 35959128kB
[ 1425.417308] Total swap = 67108860kB
[ 1425.417309] 8353655 pages RAM
[ 1425.417310] 0 pages HighMem/MovableOnly
[ 1425.417310] 226081 pages reserved
[ 1425.417311] 0 pages cma reserved
[ 1425.417312] SLUB: Unable to allocate memory on node -1, gfp=0xc0(__GFP_IO|__GFP_FS)
[ 1425.417313]   cache: radix_tree_node, object size: 576, buffer size: 584, default order: 2, min order: 0
[ 1425.417314]   node 0: slabs: 5357, objs: 149702, free: 0
[ 1426.236297] Xorg: page allocation failure: order:0, mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null),cpuset=/,mems_allowed=0
[ 1426.236306] CPU: 9 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #74
[ 1426.236307] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019
[ 1426.236308] Call Trace:
[ 1426.236318]  dump_stack+0x46/0x60
[ 1426.236322]  warn_alloc.cold+0x7b/0xfb
[ 1426.236326]  __alloc_pages_slowpath+0xbe2/0xc20
[ 1426.236328]  __alloc_pages_nodemask+0x268/0x2b0
[ 1426.236331]  new_slab+0x3ff/0x610
[ 1426.236335]  ? xas_alloc+0x9c/0xc0
[ 1426.236337]  ___slab_alloc+0x33b/0x540
[ 1426.236339]  ? xas_nomem+0x49/0x70
[ 1426.236340]  ? xas_alloc+0x9c/0xc0
[ 1426.236341]  ? xas_nomem+0x49/0x70
[ 1426.236342]  __slab_alloc+0x9/0x10
[ 1426.236343]  kmem_cache_alloc+0x118/0x150
[ 1426.236345]  xas_nomem+0x49/0x70
[ 1426.236347]  add_to_swap_cache+0x25c/0x310
[ 1426.236349]  __read_swap_cache_async+0xef/0x1e0
[ 1426.236351]  read_swap_cache_async+0x24/0x60
[ 1426.236352]  swap_cluster_readahead+0x237/0x2a0
[ 1426.236354]  ? get_page_from_freelist+0xe88/0x1330
[ 1426.236356]  ? ktime_get+0x49/0x110
[ 1426.236358]  shmem_swapin+0x55/0xa0
[ 1426.236361]  shmem_swapin_page+0x418/0x590
[ 1426.236362]  ? xas_load+0x5/0x70
[ 1426.236366]  ? find_get_entry+0x55/0x120
[ 1426.236367]  shmem_getpage_gfp.isra.0+0x37f/0x820
[ 1426.236368]  shmem_read_mapping_page_gfp+0x3b/0x70
[ 1426.236438]  i915_gem_object_get_pages_gtt+0x1f9/0x550 [i915]
[ 1426.236445]  ? set_next_entity+0x89/0x150
[ 1426.236447]  ? pick_next_task_fair+0x590/0x760
[ 1426.236482]  __i915_gem_object_get_pages+0x4e/0x60 [i915]
[ 1426.236501]  __i915_vma_do_pin+0x4bf/0x540 [i915]
[ 1426.236518]  eb_lookup_vmas+0x68b/0xb90 [i915]
[ 1426.236534]  i915_gem_do_execbuffer+0x502/0x1040 [i915]
[ 1426.236537]  ? xas_init_marks+0x19/0x40
[ 1426.236538]  ? xas_store+0x3e4/0x640
[ 1426.236540]  ? __delete_from_swap_cache+0x15a/0x190
[ 1426.236541]  ? _swap_info_get+0xa/0x30
[ 1426.236543]  ? free_swap_slot+0x43/0xc0
[ 1426.236544]  ? __swap_entry_free.constprop.0+0x92/0xa0
[ 1426.236545]  ? shmem_swapin_page+0x4fe/0x590
[ 1426.236547]  ? xas_load+0x5/0x70
[ 1426.236549]  ? find_get_entry+0x55/0x120
[ 1426.236550]  ? _cond_resched+0x10/0x20
[ 1426.236566]  i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915]
[ 1426.236577]  ? __mod_lruvec_state+0x3a/0xe0
[ 1426.236593]  ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915]
[ 1426.236618]  drm_ioctl_kernel+0xa5/0xf0 [drm]
[ 1426.236627]  drm_ioctl+0x201/0x380 [drm]
[ 1426.236645]  ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915]
[ 1426.236649]  ? __set_current_blocked+0x38/0x50
[ 1426.236650]  ? signal_setup_done+0x8b/0xa0
[ 1426.236654]  do_vfs_ioctl+0x3f4/0x660
[ 1426.236656]  ? recalc_sigpending+0xe/0x40
[ 1426.236659]  ? _copy_from_user+0x37/0x60
[ 1426.236661]  ksys_ioctl+0x59/0x90
[ 1426.236662]  __x64_sys_ioctl+0x11/0x20
[ 1426.236664]  do_syscall_64+0x50/0x170
[ 1426.236669]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1426.236671] RIP: 0033:0x78521f1b1c2b
[ 1426.236673] Code: 0f 1e fa 48 8b 05 65 d2 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 35 d2 0c 00 f7 d8 64 89 01 48
[ 1426.236674] RSP: 002b:00007ffcf8fcee18 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 1426.236676] RAX: ffffffffffffffda RBX: 000078521dd60d40 RCX: 000078521f1b1c2b
[ 1426.236677] RDX: 00007ffcf8fcee80 RSI: 0000000040406469 RDI: 000000000000000a
[ 1426.236677] RBP: 00007ffcf8fcee80 R08: 0000000000000000 R09: 00007852196c3d60
[ 1426.236678] R10: 00007852184ea000 R11: 0000000000000246 R12: 0000000040406469
[ 1426.236679] R13: 000000000000000a R14: 00007ffcf8fcee80 R15: 00007ffcf8fcee80
[ 1426.236682] SLUB: Unable to allocate memory on node -1, gfp=0xc0(__GFP_IO|__GFP_FS)
[ 1426.236683]   cache: radix_tree_node, object size: 576, buffer size: 584, default order: 2, min order: 0
[ 1426.236684]   node 0: slabs: 1195, objs: 33082, free: 0
[ 1466.913788] stress invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[ 1466.913792] CPU: 3 PID: 15698 Comm: stress Kdump: loaded Tainted: G     U            5.2.1-g527a3db363a3 #74
[ 1466.913793] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019
[ 1466.913794] Call Trace:
[ 1466.913798]  dump_stack+0x46/0x60
[ 1466.913800]  dump_header+0x4f/0x2fd
[ 1466.913801]  ? oom_unkillable_task+0x95/0xc0
[ 1466.913802]  ? find_lock_task_mm+0x2e/0x70
[ 1466.913803]  oom_kill_process.cold+0xb/0x10
[ 1466.913805]  out_of_memory+0x1bc/0x460
[ 1466.913806]  __alloc_pages_slowpath+0xb19/0xc20
[ 1466.913808]  __alloc_pages_nodemask+0x268/0x2b0
[ 1466.913810]  alloc_pages_vma+0x74/0x1c0
[ 1466.913812]  __handle_mm_fault+0xe1f/0x1310
[ 1466.913814]  handle_mm_fault+0xa9/0x1d0
[ 1466.913816]  __do_page_fault+0x237/0x480
[ 1466.913817]  do_page_fault+0x1d/0x67
[ 1466.913819]  ? page_fault+0x8/0x30
[ 1466.913821]  page_fault+0x1e/0x30
[ 1466.913822] RIP: 0033:0x56f413018c10
[ 1466.913824] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e
[ 1466.913825] RSP: 002b:00007ffd17a85910 EFLAGS: 00010206
[ 1466.913827] RAX: 00000000187e3000 RBX: 00007a864406e010 RCX: 00007a88982523db
[ 1466.913827] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 00007a864406e000
[ 1466.913828] RBP: 000056f413019a54 R08: 00007a864406e010 R09: 0000000000000000
[ 1466.913829] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
[ 1466.913830] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
[ 1466.913831] Mem-Info:
[ 1466.913834] active_anon:6889086 inactive_anon:437196 isolated_anon:44189
                active_file:210 inactive_file:248 isolated_file:32
                unevictable:16068 dirty:0 writeback:0 unstable:0
                slab_reclaimable:10669 slab_unreclaimable:55287
                mapped:492 shmem:16679 pagetables:49030 bounce:0
                free:85711 free_pcp:1702 free_cma:0
[ 1466.913837] Node 0 active_anon:27556344kB inactive_anon:1748784kB active_file:840kB inactive_file:992kB unevictable:64272kB isolated(anon):176756kB isolated(file):128kB mapped:1968kB dirty:0kB writeback:0kB shmem:66716kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 18806784kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1466.913838] Node 0 DMA free:15892kB min:100kB low:124kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15892kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1466.913841] lowmem_reserve[]: 0 2346 31714 31714
[ 1466.913842] Node 0 DMA32 free:133064kB min:15608kB low:19508kB high:23408kB active_anon:2246984kB inactive_anon:516kB active_file:32kB inactive_file:20kB unevictable:56kB writepending:0kB present:2744176kB managed:2416496kB mlocked:0kB kernel_stack:16kB pagetables:6120kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB
[ 1466.913845] lowmem_reserve[]: 0 0 29368 29368
[ 1466.913847] Node 0 Normal free:193888kB min:194288kB low:242860kB high:291432kB active_anon:25307420kB inactive_anon:1748072kB active_file:2148kB inactive_file:2432kB unevictable:62728kB writepending:0kB present:30654464kB managed:30077908kB mlocked:32kB kernel_stack:9392kB pagetables:190000kB bounce:0kB free_pcp:6804kB local_pcp:344kB free_cma:0kB
[ 1466.913849] lowmem_reserve[]: 0 0 0 0
[ 1466.913851] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[ 1466.913856] Node 0 DMA32: 117*4kB (UME) 133*8kB (UME) 95*16kB (UME) 98*32kB (UME) 127*64kB (UME) 80*128kB (UME) 40*256kB (UME) 27*512kB (UME) 18*1024kB (ME) 2*2048kB (ME) 15*4096kB (ME) = 132588kB
[ 1466.913861] Node 0 Normal: 482*4kB (UME) 478*8kB (UME) 648*16kB (ME) 849*32kB (E) 661*64kB (UME) 345*128kB (UE) 132*256kB (UE) 49*512kB (UME) 4*1024kB (ME) 0*2048kB 0*4096kB = 192728kB
[ 1466.913867] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1466.913868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1466.913869] 64205 total pagecache pages
[ 1466.913875] 47161 pages in swap cache
[ 1466.913876] Swap cache stats: add 776634640, delete 776102875, find 6734206/9399597
[ 1466.913877] Free swap  = 0kB
[ 1466.913877] Total swap = 67108860kB
[ 1466.913878] 8353655 pages RAM
[ 1466.913879] 0 pages HighMem/MovableOnly
[ 1466.913879] 226081 pages reserved
[ 1466.913880] 0 pages cma reserved
[ 1466.913880] Tasks state (memory values in pages):
[ 1466.913881] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1466.913888] [    399]     0   399    21409        0   212992      330             0 systemd-journal
[ 1466.913889] [    413]     0   413     9265        0    94208      420         -1000 systemd-udevd
[ 1466.913891] [    604]    81   604     2773        0    65536      225          -900 dbus-daemon
[ 1466.913893] [    609]     0   609      649        0    45056       45             0 gpm
[ 1466.913894] [    610]     0   610     6600        0    90112      269             0 systemd-logind
[ 1466.913896] [    695]     0   695    11319        0   131072      157             0 login
[ 1466.913898] [    799]  1000   799     7472        1   102400      377             0 systemd
[ 1466.913899] [    800]  1000   800    16164      131   159744      463             0 (sd-pam)
[ 1466.913900] [    812]  1000   812     3407        1    69632     1626             0 bash
[ 1466.913902] [   1001]  1000  1001     1895        1    57344      143             0 startx
[ 1466.913903] [   1023]  1000  1023     1006        0    49152       51             0 xinit
[ 1466.913905] [   1024]  1000  1024   179163      512   634880     3292             0 Xorg
[ 1466.913906] [   1033]  1000  1033    92987        0   217088     1934             0 xfce4-session
[ 1466.913908] [   1037]  1000  1037     2758        0    61440      241             0 dbus-daemon
[ 1466.913909] [   1040]  1000  1040    76635        1    98304      232             0 at-spi-bus-laun
[ 1466.913911] [   1044]  1000  1044    57827        0    81920      290             0 xfconfd
[ 1466.913912] [   1051]  1000  1051     2716        1    65536      165             0 dbus-daemon
[ 1466.913914] [   1054]  1000  1054    40960        1    81920      256             0 at-spi2-registr
[ 1466.913915] [   1057]   102  1057   483104        0   233472     1331             0 polkitd
[ 1466.913916] [   1075]  1000  1075     1498        0    49152      120             0 ssh-agent
[ 1466.913918] [   1080]  1000  1080    39233       18    69632       57             0 gpg-agent
[ 1466.913919] [   1082]  1000  1082    90685      109   274432     4696             0 xfwm4
[ 1466.913921] [   1088]  1000  1088    58565        1   176128     1375             0 xfsettingsd
[ 1466.913922] [   1089]  1000  1089    69802        0   184320     1784             0 xfce4-panel
[ 1466.913924] [   1092]     0  1092    65176        0   126976      419             0 upowerd
[ 1466.913925] [   1097]  1000  1097    87154        0   184320     1268             0 Thunar
[ 1466.913926] [   1118]  1000  1118    78926        0   258048    10182             0 xfdesktop
[ 1466.913928] [   1123]  1000  1123    69044        0   167936     1478             0 panel-16-whiske
[ 1466.913929] [   1126]  1000  1126    60071        0   196608     1436             0 panel-22-screen
[ 1466.913931] [   1127]  1000  1127    49794        0   163840     1227             0 panel-12-system
[ 1466.913932] [   1128]  1000  1128    49428        0   159744     1239             0 panel-24-cpugra
[ 1466.913933] [   1129]  1000  1129    50483        0   167936     1603             0 panel-20-cpufre
[ 1466.913935] [   1130]  1000  1130    49795        0   155648     1230             0 panel-13-netloa
[ 1466.913936] [   1131]  1000  1131    49794        0   159744     1267             0 panel-10-diskpe
[ 1466.913938] [   1132]  1000  1132    49795        0   163840     1225             0 panel-4-diskper
[ 1466.913939] [   1133]  1000  1133    49928        0   159744     1242             0 panel-8-xfce4-s
[ 1466.913940] [   1134]  1000  1134    59673        0   192512     1491             0 panel-3-power-m
[ 1466.913942] [   1137]  1000  1137    49443        0   159744     1225             0 panel-6-systray
[ 1466.913943] [   1140]  1000  1140    59653        0   192512     1497             0 panel-11-notifi
[ 1466.913944] [   1141]  1000  1141    58452        4   176128      695             0 panel-17-xfce4-
[ 1466.913946] [   1142]  1000  1142    49851        0   159744     1243             0 panel-2-actions
[ 1466.913947] [   1145]  1000  1145    48311      160   143360      350             0 panel-25-xfce4-
[ 1466.913949] [   1174]  1000  1174    95427        0   212992     2255             0 xfce4-terminal
[ 1466.913950] [   1176]  1000  1176   116553        0   258048     4697             0 xfce4-terminal
[ 1466.913951] [   1189]  1000  1189    48086        0   139264      467             0 polkit-gnome-au
[ 1466.913953] [   1197]  1000  1197    50286        0   159744     1402             0 xfce4-power-man
[ 1466.913954] [   1200]  1000  1200    42772        1   114688      874             0 pulseaudio
[ 1466.913956] [   1206]  1000  1206    67174        0   155648      541             0 xfce4-notifyd
[ 1466.913957] [   1216]  1000  1216     1895        1    53248      121             0 showdns
[ 1466.913958] [   1223]  1000  1223     1862        0    57344      107             0 logw
[ 1466.913960] [   1224]  1000  1224     1895        0    53248      124             0 showdns
[ 1466.913961] [   1225]  1000  1225     1895        1    53248      121             0 showdns_success
[ 1466.913963] [   1226]  1000  1226     1895        1    57344      120             0 showdns_fails
[ 1466.913964] [   1227]  1000  1227     1862        1    53248      120             0 showdns_from_fi
[ 1466.913965] [   1228]  1000  1228     1862        1    57344      120             0 showdns_from_ch
[ 1466.913967] [   1231]  1000  1231     1528        0    53248       82             0 dmesg
[ 1466.913968] [   1254]  1000  1254     1528        0    53248       77             0 dmesg
[ 1466.913970] [   1255]  1000  1255     1862        0    53248      108             0 stripcolors
[ 1466.913971] [   1256]  1000  1256     1625        0    53248      121             0 grep
[ 1466.913972] [   1257]  1000  1257     2479        1    53248       78             0 sed
[ 1466.913974] [   1260]  1000  1260     1528        0    53248       74             0 dmesg
[ 1466.913975] [   1261]  1000  1261     1862        0    57344      107             0 stripcolors
[ 1466.913977] [   1262]  1000  1262     1528        0    53248       87             0 dmesg
[ 1466.913978] [   1263]  1000  1263     1625        0    53248      119             0 grep
[ 1466.913979] [   1264]  1000  1264     1862        0    53248      108             0 stripcolors
[ 1466.913981] [   1265]  1000  1265     1635        0    53248      108             0 grep
[ 1466.913982] [   1266]  1000  1266     1624        0    53248      169             0 grep
[ 1466.913984] [   1267]  1000  1267     2545        0    61440      213             0 sed
[ 1466.913985] [   1268]  1000  1268     1623        0    53248      103             0 grep
[ 1466.913986] [   1269]  1000  1269     1528        0    53248       93             0 dmesg
[ 1466.913988] [   1270]  1000  1270     1862        0    49152      106             0 stripcolors
[ 1466.913989] [   1271]  1000  1271     1627        0    53248      143             0 grep
[ 1466.913991] [   1272]  1000  1272     2512        0    61440      154             0 sed
[ 1466.913992] [   1273]  1000  1273     2479        0    57344      101             0 sed
[ 1466.913993] [   1274]  1000  1274     2479        0    61440       94             0 sed
[ 1466.913995] [   1275]  1000  1275     2479        0    57344       82             0 sed
[ 1466.913996] [   1276]  1000  1276     2479        0    57344      114             0 sed
[ 1466.913998] [   1325]   133  1325    38725        0    73728      106             0 rtkit-daemon
[ 1466.913999] [   1336]  1000  1336     2649        0    69632       99             0 dbus-daemon
[ 1466.914000] [   1364]  1000  1364    61444        0   106496      288             0 gsettings-helpe
[ 1466.914002] [   1369]  1000  1369    57721        0    81920      137             0 xfconfd
[ 1466.914003] [   1765]  1000  1765    39295        0    69632      188             0 dconf-service
[ 1466.914005] [   3405]  1000  3405     3410        1    69632     1632             0 bash
[ 1466.914007] [   3472]  1000  3472     3487        0    61440      277             0 top
[ 1466.914008] [   3824]  1000  3824     3411        1    69632     1656             0 bash
[ 1466.914010] [   5947]  1000  5947     3410        1    73728     1633             0 bash
[ 1466.914012] [  11271]  1000 11271     3410        1    73728     1669             0 bash
[ 1466.914014] [  11355]  1000 11355     1862        1    53248      113             0 v
[ 1466.914015] [  11363]  1000 11363    16261        0   167936     1709             0 vim
[ 1466.914017] [  14553]  1000 14553     3501        1    73728     1714             0 bash
[ 1466.914019] [  14624]  1000 14624     1646      131    65536       82             0 watch
[ 1466.914020] [  15658]  1000 15658      947        1    49152       26             0 stress
[ 1466.914022] [  15659]  1000 15659  2442354    35472   733184    49380             0 stress
[ 1466.914023] [  15660]  1000 15660  2442354    17461   712704    64448             0 stress
[ 1466.914025] [  15661]  1000 15661  2442354    31447  1048576    92880             0 stress
[ 1466.914026] [  15662]  1000 15662  2442354    24856   692224    54878             0 stress
[ 1466.914027] [  15663]  1000 15663  2442354    78517  1007616    40479             0 stress
[ 1466.914029] [  15664]  1000 15664  2442354    28535   946176    82962             0 stress
[ 1466.914030] [  15665]  1000 15665  2442354      763   458752    49668             0 stress
[ 1466.914032] [  15666]  1000 15666  2442354    33217   905216    73150             0 stress
[ 1466.914033] [  15667]  1000 15667  2442354    27000  1216512   118297             0 stress
[ 1466.914034] [  15668]  1000 15668  2442354    30129   868352    71648             0 stress
[ 1466.914036] [  15669]  1000 15669  2442354    58911  1298432    96600             0 stress
[ 1466.914037] [  15670]  1000 15670  2442354    34067  1028096    87659             0 stress
[ 1466.914038] [  15671]  1000 15671  2442354    37665  1290240   116863             0 stress
[ 1466.914040] [  15672]  1000 15672  2442354    27605  1064960    98712             0 stress
[ 1466.914041] [  15673]  1000 15673  2442354    72245  1294336    82778             0 stress
[ 1466.914042] [  15674]  1000 15674  2442354    32309  1101824    98646             0 stress
[ 1466.914044] [  15675]  1000 15675  2442354    16205   659456    59473             0 stress
[ 1466.914045] [  15676]  1000 15676  2442354    75163  1179648    65513             0 stress
[ 1466.914047] [  15677]  1000 15677  2442354        1   577536    65327             0 stress
[ 1466.914048] [  15678]  1000 15678  2442354       87   643072    73506             0 stress
[ 1466.914050] [  15679]  1000 15679  2442354    21381   598016    46576             0 stress
[ 1466.914051] [  15680]  1000 15680  2442354    27317   925696    81287             0 stress
[ 1466.914052] [  15681]  1000 15681  2442354    53527  1007616    65673             0 stress
[ 1466.914054] [  15682]  1000 15682  2442354    33620   790528    58401             0 stress
[ 1466.914055] [  15683]  1000 15683  2442354    22478   688128    56765             0 stress
[ 1466.914056] [  15684]  1000 15684  2442354    45031   774144    44974             0 stress
[ 1466.914058] [  15685]  1000 15685  2442354      167   466944    51456             0 stress
[ 1466.914059] [  15686]  1000 15686  2442354    30506   684032    48204             0 stress
[ 1466.914060] [  15687]  1000 15687  2442354       76   360448    38174             0 stress
[ 1466.914062] [  15688]  1000 15688  2442354    27924   716800    54795             0 stress
[ 1466.914063] [  15689]  1000 15689  2442354    24105   823296    72068             0 stress
[ 1466.914064] [  15690]  1000 15690  2442354    13187   651264    60993             0 stress
[ 1466.914066] [  15691]  1000 15691  2442354    20202   675840    57520             0 stress
[ 1466.914067] [  15692]  1000 15692  2442354    37068  1036288    85690             0 stress
[ 1466.914069] [  15693]  1000 15693  2442354    34960   753664    52047             0 stress
[ 1466.914070] [  15694]  1000 15694  2442354    33364  1003520    85280             0 stress
[ 1466.914071] [  15695]  1000 15695  2442354    37500  1122304    95993             0 stress
[ 1466.914073] [  15696]  1000 15696  2442354    40010   774144    49648             0 stress
[ 1466.914074] [  15697]  1000 15697  2442354    25200   733184    59669             0 stress
[ 1466.914075] [  15698]  1000 15698  2442354    33928   860160    66367             0 stress
[ 1466.914077] [  15699]  1000 15699  2442354    28883   696320    51372             0 stress
[ 1466.914078] [  15700]  1000 15700  2442354    63688  1032192    58535             0 stress
[ 1466.914079] [  15701]  1000 15701  2442354      206   462848    50676             0 stress
[ 1466.914081] [  15702]  1000 15702  2442354    37586   978944    78007             0 stress
[ 1466.914082] [  15703]  1000 15703  2442354    57038   950272    54697             0 stress
[ 1466.914083] [  15704]  1000 15704  2442354    23691   688128    55570             0 stress
[ 1466.914085] [  15705]  1000 15705  2442354    24519   704512    56769             0 stress
[ 1466.914086] [  15706]  1000 15706  2442354    41070   831488    56065             0 stress
[ 1466.914087] [  15707]  1000 15707  2442354    16687   950272    95308             0 stress
[ 1466.914089] [  15708]  1000 15708  2442354    72969  1179648    67689             0 stress
[ 1466.914090] [  15709]  1000 15709  2442354    46107   962560    67440             0 stress
[ 1466.914091] [  15710]  1000 15710  2442354    30480   733184    54368             0 stress
[ 1466.914093] [  15711]  1000 15711  2442354    28799   757760    58943             0 stress
[ 1466.914094] [  15712]  1000 15712  2442354    24325   806912    69759             0 stress
[ 1466.914095] [  15713]  1000 15713  2442354    17610   737280    67758             0 stress
[ 1466.914097] [  15714]  1000 15714  2442354    26403   917504    81508             0 stress
[ 1466.914098] [  15715]  1000 15715  2442354     1099   577536    64016             0 stress
[ 1466.914099] [  15716]  1000 15716  2442354    36219   880640    67074             0 stress
[ 1466.914101] [  15717]  1000 15717  2442354    47198  1003520    71486             0 stress
[ 1466.914102] [  15718]  1000 15718  2442354    22209   798720    70805             0 stress
[ 1466.914103] [  15719]  1000 15719  2442354    36188   729088    48179             0 stress
[ 1466.914105] [  15720]  1000 15720  2442354    48222  1081344    80144             0 stress
[ 1466.914106] [  15721]  1000 15721  2442354   114843  1372160    49891             0 stress
[ 1466.914108] [  15722]  1000 15722  2442354      162   557056    62352             0 stress
[ 1466.914109] [  15723]  1000 15723  2442354    34078   843776    64635             0 stress
[ 1466.914110] [  15724]  1000 15724  2442354        1   516096    57478             0 stress
[ 1466.914112] [  15725]  1000 15725  2442354    33713   684032    45031             0 stress
[ 1466.914113] [  15726]  1000 15726  2442354    70880  1007616    48303             0 stress
[ 1466.914114] [  15727]  1000 15727  2442354    33070   794624    59294             0 stress
[ 1466.914116] [  15728]  1000 15728  2442354    49921   884736    53910             0 stress
[ 1466.914117] [  15729]  1000 15729  2442354    37100   786432    54438             0 stress
[ 1466.914118] [  15730]  1000 15730  2442354      772   745472    85673             0 stress
[ 1466.914120] [  15731]  1000 15731  2442354    20798   647168    53336             0 stress
[ 1466.914121] [  15732]  1000 15732  2442354    66860  1286144    87132             0 stress
[ 1466.914122] [  15733]  1000 15733  2442354    58022  1003520    60648             0 stress
[ 1466.914124] [  15734]  1000 15734  2442354    40229   794624    52337             0 stress
[ 1466.914125] [  15735]  1000 15735  2442354    52416   831488    44743             0 stress
[ 1466.914127] [  15736]  1000 15736  2442354    22784  1220608   123039             0 stress
[ 1466.914128] [  15737]  1000 15737  2442354    47752  1228800    99082             0 stress
[ 1466.914130] [  15738]  1000 15738  2442354     5388   598016    62219             0 stress
[ 1466.914131] [  15739]  1000 15739  2442354    42512   991232    74612             0 stress
[ 1466.914132] [  15740]  1000 15740  2442354    30409   843776    68268             0 stress
[ 1466.914134] [  15741]  1000 15741  2442354       61   385024    41076             0 stress
[ 1466.914135] [  15742]  1000 15742  2442354    22049   688128    57225             0 stress
[ 1466.914136] [  15743]  1000 15743  2442354    11593  1163264   127053             0 stress
[ 1466.914138] [  15744]  1000 15744  2442354    12709   700416    68066             0 stress
[ 1466.914139] [  15745]  1000 15745  2442354    58506   966656    55522             0 stress
[ 1466.914140] [  15746]  1000 15746  2442354    37704   978944    77912             0 stress
[ 1466.914142] [  15747]  1000 15747  2442354      557   430080    46244             0 stress
[ 1466.914143] [  15748]  1000 15748  2442354    47961   929792    61466             0 stress
[ 1466.914144] [  15749]  1000 15749  2442354    35600   880640    67733             0 stress
[ 1466.914146] [  15750]  1000 15750  2442354    36018   897024    69355             0 stress
[ 1466.914147] [  15751]  1000 15751  2442354      397   552960    61659             0 stress
[ 1466.914149] [  15752]  1000 15752  2442354     9336   745472    77088             0 stress
[ 1466.914150] [  15753]  1000 15753  2442354    57814  1249280    91607             0 stress
[ 1466.914151] [  15754]  1000 15754  2442354    46202  1044480    77582             0 stress
[ 1466.914153] [  15755]  1000 15755  2442354    63801  1069056    63083             0 stress
[ 1466.914154] [  15756]  1000 15756  2442354    21609   765952    67300             0 stress
[ 1466.914155] [  15757]  1000 15757  2442354    59274   892928    45599             0 stress
[ 1466.914157] [  15758]  1000 15758  2442354        1   479232    52862             0 stress
[ 1466.914158] [  15759]  1000 15759  2442354    76444  1163264    62175             0 stress
[ 1466.914160] [  15760]  1000 15760  2442354    31653   933888    78283             0 stress
[ 1466.914161] [  15761]  1000 15761  2442354    30021   790528    62026             0 stress
[ 1466.914163] [  15762]  1000 15762  2442354    45007   901120    60875             0 stress
[ 1466.914164] [  15763]  1000 15763  2442354    21275   659456    54316             0 stress
[ 1466.914165] [  15764]  1000 15764  2442354    29773   778240    60715             0 stress
[ 1466.914167] [  15765]  1000 15765  2442354    34685   815104    60423             0 stress
[ 1466.914168] [  15766]  1000 15766  2442354    19320   741376    66595             0 stress
[ 1466.914169] [  15767]  1000 15767  2442354    26619   630784    45137             0 stress
[ 1466.914171] [  15768]  1000 15768  2442354    65238  1044480    58545             0 stress
[ 1466.914172] [  15769]  1000 15769  2442354    34694   851968    65018             0 stress
[ 1466.914174] [  15770]  1000 15770  2442354     7212   589824    59319             0 stress
[ 1466.914175] [  15771]  1000 15771  2442354    35594   729088    48796             0 stress
[ 1466.914177] [  15772]  1000 15772  2442354    17326   811008    77258             0 stress
[ 1466.914178] [  15773]  1000 15773  2442354    32841  1118208   100160             0 stress
[ 1466.914180] [  15774]  1000 15774  2442354      202   847872    98614             0 stress
[ 1466.914181] [  15775]  1000 15775  2442354       31   651264    74185             0 stress
[ 1466.914182] [  15776]  1000 15776  2442354    44425  1032192    77815             0 stress
[ 1466.914184] [  15777]  1000 15777  2442354    23280   675840    54459             0 stress
[ 1466.914185] [  15778]  1000 15778  2442354    20222   839680    77633             0 stress
[ 1466.914187] [  15779]  1000 15779  2442354    34766  1187840   106667             0 stress
[ 1466.914188] [  15780]  1000 15780  2442354    18561   540672    42251             0 stress
[ 1466.914189] [  15781]  1000 15781  2442354      157   360448    37959             0 stress
[ 1466.914191] [  15782]  1000 15782  2442354    31124   733184    53724             0 stress
[ 1466.914209] [  15783]  1000 15783  2442354    25112   638976    48009             0 stress
[ 1466.914210] [  15784]  1000 15784  2442354     9932   495616    45085             0 stress
[ 1466.914212] [  15785]  1000 15785  2442354    33793   724992    50071             0 stress
[ 1466.914213] [  15786]  1000 15786  2442354    20149   860160    80493             0 stress
[ 1466.914214] [  15787]  1000 15787  2442354    49386  1040384    73876             0 stress
[ 1466.914216] [  15788]  1000 15788  2442354    26792   794624    65738             0 stress
[ 1466.914217] [  15789]  1000 15789  2442354      463   647168    73278             0 stress
[ 1466.914219] [  15790]  1000 15790  2442354    27350   651264    47280             0 stress
[ 1466.914220] [  15791]  1000 15791  2442354    27893   974848    87221             0 stress
[ 1466.914221] [  15792]  1000 15792  2442354    51549   917504    56361             0 stress
[ 1466.914223] [  15793]  1000 15793  2442354    31852   860160    68872             0 stress
[ 1466.914224] [  15794]  1000 15794  2442354    28996   827392    67626             0 stress
[ 1466.914226] [  15795]  1000 15795  2442354    49525   806912    44556             0 stress
[ 1466.914227] [  15796]  1000 15796  2442354    59806   819200    35827             0 stress
[ 1466.914228] [  15797]  1000 15797  2442354    42778   966656    71305             0 stress
[ 1466.914230] [  15798]  1000 15798  2442354    29384   745472    57056             0 stress
[ 1466.914231] [  15799]  1000 15799  2442354    29609   655360    45554             0 stress
[ 1466.914232] [  15800]  1000 15800  2442354    19677   602112    48838             0 stress
[ 1466.914234] [  15801]  1000 15801  2442354    91631  1261568    59202             0 stress
[ 1466.914235] [  15802]  1000 15802  2442354    37784   708608    44013             0 stress
[ 1466.914236] [  15803]  1000 15803  2442354    38862   843776    59834             0 stress
[ 1466.914238] [  15804]  1000 15804  2442354    53364  1110016    78641             0 stress
[ 1466.914239] [  15805]  1000 15805  2442354       93   380928    40778             0 stress
[ 1466.914241] [  15806]  1000 15806  2442354    38511  1003520    80159             0 stress
[ 1466.914242] [  15807]  1000 15807  2442354    73352  1150976    63780             0 stress
[ 1466.914243] [  15808]  1000 15808  2442354    41336   745472    45096             0 stress
[ 1466.914245] [  15809]  1000 15809  2442354    22708   831488    74460             0 stress
[ 1466.914246] [  15810]  1000 15810  2442354    30775   716800    52072             0 stress
[ 1466.914247] [  15811]  1000 15811  2442354    28548  1024000    92686             0 stress
[ 1466.914249] [  15812]  1000 15812  2442354    45063  1224704   101243             0 stress
[ 1466.914250] [  15813]  1000 15813  2442354    41298   786432    50210             0 stress
[ 1466.914252] [  15814]  1000 15814  2442354    16514   557056    45980             0 stress
[ 1466.914253] [  15815]  1000 15815  2442354    38228   950272    73777             0 stress
[ 1466.914255] [  15816]  1000 15816  2442354     3546  1073152   123763             0 stress
[ 1466.914256] [  15817]  1000 15817  2442354      228   774144    89437             0 stress
[ 1466.914258] [  15818]  1000 15818  2442354   147848  2117632   109587             0 stress
[ 1466.914259] [  15819]  1000 15819  2442354    30535  1105920   100966             0 stress
[ 1466.914260] [  15820]  1000 15820  2442354    35952  1097728    94515             0 stress
[ 1466.914262] [  15821]  1000 15821  2442354        1   716800    82464             0 stress
[ 1466.914263] [  15822]  1000 15822  2442354    24288  1056768   101008             0 stress
[ 1466.914265] [  15823]  1000 15823  2442354       82   753664    87027             0 stress
[ 1466.914266] [  15824]  1000 15824  2442354    46572  1122304    86919             0 stress
[ 1466.914268] [  15825]  1000 15825  2442354    27250  1073152   100103             0 stress
[ 1466.914269] [  15826]  1000 15826  2442354    16521   851968    83211             0 stress
[ 1466.914270] [  15827]  1000 15827  2442354    65108  1241088    83270             0 stress
[ 1466.914272] [  15828]  1000 15828  2442354    42946   995328    74688             0 stress
[ 1466.914273] [  15829]  1000 15829  2442354    29091  1073152    97972             0 stress
[ 1466.914275] [  15830]  1000 15830  2442354    43149  1134592    91934             0 stress
[ 1466.914276] [  15831]  1000 15831  2442354    20040   962560    93516             0 stress
[ 1466.914277] [  15832]  1000 15832  2442354    39625  1032192    82643             0 stress
[ 1466.914279] [  15833]  1000 15833  2442354    28676   802816    64882             0 stress
[ 1466.914280] [  15834]  1000 15834  2442354    32089  1122304   101453             0 stress
[ 1466.914281] [  15835]  1000 15835  2442354    53649  1032192    68572             0 stress
[ 1466.914283] [  15836]  1000 15836  2442354      176   585728    66213             0 stress
[ 1466.914284] [  15837]  1000 15837  2442354       16   487424    53694             0 stress
[ 1466.914286] [  15838]  1000 15838  2442354    34092   753664    53328             0 stress
[ 1466.914287] [  15839]  1000 15839  2442354    24923   733184    59956             0 stress
[ 1466.914288] [  15840]  1000 15840  2442354    75766  1343488    85385             0 stress
[ 1466.914290] [  15841]  1000 15841  2442354    24663   819200    70933             0 stress
[ 1466.914291] [  15842]  1000 15842  2442354    53107   966656    60930             0 stress
[ 1466.914292] [  15843]  1000 15843  2442354    38419   925696    70548             0 stress
[ 1466.914294] [  15844]  1000 15844  2442354    34410   823296    61763             0 stress
[ 1466.914295] [  15845]  1000 15845  2442354    18088   724992    65749             0 stress
[ 1466.914297] [  15846]  1000 15846  2442354    48202  1228800    98646             0 stress
[ 1466.914298] [  15847]  1000 15847  2442354    82782  1212416    61974             0 stress
[ 1466.914299] [  15848]  1000 15848  2442354    19695   872448    82357             0 stress
[ 1466.914301] [  15849]  1000 15849  2442354    34530   860160    66229             0 stress
[ 1466.914302] [  15850]  1000 15850  2442354    40215   819200    55441             0 stress
[ 1466.914303] [  15851]  1000 15851  2442354    33624   716800    49227             0 stress
[ 1466.914305] [  15852]  1000 15852  2442354    55436   995328    62192             0 stress
[ 1466.914306] [  15853]  1000 15853  2442354    21205   929792    88066             0 stress
[ 1466.914308] [  15854]  1000 15854  2442354    52815  1171456    86823             0 stress
[ 1466.914309] [  15855]  1000 15855  2442354    16708   667648    59546             0 stress
[ 1466.914310] [  15856]  1000 15856  2442354    33506   925696    75409             0 stress
[ 1466.914312] [  15857]  1000 15857  2442354     4436   417792    41037             0 stress
[ 1466.914313] [  15858]  1000 15858  2442354    41905   839680    56283             0 stress
[ 1466.914314] [  15859]  1000 15859  2442354    16448   802816    77136             0 stress
[ 1466.914316] [  15860]  1000 15860  2442354    48905   950272    63128             0 stress
[ 1466.914317] [  15861]  1000 15861  2442354    14641   704512    66276             0 stress
[ 1466.914319] [  15862]  1000 15862  2442354      244   638976    72828             0 stress
[ 1466.914320] [  15863]  1000 15863  2442354    37726   978944    77901             0 stress
[ 1466.914321] [  15864]  1000 15864  2442354    69044  1224704    77302             0 stress
[ 1466.914323] [  15865]  1000 15865  2442354    50818   892928    54057             0 stress
[ 1466.914324] [  15866]  1000 15866  2442354      149   815104    94511             0 stress
[ 1466.914326] [  15867]  1000 15867  2442354    21797   753664    65579             0 stress
[ 1466.914327] [  15868]  1000 15868  2442354    86029  3780608   379238             0 stress
[ 1466.914329] [  15869]  1000 15869  2442354    35631  1433600   136771             0 stress
[ 1466.914330] [  15870]  1000 15870  2442354    73646  1695744   131540             0 stress
[ 1466.914332] [  15871]  1000 15871  2442354    70948  1466368   105576             0 stress
[ 1466.914333] [  15872]  1000 15872  2442354      627  2543616   309624             0 stress
[ 1466.914335] [  15873]  1000 15873  2442354      592   802816    93005             0 stress
[ 1466.914336] [  15874]  1000 15874  2442354    50241  3207168   343395             0 stress
[ 1466.914337] [  15875]  1000 15875  2442354    33324  1392640   133978             0 stress
[ 1466.914339] [  15876]  1000 15876  2442354    75920  1761280   137479             0 stress
[ 1466.914340] [  15877]  1000 15877  2442354    36961  3280896   365416             0 stress
[ 1466.914341] [  15878]  1000 15878  2442354    33187  1441792   140260             0 stress
[ 1466.914342] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=stress,pid=15868,uid=1000
[ 1466.914349] Out of memory: Killed process 15868 (stress) total-vm:9769416kB, anon-rss:344112kB, file-rss:4kB, shmem-rss:0kB
[ 1467.129760] oom_reaper: reaped process 15868 (stress), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB



I got one at 19.2sec
$ time stress -m 220 --vm-bytes 10000000000 --timeout 10
stress: info: [21374] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: FAIL: [21374] (415) <-- worker 21585 got signal 9
stress: WARN: [21374] (417) now reaping child worker processes
stress: FAIL: [21374] (451) failed run completed in 19s

real	0m19.223s
user	0m1.077s
sys	2m53.496s

I did 45 tries without tracers.
Recompiled kernel just like the last time(when I was able to trigger it on the third try) aka with tracers:
I did 15 tries and they were all successfull(at full CPU speed) with durations between 17-22 sec.
I guess all that tracing code in kernel introduced some minor lag which is enough for this to never trigger OOM-killer, just like when I limit cpu max to 800Mhz.

I even tried running two in parallel, no OOM-killer.
Or this:
$ time stress -m 220 --vm-bytes 10000000000 --timeout 30
stress: info: [12381] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: info: [12381] successful run completed in 56s

real	0m55.957s
user	0m1.908s
sys	9m20.969s
Longer timeout I could see 65000MiB of swap were used at some point! (in 'top')
still nothing triggered OOM-killer :D
ok, a timeout of 60 did it(triggered OOM):
$ time stress -m 220 --vm-bytes 10000000000 --timeout 60
stress: info: [13225] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
stress: FAIL: [13225] (415) <-- worker 13445 got signal 9
stress: WARN: [13225] (417) now reaping child worker processes
stress: FAIL: [13225] (451) failed run completed in 59s

real	0m58.812s
user	0m2.312s
sys	9m53.327s

But anyway, since last time I was able to trigger it with the normal(timeout 10) command on the third try, I've decided to keep trying that:
after 170 more tries via `$ while true; do time stress -m 220 --vm-bytes 10000000000 --timeout 10; done`
I saw no hangs or any runs taking more time than the usual 16-26 sec.
So the patch must be working as intended. Thanks very much. Let me know if you want me to do anything else.

Cheers!
Comment 24 mgorman 2019-07-18 08:37:15 UTC
On Wed, Jul 17, 2019 at 10:00:18PM +0000, howaboutsynergy@protonmail.com wrote:
> tl;dr: patch seems to work, thank you very much!
> 

\o/

> ????????????????????? Original Message ?????????????????????
> On Wednesday, July 17, 2019 7:53 PM, Mel Gorman <mgorman@techsingularity.net>
> wrote:
> 
> > Ok, great. From the trace, it was obvious that the scanner is making no
> > progress. I don't think zswap is involved as such but it may be making
> > it easier to trigger due to altering timing. At least, I see no reason
> > why zswap would materially affect the termination conditions.
>
> I don't know if it matters in this context, but I've been using the term
> `zswap`(somewhere else I think) to (wrongly)refer to swap in zram (and
> even sometimes called it ext4(in this bug report too) without realizing
> at the time that ext4 is only for /tmp and /var/tmp instead! they are
> ext4 in zram) but in fact this isn't zswap that I have been using (even
> though I have CONFIG_ZSWAP=y in .config) but it's just CONFIG_ZRAM=y with
> CONFIG_SWAP=y (and probably a bunch of others being needed too).

For other bugs, it would matter a lot. In this specific case, all that
was altered was the timing. With enough effort I was able to reproduce
the problem reliably within 30 minutes. With the patch, it ran without
hitting the problem for hours. I had tracing patches applied to log when
the specific problem occurred.

Without the patch, when the problem occurred, it hit 197009 times in a tight
loop driving up CPU usage. With the patch, the same condition was hit 4
times and in each case the task exited immediately as expected so I
think we're good.

> > a proper abort. I think it ends up looping in compaction instead of dying
> > without either aborting or progressing the scanner. It might explain why
> > stress-ng is hitting is as it is probably sending fatal signals on timeout
> > (I didn't check the source).
> Ah I didn't know there are multiple `stress` versions, here's what I used:
> 
> /usr/bin/stress is owned by stress 1.0.4-5
> 

Fortunately, they were all equivalent. The key was getting a task to
receive a fatal signal while compacting and while scanning a PFN that
was not aligned to SWAP_CLUSTER_MAX. Tricky to hit but fortunately your
test case was exactly what we needed.

> > <SNIP>
> 
> Now the "problem" is I can't tell if it would get stuck :D but it usually
> ends in no more than 17 sec:
> $ time stress -m 220 --vm-bytes 10000000000 --timeout 10
> stress: info: [7981] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd
> stress: FAIL: [7981] (415) <-- worker 8202 got signal 9
> stress: WARN: [7981] (417) now reaping child worker processes
> stress: FAIL: [7981] (415) <-- worker 8199 got signal 9
> stress: WARN: [7981] (417) now reaping child worker processes
> stress: FAIL: [7981] (451) failed run completed in 18s
> 

That's fine. Fortunately my own debug tracing added based on your trace
gives me enough confidence.

> <SNIP>
>
> (probably irrelevant)Sometimes Xorg says it can't allocate any more memory
> but stacktrace looks like it's inside some zram i915 kernel stuff:
> 
> [ 1416.842931] [drm] Atomic update on pipe (A) took 188 us, max time under
> evasion is 100 us
> [ 1425.416979] Xorg: page allocation failure: order:0,
> mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE),
> nodemask=(null),cpuset=/,mems_allowed=0
> [ 1425.416984] CPU: 1 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G     U    
>        5.2.1-g527a3db363a3 #74
> [ 1425.416985] Hardware name: System manufacturer System Product Name/PRIME
> Z370-A, BIOS 2201 05/27/2019
> [ 1425.416986] Call Trace:

So this looks like the system is still under a lot of stress trying to
swap. It's unfortunate but unrelated and relatively benign given the
level of stress the system is under.

> <SNIP>
>
> But anyway, since last time I was able to trigger it with the normal(timeout
> 10) command on the third try, I've decided to keep trying that:
> after 170 more tries via `$ while true; do time stress -m 220 --vm-bytes
> 10000000000 --timeout 10; done`
> I saw no hangs or any runs taking more time than the usual 16-26 sec.
> So the patch must be working as intended. Thanks very much. Let me know if
> you want me to do anything else.
> 

Perfect. Thanks a million for your patience, testing and tracing (the
tracing pinpointed exactly where I needed to look -- of 3 potential
candidates, only one really made sense). I'll put a proper changelog on
this and send it out where it should get picked up for 5.3 and -stable.
Comment 25 GYt2bW 2019-07-19 22:13:11 UTC
For completeness this is what I've had to change in kernel .config to get trace-cmd to work:
seen here https://bugzilla.kernel.org/show_bug.cgi?id=204231#c3

Probably less CONFIG_* settings needed to be added than that, but I just made sure and added a bunch(if not all) so that trace-cmd didn't complain.

Note You need to log in before you can comment on or make changes to this bug.