Bug 204165
| Summary: | 100% CPU usage in compact_zone | ||
|---|---|---|---|
| Product: | Memory Management | Reporter: | GYt2bW (howaboutsynergy) |
| Component: | Page Allocator | Assignee: | Andrew Morton (akpm) |
| Status: | NEW --- | ||
| Severity: | normal | CC: | howaboutsynergy |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 5.2.0-g0ecfebd2b524 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
zcat-ed kernel .config from /proc
publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc signature.asc publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc signature.asc |
||
|
Description
GYt2bW
2019-07-13 19:20:21 UTC
``` PID %CPU COMMAND PR NI VIRT RES S USER 25677 100.0 stress 20 0 980352 116676 R user 25752 100.0 stress 20 0 980352 122820 R user 25800 100.0 stress 20 0 980352 0 R user 25727 98.1 stress 20 0 980352 96196 R user 25777 98.1 stress 20 0 980352 124868 R user ``` ``` [ 1044.371191] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(V) show-blocked-tasks(w) [ 1073.867794] sysrq: Show Blocked State [ 1073.867806] task PC stack pid father [ 1094.149119] sysrq: Show backtrace of all active CPUs [ 1094.149129] NMI backtrace for cpu 8 [ 1094.149138] CPU: 8 PID: 14722 Comm: bash Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.149142] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.149144] Call Trace: [ 1094.149157] dump_stack+0x46/0x60 [ 1094.149163] nmi_cpu_backtrace.cold+0x14/0x53 [ 1094.149170] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 1094.149174] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 1094.149181] __handle_sysrq.cold+0x48/0x102 [ 1094.149187] write_sysrq_trigger+0x1f/0x30 [ 1094.149193] proc_reg_write+0x37/0x60 [ 1094.149201] vfs_write+0xb1/0x190 [ 1094.149207] ksys_write+0x5d/0xe0 [ 1094.149213] do_syscall_64+0x50/0x170 [ 1094.149220] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1094.149225] RIP: 0033:0x79787f47a4ba [ 1094.149230] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 [ 1094.149234] RSP: 002b:00007ffc61be1698 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1094.149240] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000079787f47a4ba [ 1094.149242] RDX: 0000000000000002 RSI: 000055f82dde0ee0 RDI: 0000000000000001 [ 1094.149245] RBP: 00007ffc61be16c0 R08: 000000000000000a R09: 0000000000000001 [ 1094.149248] R10: 000055f82d3c305e R11: 0000000000000246 R12: 0000000000000002 [ 1094.149251] R13: 000055f82dde0ee0 R14: 000079787f54f500 R15: 000079787f550300 [ 1094.149256] Sending NMI from CPU 8 to CPUs 0-7,9-11: [ 1094.149387] NMI backtrace for cpu 0 [ 1094.149389] CPU: 0 PID: 25727 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.149391] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.149392] RIP: 0010:isolate_migratepages_block+0x3b/0xa50 [ 1094.149394] Code: 49 89 fe 48 89 f3 48 89 54 24 08 89 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 c0 48 8b 47 40 48 8b 68 50 <48> 89 6c 24 20 48 c7 44 24 78 00 00 00 00 c6 44 24 77 00 65 48 8b [ 1094.149395] RSP: 0018:ffffb2d88badb958 EFLAGS: 00000246 [ 1094.149397] RAX: ffff99f64dfded00 RBX: 000000000058fcc0 RCX: 000000000000000c [ 1094.149398] RDX: 000000000058fe00 RSI: 000000000058fcc0 RDI: ffffb2d88badbad0 [ 1094.149399] RBP: ffff99f64dfde000 R08: ffff99f64dfd9b10 R09: ffff99f62d821450 [ 1094.149400] R10: 0000000000000001 R11: ffff99f62d820a80 R12: 000000000058fe00 [ 1094.149401] R13: ffff99f64dfded00 R14: ffffb2d88badbad0 R15: ffffb2d88badbad0 [ 1094.149402] FS: 0000720b85407740(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000 [ 1094.149403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.149404] CR2: 0000631d9a784fe0 CR3: 00000007c23a4003 CR4: 00000000003606f0 [ 1094.149405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.149406] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.149406] Call Trace: [ 1094.149407] compact_zone+0x577/0xa70 [ 1094.149408] compact_zone_order+0xde/0x120 [ 1094.149409] try_to_compact_pages+0x187/0x240 [ 1094.149409] __alloc_pages_direct_compact+0x87/0x170 [ 1094.149410] __alloc_pages_slowpath+0x20e/0xc20 [ 1094.149411] ? release_pages+0x348/0x3b0 [ 1094.149412] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1094.149413] __alloc_pages_nodemask+0x268/0x2b0 [ 1094.149413] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1094.149414] __handle_mm_fault+0xc11/0x1890 [ 1094.149415] handle_mm_fault+0xa9/0x1d0 [ 1094.149416] __do_page_fault+0x237/0x480 [ 1094.149416] do_page_fault+0x1d/0x67 [ 1094.149417] ? page_fault+0x8/0x30 [ 1094.149418] page_fault+0x1e/0x30 [ 1094.149419] RIP: 0033:0x5c2e1a191c10 [ 1094.149419] Code: Bad RIP value. [ 1094.149420] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1094.149422] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1094.149423] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1094.149424] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1094.149425] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1094.149426] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1094.149478] NMI backtrace for cpu 2 skipped: idling at intel_idle+0x7d/0x120 [ 1094.149483] NMI backtrace for cpu 1 [ 1094.149487] CPU: 1 PID: 25677 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.149490] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.149493] RIP: 0010:compact_zone+0x499/0xa70 [ 1094.149497] Code: f6 48 2b 35 29 7e d1 00 48 c1 fe 06 b9 01 00 00 00 ba 03 00 00 00 4c 89 f7 e8 93 10 02 00 48 85 c0 0f 85 93 00 00 00 4c 89 f7 <e8> 62 d8 ff ff 84 c0 75 56 44 8b 45 70 45 85 c0 0f 85 af 00 00 00 [ 1094.149501] RSP: 0018:ffffb2d88b94ba18 EFLAGS: 00000202 [ 1094.149504] RAX: 0000000000718420 RBX: 0000000000718800 RCX: 00000000000038c2 [ 1094.149507] RDX: 0000000000100000 RSI: 000000000000000c RDI: ffffde0b9c610000 [ 1094.149511] RBP: ffffb2d88b94bad0 R08: 0000000000000001 R09: ffff99f62d861450 [ 1094.149525] R10: 0000000000000001 R11: ffff99f62d860a80 R12: 0000000000718600 [ 1094.149538] R13: ffff99f64dfded00 R14: ffffde0b9c610000 R15: 0000000000000000 [ 1094.149541] FS: 0000720b85407740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000 [ 1094.149544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.149547] CR2: 0000631d9a784fe0 CR3: 00000007c8b04001 CR4: 00000000003606e0 [ 1094.149550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.149552] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.149555] Call Trace: [ 1094.149558] compact_zone_order+0xde/0x120 [ 1094.149560] try_to_compact_pages+0x187/0x240 [ 1094.149563] __alloc_pages_direct_compact+0x87/0x170 [ 1094.149569] __alloc_pages_slowpath+0x454/0xc20 [ 1094.149575] ? release_pages+0x348/0x3b0 [ 1094.149577] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1094.149589] __alloc_pages_nodemask+0x268/0x2b0 [ 1094.149600] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1094.149606] __handle_mm_fault+0xc11/0x1890 [ 1094.149609] handle_mm_fault+0xa9/0x1d0 [ 1094.149612] __do_page_fault+0x237/0x480 [ 1094.149615] do_page_fault+0x1d/0x67 [ 1094.149620] ? page_fault+0x8/0x30 [ 1094.149622] page_fault+0x1e/0x30 [ 1094.149625] RIP: 0033:0x5c2e1a191c10 [ 1094.149630] Code: Bad RIP value. [ 1094.149636] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1094.149645] RAX: 00000000071a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1094.149650] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1094.149654] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1094.149657] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1094.149663] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1094.149704] NMI backtrace for cpu 4 [ 1094.149708] CPU: 4 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.149710] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.149712] RIP: 0033:0x571ede0e4420 [ 1094.149716] Code: 44 24 2c 48 89 44 24 10 48 85 f6 0f 84 97 00 00 00 49 89 cc 31 db 4c 8d 6c 24 30 4c 8d 35 59 6c 00 00 0f 1f 84 00 00 00 00 00 <48> 8b 44 24 08 48 8d 2c 18 48 8b 14 24 48 29 da 4c 89 e9 48 89 ee [ 1094.149720] RSP: 002b:00007ffe25d18460 EFLAGS: 00000202 [ 1094.149726] RAX: 0000000000000001 RBX: 0000000000000046 RCX: 0000000000000000 [ 1094.149728] RDX: 0000000000000001 RSI: 0000571ede0f049a RDI: 00007c7ea567b320 [ 1094.149729] RBP: 0000571ede0f049a R08: 0000000000000001 R09: 0000000000000000 [ 1094.149731] R10: 00007ffe25d1848c R11: 0000571ede0f049a R12: 00007c7ea5679500 [ 1094.149735] R13: 00007ffe25d18490 R14: 0000571ede0eb071 R15: 0000000000000001 [ 1094.149741] FS: 00007c7ea548a740 GS: 0000000000000000 [ 1094.149752] NMI backtrace for cpu 3 [ 1094.149758] CPU: 3 PID: 25800 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.149767] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.149772] RIP: 0010:isolate_migratepages_block+0x1c/0xa50 [ 1094.149777] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 <89> 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 [ 1094.149783] RSP: 0018:ffffb2d88bd23958 EFLAGS: 00000292 [ 1094.149792] RAX: 0000000000000000 RBX: 000000000016b420 RCX: 000000000000000c [ 1094.149800] RDX: 000000000016b600 RSI: 000000000016b420 RDI: ffffb2d88bd23ad0 [ 1094.149803] RBP: ffffb2d88bd23ad0 R08: 0000000000000001 R09: ffff99f62d8e1450 [ 1094.149808] R10: 0000000000000001 R11: ffff99f62d8e0a80 R12: 000000000016b600 [ 1094.149816] R13: ffff99f64dfded00 R14: ffffb2d88bd23ad0 R15: ffffb2d88bd23ad0 [ 1094.149822] FS: 0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000 [ 1094.149825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.149831] CR2: 00007210ba3fc578 CR3: 00000007ec28a005 CR4: 00000000003606e0 [ 1094.149838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.149842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.149848] Call Trace: [ 1094.149853] compact_zone+0x577/0xa70 [ 1094.149859] compact_zone_order+0xde/0x120 [ 1094.149863] try_to_compact_pages+0x187/0x240 [ 1094.149868] __alloc_pages_direct_compact+0x87/0x170 [ 1094.149871] __alloc_pages_slowpath+0x454/0xc20 [ 1094.149875] ? release_pages+0x348/0x3b0 [ 1094.149878] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1094.149883] __alloc_pages_nodemask+0x268/0x2b0 [ 1094.149889] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1094.149894] __handle_mm_fault+0xc11/0x1890 [ 1094.149897] handle_mm_fault+0xa9/0x1d0 [ 1094.149900] __do_page_fault+0x237/0x480 [ 1094.149905] do_page_fault+0x1d/0x67 [ 1094.149907] ? page_fault+0x8/0x30 [ 1094.149910] page_fault+0x1e/0x30 [ 1094.149915] RIP: 0033:0x5c2e1a191c10 [ 1094.149920] Code: Bad RIP value. [ 1094.149923] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1094.149932] RAX: 00000000183a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1094.149935] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1094.149943] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1094.149946] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1094.149949] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1094.149954] NMI backtrace for cpu 5 [ 1094.149960] CPU: 5 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150093] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150099] RIP: 0033:0x77d47d3fd585 [ 1094.150104] Code: 8b 04 24 25 00 0a 00 00 3d 00 0a 00 00 74 66 49 8b 44 24 30 48 39 f8 76 7c 48 29 f8 48 89 c2 31 c9 48 85 d2 74 6f 48 89 4d c8 <48> 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24 [ 1094.150107] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000246 [ 1094.150114] RAX: 00005ffb5747045f RBX: 0000000000000001 RCX: 0000000000000000 [ 1094.150117] RDX: 0000000000000ff6 RSI: 00005ffb5747045f RDI: 00005ffb591e43ca [ 1094.150121] RBP: 00007ffe06db5e30 R08: 00005ffb57470460 R09: 0000000000000000 [ 1094.150124] R10: 00007ffe06db5ebc R11: 00005ffb5747045f R12: 000077d47d554500 [ 1094.150127] R13: 00005ffb5747045f R14: 0000000000000000 R15: 00005ffb5747045f [ 1094.150131] FS: 000077d47d365740 GS: 0000000000000000 [ 1094.150134] NMI backtrace for cpu 6 [ 1094.150139] CPU: 6 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150143] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150151] RIP: 0010:ahci_scr_read+0x41/0x60 [ 1094.150156] Code: 07 48 05 00 01 00 00 41 b8 ea ff ff ff 83 fe 04 77 1d 74 1f 89 f6 48 63 0c b5 f0 84 cc a2 85 c9 74 0d 48 01 c8 48 01 f8 8b 00 <89> 02 45 31 c0 44 89 c0 c3 f6 41 1b 20 74 f6 b9 3c 00 00 00 eb e2 [ 1094.150159] RSP: 0000:ffffb2d883340e60 EFLAGS: 00000082 [ 1094.150171] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 000000000000003c [ 1094.150174] RDX: ffffb2d883340e6c RSI: 0000000000000004 RDI: ffffb2d883189000 [ 1094.150180] RBP: ffff99f628f28000 R08: 00000000ffffffea R09: 0000000000000000 [ 1094.150184] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f628f2a040 [ 1094.150190] R13: ffffb2d883189180 R14: ffff99f628f2a040 R15: ffff99f628f22018 [ 1094.150196] FS: 00007c32c37f6740(0000) GS:ffff99f62d980000(0000) knlGS:0000000000000000 [ 1094.150200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.150204] CR2: 00007da5da76cb50 CR3: 000000077cd3c003 CR4: 00000000003606e0 [ 1094.150209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.150216] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.150219] Call Trace: [ 1094.150222] <IRQ> [ 1094.150226] sata_async_notification+0x59/0x130 [ 1094.150231] ahci_handle_port_interrupt+0xf2/0x590 [ 1094.150236] ahci_handle_port_intr+0x52/0xa0 [ 1094.150240] ahci_single_level_irq_intr+0x34/0x50 [ 1094.150243] __handle_irq_event_percpu+0x2f/0xc0 [ 1094.150248] handle_irq_event_percpu+0x2c/0x80 [ 1094.150255] handle_irq_event+0x23/0x43 [ 1094.150258] handle_edge_irq+0x78/0x190 [ 1094.150261] handle_irq+0x17/0x20 [ 1094.150264] do_IRQ+0x3e/0xd0 [ 1094.150269] common_interrupt+0xf/0xf [ 1094.150272] </IRQ> [ 1094.150275] RIP: 0033:0x58517a058690 [ 1094.150280] Code: 68 63 00 00 00 e9 b0 f9 ff ff ff 25 c2 dc 00 00 68 64 00 00 00 e9 a0 f9 ff ff ff 25 ba dc 00 00 68 65 00 00 00 e9 90 f9 ff ff <ff> 25 b2 dc 00 00 68 66 00 00 00 e9 80 f9 ff ff ff 25 aa dc 00 00 [ 1094.150283] RSP: 002b:00007ffe934c7338 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffde [ 1094.150291] RAX: 0000000000000001 RBX: 0000000000000012 RCX: 00007c32c39e5500 [ 1094.150295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 000058517a066466 [ 1094.150300] RBP: 000058517a066466 R08: 000058517a066467 R09: 0000000000000000 [ 1094.150303] R10: 00007ffe934c736c R11: 000058517a066466 R12: 00007c32c39e5500 [ 1094.150306] R13: 00007ffe934c7370 R14: 000058517a061071 R15: 0000000000000001 [ 1094.150309] NMI backtrace for cpu 7 [ 1094.150313] CPU: 7 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150319] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150322] RIP: 0010:format_decode+0x91/0x3f0 [ 1094.150329] Code: 01 00 00 80 4d 04 40 48 89 c8 48 8d 48 01 48 89 0c 24 0f b6 50 01 80 fa 2b 0f 84 af 01 00 00 7e d0 80 fa 2d 0f 84 e9 01 00 00 <80> fa 30 0f 85 a4 01 00 00 80 4d 04 10 eb ce 29 d8 48 83 c4 08 5b [ 1094.150331] RSP: 0018:ffffb2d88413fd40 EFLAGS: 00000016 [ 1094.150337] RAX: ffffffffa2e31775 RBX: ffffffffa2e31775 RCX: ffffffffa2e31776 [ 1094.150340] RDX: 000000000000006c RSI: ffffb2d88413fd70 RDI: ffffffffa2e31775 [ 1094.150343] RBP: ffffb2d88413fd70 R08: ffff99f5c714405f R09: 0000000000000004 [ 1094.150349] R10: 0000000000000004 R11: ffff99f5c714405f R12: ffffffffa2e31775 [ 1094.150355] R13: 0000000000002000 R14: ffffffffa2e31775 R15: ffff99f5c7144060 [ 1094.150358] FS: 000074e35af43740(0000) GS:ffff99f62d9c0000(0000) knlGS:0000000000000000 [ 1094.150365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.150371] CR2: 00007b610f6c0020 CR3: 000000077cedc006 CR4: 00000000003606e0 [ 1094.150376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.150383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.150386] Call Trace: [ 1094.150389] vsnprintf+0x6e/0x5b0 [ 1094.150391] scnprintf+0x4d/0x90 [ 1094.150394] msg_print_ext_header.constprop.0+0x7f/0xa0 [ 1094.150397] ? _cond_resched+0x10/0x20 [ 1094.150403] ? mutex_lock_interruptible+0x9/0x30 [ 1094.150405] devkmsg_read+0x17f/0x270 [ 1094.150408] vfs_read+0x98/0x150 [ 1094.150410] ksys_read+0x5d/0xe0 [ 1094.150413] do_syscall_64+0x50/0x170 [ 1094.150418] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1094.150420] RIP: 0033:0x74e35b05d415 [ 1094.150427] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1094.150432] RSP: 002b:00007ffda01e7bc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1094.150438] RAX: ffffffffffffffda RBX: 00005d76c75ae440 RCX: 000074e35b05d415 [ 1094.150441] RDX: 0000000000001fff RSI: 00005d76c75ae440 RDI: 0000000000000003 [ 1094.150446] RBP: 00005d76c75aa7fa R08: 000000000000000a R09: 0000000000000000 [ 1094.150454] R10: 00007ffda01e79fc R11: 0000000000000246 R12: 0000000000000029 [ 1094.150457] R13: 00005d76c75a9377 R14: 00005d76c75ae440 R15: 00005d76c75ae455 [ 1094.150462] NMI backtrace for cpu 9 [ 1094.150466] CPU: 9 PID: 1254 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150469] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150474] RIP: 0010:do_syscall_64+0x1c/0x170 [ 1094.150484] Code: da fa ff ff eb 94 0f 1f 84 00 00 00 00 00 55 53 48 89 fb 48 89 f5 0f 1f 44 00 00 0f 1f 44 00 00 fb 65 48 8b 04 25 00 5d 01 00 <48> 8b 00 a9 c1 01 08 10 0f 85 f4 00 00 00 48 81 fb b1 01 00 00 77 [ 1094.150492] RSP: 0018:ffffb2d883f1ff40 EFLAGS: 00000246 [ 1094.150500] RAX: ffff99f57cc38000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1094.150504] RDX: 0000000000000000 RSI: ffffb2d883f1ff58 RDI: 0000000000000000 [ 1094.150507] RBP: ffffb2d883f1ff58 R08: 0000000000000000 R09: 0000000000000000 [ 1094.150510] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1094.150513] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1094.150516] FS: 0000705c7f390740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000 [ 1094.150519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.150524] CR2: 00007d2b869ac010 CR3: 00000007c71a2004 CR4: 00000000003606e0 [ 1094.150532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.150535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.150537] Call Trace: [ 1094.150541] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1094.150544] RIP: 0033:0x705c7f6af415 [ 1094.150552] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1094.150557] RSP: 002b:00007ffc9e431508 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1094.150564] RAX: ffffffffffffffda RBX: 0000705c7f7837e0 RCX: 0000705c7f6af415 [ 1094.150570] RDX: 0000000000000001 RSI: 0000705c7f783863 RDI: 0000000000000000 [ 1094.150577] RBP: 00007ffc9e431550 R08: 0000000000000000 R09: 0000000000000000 [ 1094.150583] R10: 00005c354d1911f7 R11: 0000000000000246 R12: 0000705c7f785300 [ 1094.150586] R13: 0000705c7f784500 R14: 0000705c7f784700 R15: 0000705c7f785468 [ 1094.150589] NMI backtrace for cpu 10 [ 1094.150595] CPU: 10 PID: 25752 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150597] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150602] RIP: 0010:compact_zone+0x3a3/0xa70 [ 1094.150609] Code: 48 31 c2 48 f7 c2 00 fe ff ff 41 0f 94 47 7a 49 8b 6f 40 31 f6 44 8b 15 fb 6a ef 00 45 85 d2 40 0f 95 c6 c1 e6 03 89 74 24 50 <41> 83 7f 70 02 74 07 83 ce 04 89 74 24 50 45 0f b6 6f 74 48 8b 55 [ 1094.150614] RSP: 0018:ffffb2d88bba3a18 EFLAGS: 00000202 [ 1094.150620] RAX: 00000000004f3c60 RBX: 00000000004f4000 RCX: 000000000000279e [ 1094.150622] RDX: 00000000004f3c60 RSI: 0000000000000008 RDI: ffff99f64dfffb00 [ 1094.150625] RBP: ffff99f64dfded00 R08: 0000000000000001 R09: ffff99f62daa1450 [ 1094.150627] R10: 0000000000000001 R11: ffff99f62daa0a80 R12: 0000000000000009 [ 1094.150630] R13: ffff99f64dfded00 R14: 00000000004f3c60 R15: ffffb2d88bba3ad0 [ 1094.150642] FS: 0000720b85407740(0000) GS:ffff99f62da80000(0000) knlGS:0000000000000000 [ 1094.150644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.150647] CR2: 00007f962a63c020 CR3: 000000081439e005 CR4: 00000000003606e0 [ 1094.150649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.150653] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.150658] Call Trace: [ 1094.150662] compact_zone_order+0xde/0x120 [ 1094.150664] try_to_compact_pages+0x187/0x240 [ 1094.150666] __alloc_pages_direct_compact+0x87/0x170 [ 1094.150671] __alloc_pages_slowpath+0x454/0xc20 [ 1094.150680] ? release_pages+0x348/0x3b0 [ 1094.150684] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1094.150686] __alloc_pages_nodemask+0x268/0x2b0 [ 1094.150688] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1094.150692] __handle_mm_fault+0xc11/0x1890 [ 1094.150696] handle_mm_fault+0xa9/0x1d0 [ 1094.150698] __do_page_fault+0x237/0x480 [ 1094.150701] do_page_fault+0x1d/0x67 [ 1094.150702] ? page_fault+0x8/0x30 [ 1094.150704] page_fault+0x1e/0x30 [ 1094.150706] RIP: 0033:0x5c2e1a191c10 [ 1094.150717] Code: Bad RIP value. [ 1094.150720] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1094.150724] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1094.150728] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1094.150730] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1094.150733] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1094.150738] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1094.150746] NMI backtrace for cpu 11 [ 1094.150752] CPU: 11 PID: 25777 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1094.150758] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1094.150763] RIP: 0010:compact_zone+0x5db/0xa70 [ 1094.150770] Code: 8b 47 60 85 c0 7e 23 48 8b 4c 24 08 48 85 c9 74 19 48 c7 c2 ff ff ff ff c4 e2 f9 f7 c2 49 23 47 30 48 39 c1 0f 82 63 03 00 00 <48> 8b 44 24 28 48 85 c0 0f 84 14 fb ff ff 48 83 78 08 00 0f 84 09 [ 1094.150777] RSP: 0018:ffffb2d88bc6ba18 EFLAGS: 00000246 [ 1094.150786] RAX: 0000000000000009 RBX: 0000000000718800 RCX: 0000000000000000 [ 1094.150789] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 1094.150792] RBP: ffffb2d88bc6bad0 R08: 0000000000000001 R09: ffff99f62dae1450 [ 1094.150800] R10: 0000000000000000 R11: ffff99f62dae0a80 R12: 0000000000718440 [ 1094.150805] R13: ffff99f64dfded00 R14: 0000000000718440 R15: ffffb2d88bc6bad0 [ 1094.150809] FS: 0000720b85407740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000 [ 1094.150814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.150818] CR2: 0000700c4abe4000 CR3: 00000007ccf6a003 CR4: 00000000003606e0 [ 1094.150823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1094.150830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1094.150834] Call Trace: [ 1094.150837] compact_zone_order+0xde/0x120 [ 1094.150847] try_to_compact_pages+0x187/0x240 [ 1094.150853] __alloc_pages_direct_compact+0x87/0x170 [ 1094.150856] __alloc_pages_slowpath+0x454/0xc20 [ 1094.150862] ? release_pages+0x348/0x3b0 [ 1094.150869] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1094.150874] __alloc_pages_nodemask+0x268/0x2b0 [ 1094.150879] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1094.150882] __handle_mm_fault+0xc11/0x1890 [ 1094.150885] handle_mm_fault+0xa9/0x1d0 [ 1094.150892] __do_page_fault+0x237/0x480 [ 1094.150900] do_page_fault+0x1d/0x67 [ 1094.150904] ? page_fault+0x8/0x30 [ 1094.150909] page_fault+0x1e/0x30 [ 1094.150912] RIP: 0033:0x5c2e1a191c10 [ 1094.150915] Code: Bad RIP value. [ 1094.150918] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1094.150933] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1094.150938] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1094.150941] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1094.150947] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1094.150952] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1096.504728] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started [ 1096.645895] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated ``` ``` [ 1319.077352] sysrq: Show backtrace of all active CPUs [ 1319.077365] NMI backtrace for cpu 7 [ 1319.077376] CPU: 7 PID: 14722 Comm: bash Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.077381] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.077386] Call Trace: [ 1319.077415] dump_stack+0x46/0x60 [ 1319.077426] nmi_cpu_backtrace.cold+0x14/0x53 [ 1319.077440] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 1319.077447] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 1319.077458] __handle_sysrq.cold+0x48/0x102 [ 1319.077467] write_sysrq_trigger+0x1f/0x30 [ 1319.077481] proc_reg_write+0x37/0x60 [ 1319.077497] vfs_write+0xb1/0x190 [ 1319.077506] ksys_write+0x5d/0xe0 [ 1319.077516] do_syscall_64+0x50/0x170 [ 1319.077529] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1319.077537] RIP: 0033:0x79787f47a4ba [ 1319.077546] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 [ 1319.077553] RSP: 002b:00007ffc61be1698 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1319.077561] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000079787f47a4ba [ 1319.077567] RDX: 0000000000000002 RSI: 000055f82dde0ee0 RDI: 0000000000000001 [ 1319.077572] RBP: 00007ffc61be16c0 R08: 000000000000000a R09: 0000000000000001 [ 1319.077576] R10: 000055f82d3c305e R11: 0000000000000246 R12: 0000000000000002 [ 1319.077582] R13: 000055f82dde0ee0 R14: 000079787f54f500 R15: 000079787f550300 [ 1319.077680] Sending NMI from CPU 7 to CPUs 0-6,8-11: [ 1319.077882] NMI backtrace for cpu 0 [ 1319.077890] CPU: 0 PID: 25727 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.077892] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.077894] RIP: 0010:get_pfnblock_flags_mask+0x1e/0x70 [ 1319.077897] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 90 49 89 f0 49 c1 e8 0f 48 8b 3d ea 5b 5f 01 48 85 ff 74 1d 48 89 f0 48 c1 e8 17 48 8b 3c c7 <45> 0f b6 c0 49 c1 e0 04 49 01 f8 48 85 ff 49 0f 45 f8 48 c1 ee 09 [ 1319.077899] RSP: 0018:ffffb2d88badba10 EFLAGS: 00000247 [ 1319.077910] RAX: 0000000000000000 RBX: 0000000000590000 RCX: 0000000000000007 [ 1319.077912] RDX: 0000000000000002 RSI: 000000000058fc00 RDI: ffff99f64dfd9000 [ 1319.077914] RBP: ffffb2d88badbad0 R08: 00000000000000b1 R09: ffff99f62d821450 [ 1319.077917] R10: 0000000000000001 R11: ffff99f57cfb5c00 R12: 000000000058fe00 [ 1319.077920] R13: ffff99f64dfded00 R14: ffffde0b963f0000 R15: 0000000000000000 [ 1319.077924] FS: 0000720b85407740(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000 [ 1319.077926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.077931] CR2: 0000631d9a780ac0 CR3: 00000007c23a4006 CR4: 00000000003606f0 [ 1319.077933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.077935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.077939] Call Trace: [ 1319.077945] compact_zone+0x4d9/0xa70 [ 1319.077948] compact_zone_order+0xde/0x120 [ 1319.077950] try_to_compact_pages+0x187/0x240 [ 1319.077952] __alloc_pages_direct_compact+0x87/0x170 [ 1319.077953] __alloc_pages_slowpath+0x20e/0xc20 [ 1319.077957] ? release_pages+0x348/0x3b0 [ 1319.077960] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1319.077965] __alloc_pages_nodemask+0x268/0x2b0 [ 1319.077967] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1319.077971] __handle_mm_fault+0xc11/0x1890 [ 1319.077977] handle_mm_fault+0xa9/0x1d0 [ 1319.077978] __do_page_fault+0x237/0x480 [ 1319.077980] do_page_fault+0x1d/0x67 [ 1319.077981] ? page_fault+0x8/0x30 [ 1319.077983] page_fault+0x1e/0x30 [ 1319.077985] RIP: 0033:0x5c2e1a191c10 [ 1319.077988] Code: Bad RIP value. [ 1319.077991] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1319.077999] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1319.078000] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1319.078003] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1319.078005] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1319.078008] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1319.078016] NMI backtrace for cpu 1 [ 1319.078023] CPU: 1 PID: 25752 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078033] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078039] RIP: 0010:node_page_state+0x10/0x20 [ 1319.078044] Code: 49 39 ee 75 b3 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 80 00 00 00 00 89 f6 48 8d 84 f7 c8 1b 02 00 48 8b 00 48 85 c0 <ba> 00 00 00 00 48 0f 48 c2 c3 66 0f 1f 44 00 00 48 83 ec 20 41 89 [ 1319.078047] RSP: 0018:ffffb2d88bba3950 EFLAGS: 00000246 [ 1319.078056] RAX: 0000000000000000 RBX: 00000000004f3c60 RCX: 000000000000000c [ 1319.078059] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfde000 [ 1319.078061] RBP: ffff99f64dfde000 R08: 0000000000000001 R09: ffff99f62d861450 [ 1319.078064] R10: 0000000000000001 R11: ffff99f57cfb6200 R12: 0000000000000000 [ 1319.078068] R13: 0000000000048a59 R14: ffffb2d88bba3ad0 R15: 0000000000007c63 [ 1319.078071] FS: 0000720b85407740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000 [ 1319.078076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.078079] CR2: 00007210ba498ff8 CR3: 000000081439e006 CR4: 00000000003606e0 [ 1319.078084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.078087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.078090] Call Trace: [ 1319.078092] isolate_migratepages_block+0xba/0xa50 [ 1319.078098] compact_zone+0x577/0xa70 [ 1319.078101] compact_zone_order+0xde/0x120 [ 1319.078104] try_to_compact_pages+0x187/0x240 [ 1319.078108] __alloc_pages_direct_compact+0x87/0x170 [ 1319.078111] __alloc_pages_slowpath+0x454/0xc20 [ 1319.078114] ? release_pages+0x348/0x3b0 [ 1319.078120] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1319.078123] __alloc_pages_nodemask+0x268/0x2b0 [ 1319.078126] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1319.078131] __handle_mm_fault+0xc11/0x1890 [ 1319.078133] handle_mm_fault+0xa9/0x1d0 [ 1319.078136] __do_page_fault+0x237/0x480 [ 1319.078139] do_page_fault+0x1d/0x67 [ 1319.078142] ? page_fault+0x8/0x30 [ 1319.078148] page_fault+0x1e/0x30 [ 1319.078151] RIP: 0033:0x5c2e1a191c10 [ 1319.078153] Code: Bad RIP value. [ 1319.078159] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1319.078172] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1319.078176] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1319.078181] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1319.078190] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1319.078199] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1319.078206] NMI backtrace for cpu 2 [ 1319.078212] CPU: 2 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078217] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078221] RIP: 0010:new_sync_write+0x57/0x1b0 [ 1319.078229] Code: f6 c6 04 0f 95 c0 01 c0 89 c1 83 c9 04 f6 c6 40 0f 45 c1 f6 c6 10 75 18 48 8b 8f f0 00 00 00 48 8b 09 48 8b 71 28 f6 46 50 10 <0f> 84 f9 00 00 00 83 c8 10 89 c1 83 c9 20 81 e2 00 00 10 00 0f 45 [ 1319.078233] RSP: 0018:ffffb2d884117e40 EFLAGS: 00000246 [ 1319.078246] RAX: 0000000000000000 RBX: ffff99f5c67e9e00 RCX: ffff99f5c7d50de0 [ 1319.078250] RDX: 0000000000000001 RSI: ffff99f629132800 RDI: ffff99f5c67e9e00 [ 1319.078253] RBP: ffffb2d884117f08 R08: 0000000000000020 R09: 0000000000000000 [ 1319.078256] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020 [ 1319.078260] R13: ffffb2d884117f08 R14: 000058517b6f83c0 R15: 0000000000000000 [ 1319.078266] FS: 00007c32c37f6740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000 [ 1319.078271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.078283] CR2: 0000631d9a77fcc0 CR3: 000000077cd3c002 CR4: 00000000003606e0 [ 1319.078288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.078292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.078296] Call Trace: [ 1319.078298] vfs_write+0xb1/0x190 [ 1319.078302] ksys_write+0x5d/0xe0 [ 1319.078305] do_syscall_64+0x50/0x170 [ 1319.078309] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1319.078313] RIP: 0033:0x7c32c39104ba [ 1319.078319] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 [ 1319.078324] RSP: 002b:00007ffe934c7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1319.078332] RAX: ffffffffffffffda RBX: 0000000000000020 RCX: 00007c32c39104ba [ 1319.078336] RDX: 0000000000000020 RSI: 000058517b6f83c0 RDI: 0000000000000001 [ 1319.078340] RBP: 00007ffe934c7340 R08: 000000000000000a R09: 0000000000000000 [ 1319.078344] R10: 00007ffe934c736c R11: 0000000000000246 R12: 0000000000000020 [ 1319.078348] R13: 000058517b6f83c0 R14: 00007c32c39e5500 R15: 00007c32c39e6300 [ 1319.078355] NMI backtrace for cpu 4 [ 1319.078363] CPU: 4 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078369] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078372] RIP: 0033:0x70f212b62cff [ 1319.078377] Code: 64 8b 04 25 18 00 00 00 85 c0 0f 85 0b 01 00 00 ba 01 00 00 00 0f b1 17 48 8b bb 88 00 00 00 48 8b 45 c8 48 89 47 08 ff 47 04 <8b> 83 c0 00 00 00 85 c0 0f 85 a3 00 00 00 c7 83 c0 00 00 00 ff ff [ 1319.078379] RSP: 002b:00007fffa4d872b0 EFLAGS: 00000202 [ 1319.078385] RAX: 000070f212ad7740 RBX: 000070f212cc6500 RCX: 000070f212cc6500 [ 1319.078388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 000070f212cc8320 [ 1319.078396] RBP: 00007fffa4d872f0 R08: 00005acd2cb39459 R09: 0000000000000000 [ 1319.078402] R10: 00007fffa4d8732c R11: 00005acd2cb39458 R12: 0000000000000001 [ 1319.078405] R13: 0000000000000001 R14: 0000000000000001 R15: 00005acd2cb39458 [ 1319.078410] FS: 000070f212ad7740 GS: 0000000000000000 [ 1319.078416] NMI backtrace for cpu 3 [ 1319.078421] CPU: 3 PID: 25777 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078425] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078428] RIP: 0010:compact_unlock_should_abort.isra.0+0x3b/0x50 [ 1319.078440] Code: 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 74 e0 <c6> 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00 00 41 [ 1319.078442] RSP: 0018:ffffb2d88bc6b940 EFLAGS: 00000202 [ 1319.078448] RAX: ffff99f370429e40 RBX: 0000000000718440 RCX: ffffb2d88bc6bb49 [ 1319.078453] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 1319.078455] RBP: ffffb2d88bc6bb49 R08: 0000000000000001 R09: 0000000000000000 [ 1319.078457] R10: 0000000000000000 R11: ffff99f57cfb4e00 R12: ffffb2d88bc6bad0 [ 1319.078460] R13: 0000000000000000 R14: ffffb2d88bc6bad0 R15: 0000000000000000 [ 1319.078465] FS: 0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000 [ 1319.078471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.078473] CR2: 000076e9fabf2020 CR3: 00000007ccf6a001 CR4: 00000000003606e0 [ 1319.078476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.078479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.078486] Call Trace: [ 1319.078490] isolate_migratepages_block+0x2ed/0xa50 [ 1319.078494] compact_zone+0x577/0xa70 [ 1319.078628] compact_zone_order+0xde/0x120 [ 1319.078637] try_to_compact_pages+0x187/0x240 [ 1319.078641] __alloc_pages_direct_compact+0x87/0x170 [ 1319.078644] __alloc_pages_slowpath+0x454/0xc20 [ 1319.078649] ? release_pages+0x348/0x3b0 [ 1319.078653] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1319.078655] __alloc_pages_nodemask+0x268/0x2b0 [ 1319.078659] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1319.078665] __handle_mm_fault+0xc11/0x1890 [ 1319.078670] handle_mm_fault+0xa9/0x1d0 [ 1319.078673] __do_page_fault+0x237/0x480 [ 1319.078678] do_page_fault+0x1d/0x67 [ 1319.078680] ? page_fault+0x8/0x30 [ 1319.078683] page_fault+0x1e/0x30 [ 1319.078688] RIP: 0033:0x5c2e1a191c10 [ 1319.078697] Code: Bad RIP value. [ 1319.078700] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1319.078707] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1319.078714] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1319.078720] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1319.078727] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1319.078729] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1319.078735] NMI backtrace for cpu 5 [ 1319.078740] CPU: 5 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078743] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078747] RIP: 0033:0x7c7ea55225a9 [ 1319.078760] Code: 74 6f 48 89 4d c8 48 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24 28 4d 01 f5 49 89 df 4d 29 f7 48 8b 4d c8 <4c> 01 f9 75 45 4c 29 fb 49 89 de 4c 89 f0 48 83 c4 18 5b 41 5c 41 [ 1319.078766] RSP: 002b:00007ffe25d183c0 EFLAGS: 00000246 [ 1319.078772] RAX: 0000571ede4bd476 RBX: 0000000000000001 RCX: 0000000000000000 [ 1319.078775] RDX: 0000000000000001 RSI: 0000571ede0f050a RDI: 0000571ede4bd475 [ 1319.078784] RBP: 00007ffe25d18400 R08: 0000571ede0f050b R09: 0000000000000000 [ 1319.078792] R10: 00007ffe25d1848c R11: 0000571ede0f050a R12: 00007c7ea5679500 [ 1319.078796] R13: 0000571ede0f050b R14: 0000000000000001 R15: 0000000000000000 [ 1319.078801] FS: 00007c7ea548a740 GS: 0000000000000000 [ 1319.078806] NMI backtrace for cpu 6 [ 1319.078809] CPU: 6 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078812] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078821] RIP: 0033:0x77d47d3fd5a2 [ 1319.078826] Code: 89 c2 31 c9 48 85 d2 74 6f 48 89 4d c8 48 39 d3 48 0f 46 d3 49 89 d6 4c 89 ee e8 e9 00 f9 ff 49 89 44 24 28 4d 01 f5 49 89 df <4d> 29 f7 48 8b 4d c8 4c 01 f9 75 45 4c 29 fb 49 89 de 4c 89 f0 48 [ 1319.078829] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000206 [ 1319.078834] RAX: 00005ffb591e43c1 RBX: 0000000000000001 RCX: 0000000000000052 [ 1319.078841] RDX: 0000000000000001 RSI: 00005ffb57470455 RDI: 00005ffb591e43c0 [ 1319.078847] RBP: 00007ffe06db5e30 R08: 00005ffb57470456 R09: 0000000000000000 [ 1319.078852] R10: 00007ffe06db5ebc R11: 00005ffb57470455 R12: 000077d47d554500 [ 1319.078856] R13: 00005ffb57470456 R14: 0000000000000001 R15: 0000000000000001 [ 1319.078860] FS: 000077d47d365740 GS: 0000000000000000 [ 1319.078867] NMI backtrace for cpu 8 [ 1319.078876] CPU: 8 PID: 25800 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.078880] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.078882] RIP: 0010:compact_zone+0x587/0xa70 [ 1319.078887] Code: ff 4c 8b 34 24 49 89 ef 8b 4c 24 50 4c 89 e2 4c 89 f6 48 89 ef e8 c9 e8 ff ff 49 89 c4 48 85 c0 0f 84 c3 02 00 00 4d 89 67 30 <41> 8b 77 24 85 f6 0f 85 9f 00 00 00 80 7c 24 56 00 0f 84 c3 02 00 [ 1319.078896] RSP: 0018:ffffb2d88bd23a18 EFLAGS: 00000202 [ 1319.078905] RAX: 000000000016b420 RBX: 000000000016b800 RCX: 0000000000000000 [ 1319.078908] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 1319.078915] RBP: ffffb2d88bd23ad0 R08: 0000000000000001 R09: ffff99f62da21450 [ 1319.078923] R10: 0000000000000000 R11: ffff99f62da20a80 R12: 000000000016b420 [ 1319.078930] R13: ffff99f64dfded00 R14: 000000000016b420 R15: ffffb2d88bd23ad0 [ 1319.078936] FS: 0000720b85407740(0000) GS:ffff99f62da00000(0000) knlGS:0000000000000000 [ 1319.078940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.078944] CR2: 00007ecf94de9578 CR3: 00000007ec28a006 CR4: 00000000003606e0 [ 1319.078952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.078961] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.078964] Call Trace: [ 1319.078969] compact_zone_order+0xde/0x120 [ 1319.078974] try_to_compact_pages+0x187/0x240 [ 1319.078978] __alloc_pages_direct_compact+0x87/0x170 [ 1319.078985] __alloc_pages_slowpath+0x454/0xc20 [ 1319.078993] ? release_pages+0x348/0x3b0 [ 1319.078997] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1319.079003] __alloc_pages_nodemask+0x268/0x2b0 [ 1319.079007] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1319.079012] __handle_mm_fault+0xc11/0x1890 [ 1319.079017] handle_mm_fault+0xa9/0x1d0 [ 1319.079030] __do_page_fault+0x237/0x480 [ 1319.079034] do_page_fault+0x1d/0x67 [ 1319.079041] ? page_fault+0x8/0x30 [ 1319.079047] page_fault+0x1e/0x30 [ 1319.079052] RIP: 0033:0x5c2e1a191c10 [ 1319.079055] Code: Bad RIP value. [ 1319.079060] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 1319.079074] RAX: 00000000183a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 1319.079079] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 1319.079084] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 1319.079088] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 1319.079093] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 1319.079103] NMI backtrace for cpu 9 [ 1319.079109] CPU: 9 PID: 1238 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.079113] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.079121] RIP: 0010:touch_atime+0x26/0xe0 [ 1319.079133] Code: 1f 44 00 00 55 48 89 e5 41 54 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 4c 8b 27 48 8b 47 08 48 8b 58 30 <48> 89 de e8 02 ff ff ff 84 c0 75 1d 48 8b 44 24 10 65 48 33 04 25 [ 1319.079141] RSP: 0018:ffffb2d884137db0 EFLAGS: 00000246 [ 1319.079151] RAX: ffff99f594bc4840 RBX: ffff99f5c7ec6750 RCX: 0000000000000000 [ 1319.079158] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff99f5c8398810 [ 1319.079164] RBP: ffffb2d884137dd8 R08: 0000000000000001 R09: 0000000000000000 [ 1319.079170] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f629139220 [ 1319.079186] R13: ffff99f5c92c4d68 R14: 0000000000000001 R15: ffff99f5c75b0180 [ 1319.079197] FS: 00007d2186c2e740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000 [ 1319.079202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.079209] CR2: 00005892d7c57e02 CR3: 0000000788c3c002 CR4: 00000000003606e0 [ 1319.079224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.079234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.079241] Call Trace: [ 1319.079250] pipe_read+0x2aa/0x2c0 [ 1319.079258] new_sync_read+0x10f/0x1a0 [ 1319.079264] vfs_read+0x98/0x150 [ 1319.079270] ksys_read+0x5d/0xe0 [ 1319.079277] do_syscall_64+0x50/0x170 [ 1319.079293] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1319.079302] RIP: 0033:0x7d2186f4d415 [ 1319.079308] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1319.079317] RSP: 002b:00007ffd33329a38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1319.079337] RAX: ffffffffffffffda RBX: 00007d21870217e0 RCX: 00007d2186f4d415 [ 1319.079346] RDX: 0000000000000001 RSI: 00007d2187021863 RDI: 0000000000000000 [ 1319.079351] RBP: 00007ffd33329a80 R08: 0000000000000000 R09: 0000000000000000 [ 1319.079360] R10: 0000647dee70b1f7 R11: 0000000000000246 R12: 00007d2187023300 [ 1319.079367] R13: 00007d2187022500 R14: 00007d2187022700 R15: 00007d2187023468 [ 1319.079372] NMI backtrace for cpu 10 [ 1319.079384] CPU: 10 PID: 1053 Comm: xfwm4 Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.079392] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.079395] RIP: 0033:0x624fe3bc1622 [ 1319.079403] Code: 8b 85 24 04 00 00 ff c0 41 39 45 00 0f 85 1c fc ff ff 49 8b 75 20 48 89 ef e8 2a dc ff ff 48 89 c7 48 85 c0 0f 84 fe fb ff ff <49> 8b 75 28 e8 d5 5d 02 00 e9 f0 fb ff ff 48 8b 73 08 ba 06 00 00 [ 1319.079412] RSP: 002b:00007ffdd444b6f0 EFLAGS: 00000206 [ 1319.079428] RAX: 0000624fe5f3ce00 RBX: 0000624fe5a93830 RCX: 0000000000000001 [ 1319.079433] RDX: 0000000000000000 RSI: 00000000008004bd RDI: 0000624fe5f3ce00 [ 1319.079441] RBP: 0000624fe597a100 R08: 0000624fe5f3ce00 R09: 0000000000000020 [ 1319.079449] R10: 00007388623d19e0 R11: 0000000000000058 R12: 0000000000000000 [ 1319.079454] R13: 00007ffdd444b810 R14: 00007ffdd444b810 R15: 0000624fe5ea6da0 [ 1319.079465] FS: 0000738860b0c980 GS: 0000000000000000 [ 1319.079473] NMI backtrace for cpu 11 [ 1319.079484] CPU: 11 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 1319.079491] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1319.079495] RIP: 0010:queued_spin_lock_slowpath+0x110/0x1a0 [ 1319.079505] Code: 85 c0 74 f2 eb f6 41 8d 40 01 c1 e0 12 c1 e1 10 09 c1 89 c8 c1 e8 10 66 87 47 02 89 c6 c1 e6 10 75 3c 31 f6 eb 02 f3 90 8b 07 <66> 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 74 63 c6 07 01 48 85 [ 1319.079508] RSP: 0018:ffffb2d88413fe50 EFLAGS: 00000002 [ 1319.079517] RAX: 0000000000100000 RBX: ffff99f5c7144000 RCX: 0000000000300000 [ 1319.079520] RDX: ffff99f62dae16c0 RSI: 0000000000000000 RDI: ffffffffa355d2ac [ 1319.079527] RBP: ffff99f57cd9b400 R08: 000000000000000b R09: 0000000000000000 [ 1319.079531] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001fff [ 1319.079535] R13: 00005d76c75ae440 R14: ffff99f5c7144038 R15: ffff99f57cd9b400 [ 1319.079545] FS: 000074e35af43740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000 [ 1319.079552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1319.079563] CR2: 00007ecf94e85ff8 CR3: 000000077cedc004 CR4: 00000000003606e0 [ 1319.079568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1319.079573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1319.079580] Call Trace: [ 1319.079589] devkmsg_read+0x7f/0x270 [ 1319.079594] vfs_read+0x98/0x150 [ 1319.079597] ksys_read+0x5d/0xe0 [ 1319.079610] do_syscall_64+0x50/0x170 [ 1319.079616] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1319.079622] RIP: 0033:0x74e35b05d415 [ 1319.079630] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1319.079637] RSP: 002b:00007ffda01e7bc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1319.079648] RAX: ffffffffffffffda RBX: 00005d76c75ae440 RCX: 000074e35b05d415 [ 1319.079656] RDX: 0000000000001fff RSI: 00005d76c75ae440 RDI: 0000000000000003 [ 1319.079661] RBP: 00005d76c75aa7fa R08: 000000000000000a R09: 0000000000000000 [ 1319.079664] R10: 00007ffda01e79fc R11: 0000000000000246 R12: 0000000000000041 [ 1319.079676] R13: 00005d76c75a9377 R14: 00005d76c75ae440 R15: 00005d76c75ae455 [ 1336.744759] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started [ 1336.846541] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated [ 1343.485167] dbus-daemon[653]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.111' (uid=1000 pid=523 comm="mousepad /tmp/dmesg.log ") [ 1343.505410] systemd[1]: Starting Hostname Service... [ 1343.507309] device: 'lo': device_add [ 1343.507448] PM: Adding info for No Bus:lo [ 1344.616004] systemd[1]: Started Hostname Service. ``` some of them finished/disappeared , only 3 are still running now: ``` PID %CPU COMMAND PR NI VIRT RES S USER 25777 100.0 stress 20 0 980352 124868 R user 25727 100.0 stress 20 0 980352 96196 R user 25752 100.0 stress 20 0 980352 122820 R user ``` ``` [ 4245.323214] snd_hda_intel 0000:00:1f.3: PME# enabled [ 4245.334733] snd_hda_intel 0000:00:1f.3: power state changed by ACPI to D3hot [ 4279.678617] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started [ 4279.765483] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated [ 4324.861160] i2c i2c-2: NAK from device addr 0x50 msg #0 [ 4324.864949] i2c i2c-1: NAK from device addr 0x50 msg #0 [ 4324.868589] i2c i2c-3: NAK from device addr 0x50 msg #0 [ 4339.738573] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started [ 4339.811766] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated [ 4399.798582] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started [ 4399.886406] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated [ 4459.859214] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 started [ 4459.952334] gpg-agent[1051]: handler 0x7b1efceeb700 for fd 10 terminated [ 4519.918645] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 started [ 4520.000915] gpg-agent[1051]: handler 0x7b1ef7fff700 for fd 10 terminated [ 4537.428195] sysrq: Show backtrace of all active CPUs [ 4537.428204] NMI backtrace for cpu 8 [ 4537.428214] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.428219] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.428223] Call Trace: [ 4537.428230] <IRQ> [ 4537.428243] dump_stack+0x46/0x60 [ 4537.428251] nmi_cpu_backtrace.cold+0x14/0x53 [ 4537.428260] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 4537.428267] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 4537.428275] __handle_sysrq.cold+0x48/0x102 [ 4537.428283] sysrq_filter+0x2ea/0x3b0 [ 4537.428291] input_to_handler+0x4d/0xf0 [ 4537.428300] input_pass_values.part.0+0x109/0x130 [ 4537.428308] input_handle_event+0x171/0x5a0 [ 4537.428316] input_event+0x4d/0x70 [ 4537.428325] hidinput_report_event+0x2e/0x40 [ 4537.428333] hid_report_raw_event+0x260/0x430 [ 4537.428341] hid_input_report+0xfb/0x150 [ 4537.428350] hid_irq_in+0x168/0x190 [ 4537.428360] __usb_hcd_giveback_urb+0x77/0xe0 [ 4537.428383] xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd] [ 4537.428406] xhci_td_cleanup+0xf7/0x140 [xhci_hcd] [ 4537.428428] xhci_irq+0x7e8/0x1be0 [xhci_hcd] [ 4537.428438] __handle_irq_event_percpu+0x2f/0xc0 [ 4537.428445] handle_irq_event_percpu+0x2c/0x80 [ 4537.428453] handle_irq_event+0x23/0x43 [ 4537.428462] handle_edge_irq+0x78/0x190 [ 4537.428470] handle_irq+0x17/0x20 [ 4537.428476] do_IRQ+0x3e/0xd0 [ 4537.428485] common_interrupt+0xf/0xf [ 4537.428490] </IRQ> [ 4537.428500] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0 [ 4537.428508] Code: e8 d6 f9 a1 ff 49 89 c6 31 ff e8 fc 09 a2 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 f5 f3 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c [ 4537.428514] RSP: 0018:ffffb2d883217e80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde [ 4537.428521] RAX: ffff99f62da20a00 RBX: ffffffffa3061940 RCX: 000000000000001f [ 4537.428526] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000 [ 4537.428531] RBP: 0000042073c1a3e8 R08: 0000042073cb420a R09: 0000000000000f70 [ 4537.428535] R10: ffff99f62da1fb44 R11: ffff99f62da1fb24 R12: ffff99f62da29a00 [ 4537.428540] R13: 0000000000000004 R14: 0000042073cb420a R15: 0000000000000000 [ 4537.428551] cpuidle_enter+0x24/0x40 [ 4537.428559] do_idle+0x1c1/0x230 [ 4537.428567] cpu_startup_entry+0x14/0x20 [ 4537.428575] start_secondary+0x168/0x1b0 [ 4537.428585] secondary_startup_64+0xa4/0xb0 [ 4537.428594] Sending NMI from CPU 8 to CPUs 0-7,9-11: [ 4537.428681] NMI backtrace for cpu 0 [ 4537.428683] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.428685] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.428686] RIP: 0010:menu_select+0x283/0x600 [ 4537.428688] Code: 00 85 c9 0f 8e 26 01 00 00 48 8d 45 4c 48 8b 3c 24 48 8d 57 18 41 bf ff ff ff ff 45 31 f6 80 78 0c 00 75 25 48 83 3a 00 75 1f <41> 83 ff ff 45 0f 44 fe 8b 70 08 44 39 ee 0f 87 70 02 00 00 39 18 [ 4537.428689] RSP: 0018:ffffffffa3003e60 EFLAGS: 00000046 [ 4537.428691] RAX: ffffffffa3061aac RBX: 0000000000000940 RCX: 0000000000000009 [ 4537.428692] RDX: ffff99f62d829ac0 RSI: 0000000000000014 RDI: ffff99f62d829a00 [ 4537.428693] RBP: ffffffffa3061940 R08: 0000000000001b9a R09: 0000000000001f67 [ 4537.428694] R10: ffff99f62d81fb44 R11: ffff99f62d81fb24 R12: ffff99f62d81fae0 [ 4537.428695] R13: 0000000000000940 R14: 0000000000000003 R15: 0000000000000002 [ 4537.428696] FS: 0000000000000000(0000) GS:ffff99f62d800000(0000) knlGS:0000000000000000 [ 4537.428697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4537.428698] CR2: 0000787391901145 CR3: 000000080600a005 CR4: 00000000003606f0 [ 4537.428699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4537.428700] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4537.428700] Call Trace: [ 4537.428701] do_idle+0x19d/0x230 [ 4537.428702] cpu_startup_entry+0x14/0x20 [ 4537.428702] start_kernel+0x453/0x470 [ 4537.428703] secondary_startup_64+0xa4/0xb0 [ 4537.428761] NMI backtrace for cpu 1 skipped: idling at intel_idle+0x7d/0x120 [ 4537.428927] NMI backtrace for cpu 2 [ 4537.428930] CPU: 2 PID: 25777 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.428936] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.428940] RIP: 0010:compact_zone+0x411/0xa70 [ 4537.428942] Code: 48 0f 42 fa 45 31 ed 49 8d 9c 24 00 02 00 00 48 81 e3 00 fe ff ff 49 3b 5f 28 0f 87 7c 01 00 00 48 81 c3 00 02 00 00 49 89 fe <4c> 89 f8 45 89 ef 49 89 ed 48 89 c5 4c 89 e0 4c 89 24 24 4c 8d a3 [ 4537.428946] RSP: 0018:ffffb2d88bc6ba18 EFLAGS: 00000206 [ 4537.428954] RAX: 0000000000718440 RBX: 0000000000718800 RCX: 00000000000038c2 [ 4537.428957] RDX: 0000000000100000 RSI: 000000000000000c RDI: 0000000000718400 [ 4537.428959] RBP: ffff99f64dfded00 R08: 0000000000000001 R09: ffff99f62d8a1450 [ 4537.428962] R10: 0000000000000001 R11: ffff99f62d8a0a80 R12: 0000000000718440 [ 4537.428966] R13: 0000000000000000 R14: 0000000000718400 R15: ffffb2d88bc6bad0 [ 4537.428969] FS: 0000720b85407740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000 [ 4537.428973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4537.428974] CR2: 000057450363c268 CR3: 00000007ccf6a003 CR4: 00000000003606e0 [ 4537.428976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4537.428977] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4537.428980] Call Trace: [ 4537.428984] compact_zone_order+0xde/0x120 [ 4537.428990] try_to_compact_pages+0x187/0x240 [ 4537.428991] __alloc_pages_direct_compact+0x87/0x170 [ 4537.428992] __alloc_pages_slowpath+0x454/0xc20 [ 4537.428993] ? release_pages+0x348/0x3b0 [ 4537.428995] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 4537.428998] __alloc_pages_nodemask+0x268/0x2b0 [ 4537.429001] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 4537.429005] __handle_mm_fault+0xc11/0x1890 [ 4537.429008] handle_mm_fault+0xa9/0x1d0 [ 4537.429009] __do_page_fault+0x237/0x480 [ 4537.429013] do_page_fault+0x1d/0x67 [ 4537.429014] ? page_fault+0x8/0x30 [ 4537.429017] page_fault+0x1e/0x30 [ 4537.429018] RIP: 0033:0x5c2e1a191c10 [ 4537.429019] Code: Bad RIP value. [ 4537.429023] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 4537.429029] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 4537.429033] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 4537.429037] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 4537.429038] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 4537.429039] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 4537.429048] NMI backtrace for cpu 3 [ 4537.429056] CPU: 3 PID: 1252 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429060] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429064] RIP: 0010:mutex_lock+0x1a/0x30 [ 4537.429073] Code: be 02 00 00 00 e9 46 fb ff ff 66 0f 1f 44 00 00 55 48 89 fd e8 97 df ff ff 65 48 8b 14 25 00 5d 01 00 31 c0 f0 48 0f b1 55 00 <74> 06 48 89 ef 5d eb ce 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 [ 4537.429078] RSP: 0018:ffffb2d883effdd8 EFLAGS: 00000246 [ 4537.429091] RAX: 0000000000000000 RBX: ffff99f588eb2d00 RCX: 0000000000000001 [ 4537.429096] RDX: ffff99f57cc3dac0 RSI: ffffb2d883effe50 RDI: ffff99f5c8094c00 [ 4537.429103] RBP: ffff99f5c8094c00 R08: 0000000000000001 R09: 0000000000000000 [ 4537.429110] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4537.429115] R13: ffffb2d883efff08 R14: 00007c3fc3975863 R15: ffff99f5c8094c00 [ 4537.429121] FS: 00007c3fc3582740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000 [ 4537.429127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4537.429130] CR2: 000078739192f578 CR3: 000000077ce7c006 CR4: 00000000003606e0 [ 4537.429135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4537.429141] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4537.429147] Call Trace: [ 4537.429152] pipe_read+0x37/0x2c0 [ 4537.429157] new_sync_read+0x10f/0x1a0 [ 4537.429164] vfs_read+0x98/0x150 [ 4537.429168] ksys_read+0x5d/0xe0 [ 4537.429173] do_syscall_64+0x50/0x170 [ 4537.429180] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 4537.429184] RIP: 0033:0x7c3fc38a1415 [ 4537.429192] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 4537.429199] RSP: 002b:00007ffdfcb6a818 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 4537.429207] RAX: ffffffffffffffda RBX: 00007c3fc39757e0 RCX: 00007c3fc38a1415 [ 4537.429210] RDX: 0000000000000001 RSI: 00007c3fc3975863 RDI: 0000000000000000 [ 4537.429215] RBP: 00007ffdfcb6a860 R08: 0000000000000000 R09: 0000000000000000 [ 4537.429222] R10: 000063939c42a58c R11: 0000000000000246 R12: 00007c3fc3977300 [ 4537.429226] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468 [ 4537.429229] NMI backtrace for cpu 4 [ 4537.429235] CPU: 4 PID: 25727 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429245] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429249] RIP: 0010:rcu_all_qs+0x7/0x50 [ 4537.429256] Code: 43 30 48 83 7b 20 00 74 d2 0f 0b 48 85 c0 74 d0 c6 03 00 5b 5d c3 0f 0b c3 66 2e 0f 1f 84 00 00 00 00 00 65 8a 05 16 27 f2 5d <84> c0 74 2e 48 c7 c0 bd 17 02 00 65 48 03 05 7e 10 f1 5d 0f b6 00 [ 4537.429261] RSP: 0018:ffffb2d88badb948 EFLAGS: 00000286 [ 4537.429276] RAX: 0000000080000000 RBX: 000000000058fcc0 RCX: 000000000000000c [ 4537.429279] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfde000 [ 4537.429282] RBP: ffff99f64dfde000 R08: ffff99f64dfd9b10 R09: ffff99f62d921450 [ 4537.429289] R10: 0000000000000001 R11: ffff99f62d920a80 R12: 0000000000000000 [ 4537.429295] R13: 000000000003389a R14: ffffb2d88badbad0 R15: 000000000001f5a8 [ 4537.429302] FS: 0000720b85407740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000 [ 4537.429307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4537.429309] CR2: 000072d5e050ac40 CR3: 00000007c23a4002 CR4: 00000000003606e0 [ 4537.429312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4537.429319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4537.429324] Call Trace: [ 4537.429331] _cond_resched+0x10/0x20 [ 4537.429334] isolate_migratepages_block+0xd5/0xa50 [ 4537.429337] compact_zone+0x577/0xa70 [ 4537.429343] compact_zone_order+0xde/0x120 [ 4537.429348] try_to_compact_pages+0x187/0x240 [ 4537.429353] __alloc_pages_direct_compact+0x87/0x170 [ 4537.429357] __alloc_pages_slowpath+0x20e/0xc20 [ 4537.429361] ? release_pages+0x348/0x3b0 [ 4537.429364] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 4537.429367] __alloc_pages_nodemask+0x268/0x2b0 [ 4537.429373] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 4537.429379] __handle_mm_fault+0xc11/0x1890 [ 4537.429384] handle_mm_fault+0xa9/0x1d0 [ 4537.429386] __do_page_fault+0x237/0x480 [ 4537.429389] do_page_fault+0x1d/0x67 [ 4537.429391] ? page_fault+0x8/0x30 [ 4537.429398] page_fault+0x1e/0x30 [ 4537.429401] RIP: 0033:0x5c2e1a191c10 [ 4537.429404] Code: Bad RIP value. [ 4537.429406] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 4537.429414] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 4537.429417] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 4537.429419] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 4537.429426] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 4537.429429] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 4537.429437] NMI backtrace for cpu 5 [ 4537.429440] CPU: 5 PID: 25752 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429443] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429450] RIP: 0010:isolate_migratepages_block+0x2de/0xa50 [ 4537.429455] Code: 00 00 48 81 c4 88 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 4c 89 54 24 18 48 8b 44 24 20 48 8d b8 00 1b 02 00 48 8b 4c 24 28 <48> 8d 54 24 77 48 8b 74 24 78 e8 33 f0 ff ff 84 c0 4c 8b 54 24 18 [ 4537.429458] RSP: 0018:ffffb2d88bba3958 EFLAGS: 00000246 [ 4537.429468] RAX: ffff99f64dfde000 RBX: 00000000004f3c60 RCX: ffffb2d88bba3b49 [ 4537.429472] RDX: 0000000000000000 RSI: 0000000000000007 RDI: ffff99f64dfffb00 [ 4537.429475] RBP: ffff99f64dfde000 R08: 0000000000000001 R09: ffff99f62d961450 [ 4537.429478] R10: 0000000000000000 R11: ffff99f57cfb7400 R12: ffffb2d88bba3ad0 [ 4537.429480] R13: 0000000000000000 R14: ffffb2d88bba3ad0 R15: 0000000000000000 [ 4537.429485] FS: 0000720b85407740(0000) GS:ffff99f62d940000(0000) knlGS:0000000000000000 [ 4537.429490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4537.429493] CR2: 00007ac70ee0a000 CR3: 000000081439e001 CR4: 00000000003606e0 [ 4537.429496] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4537.429500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4537.429503] Call Trace: [ 4537.429506] compact_zone+0x577/0xa70 [ 4537.429510] compact_zone_order+0xde/0x120 [ 4537.429515] try_to_compact_pages+0x187/0x240 [ 4537.429518] __alloc_pages_direct_compact+0x87/0x170 [ 4537.429520] __alloc_pages_slowpath+0x454/0xc20 [ 4537.429523] ? release_pages+0x348/0x3b0 [ 4537.429526] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 4537.429530] __alloc_pages_nodemask+0x268/0x2b0 [ 4537.429533] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 4537.429536] __handle_mm_fault+0xc11/0x1890 [ 4537.429538] handle_mm_fault+0xa9/0x1d0 [ 4537.429543] __do_page_fault+0x237/0x480 [ 4537.429546] do_page_fault+0x1d/0x67 [ 4537.429548] ? page_fault+0x8/0x30 [ 4537.429551] page_fault+0x1e/0x30 [ 4537.429554] RIP: 0033:0x5c2e1a191c10 [ 4537.429561] Code: Bad RIP value. [ 4537.429564] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 4537.429574] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 4537.429579] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 4537.429582] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 4537.429585] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 4537.429590] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 4537.429595] NMI backtrace for cpu 6 [ 4537.429601] CPU: 6 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429604] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429609] RIP: 0033:0x7c32c38bcbd3 [ 4537.429614] Code: 00 31 c0 eb 09 66 90 48 c7 c0 fe ff ff ff 48 8b 7d c8 64 48 33 3c 25 28 00 00 00 0f 85 7c 00 00 00 48 8d 65 d8 5b 41 5c 41 5d <41> 5e 41 5f 5d c3 0f 1f 80 00 00 00 00 48 8b 45 80 4c 29 e0 eb ce [ 4537.429617] RSP: 002b:00007ffe934c7320 EFLAGS: 00000246 [ 4537.429624] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000000 [ 4537.429627] RDX: 0000000000000068 RSI: 0000000000000001 RDI: 0000000000000000 [ 4537.429630] RBP: 00007ffe934c7330 R08: 000058517a066458 R09: 0000000000000000 [ 4537.429633] R10: 00007ffe934c736c R11: 000058517a066457 R12: 00007c32c39e5500 [ 4537.429636] R13: 00007ffe934c7370 R14: 000058517a06648a R15: 000058517b6d4b10 [ 4537.429640] FS: 00007c32c37f6740 GS: 0000000000000000 [ 4537.429644] NMI backtrace for cpu 7 [ 4537.429647] CPU: 7 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429653] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429656] RIP: 0033:0x70f212b62d36 [ 4537.429661] Code: 00 c7 83 c0 00 00 00 ff ff ff ff 48 8b 8b d8 00 00 00 48 89 4d c8 48 8d 15 d7 39 16 00 48 8d 05 38 47 16 00 48 29 d0 48 29 d1 <48> 39 c8 0f 86 d1 00 00 00 4c 89 ea 4c 89 fe 48 89 df 48 8b 45 c8 [ 4537.429664] RSP: 002b:00007fffa4d872b0 EFLAGS: 00000206 [ 4537.429670] RAX: 0000000000000d68 RBX: 000070f212cc6500 RCX: 0000000000000c00 [ 4537.429675] RDX: 000070f212cc6700 RSI: 0000000000000001 RDI: 000070f212cc8320 [ 4537.429678] RBP: 00007fffa4d872f0 R08: 00005acd2cb3946a R09: 0000000000000000 [ 4537.429681] R10: 00007fffa4d8732c R11: 00005acd2cb39469 R12: 0000000000000001 [ 4537.429684] R13: 0000000000000001 R14: 0000000000000001 R15: 00005acd2cb39469 [ 4537.429690] FS: 000070f212ad7740 GS: 0000000000000000 [ 4537.429696] NMI backtrace for cpu 9 [ 4537.429699] CPU: 9 PID: 1266 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429702] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429706] RIP: 0033:0x74e35afcec9c [ 4537.429712] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 18 49 89 f5 4c 0f af ea <4d> 85 ed 0f 84 f0 00 00 00 49 89 ff 49 89 f6 49 89 d4 48 89 cb 8b [ 4537.429714] RSP: 002b:00007ffda01e7980 EFLAGS: 00000202 [ 4537.429729] RAX: 0000000000000001 RBX: 000000000000000a RCX: 000074e35b132500 [ 4537.429732] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00005d76c75ae45e [ 4537.429737] RBP: 00007ffda01e79c0 R08: 00005d76c75ae45f R09: 0000000000000000 [ 4537.429743] R10: 00007ffda01e79fc R11: 00005d76c75ae45e R12: 000074e35b132500 [ 4537.429748] R13: 0000000000000001 R14: 00005d76c75a9071 R15: 0000000000000001 [ 4537.429753] FS: 000074e35af43740 GS: 0000000000000000 [ 4537.429757] NMI backtrace for cpu 10 skipped: idling at intel_idle+0x7d/0x120 [ 4537.429764] NMI backtrace for cpu 11 [ 4537.429770] CPU: 11 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 4537.429774] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 4537.429779] RIP: 0033:0x77d47d42bb16 [ 4537.429790] Code: 12 00 64 48 8b 00 4c 8b 28 4d 8b 7d 28 4d 85 ff 0f 84 06 01 00 00 4c 89 65 80 4d 01 e6 0f 82 21 01 00 00 49 8b 07 4c 8b 68 28 <48> 83 38 00 74 0d 49 c1 cd 11 64 4c 33 2c 25 30 00 00 00 4c 89 ef [ 4537.429797] RSP: 002b:00007ffe06db5df0 EFLAGS: 00000216 [ 4537.429811] RAX: 00005ffb591c31d0 RBX: 00007ffe06db5ebc RCX: 00007ffe06db5ec0 [ 4537.429814] RDX: 000000000000000d RSI: 000077d47d556b60 RDI: 00007ffe06db5ebc [ 4537.429820] RBP: 00007ffe06db5e80 R08: 0000000000000001 R09: 0000000000000000 [ 4537.429827] R10: 00007ffe06db5ebc R11: 00005ffb57470458 R12: 00005ffb57470459 [ 4537.429833] R13: 000077d47d3962d0 R14: 00005ffb57470466 R15: 00005ffb591c0b10 [ 4537.429838] FS: 000077d47d365740 GS: 0000000000000000 ``` Created attachment 283669 [details]
zcat-ed kernel .config from /proc
the initial command was this: ``` $ time stress -m 220 --vm-bytes 1000000000 --timeout 10 stress: info: [25580] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: FAIL: [25580] (415) <-- worker 25793 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25795 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25799 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25792 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25797 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25798 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25791 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes stress: FAIL: [25580] (415) <-- worker 25800 got signal 9 stress: WARN: [25580] (417) now reaping child worker processes ``` (and it's still running 3 `stress` processes at the moment) ``` $ cat /proc/uptime 5490.76 43065.45 ``` omg that bugzilla bug again :((((((( that's not what I pasted in the previous comment!!!!!!!!!!11 oh great, it got partially fixed as I posted my previous comment! ^C real 80m19.315s user 0m0.464s sys 50m33.525s That stress command has been running for 80mins, but it was supposed to exit after 10 seconds! There are still 3 stress processes running. ``` [ 5905.542236] sysrq: Show backtrace of all active CPUs [ 5905.542244] NMI backtrace for cpu 8 [ 5905.542250] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.542253] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.542256] Call Trace: [ 5905.542260] <IRQ> [ 5905.542271] dump_stack+0x46/0x60 [ 5905.542276] nmi_cpu_backtrace.cold+0x14/0x53 [ 5905.542283] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 5905.542287] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 5905.542294] __handle_sysrq.cold+0x48/0x102 [ 5905.542299] sysrq_filter+0x2ea/0x3b0 [ 5905.542306] input_to_handler+0x4d/0xf0 [ 5905.542312] input_pass_values.part.0+0x109/0x130 [ 5905.542319] input_handle_event+0x171/0x5a0 [ 5905.542324] input_event+0x4d/0x70 [ 5905.542331] hidinput_report_event+0x2e/0x40 [ 5905.542337] hid_report_raw_event+0x260/0x430 [ 5905.542343] hid_input_report+0xfb/0x150 [ 5905.542350] hid_irq_in+0x168/0x190 [ 5905.542358] __usb_hcd_giveback_urb+0x77/0xe0 [ 5905.542376] xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd] [ 5905.542391] xhci_td_cleanup+0xf7/0x140 [xhci_hcd] [ 5905.542406] xhci_irq+0x7e8/0x1be0 [xhci_hcd] [ 5905.542413] __handle_irq_event_percpu+0x2f/0xc0 [ 5905.542419] handle_irq_event_percpu+0x2c/0x80 [ 5905.542424] handle_irq_event+0x23/0x43 [ 5905.542431] handle_edge_irq+0x78/0x190 [ 5905.542437] handle_irq+0x17/0x20 [ 5905.542442] do_IRQ+0x3e/0xd0 [ 5905.542450] common_interrupt+0xf/0xf [ 5905.542453] </IRQ> [ 5905.542460] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0 [ 5905.542466] Code: e8 d6 f9 a1 ff 49 89 c6 31 ff e8 fc 09 a2 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 f5 f3 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c [ 5905.542470] RSP: 0018:ffffb2d883217e80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde [ 5905.542475] RAX: ffff99f62da20a00 RBX: ffffffffa3061940 RCX: 000000000000001f [ 5905.542478] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000 [ 5905.542482] RBP: 0000055efdb5caf8 R08: 0000055efdbed96b R09: 0000000000000384 [ 5905.542484] R10: ffff99f62da1fb44 R11: ffff99f62da1fb24 R12: ffff99f62da29a00 [ 5905.542487] R13: 0000000000000004 R14: 0000055efdbed96b R15: 0000000000000000 [ 5905.542495] cpuidle_enter+0x24/0x40 [ 5905.542501] do_idle+0x1c1/0x230 [ 5905.542507] cpu_startup_entry+0x14/0x20 [ 5905.542514] start_secondary+0x168/0x1b0 [ 5905.542522] secondary_startup_64+0xa4/0xb0 [ 5905.542528] Sending NMI from CPU 8 to CPUs 0-7,9-11: [ 5905.542583] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x7d/0x120 [ 5905.542835] NMI backtrace for cpu 1 [ 5905.542840] CPU: 1 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.542843] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.542844] RIP: 0010:queued_spin_lock_slowpath+0x3d/0x1a0 [ 5905.542848] Code: 41 f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 1b 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 f6 c4 01 75 04 c6 47 [ 5905.542850] RSP: 0018:ffffb2d883fc7e50 EFLAGS: 00000002 [ 5905.542858] RAX: 00000000000c0101 RBX: ffff99f57ce54000 RCX: 0000000000080000 [ 5905.542860] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa355d2ac [ 5905.542862] RBP: ffff99f57cd9e300 R08: ffffb2d883fa7e78 R09: ffffb2d883f8fe78 [ 5905.542864] R10: 0000000000000004 R11: ffff99f588c48600 R12: 0000000000001fff [ 5905.542866] R13: 00005acd2cb39440 R14: ffff99f57ce54038 R15: ffff99f57cd9e300 [ 5905.542868] FS: 000070f212ad7740(0000) GS:ffff99f62d840000(0000) knlGS:0000000000000000 [ 5905.542872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.542874] CR2: 000078739192f578 CR3: 000000077cdac005 CR4: 00000000003606e0 [ 5905.542876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.542878] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.542879] Call Trace: [ 5905.542880] devkmsg_read+0xa2/0x270 [ 5905.542881] ? finish_wait+0x80/0x80 [ 5905.542883] vfs_read+0x98/0x150 [ 5905.542884] ksys_read+0x5d/0xe0 [ 5905.542887] do_syscall_64+0x50/0x170 [ 5905.542895] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 5905.542896] RIP: 0033:0x70f212bf1415 [ 5905.542900] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 5905.542901] RSP: 002b:00007fffa4d874f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 5905.542905] RAX: ffffffffffffffda RBX: 00005acd2cb39440 RCX: 000070f212bf1415 [ 5905.542907] RDX: 0000000000001fff RSI: 00005acd2cb39440 RDI: 0000000000000003 [ 5905.542908] RBP: 00005acd2cb357fa R08: 000000000000000a R09: 0000000000000000 [ 5905.542913] R10: 00007fffa4d8732c R11: 0000000000000246 R12: 000000000000003c [ 5905.542916] R13: 00005acd2cb34377 R14: 00005acd2cb39440 R15: 00005acd2cb39456 [ 5905.542919] NMI backtrace for cpu 2 [ 5905.542924] CPU: 2 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.542932] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.542936] RIP: 0010:queued_spin_lock_slowpath+0x16d/0x1a0 [ 5905.542944] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 16 02 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 e5 ec a2 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb [ 5905.542948] RSP: 0018:ffffb2d884117e50 EFLAGS: 00000046 [ 5905.542963] RAX: 0000000000000000 RBX: ffff99f57ce60000 RCX: 00000000000c0000 [ 5905.542969] RDX: ffff99f62d8a16c0 RSI: 000000000000000b RDI: ffffffffa355d2ac [ 5905.542975] RBP: ffff99f588e03a00 R08: 0000000000000002 R09: ffffb2d883fa7e78 [ 5905.542979] R10: 0000000000000002 R11: ffff99f588c49800 R12: 0000000000001fff [ 5905.542982] R13: 000058517a066440 R14: ffff99f57ce60038 R15: ffff99f588e03a00 [ 5905.542988] FS: 00007c32c37f6740(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000 [ 5905.542993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.542997] CR2: 00007ecf94de9578 CR3: 000000077cd3c003 CR4: 00000000003606e0 [ 5905.543002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.543008] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.543013] Call Trace: [ 5905.543018] devkmsg_read+0xa2/0x270 [ 5905.543021] ? finish_wait+0x80/0x80 [ 5905.543023] vfs_read+0x98/0x150 [ 5905.543029] ksys_read+0x5d/0xe0 [ 5905.543036] do_syscall_64+0x50/0x170 [ 5905.543039] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 5905.543042] RIP: 0033:0x7c32c3910415 [ 5905.543048] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 5905.543054] RSP: 002b:00007ffe934c7538 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 5905.543066] RAX: ffffffffffffffda RBX: 000058517a066440 RCX: 00007c32c3910415 [ 5905.543072] RDX: 0000000000001fff RSI: 000058517a066440 RDI: 0000000000000003 [ 5905.543076] RBP: 000058517a0627fa R08: 000000000000000a R09: 0000000000000000 [ 5905.543084] R10: 00007ffe934c736c R11: 0000000000000246 R12: 000000000000003c [ 5905.543087] R13: 000058517a061377 R14: 000058517a066440 R15: 000058517a066456 [ 5905.543094] NMI backtrace for cpu 3 [ 5905.543100] CPU: 3 PID: 25752 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543106] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543112] RIP: 0010:isolate_migratepages_block+0x6/0xa50 [ 5905.543119] Code: ff 0f b7 43 60 ff c8 66 89 43 5a e9 8b fa ff ff e8 1f 82 ee ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 <41> 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 [ 5905.543122] RSP: 0018:ffffb2d88bba39f8 EFLAGS: 00000202 [ 5905.543136] RAX: 0000000000000000 RBX: 00000000004f4000 RCX: 000000000000000c [ 5905.543142] RDX: 00000000004f3e00 RSI: 00000000004f3c60 RDI: ffffb2d88bba3ad0 [ 5905.543149] RBP: ffffb2d88bba3ad0 R08: 0000000000000001 R09: ffff99f62d8e1450 [ 5905.543158] R10: 0000000000000001 R11: ffff99f62d8e0a80 R12: 00000000004f3e00 [ 5905.543163] R13: ffff99f64dfded00 R14: 00000000004f3c60 R15: ffffb2d88bba3ad0 [ 5905.543169] FS: 0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000 [ 5905.543174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.543182] CR2: 00007ecf94e85ff8 CR3: 000000081439e002 CR4: 00000000003606e0 [ 5905.543190] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.543195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.543200] Call Trace: [ 5905.543205] compact_zone+0x577/0xa70 [ 5905.543210] compact_zone_order+0xde/0x120 [ 5905.543217] try_to_compact_pages+0x187/0x240 [ 5905.543224] __alloc_pages_direct_compact+0x87/0x170 [ 5905.543230] __alloc_pages_slowpath+0x454/0xc20 [ 5905.543236] ? release_pages+0x348/0x3b0 [ 5905.543243] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 5905.543250] __alloc_pages_nodemask+0x268/0x2b0 [ 5905.543255] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 5905.543260] __handle_mm_fault+0xc11/0x1890 [ 5905.543267] handle_mm_fault+0xa9/0x1d0 [ 5905.543275] __do_page_fault+0x237/0x480 [ 5905.543281] do_page_fault+0x1d/0x67 [ 5905.543286] ? page_fault+0x8/0x30 [ 5905.543294] page_fault+0x1e/0x30 [ 5905.543299] RIP: 0033:0x5c2e1a191c10 [ 5905.543304] Code: Bad RIP value. [ 5905.543310] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 5905.543324] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 5905.543331] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 5905.543337] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 5905.543343] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 5905.543348] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 5905.543360] NMI backtrace for cpu 4 [ 5905.543367] CPU: 4 PID: 25727 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543374] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543381] RIP: 0010:isolate_migratepages_block+0x20/0xa50 [ 5905.543388] Code: 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 89 4c 24 3c <65> 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 c0 48 8b 47 [ 5905.543393] RSP: 0018:ffffb2d88badb958 EFLAGS: 00000292 [ 5905.543403] RAX: 0000000000000001 RBX: 000000000058fcc0 RCX: 000000000000000c [ 5905.543406] RDX: 000000000058fe00 RSI: 000000000058fcc0 RDI: ffffb2d88badbad0 [ 5905.543411] RBP: ffffb2d88badbad0 R08: ffff99f64dfd9b10 R09: ffff99f62d921450 [ 5905.543414] R10: 0000000000000001 R11: ffff99f62d920a80 R12: 000000000058fe00 [ 5905.543419] R13: ffff99f64dfded00 R14: ffffb2d88badbad0 R15: ffffb2d88badbad0 [ 5905.543423] FS: 0000720b85407740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000 [ 5905.543426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.543431] CR2: 000079ad89768970 CR3: 00000007c23a4003 CR4: 00000000003606e0 [ 5905.543440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.543443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.543445] Call Trace: [ 5905.543451] compact_zone+0x577/0xa70 [ 5905.543454] compact_zone_order+0xde/0x120 [ 5905.543459] try_to_compact_pages+0x187/0x240 [ 5905.543463] __alloc_pages_direct_compact+0x87/0x170 [ 5905.543466] __alloc_pages_slowpath+0x20e/0xc20 [ 5905.543471] ? release_pages+0x348/0x3b0 [ 5905.543475] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 5905.543480] __alloc_pages_nodemask+0x268/0x2b0 [ 5905.543485] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 5905.543487] __handle_mm_fault+0xc11/0x1890 [ 5905.543492] handle_mm_fault+0xa9/0x1d0 [ 5905.543496] __do_page_fault+0x237/0x480 [ 5905.543503] do_page_fault+0x1d/0x67 [ 5905.543506] ? page_fault+0x8/0x30 [ 5905.543510] page_fault+0x1e/0x30 [ 5905.543513] RIP: 0033:0x5c2e1a191c10 [ 5905.543515] Code: Bad RIP value. [ 5905.543520] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 5905.543529] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 5905.543532] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 5905.543636] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 5905.543642] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 5905.543645] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 5905.543648] NMI backtrace for cpu 6 [ 5905.543652] CPU: 6 PID: 1252 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543655] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543658] RIP: 0033:0x7c3fc3821982 [ 5905.543664] Code: 57 58 48 8b 57 18 48 8b 4f 48 48 89 4f 18 48 89 57 48 48 89 47 08 c3 90 f3 0f 1e fa 55 48 89 e5 41 54 53 48 89 fb 48 8b 47 28 <48> 3b 47 20 76 39 4c 8b a7 d8 00 00 00 48 8d 15 6a 4d 15 00 48 8d [ 5905.543667] RSP: 002b:00007ffdfcb6a800 EFLAGS: 00000246 [ 5905.543677] RAX: 00007c3fc3975863 RBX: 00007c3fc39757e0 RCX: 0000000000000000 [ 5905.543680] RDX: 0000000000000000 RSI: 00007c3fc3975864 RDI: 00007c3fc39757e0 [ 5905.543685] RBP: 00007ffdfcb6a810 R08: 00007c3fc3976500 R09: 0000000000000000 [ 5905.543688] R10: 000063939c42a5af R11: 0000000000000246 R12: 00007c3fc3976700 [ 5905.543693] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468 [ 5905.543695] FS: 00007c3fc3582740 GS: 0000000000000000 [ 5905.543701] NMI backtrace for cpu 5 [ 5905.543707] CPU: 5 PID: 396 Comm: systemd-journal Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543711] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543713] RIP: 0010:find_get_pages_range_tag+0xbf/0x2a0 [ 5905.543718] Code: 49 c7 c5 ff ff ff ff eb 75 48 8b 42 08 48 8d 78 ff a8 01 48 0f 44 fa 8b 47 34 85 c0 74 3c 8d 48 01 4c 8d 47 34 f0 0f b1 4f 34 <75> ee 48 8b 4a 08 48 8d 41 ff 83 e1 01 48 0f 44 c2 48 39 c7 0f 84 [ 5905.543720] RSP: 0018:ffffb2d883503b68 EFLAGS: 00000246 [ 5905.543726] RAX: 0000000000000003 RBX: 0000000000000002 RCX: 0000000000000004 [ 5905.543731] RDX: ffffde0b9dcd6180 RSI: ffffb2d883503c38 RDI: ffffde0b9dcd6180 [ 5905.543734] RBP: 0000000000000238 R08: ffffde0b9dcd61b4 R09: ffffffffffffffff [ 5905.543737] R10: ffffffffffffffc0 R11: 0000000000000000 R12: 0000000000000009 [ 5905.543740] R13: ffffffffffffffff R14: 0007ffffffffffff R15: ffffb2d883503c28 [ 5905.543743] FS: 00007d2b87dae840(0000) GS:ffff99f62d940000(0000) knlGS:0000000000000000 [ 5905.543746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.543748] CR2: 00007d2b896b2010 CR3: 0000000826cce003 CR4: 00000000003606e0 [ 5905.543752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.543757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.543759] Call Trace: [ 5905.543765] pagevec_lookup_range_tag+0x1f/0x30 [ 5905.543767] extent_write_cache_pages+0xe1/0x3a0 [ 5905.543770] ? clear_state_bit+0xde/0x190 [ 5905.543773] extent_writepages+0x4c/0xa0 [ 5905.543778] do_writepages+0x3c/0xd0 [ 5905.543780] ? btrfs_page_mkwrite+0x38e/0x480 [ 5905.543783] __filemap_fdatawrite_range+0xec/0x130 [ 5905.543786] btrfs_fdatawrite_range+0x16/0x40 [ 5905.543789] start_ordered_ops+0x47/0x80 [ 5905.543792] btrfs_sync_file+0xa1/0x3b0 [ 5905.543799] do_fsync+0x33/0x70 [ 5905.543802] __x64_sys_fsync+0xb/0x10 [ 5905.543805] do_syscall_64+0x50/0x170 [ 5905.543807] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 5905.543810] RIP: 0033:0x7d2b895b9c38 [ 5905.543813] Code: ff ff 77 3e c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 89 7d fc e8 58 bd f7 ff 41 89 c0 8b 7d fc b8 4a 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 22 89 45 fc 44 89 c7 e8 95 bd f7 ff 8b 45 fc [ 5905.543818] RSP: 002b:00007fff5ff3eba0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [ 5905.543824] RAX: ffffffffffffffda RBX: 00005dc473d022d0 RCX: 00007d2b895b9c38 [ 5905.543829] RDX: 00007d2b896b2000 RSI: 00005dc473d2f550 RDI: 0000000000000024 [ 5905.543832] RBP: 00007fff5ff3ebb0 R08: 0000000000000000 R09: 0000000000000014 [ 5905.543835] R10: 00007fff5ff3eb80 R11: 0000000000000293 R12: 00007fff5ff3ecf0 [ 5905.543839] R13: 00005dc473d022d0 R14: 00007fff5ff3f250 R15: 0000000000000180 [ 5905.543843] NMI backtrace for cpu 7 [ 5905.543848] CPU: 7 PID: 25777 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543851] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543853] RIP: 0010:compact_unlock_should_abort.isra.0+0x39/0x50 [ 5905.543858] Code: 00 5d 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 <74> e0 c6 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00 [ 5905.543860] RSP: 0018:ffffb2d88bc6b940 EFLAGS: 00000202 [ 5905.543868] RAX: ffff99f370429e40 RBX: 0000000000718440 RCX: ffffb2d88bc6bb49 [ 5905.543870] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 5905.543872] RBP: ffffb2d88bc6bb49 R08: 0000000000000001 R09: ffff99f62d9e1450 [ 5905.543876] R10: 0000000000000000 R11: ffff99f62d9e0a80 R12: ffffb2d88bc6bad0 [ 5905.543878] R13: 0000000000000000 R14: ffffb2d88bc6bad0 R15: 0000000000000000 [ 5905.543881] FS: 0000720b85407740(0000) GS:ffff99f62d9c0000(0000) knlGS:0000000000000000 [ 5905.543886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.543891] CR2: 00007873919cbff8 CR3: 00000007ccf6a003 CR4: 00000000003606e0 [ 5905.543893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.543895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.543898] Call Trace: [ 5905.543903] isolate_migratepages_block+0x2ed/0xa50 [ 5905.543907] compact_zone+0x577/0xa70 [ 5905.543909] compact_zone_order+0xde/0x120 [ 5905.543913] try_to_compact_pages+0x187/0x240 [ 5905.543915] __alloc_pages_direct_compact+0x87/0x170 [ 5905.543918] __alloc_pages_slowpath+0x454/0xc20 [ 5905.543920] ? release_pages+0x348/0x3b0 [ 5905.543925] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 5905.543930] __alloc_pages_nodemask+0x268/0x2b0 [ 5905.543932] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 5905.543934] __handle_mm_fault+0xc11/0x1890 [ 5905.543938] handle_mm_fault+0xa9/0x1d0 [ 5905.543940] __do_page_fault+0x237/0x480 [ 5905.543942] do_page_fault+0x1d/0x67 [ 5905.543945] ? page_fault+0x8/0x30 [ 5905.543947] page_fault+0x1e/0x30 [ 5905.543949] RIP: 0033:0x5c2e1a191c10 [ 5905.543953] Code: Bad RIP value. [ 5905.543958] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 5905.543962] RAX: 00000000079a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 5905.543964] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 5905.543967] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 5905.543969] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 5905.543973] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 5905.543976] NMI backtrace for cpu 9 [ 5905.543980] CPU: 9 PID: 1254 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.543985] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.543991] RIP: 0010:ktime_get_coarse_real_ts64+0x0/0x40 [ 5905.543996] Code: ef e8 c4 fb ff ff 85 c0 75 0f 48 8b 74 24 28 48 8b 44 24 30 e9 5d ff ff ff 41 89 c6 e9 6a ff ff ff e8 74 fa f8 ff 0f 1f 40 00 <8b> 15 7a 85 65 01 f6 c2 01 75 26 8b 05 93 85 65 01 c4 e2 fb f7 05 [ 5905.543999] RSP: 0018:ffffb2d883f1fd58 EFLAGS: 00000246 [ 5905.544005] RAX: 0000000000000000 RBX: ffff99f5c7d50250 RCX: 0000000000000000 [ 5905.544011] RDX: 0000000000000000 RSI: ffff99f5c7d50250 RDI: ffffb2d883f1fd60 [ 5905.544016] RBP: ffffb2d883f1fd80 R08: 0000000000000001 R09: 0000000000000000 [ 5905.544019] R10: 0000000000000000 R11: 0000000000000000 R12: ffff99f629139220 [ 5905.544022] R13: ffff99f5c8648878 R14: 0000000000000001 R15: ffff99f5c86a8cc0 [ 5905.544026] FS: 0000705c7f390740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000 [ 5905.544029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.544032] CR2: 00007e392404fb50 CR3: 00000007c71a2004 CR4: 00000000003606e0 [ 5905.544035] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.544040] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.544044] Call Trace: [ 5905.544047] current_time+0x24/0x80 [ 5905.544053] atime_needs_update+0x6e/0xd0 [ 5905.544059] touch_atime+0x2e/0xe0 [ 5905.544062] pipe_read+0x2aa/0x2c0 [ 5905.544065] new_sync_read+0x10f/0x1a0 [ 5905.544069] vfs_read+0x98/0x150 [ 5905.544072] ksys_read+0x5d/0xe0 [ 5905.544075] do_syscall_64+0x50/0x170 [ 5905.544078] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 5905.544081] RIP: 0033:0x705c7f6af415 [ 5905.544085] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 5905.544091] RSP: 002b:00007ffc9e431508 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 5905.544099] RAX: ffffffffffffffda RBX: 0000705c7f7837e0 RCX: 0000705c7f6af415 [ 5905.544104] RDX: 0000000000000001 RSI: 0000705c7f783863 RDI: 0000000000000000 [ 5905.544108] RBP: 00007ffc9e431550 R08: 0000000000000000 R09: 0000000000000000 [ 5905.544111] R10: 00005c354d18d56f R11: 0000000000000246 R12: 0000705c7f785300 [ 5905.544114] R13: 0000705c7f784500 R14: 0000705c7f784700 R15: 0000705c7f785468 [ 5905.544123] NMI backtrace for cpu 10 [ 5905.544127] CPU: 10 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.544130] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.544133] RIP: 0033:0x77d47d3fd535 [ 5905.544139] Code: 02 44 89 c7 75 0f e9 ea 1e 08 00 66 2e 0f 1f 84 00 00 00 00 00 e9 1b 7a 08 00 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 18 45 31 f6 48 85 d2 [ 5905.544144] RSP: 002b:00007ffe06db5e30 EFLAGS: 00000202 [ 5905.544151] RAX: 000077d47d555300 RBX: 000077d47d554500 RCX: 0000000000000c00 [ 5905.544154] RDX: 0000000000000001 RSI: 00005ffb57470462 RDI: 000077d47d554500 [ 5905.544158] RBP: 00007ffe06db5e80 R08: 00005ffb57470463 R09: 0000000000000000 [ 5905.544161] R10: 00007ffe06db5ebc R11: 00005ffb57470462 R12: 0000000000000001 [ 5905.544167] R13: 0000000000000001 R14: 0000000000000001 R15: 00005ffb57470462 [ 5905.544170] FS: 000077d47d365740 GS: 0000000000000000 [ 5905.544174] NMI backtrace for cpu 11 [ 5905.544178] CPU: 11 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 5905.544182] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 5905.544187] RIP: 0010:format_decode+0x3df/0x3f0 [ 5905.544194] Code: e9 58 fe ff ff 31 c0 e9 e1 fc ff ff c6 45 00 02 48 8d 41 02 29 d8 e9 d2 fc ff ff 83 c2 0b 88 55 00 e9 37 fe ff ff 48 8d 41 02 <48> 89 04 24 be 4c 00 00 00 e9 36 fd ff ff 0f 1f 00 41 54 55 53 49 [ 5905.544198] RSP: 0018:ffffb2d883fa7d40 EFLAGS: 00000046 [ 5905.544208] RAX: ffffffffa2e31773 RBX: ffffffffa2e31770 RCX: ffffffffa2e31771 [ 5905.544213] RDX: 0000000000000004 RSI: 000000000000006c RDI: ffffffffa2e31770 [ 5905.544219] RBP: ffffb2d883fa7d70 R08: ffff99f61c780059 R09: 0000000000000000 [ 5905.544223] R10: 0000000000000004 R11: ffff99f61c780059 R12: ffffffffa2e31770 [ 5905.544226] R13: 0000000000002000 R14: ffffffffa2e31770 R15: ffff99f61c78005a [ 5905.544231] FS: 00007c7ea548a740(0000) GS:ffff99f62dac0000(0000) knlGS:0000000000000000 [ 5905.544238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5905.544241] CR2: 000056eb222ca148 CR3: 00000007c97bc001 CR4: 00000000003606e0 [ 5905.544244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5905.544247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 5905.544253] Call Trace: [ 5905.544258] vsnprintf+0x6e/0x5b0 [ 5905.544262] scnprintf+0x4d/0x90 [ 5905.544268] msg_print_ext_header.constprop.0+0x7f/0xa0 [ 5905.544273] ? _cond_resched+0x10/0x20 [ 5905.544277] ? mutex_lock_interruptible+0x9/0x30 [ 5905.544280] devkmsg_read+0x17f/0x270 [ 5905.544286] vfs_read+0x98/0x150 [ 5905.544289] ksys_read+0x5d/0xe0 [ 5905.544297] do_syscall_64+0x50/0x170 [ 5905.544300] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 5905.544303] RIP: 0033:0x7c7ea55a4415 [ 5905.544307] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 5905.544310] RSP: 002b:00007ffe25d18658 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 5905.544321] RAX: ffffffffffffffda RBX: 0000571ede0f0440 RCX: 00007c7ea55a4415 [ 5905.544327] RDX: 0000000000001fff RSI: 0000571ede0f0440 RDI: 0000000000000003 [ 5905.544330] RBP: 0000571ede0ec7fa R08: 000000000000000a R09: 0000000000000000 [ 5905.544333] R10: 00007ffe25d1848c R11: 0000000000000246 R12: 0000000000000041 [ 5905.544338] R13: 0000571ede0eb377 R14: 0000571ede0f0440 R15: 0000571ede0f0455 ``` Now only two stress processes left: (I actually tried to gdb attach to one this time, then tried to kill -9 another, and tried to kill -11 another, no effect - but eventually, on its own(!), one process/thread went away, so two left now) ``` MiB Swap: 0.0 total, 0.0 free, 0.0 used. 27127.4 avail Mem PID %CPU COMMAND PR NI VIRT RES S USER 25727 98.1 stress 20 0 980352 96196 R user 25752 98.1 stress 20 0 980352 122820 R user ``` ``` [ 7163.859183] sysrq: Show backtrace of all active CPUs [ 7163.859189] NMI backtrace for cpu 8 [ 7163.859196] CPU: 8 PID: 25727 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.859199] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.859202] Call Trace: [ 7163.859206] <IRQ> [ 7163.859216] dump_stack+0x46/0x60 [ 7163.859222] nmi_cpu_backtrace.cold+0x14/0x53 [ 7163.859228] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 7163.859233] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 7163.859239] __handle_sysrq.cold+0x48/0x102 [ 7163.859244] sysrq_filter+0x2ea/0x3b0 [ 7163.859250] input_to_handler+0x4d/0xf0 [ 7163.859257] input_pass_values.part.0+0x109/0x130 [ 7163.859263] input_handle_event+0x171/0x5a0 [ 7163.859269] input_event+0x4d/0x70 [ 7163.859275] hidinput_report_event+0x2e/0x40 [ 7163.859280] hid_report_raw_event+0x260/0x430 [ 7163.859286] hid_input_report+0xfb/0x150 [ 7163.859293] hid_irq_in+0x168/0x190 [ 7163.859300] __usb_hcd_giveback_urb+0x77/0xe0 [ 7163.859316] xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd] [ 7163.859331] xhci_td_cleanup+0xf7/0x140 [xhci_hcd] [ 7163.859346] xhci_irq+0x7e8/0x1be0 [xhci_hcd] [ 7163.859353] __handle_irq_event_percpu+0x2f/0xc0 [ 7163.859359] handle_irq_event_percpu+0x2c/0x80 [ 7163.859364] handle_irq_event+0x23/0x43 [ 7163.859370] handle_edge_irq+0x78/0x190 [ 7163.859376] handle_irq+0x17/0x20 [ 7163.859381] do_IRQ+0x3e/0xd0 [ 7163.859388] common_interrupt+0xf/0xf [ 7163.859391] </IRQ> [ 7163.859397] RIP: 0010:compact_zone+0x101/0xa70 [ 7163.859403] Code: 98 90 05 00 00 0f 94 44 24 56 e8 ca 0e 04 00 48 c7 44 24 08 00 00 00 00 65 48 8b 04 25 00 5d 01 00 48 89 44 24 58 49 8b 57 30 <b8> 09 00 00 00 c4 c2 fb f7 47 28 48 89 d1 48 c1 e9 09 48 39 c8 0f [ 7163.859407] RSP: 0018:ffffb2d88badba18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffde [ 7163.859412] RAX: ffffb2d88badbac0 RBX: 0000000000590000 RCX: 000000000058fe00 [ 7163.859415] RDX: 000000000058fcc0 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 7163.859418] RBP: ffffb2d88badbad0 R08: ffff99f64dfd9b10 R09: ffff99f62da21450 [ 7163.859420] R10: 0000000000000000 R11: ffff99f62da20a80 R12: 000000000058fcc0 [ 7163.859423] R13: ffff99f64dfded00 R14: 000000000058fcc0 R15: ffffb2d88badbad0 [ 7163.859429] ? compact_zone+0x577/0xa70 [ 7163.859434] compact_zone_order+0xde/0x120 [ 7163.859440] try_to_compact_pages+0x187/0x240 [ 7163.859445] __alloc_pages_direct_compact+0x87/0x170 [ 7163.859450] __alloc_pages_slowpath+0x20e/0xc20 [ 7163.859456] ? release_pages+0x348/0x3b0 [ 7163.859461] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 7163.859465] __alloc_pages_nodemask+0x268/0x2b0 [ 7163.859470] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 7163.859476] __handle_mm_fault+0xc11/0x1890 [ 7163.859483] handle_mm_fault+0xa9/0x1d0 [ 7163.859489] __do_page_fault+0x237/0x480 [ 7163.859495] do_page_fault+0x1d/0x67 [ 7163.859499] ? page_fault+0x8/0x30 [ 7163.859502] page_fault+0x1e/0x30 [ 7163.859506] RIP: 0033:0x5c2e1a191c10 [ 7163.859514] Code: Bad RIP value. [ 7163.859518] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 7163.859521] RAX: 0000000005da6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 7163.859524] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 7163.859527] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 7163.859530] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 7163.859533] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 7163.859539] Sending NMI from CPU 8 to CPUs 0-7,9-11: [ 7163.859585] NMI backtrace for cpu 0 [ 7163.859588] CPU: 0 PID: 6555 Comm: xz Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.859591] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.859592] RIP: 0033:0x719ad7a360f0 [ 7163.859595] Code: 85 1e 02 00 00 bf fe ff ff ff 8d 42 02 29 d7 01 f7 48 01 c7 eb 14 0f 1f 40 00 41 0f b6 34 04 48 83 c0 01 40 3a 74 01 ff 75 08 <48> 39 c7 41 89 c0 75 e8 41 29 d0 45 8d 68 ff 44 89 44 24 30 41 83 [ 7163.859597] RSP: 002b:0000719ab1d1bb50 EFLAGS: 00000246 [ 7163.859600] RAX: 0000000000000010 RBX: 0000000000000acc RCX: 0000719ab1430869 [ 7163.859602] RDX: 000000000000000b RSI: 0000000000000000 RDI: 000000000000004c [ 7163.859604] RBP: 000000000000000c R08: 000000000000000f R09: 000000000000003a [ 7163.859606] R10: 0000719ab1471163 R11: 00000000000408f9 R12: 0000719ab1471163 [ 7163.859607] R13: 00000000000009a5 R14: 0000719ab14df010 R15: 0000000000000032 [ 7163.859608] FS: 0000719ab1d1c700 GS: 0000000000000000 [ 7163.859684] NMI backtrace for cpu 1 [ 7163.859687] CPU: 1 PID: 6565 Comm: xz Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.859689] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.859691] RIP: 0033:0x719ad7a36320 [ 7163.859694] Code: 0f 84 2c f6 ff ff 89 c1 89 ea e9 c3 fc ff ff 0f 1f 00 44 89 de 48 c1 e0 07 48 8d 84 06 a4 41 00 00 41 8b 04 86 e9 0e fd ff ff <45> 85 db 78 1b 4c 8d 05 34 aa 00 00 c1 ee 12 41 0f b6 34 30 83 c6 [ 7163.859695] RSP: 002b:0000719ab0859b50 EFLAGS: 00000216 [ 7163.859698] RAX: 0000000000000002 RBX: 0000000000000591 RCX: 0000000000000001 [ 7163.859705] RDX: 0000000000000004 RSI: 0000000000570d43 RDI: 0000719ab001d018 [ 7163.859706] RBP: 0000000000000004 R08: 0000000000000000 R09: 00000000000003f3 [ 7163.859708] R10: 0000719aa56925b4 R11: 0000000000570d43 R12: 0000719aa56925b4 [ 7163.859709] R13: 00000000000003ed R14: 0000719ab001d010 R15: 0000000000000021 [ 7163.859711] FS: 0000719ab085a700 GS: 0000000000000000 [ 7163.859719] NMI backtrace for cpu 2 [ 7163.859726] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.859734] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.859745] RIP: 0010:tick_nohz_idle_exit+0x26/0xa0 [ 7163.859754] Code: 00 0f 1f 00 41 55 41 54 55 53 48 c7 c5 80 cf 01 00 65 48 03 2d e3 65 ef 5d fa f6 45 4c 01 74 7a 48 83 bd b0 00 00 00 00 75 6c <0f> b6 5d 4c 83 e3 fe 88 5d 4c 41 89 dc 41 c0 ec 02 d0 eb 83 e3 01 [ 7163.859757] RSP: 0018:ffffb2d8831e7eb0 EFLAGS: 00000046 [ 7163.859765] RAX: 0000000080204008 RBX: 0000000000000002 RCX: ffff99f62d8a1700 [ 7163.859771] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff99f62d8a9a00 [ 7163.859773] RBP: ffff99f62d89cf80 R08: 0000000000000000 R09: 0000000000000daf [ 7163.859776] R10: ffff99f62d89fb44 R11: ffff99f62d89fb24 R12: ffff99f62a20bc80 [ 7163.859782] R13: ffff99f62a20bc80 R14: ffff99f62d8a9a00 R15: ffffffffa3061940 [ 7163.859785] FS: 0000000000000000(0000) GS:ffff99f62d880000(0000) knlGS:0000000000000000 [ 7163.859787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7163.859795] CR2: 00007112d0bcfb50 CR3: 000000080600a003 CR4: 00000000003606e0 [ 7163.859798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7163.859805] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7163.859809] Call Trace: [ 7163.859812] do_idle+0x137/0x230 [ 7163.859817] cpu_startup_entry+0x14/0x20 [ 7163.859821] start_secondary+0x168/0x1b0 [ 7163.859824] secondary_startup_64+0xa4/0xb0 [ 7163.859967] NMI backtrace for cpu 3 [ 7163.859972] CPU: 3 PID: 25752 Comm: stress Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.859978] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.859983] RIP: 0010:compact_unlock_should_abort.isra.0+0x39/0x50 [ 7163.859988] Code: 00 5d 01 00 48 8b 10 83 e2 04 75 17 e8 e0 6d 63 00 31 c0 5b 5d c3 48 89 d3 e8 93 b1 63 00 c6 03 00 eb d8 f6 80 f1 06 00 00 01 <74> e0 c6 45 00 01 b8 01 00 00 00 eb dc 66 2e 0f 1f 84 00 00 00 00 [ 7163.859993] RSP: 0018:ffffb2d88bba3940 EFLAGS: 00000202 [ 7163.860000] RAX: ffff99f517c53c80 RBX: 00000000004f3c60 RCX: ffffb2d88bba3b49 [ 7163.860004] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff99f64dfffb00 [ 7163.860009] RBP: ffffb2d88bba3b49 R08: 0000000000000001 R09: ffff99f62d8e1450 [ 7163.860013] R10: 0000000000000000 R11: ffff99f57cfb4e00 R12: ffffb2d88bba3ad0 [ 7163.860017] R13: 0000000000000000 R14: ffffb2d88bba3ad0 R15: 0000000000000000 [ 7163.860022] FS: 0000720b85407740(0000) GS:ffff99f62d8c0000(0000) knlGS:0000000000000000 [ 7163.860027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7163.860034] CR2: 000078739192f578 CR3: 000000081439e002 CR4: 00000000003606e0 [ 7163.860039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7163.860043] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7163.860046] Call Trace: [ 7163.860051] isolate_migratepages_block+0x2ed/0xa50 [ 7163.860055] compact_zone+0x577/0xa70 [ 7163.860059] compact_zone_order+0xde/0x120 [ 7163.860064] try_to_compact_pages+0x187/0x240 [ 7163.860068] __alloc_pages_direct_compact+0x87/0x170 [ 7163.860073] __alloc_pages_slowpath+0x454/0xc20 [ 7163.860077] ? release_pages+0x348/0x3b0 [ 7163.860081] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 7163.860086] __alloc_pages_nodemask+0x268/0x2b0 [ 7163.860090] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 7163.860094] __handle_mm_fault+0xc11/0x1890 [ 7163.860101] handle_mm_fault+0xa9/0x1d0 [ 7163.860103] __do_page_fault+0x237/0x480 [ 7163.860108] do_page_fault+0x1d/0x67 [ 7163.860110] ? page_fault+0x8/0x30 [ 7163.860115] page_fault+0x1e/0x30 [ 7163.860119] RIP: 0033:0x5c2e1a191c10 [ 7163.860124] Code: Bad RIP value. [ 7163.860128] RSP: 002b:00007fff325961b0 EFLAGS: 00010206 [ 7163.860136] RAX: 00000000077a6000 RBX: 0000720b49a5a010 RCX: 0000720b8552c3db [ 7163.860140] RDX: 0000000000000000 RSI: 000000003b9ad000 RDI: 0000720b49a5a000 [ 7163.860145] RBP: 00005c2e1a192a54 R08: 0000720b49a5a010 R09: 0000000000000000 [ 7163.860147] R10: 0000000000000022 R11: 000000003b9aca00 R12: ffffffffffffffff [ 7163.860152] R13: 0000000000000002 R14: 0000000000001000 R15: 000000003b9aca00 [ 7163.860159] NMI backtrace for cpu 4 [ 7163.860164] CPU: 4 PID: 1234 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860167] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860172] RIP: 0010:queued_spin_lock_slowpath+0x16d/0x1a0 [ 7163.860176] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 16 02 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 e5 ec a2 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb [ 7163.860180] RSP: 0018:ffffb2d884117e50 EFLAGS: 00000046 [ 7163.860188] RAX: 0000000000000000 RBX: ffff99f57ce60000 RCX: 0000000000140000 [ 7163.860191] RDX: ffff99f62d9216c0 RSI: 0000000000000001 RDI: ffffffffa355d2ac [ 7163.860196] RBP: ffff99f588e03a00 R08: 0000000000000004 R09: 0000000000000000 [ 7163.860198] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000001fff [ 7163.860203] R13: 000058517a066440 R14: ffff99f57ce60038 R15: ffff99f588e03a00 [ 7163.860207] FS: 00007c32c37f6740(0000) GS:ffff99f62d900000(0000) knlGS:0000000000000000 [ 7163.860210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7163.860213] CR2: 000078739192f578 CR3: 000000077cd3c006 CR4: 00000000003606e0 [ 7163.860215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7163.860221] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7163.860224] Call Trace: [ 7163.860227] devkmsg_read+0x7f/0x270 [ 7163.860229] vfs_read+0x98/0x150 [ 7163.860233] ksys_read+0x5d/0xe0 [ 7163.860237] do_syscall_64+0x50/0x170 [ 7163.860240] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 7163.860244] RIP: 0033:0x7c32c3910415 [ 7163.860248] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 7163.860254] RSP: 002b:00007ffe934c7538 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 7163.860260] RAX: ffffffffffffffda RBX: 000058517a066440 RCX: 00007c32c3910415 [ 7163.860265] RDX: 0000000000001fff RSI: 000058517a066440 RDI: 0000000000000003 [ 7163.860269] RBP: 000058517a0627fa R08: 000000000000000a R09: 0000000000000000 [ 7163.860274] R10: 00007ffe934c736c R11: 0000000000000246 R12: 000000000000000b [ 7163.860276] R13: 000058517a061377 R14: 000058517a066440 R15: 000058517a066455 [ 7163.860280] NMI backtrace for cpu 5 [ 7163.860287] CPU: 5 PID: 1098 Comm: panel-24-cpugra Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860290] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860297] RIP: 0033:0x7b5fc1db0606 [ 7163.860304] Code: 28 f2 0f 11 47 28 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f2 0f 10 06 f2 0f 10 0a f2 0f 10 57 08 f2 0f 10 5f 18 f2 0f 59 d0 <f2> 0f 59 d9 f2 0f 59 07 f2 0f 59 4f 10 f2 0f 58 d3 f2 0f 58 c1 f2 [ 7163.860309] RSP: 002b:00007ffd6a2c6e88 EFLAGS: 00000206 [ 7163.860317] RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000000076 [ 7163.860320] RDX: 00007ffd6a2c6ed8 RSI: 00007ffd6a2c6ed0 RDI: 00007ffd6a2c7f80 [ 7163.860323] RBP: 00007ffd6a2c7f80 R08: 00007ffd6a2c76c0 R09: 0000000000000000 [ 7163.860328] R10: 0000000000000083 R11: 000055caec6febb8 R12: 00007ffd6a2c6fd8 [ 7163.860331] R13: 00007ffd6a2c7058 R14: 00007ffd6a2c6eb0 R15: 00007ffd6a2c6eb8 [ 7163.860335] FS: 00007b5fbfea8980 GS: 0000000000000000 [ 7163.860339] NMI backtrace for cpu 6 [ 7163.860342] CPU: 6 PID: 1240 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860347] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860350] RIP: 0033:0x77d47d3f0d6c [ 7163.860356] Code: 4c 89 fe 48 89 df 48 8b 45 c8 ff 50 38 49 89 c0 48 83 f8 ff 41 0f 94 c1 f7 03 00 80 00 00 75 26 48 8b bb 88 00 00 00 ff 4f 04 <75> 1a 48 c7 47 08 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 0f 85 [ 7163.860360] RSP: 002b:00007ffe06db5e40 EFLAGS: 00000246 [ 7163.860368] RAX: 0000000000000001 RBX: 000077d47d554500 RCX: 0000000000000000 [ 7163.860372] RDX: 0000000000000001 RSI: 00005ffb57470470 RDI: 000077d47d556320 [ 7163.860378] RBP: 00007ffe06db5e80 R08: 0000000000000001 R09: 0000000000000000 [ 7163.860383] R10: 00007ffe06db5ebc R11: 00005ffb57470470 R12: 0000000000000001 [ 7163.860386] R13: 0000000000000001 R14: 0000000000000001 R15: 00005ffb57470470 [ 7163.860390] FS: 000077d47d365740 GS: 0000000000000000 [ 7163.860395] NMI backtrace for cpu 7 [ 7163.860398] CPU: 7 PID: 1229 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860400] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860410] RIP: 0033:0x70f212b08368 [ 7163.860415] Code: 45 a0 48 83 7f 68 00 74 11 48 c1 c8 11 64 48 33 04 25 30 00 00 00 48 89 45 a0 45 85 ed 0f 85 df 07 00 00 48 8b 45 a8 4c 8b 18 <48> 8b 7d 98 48 85 ff 48 89 f8 49 0f 44 c4 4c 8b 10 4d 8b 6c 24 08 [ 7163.860417] RSP: 002b:00007fffa4d87190 EFLAGS: 00000246 [ 7163.860426] RAX: 00007fffa4d87270 RBX: 00005acd2cb3947b RCX: 00005acd2cb3947b [ 7163.860430] RDX: 00007fffa4d87270 RSI: 00007fffa4d87280 RDI: 00005acd2cc741d0 [ 7163.860434] RBP: 00007fffa4d87240 R08: 0000000000000000 R09: 0000000000000001 [ 7163.860439] R10: 00007fffa4d8732c R11: 00005acd2cb39456 R12: 00007fffa4d87280 [ 7163.860445] R13: 0000000000000000 R14: 00005acd2cb3947b R15: 00005acd2cc71b10 [ 7163.860449] FS: 000070f212ad7740 GS: 0000000000000000 [ 7163.860452] NMI backtrace for cpu 9 [ 7163.860460] CPU: 9 PID: 1238 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860465] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860472] RIP: 0010:_raw_spin_lock+0xb/0x20 [ 7163.860478] Code: 81 05 fd 94 82 5d 00 02 00 00 31 c0 ba ff 00 00 00 f0 0f b1 17 75 01 c3 e9 22 f1 8f ff 66 90 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 01 c3 89 c6 e9 db e0 8f ff 66 66 2e 0f 1f 84 00 00 00 00 00 65 [ 7163.860486] RSP: 0018:ffffb2d884137cb8 EFLAGS: 00000046 [ 7163.860491] RAX: 0000000000000000 RBX: ffff99f57ce09e40 RCX: 0000000000000009 [ 7163.860499] RDX: 0000000000000001 RSI: ffff99f62da40000 RDI: ffff99f62da60a00 [ 7163.860504] RBP: ffff99f62da60a00 R08: 0000000000000009 R09: ffffb2d884137d90 [ 7163.860507] R10: 0000000000000000 R11: 0000000000000009 R12: 0000000000000001 [ 7163.860511] R13: 0000000000000046 R14: ffff99f57ce0a584 R15: 0000000000020a00 [ 7163.860519] FS: 00007d2186c2e740(0000) GS:ffff99f62da40000(0000) knlGS:0000000000000000 [ 7163.860524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7163.860531] CR2: 00007743d494ec40 CR3: 0000000788c3c005 CR4: 00000000003606e0 [ 7163.860537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7163.860540] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7163.860546] Call Trace: [ 7163.860554] try_to_wake_up+0x18d/0x500 [ 7163.860556] autoremove_wake_function+0xc/0x50 [ 7163.860559] __wake_up_common+0x7a/0x190 [ 7163.860562] __wake_up_common_lock+0x79/0xc0 [ 7163.860565] pipe_write+0x1e3/0x410 [ 7163.860568] new_sync_write+0x116/0x1b0 [ 7163.860571] vfs_write+0xb1/0x190 [ 7163.860573] ksys_write+0x5d/0xe0 [ 7163.860580] do_syscall_64+0x50/0x170 [ 7163.860585] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 7163.860591] RIP: 0033:0x7d2186f4d4ba [ 7163.860596] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 [ 7163.860602] RSP: 002b:00007ffd33328988 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 7163.860612] RAX: ffffffffffffffda RBX: 0000000000000022 RCX: 00007d2186f4d4ba [ 7163.860618] RDX: 0000000000000022 RSI: 0000647dee7025a0 RDI: 0000000000000001 [ 7163.860622] RBP: 00007ffd333289b0 R08: 0000000000000000 R09: 0000000000000000 [ 7163.860628] R10: 0000647dee707571 R11: 0000000000000246 R12: 0000000000000022 [ 7163.860632] R13: 0000647dee7025a0 R14: 00007d2187022500 R15: 00007d2187023300 [ 7163.860640] NMI backtrace for cpu 10 [ 7163.860645] CPU: 10 PID: 1252 Comm: sed Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860648] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860651] RIP: 0033:0x7c3fc3820b9e [ 7163.860657] Code: 00 4c 89 f8 4c 29 e0 48 89 45 c8 4d 89 e6 81 e2 88 02 00 00 81 fa 80 02 00 00 0f 84 5c 01 00 00 41 f7 45 00 00 80 00 00 75 40 <49> 8b bd 88 00 00 00 8b 47 04 ff c8 89 47 04 75 2f 48 c7 47 08 00 [ 7163.860663] RSP: 002b:00007ffdfcb6a820 EFLAGS: 00000246 [ 7163.860673] RAX: 0000000000000d68 RBX: 00007c3fc39757e0 RCX: 0000000000000000 [ 7163.860678] RDX: 0000000000000080 RSI: 00007c3fc3975864 RDI: 00007c3fc3978320 [ 7163.860685] RBP: 00007ffdfcb6a860 R08: 00007c3fc3976500 R09: 0000000000000000 [ 7163.860690] R10: 000063939c42a5b0 R11: 0000000000000246 R12: 00007c3fc3976700 [ 7163.860696] R13: 00007c3fc3976500 R14: 00007c3fc3976700 R15: 00007c3fc3977468 [ 7163.860703] FS: 00007c3fc3582740 GS: 0000000000000000 [ 7163.860707] NMI backtrace for cpu 11 [ 7163.860710] CPU: 11 PID: 1244 Comm: dmesg Kdump: loaded Tainted: G U 5.2.0-g0ecfebd2b524 #22 [ 7163.860715] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 7163.860718] RIP: 0033:0x7c7ea5550a75 [ 7163.860726] Code: 8d 05 07 b1 12 00 48 0f 44 c8 48 89 f2 48 89 fe 31 ff e9 0e 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 68 48 89 fb 49 89 f4 [ 7163.860730] RSP: 002b:00007ffe25d18450 EFLAGS: 00000202 [ 7163.860737] RAX: 0000571ede0f0455 RBX: 0000000000000010 RCX: 00007ffe25d18490 [ 7163.860742] RDX: 0000000000000031 RSI: 0000571ede0f0465 RDI: 00007ffe25d1848c [ 7163.860747] RBP: 0000571ede0f0465 R08: 0000000000000001 R09: 0000000000000000 [ 7163.860751] R10: 00007ffe25d1848c R11: 0000571ede0f0464 R12: 00007c7ea5679500 [ 7163.860754] R13: 00007ffe25d18490 R14: 0000571ede0eb071 R15: 0000000000000001 [ 7163.860764] FS: 00007c7ea548a740 GS: 0000000000000000 ``` As you can see Comment 8 got hillariously botched! (must be my usage of markdown) https://bugzilla.mozilla.org/show_bug.cgi?id=1557932 (I mean, if you look at what you've received as notifications in email, for each of my comments, you see how they differ with what is in here!) ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, July 15, 2019 11:25 PM, Andrew Morton <akpm@linux-foundation.org> wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). Roger that. > > On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > > > https://bugzilla.kernel.org/show_bug.cgi?id=204165 > > > > Bug ID: 204165 > > Summary: 100% CPU usage in compact_zone_order > > > > Looks like we have a lockup in compact_zone() > > > Product: Memory Management > > Version: 2.5 > > Kernel Version: 5.2.0-g0ecfebd2b524 > > Hardware: x86-64 > > OS: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: Page Allocator > > Assignee: akpm@linux-foundation.org > > Reporter: howaboutsynergy@pm.me > > Regression: No > > > > I assume this should be "yes". Did previous kernels exhibit this > behavior or is it new in 5.2? Regression: yes? I'm not sure... tl;dr: seen with kernel linux-stable 5.1.8.r0.g937cc0cc22a2-1 And really not sure if I've seen it before. long read: At least one previous kernel did because I've seen this before in https://bugzilla.kernel.org/show_bug.cgi?id=203833#c1 where I thought it was due to 'teo' governor but it wasn't (I'm using 'menu' gov. now) I didn't mention kernel version there, but according to irc logs where I seek'd help at the time, the date was June 10th 2019 3am, checking which kernel I was running at the time (thanks to my q1q repo. git logs) it was 5.1.8 (stable)kernel to which I updated on Sun Jun 9 10:50:09 2019 +0200 and have not changed until Tue Jun 11 05:16:59 2019 +0200 local/linux-stable 5.1.8.r0.g937cc0cc22a2-1 (builtbydaddy) Side note: I've encountered something similar here https://github.com/constantoverride/qubes-linux-kernel/issues/2 where kworker would randomly start using 100% CPU and couldn't be killed. The kernel there was 4.18.7, the VM had very little RAM in Qubes there. Since it's probably a different bug due to that stacktrace being so different/unrelated, please ignore. Created attachment 283715 [details] publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, July 15, 2019 11:25 PM, Andrew Morton <akpm@linux-foundation.org> wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > > > https://bugzilla.kernel.org/show_bug.cgi?id=204165 > > > > Bug ID: 204165 > > Summary: 100% CPU usage in compact_zone_order > > > > Looks like we have a lockup in compact_zone() > > > Product: Memory Management > > Version: 2.5 > > Kernel Version: 5.2.0-g0ecfebd2b524 > > Hardware: x86-64 > > OS: Linux > > Tree: Mainline > > Status: NEW > > Severity: normal > > Priority: P1 > > Component: Page Allocator > > Assignee: akpm@linux-foundation.org > > Reporter: howaboutsynergy@pm.me > > Regression: No > > > > I assume this should be "yes". Did previous kernels exhibit this > behavior or is it new in 5.2? Is the reason that you switched to email somehow related to bugzilla not even posting this very reply of yours(seen above) that you sent via email even though you included bugzilla in the CC ?! I noticed that only mine got posted and mine was after yours in the emails. Yet there's no trace of yours when looking at the issue through the bugzilla interface. Your (emailed)comment should be right after this https://bugzilla.kernel.org/show_bug.cgi?id=204165#c11 but it's not. Anyway, I'm ready to test stuff, please let me know how I can help. Meanwhile I'll try to find a better way to replicate the issue. Created attachment 283717 [details]
signature.asc
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, July 16, 2019 1:28 AM, <howaboutsynergy@protonmail.com> wrote: > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Monday, July 15, 2019 11:25 PM, Andrew Morton akpm@linux-foundation.org > wrote: > > > (switched to email. Please respond via emailed reply-to-all, not via the > > bugzilla web interface). > > Roger that. > > > On Sat, 13 Jul 2019 19:20:21 +0000 bugzilla-daemon@bugzilla.kernel.org > wrote: > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=204165 > > > > > > Bug ID: 204165 > > > Summary: 100% CPU usage in compact_zone_order > > > > > > > Looks like we have a lockup in compact_zone() > > > > > Product: Memory Management > > > Version: 2.5 > > > Kernel Version: 5.2.0-g0ecfebd2b524 > > > Hardware: x86-64 > > > OS: Linux > > > Tree: Mainline > > > Status: NEW > > > Severity: normal > > > Priority: P1 > > > Component: Page Allocator > > > Assignee: akpm@linux-foundation.org > > > Reporter: howaboutsynergy@pm.me > > > Regression: No > > > > > > > I assume this should be "yes". Did previous kernels exhibit this > > behavior or is it new in 5.2? > > Regression: yes? > I'm not sure... > tl;dr: seen with kernel linux-stable 5.1.8.r0.g937cc0cc22a2-1 > And really not sure if I've seen it before. > > long read: > At least one previous kernel did because I've seen this before in > https://bugzilla.kernel.org/show_bug.cgi?id=203833#c1 where I thought it was > due to 'teo' governor but it wasn't (I'm using 'menu' gov. now) > > I didn't mention kernel version there, but according to irc logs where I > seek'd help at the time, the date was June 10th 2019 3am, checking which > kernel I was running at the time (thanks to my q1q repo. git logs) it was > 5.1.8 (stable)kernel to which I updated on Sun Jun 9 10:50:09 2019 +0200 and > have not changed until Tue Jun 11 05:16:59 2019 +0200 > local/linux-stable 5.1.8.r0.g937cc0cc22a2-1 (builtbydaddy) > > Side note: > I've encountered something similar here > https://github.com/constantoverride/qubes-linux-kernel/issues/2 > where kworker would randomly start using 100% CPU and couldn't be killed. The > kernel there was 4.18.7, the VM had very little RAM in Qubes there. Since > it's probably a different bug due to that stacktrace being so > different/unrelated, please ignore. Well after a maybe 50+ tries, got it triggered on kernel 5.2.1-g527a3db363a3 ``` $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [19317] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: FAIL: [19317] (415) <-- worker 19536 got signal 9 stress: WARN: [19317] (417) now reaping child worker processes stress: FAIL: [19317] (415) <-- worker 19530 got signal 9 stress: WARN: [19317] (417) now reaping child worker processes ``` MiB Mem : 31745.3 total, 28945.2 free, 2055.4 used, 744.7 buff/cache MiB Swap: 65536.0 total, 63410.3 free, 2125.7 used. 28774.5 avail Mem PID %CPU COMMAND PR NI VIRT RES S USER 19335 100.0 stress 20 0 9769416 12 R user 19376 100.0 stress 20 0 9769416 0 R user 19418 100.0 stress 20 0 9769416 12 R user 19490 100.0 stress 20 0 9769416 8 R user 21020 100.0 dmesg 20 0 6112 2636 R user ``` [ 1572.246553] sysrq: Show backtrace of all active CPUs [ 1572.246563] NMI backtrace for cpu 8 [ 1572.246572] CPU: 8 PID: 0 Comm: swapper/8 Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.246579] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.246582] Call Trace: [ 1572.246589] <IRQ> [ 1572.246603] dump_stack+0x46/0x60 [ 1572.246615] nmi_cpu_backtrace.cold+0x14/0x53 [ 1572.246623] ? lapic_can_unplug_cpu.cold+0x42/0x42 [ 1572.246634] nmi_trigger_cpumask_backtrace+0x8e/0x90 [ 1572.246641] __handle_sysrq.cold+0x48/0x102 [ 1572.246650] sysrq_filter+0x2ea/0x3b0 [ 1572.246658] input_to_handler+0x4d/0xf0 [ 1572.246668] input_pass_values.part.0+0x109/0x130 [ 1572.246675] input_handle_event+0x171/0x5a0 [ 1572.246684] input_event+0x4d/0x70 [ 1572.246694] hidinput_report_event+0x2e/0x40 [ 1572.246703] hid_report_raw_event+0x260/0x430 [ 1572.246710] hid_input_report+0xfb/0x150 [ 1572.246721] hid_irq_in+0x168/0x190 [ 1572.246730] __usb_hcd_giveback_urb+0x77/0xe0 [ 1572.246754] xhci_giveback_urb_in_irq.isra.0+0x62/0x90 [xhci_hcd] [ 1572.246777] xhci_td_cleanup+0xf7/0x140 [xhci_hcd] [ 1572.246799] xhci_irq+0x7e8/0x1be0 [xhci_hcd] [ 1572.246811] __handle_irq_event_percpu+0x2f/0xc0 [ 1572.246820] handle_irq_event_percpu+0x2c/0x80 [ 1572.246827] handle_irq_event+0x23/0x43 [ 1572.246838] handle_edge_irq+0x78/0x190 [ 1572.246846] handle_irq+0x17/0x20 [ 1572.246857] do_IRQ+0x3e/0xd0 [ 1572.246865] common_interrupt+0xf/0xf [ 1572.246872] </IRQ> [ 1572.246880] RIP: 0010:cpuidle_enter_state+0x11f/0x2a0 [ 1572.246890] Code: e8 d6 1c a1 ff 49 89 c6 31 ff e8 fc 2c a1 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 65 01 00 00 31 ff e8 d5 18 a6 ff fb 45 85 ed <0f> 88 c0 00 00 00 49 63 f5 48 8d 04 76 48 c1 e0 05 8b 7c 03 4c 4c [ 1572.246895] RSP: 0018:ffffbd38c3217e80 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffde [ 1572.246904] RAX: ffff9699ada5cb00 RBX: ffffffff93062340 RCX: 000000000000001f [ 1572.246909] RDX: 0000000000000000 RSI: 0000000022a1cd05 RDI: 0000000000000000 [ 1572.246916] RBP: 0000016e1126287d R08: 0000016e11320f1b R09: 000000007fffffff [ 1572.246921] R10: ffff9699ada5bc44 R11: ffff9699ada5bc24 R12: ffff9699ada65b00 [ 1572.246925] R13: 0000000000000006 R14: 0000016e11320f1b R15: 0000000000000000 [ 1572.246936] cpuidle_enter+0x24/0x40 [ 1572.246945] do_idle+0x1c1/0x230 [ 1572.246955] cpu_startup_entry+0x14/0x20 [ 1572.246963] start_secondary+0x168/0x1b0 [ 1572.246972] secondary_startup_64+0xa4/0xb0 [ 1572.246981] Sending NMI from CPU 8 to CPUs 0-7,9-11: [ 1572.247108] NMI backtrace for cpu 0 [ 1572.247110] CPU: 0 PID: 19376 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247111] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247112] RIP: 0010:compact_zone+0x3cf/0xa70 [ 1572.247115] Code: 7f 70 02 74 07 83 ce 04 89 74 24 50 45 0f b6 6f 74 48 8b 55 60 45 84 ed 0f 85 5d 02 00 00 48 39 d0 0f 84 3d 02 00 00 48 89 c7 <48> 81 e7 00 fe ff ff a9 ff 01 00 00 0f 84 28 02 00 00 49 89 c4 48 [ 1572.247116] RSP: 0018:ffffbd38cfca7a18 EFLAGS: 00000202 [ 1572.247118] RAX: 000000000022c720 RBX: 000000000022ca00 RCX: 0000000000001163 [ 1572.247119] RDX: 0000000000100000 RSI: 000000000000000c RDI: 000000000022c720 [ 1572.247122] RBP: ffff9699cdfded00 R08: 0000000000000001 R09: ffff9699ad65d550 [ 1572.247123] R10: 0000000000000001 R11: ffff9699ad65cb80 R12: 0000000000000009 [ 1572.247124] R13: 0000000000000000 R14: 000000000022c720 R15: ffffbd38cfca7ad0 [ 1572.247125] FS: 0000710f1082f740(0000) GS:ffff9699ad600000(0000) knlGS:0000000000000000 [ 1572.247126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.247127] CR2: 000056d077f9cea8 CR3: 000000027ed88005 CR4: 00000000003606f0 [ 1572.247128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.247129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.247130] Call Trace: [ 1572.247130] compact_zone_order+0xde/0x120 [ 1572.247131] try_to_compact_pages+0x187/0x240 [ 1572.247132] __alloc_pages_direct_compact+0x87/0x170 [ 1572.247133] __alloc_pages_slowpath+0x454/0xc20 [ 1572.247134] ? release_pages+0x348/0x3b0 [ 1572.247134] __alloc_pages_nodemask+0x268/0x2b0 [ 1572.247137] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1572.247138] __handle_mm_fault+0xc0c/0x1310 [ 1572.247139] handle_mm_fault+0xa9/0x1d0 [ 1572.247139] __do_page_fault+0x237/0x480 [ 1572.247140] do_page_fault+0x1d/0x67 [ 1572.247141] ? page_fault+0x8/0x30 [ 1572.247142] page_fault+0x1e/0x30 [ 1572.247143] RIP: 0033:0x5f7c81c80c10 [ 1572.247145] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1572.247145] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206 [ 1572.247147] RAX: 0000000016e90000 RBX: 0000710cbc770010 RCX: 0000710f109543db [ 1572.247148] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000 [ 1572.247149] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000 [ 1572.247152] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1572.247153] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1572.247212] NMI backtrace for cpu 1 [ 1572.247214] CPU: 1 PID: 19335 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247216] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247217] RIP: 0010:isolate_migratepages_block+0x267/0xa50 [ 1572.247219] Code: 00 00 00 c4 e2 f9 f7 c6 48 8d 5c 03 ff 45 84 ed 0f 85 17 ff ff ff 48 ff c3 48 39 5c 24 08 0f 87 46 ff ff ff 4d 89 d7 4d 89 e6 <48> 89 d8 0f b6 54 24 77 48 3b 5c 24 08 0f 87 25 02 00 00 84 d2 0f [ 1572.247220] RSP: 0018:ffffbd38cfb5f958 EFLAGS: 00000202 [ 1572.247223] RAX: 0000000000000001 RBX: 000000000018c060 RCX: ffffbd38cfb5fb49 [ 1572.247224] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffff9699cdfffb00 [ 1572.247225] RBP: ffff9699cdfde000 R08: 0000000000000001 R09: ffff9699ad6dd550 [ 1572.247226] R10: 0000000000000000 R11: ffff969946236600 R12: ffffbd38cfb5fad0 [ 1572.247230] R13: 0000000000000000 R14: ffffbd38cfb5fad0 R15: 0000000000000000 [ 1572.247232] FS: 0000710f1082f740(0000) GS:ffff9699ad680000(0000) knlGS:0000000000000000 [ 1572.247233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.247234] CR2: 00007f6eb6d40faa CR3: 000000080b004002 CR4: 00000000003606e0 [ 1572.247235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.247237] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.247237] Call Trace: [ 1572.247238] compact_zone+0x577/0xa70 [ 1572.247239] compact_zone_order+0xde/0x120 [ 1572.247241] try_to_compact_pages+0x187/0x240 [ 1572.247242] __alloc_pages_direct_compact+0x87/0x170 [ 1572.247248] __alloc_pages_slowpath+0x454/0xc20 [ 1572.247259] ? release_pages+0x348/0x3b0 [ 1572.247265] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1572.247266] __alloc_pages_nodemask+0x268/0x2b0 [ 1572.247267] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1572.247268] __handle_mm_fault+0xc0c/0x1310 [ 1572.247269] handle_mm_fault+0xa9/0x1d0 [ 1572.247270] __do_page_fault+0x237/0x480 [ 1572.247272] do_page_fault+0x1d/0x67 [ 1572.247273] ? page_fault+0x8/0x30 [ 1572.247275] page_fault+0x1e/0x30 [ 1572.247276] RIP: 0033:0x5f7c81c80c10 [ 1572.247280] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1572.247281] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206 [ 1572.247287] RAX: 0000000015c90000 RBX: 0000710cbc770010 RCX: 0000710f109543db [ 1572.247289] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000 [ 1572.247290] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000 [ 1572.247292] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1572.247293] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1572.247321] NMI backtrace for cpu 2 [ 1572.247328] CPU: 2 PID: 19490 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247331] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247335] RIP: 0010:isolate_migratepages_block+0x1c/0xa50 [ 1572.247339] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 49 89 fe 48 89 f3 48 89 54 24 08 <89> 4c 24 3c 65 48 8b 04 25 28 00 00 00 48 89 84 24 80 00 00 00 31 [ 1572.247346] RSP: 0018:ffffbd3929043958 EFLAGS: 00000292 [ 1572.247358] RAX: 0000000000000000 RBX: 000000000062b760 RCX: 000000000000000c [ 1572.247361] RDX: 000000000062b800 RSI: 000000000062b760 RDI: ffffbd3929043ad0 [ 1572.247364] RBP: ffffbd3929043ad0 R08: 0000000000000001 R09: ffff9699ad75d550 [ 1572.247368] R10: 0000000000000001 R11: ffff9699ad75cb80 R12: 000000000062b800 [ 1572.247371] R13: ffff9699cdfded00 R14: ffffbd3929043ad0 R15: ffffbd3929043ad0 [ 1572.247377] FS: 0000710f1082f740(0000) GS:ffff9699ad700000(0000) knlGS:0000000000000000 [ 1572.247380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.247390] CR2: 000077de6c7b3ec4 CR3: 000000080bde2005 CR4: 00000000003606e0 [ 1572.247393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.247397] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.247402] Call Trace: [ 1572.247409] compact_zone+0x577/0xa70 [ 1572.247415] compact_zone_order+0xde/0x120 [ 1572.247418] try_to_compact_pages+0x187/0x240 [ 1572.247423] __alloc_pages_direct_compact+0x87/0x170 [ 1572.247426] __alloc_pages_slowpath+0x454/0xc20 [ 1572.247432] ? release_pages+0x348/0x3b0 [ 1572.247439] ? __pagevec_lru_add_fn+0x189/0x2a0 [ 1572.247442] __alloc_pages_nodemask+0x268/0x2b0 [ 1572.247448] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1572.247450] __handle_mm_fault+0xc0c/0x1310 [ 1572.247455] handle_mm_fault+0xa9/0x1d0 [ 1572.247460] __do_page_fault+0x237/0x480 [ 1572.247464] do_page_fault+0x1d/0x67 [ 1572.247468] ? page_fault+0x8/0x30 [ 1572.247473] page_fault+0x1e/0x30 [ 1572.247476] RIP: 0033:0x5f7c81c80c10 [ 1572.247482] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1572.247488] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206 [ 1572.247494] RAX: 0000000019c90000 RBX: 0000710cbc770010 RCX: 0000710f109543db [ 1572.247500] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000 [ 1572.247504] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000 [ 1572.247507] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1572.247511] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1572.247516] NMI backtrace for cpu 5 [ 1572.247520] CPU: 5 PID: 1273 Comm: dmesg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247525] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247530] RIP: 0033:0x7de7c0c81388 [ 1572.247534] Code: 85 e4 fa ff ff 00 00 00 00 c7 85 e0 fa ff ff 00 00 00 00 e9 38 f4 ff ff 66 2e 0f 1f 84 00 00 00 00 00 48 83 bd e8 fa ff ff 00 <0f> 84 5c 0c 00 00 0f b6 85 dc fa ff ff 83 e0 01 83 bd f0 fa ff ff [ 1572.247537] RSP: 002b:00007ffeedcb2aa0 EFLAGS: 00000206 [ 1572.247546] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000004 [ 1572.247551] RDX: 0000000000000000 RSI: 00007de7c0dc95c0 RDI: 0000000000000000 [ 1572.247557] RBP: 00007ffeedcb3010 R08: 00007ffeedcb2fd4 R09: 0000000000000001 [ 1572.247560] R10: 0000560441280093 R11: 00007ffeedcb2fd4 R12: 00007ffeedcb3030 [ 1572.247563] R13: 000056044128008f R14: 00007ffeedcb31b0 R15: 0000000000000064 [ 1572.247568] FS: 00007de7c0c01740 GS: 0000000000000000 [ 1572.247575] NMI backtrace for cpu 4 [ 1572.247580] CPU: 4 PID: 1245 Comm: dmesg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247584] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247587] RIP: 0010:_raw_spin_lock+0xb/0x20 [ 1572.247690] Code: 81 05 6d b7 81 6d 00 02 00 00 31 c0 ba ff 00 00 00 f0 0f b1 17 75 01 c3 e9 92 14 8f ff 66 90 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 01 c3 89 c6 e9 1b 04 8f ff 66 66 2e 0f 1f 84 00 00 00 00 00 65 [ 1572.247801] RSP: 0018:ffffbd38c546bcb8 EFLAGS: 00000097 [ 1572.247812] RAX: 0000000000000001 RBX: ffff96994b4e5ac0 RCX: 000000000000000a [ 1572.247815] RDX: 0000000000000001 RSI: ffff9699ad800000 RDI: ffff9699adb5cb00 [ 1572.247818] RBP: ffff9699adb5cb00 R08: 000000000000000a R09: ffffbd38c546bd90 [ 1572.247821] R10: 0000000000000000 R11: 0000000000000004 R12: 0000000000000001 [ 1572.247827] R13: 0000000000000046 R14: ffff96994b4e6204 R15: 000000000005cb00 [ 1572.247830] FS: 000077a36f17e740(0000) GS:ffff9699ad800000(0000) knlGS:0000000000000000 [ 1572.247833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.247837] CR2: 000072d7ae4d9578 CR3: 00000007ca59c006 CR4: 00000000003606e0 [ 1572.247840] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.247847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.247850] Call Trace: [ 1572.247853] try_to_wake_up+0x18d/0x500 [ 1572.247855] autoremove_wake_function+0xc/0x50 [ 1572.247858] __wake_up_common+0x7a/0x190 [ 1572.247861] __wake_up_common_lock+0x79/0xc0 [ 1572.247864] pipe_write+0x1e3/0x410 [ 1572.247869] new_sync_write+0x116/0x1b0 [ 1572.247875] vfs_write+0xb1/0x190 [ 1572.247878] ksys_write+0x5d/0xe0 [ 1572.247881] do_syscall_64+0x50/0x170 [ 1572.247884] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1572.247886] RIP: 0033:0x77a36f2984ba [ 1572.247894] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 18 b8 01 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 [ 1572.247897] RSP: 002b:00007fffd15cf728 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1572.247903] RAX: ffffffffffffffda RBX: 0000000000000029 RCX: 000077a36f2984ba [ 1572.247906] RDX: 0000000000000029 RSI: 00006261bcaad3c0 RDI: 0000000000000001 [ 1572.247909] RBP: 00007fffd15cf750 R08: 000000000000000a R09: 0000000000000000 [ 1572.247912] R10: 00007fffd15cf77c R11: 0000000000000246 R12: 0000000000000029 [ 1572.247918] R13: 00006261bcaad3c0 R14: 000077a36f36d500 R15: 000077a36f36e300 [ 1572.247922] NMI backtrace for cpu 3 [ 1572.247926] CPU: 3 PID: 19418 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.247929] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.247932] RIP: 0010:isolate_migratepages_block+0xad/0xa50 [ 1572.247938] Code: 89 ef e8 06 91 ff ff 49 89 c7 be 01 00 00 00 48 89 ef e8 f6 90 ff ff 49 89 c5 be 08 00 00 00 48 89 ef e8 e6 90 ff ff 49 89 c4 <be> 07 00 00 00 48 89 ef e8 d6 90 ff ff 49 01 c4 4c 03 3c 24 4d 01 [ 1572.247944] RSP: 0018:ffffbd38cfdf7958 EFLAGS: 00000246 [ 1572.247952] RAX: 0000000000000000 RBX: 00000000006f1ba0 RCX: 000000000000000c [ 1572.247955] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9699cdfde000 [ 1572.247958] RBP: ffff9699cdfde000 R08: 0000000000000001 R09: ffff9699ad7dd550 [ 1572.247964] R10: 0000000000000001 R11: ffff9699ad7dcb80 R12: 0000000000000000 [ 1572.247969] R13: 00000000000012b2 R14: ffffbd38cfdf7ad0 R15: 00000000000018a8 [ 1572.247972] FS: 0000710f1082f740(0000) GS:ffff9699ad780000(0000) knlGS:0000000000000000 [ 1572.247974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.247977] CR2: 000072d7ae4d9578 CR3: 00000007c59c4001 CR4: 00000000003606e0 [ 1572.247980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.247983] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.247987] Call Trace: [ 1572.247990] compact_zone+0x577/0xa70 [ 1572.247992] compact_zone_order+0xde/0x120 [ 1572.247995] try_to_compact_pages+0x187/0x240 [ 1572.248000] __alloc_pages_direct_compact+0x87/0x170 [ 1572.248004] __alloc_pages_slowpath+0x454/0xc20 [ 1572.248007] __alloc_pages_nodemask+0x268/0x2b0 [ 1572.248010] do_huge_pmd_anonymous_page+0x131/0x5c0 [ 1572.248013] ? mem_cgroup_throttle_swaprate+0x20/0x113 [ 1572.248018] __handle_mm_fault+0xc0c/0x1310 [ 1572.248023] handle_mm_fault+0xa9/0x1d0 [ 1572.248026] __do_page_fault+0x237/0x480 [ 1572.248030] do_page_fault+0x1d/0x67 [ 1572.248035] ? page_fault+0x8/0x30 [ 1572.248039] page_fault+0x1e/0x30 [ 1572.248042] RIP: 0033:0x5f7c81c80c10 [ 1572.248046] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1572.248050] RSP: 002b:00007ffdb884d190 EFLAGS: 00010206 [ 1572.248058] RAX: 0000000026a90000 RBX: 0000710cbc770010 RCX: 0000710f109543db [ 1572.248060] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000710cbc770000 [ 1572.248065] RBP: 00005f7c81c81a54 R08: 0000710cbc770010 R09: 0000000000000000 [ 1572.248070] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1572.248073] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1572.248078] NMI backtrace for cpu 6 [ 1572.248082] CPU: 6 PID: 1258 Comm: dmesg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.248087] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.248092] RIP: 0033:0x7cfec2fbca97 [ 1572.248097] Code: 00 00 00 83 e1 1f 48 83 e7 e0 c5 7e 6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1 d3 f8 85 c0 74 1d f3 0f bc c0 <48> 01 c8 31 d2 48 8d 04 07 40 3a 30 48 0f 45 c2 c5 f8 77 c3 0f 1f [ 1572.248102] RSP: 002b:00007fffc45dab98 EFLAGS: 00000202 [ 1572.248110] RAX: 0000000000000001 RBX: 0000582b21b12475 RCX: 000000000000001a [ 1572.248113] RDX: 0000000000000000 RSI: 000000000000005f RDI: 0000582b21b0e7e0 [ 1572.248118] RBP: 0000582b21b0e7fa R08: 000000000000000a R09: 0000000000000000 [ 1572.248123] R10: 00007fffc45da9ac R11: 0000000000000246 R12: 0000582b21b12459 [ 1572.248126] R13: 0000582b21b0d377 R14: 0000582b21b12440 R15: 0000582b21b12456 [ 1572.248131] FS: 00007cfec2e27740 GS: 0000000000000000 [ 1572.248134] NMI backtrace for cpu 7 [ 1572.248137] CPU: 7 PID: 1249 Comm: dmesg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.248143] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.248145] RIP: 0010:number+0x234/0x360 [ 1572.248149] Code: c6 00 20 48 ff c0 4c 39 c0 75 f0 48 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 31 01 00 00 4c 89 c0 48 83 c4 38 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 83 e5 ef 44 88 7c 24 16 e9 2e fe ff ff 48 f7 [ 1572.248152] RSP: 0018:ffffbd38c541bd40 EFLAGS: 00000086 [ 1572.248163] RAX: ffff96994b6f0059 RBX: ffffbd38c541bdb8 RCX: 00000000fffffffb [ 1572.248168] RDX: ffffbd38c541bd07 RSI: 0000000000000034 RDI: ffff969a4b6f0056 [ 1572.248170] RBP: ffff96994b6f2058 R08: ffff96994b6f0059 R09: 0000000000000000 [ 1572.248172] R10: 0000000000000004 R11: ffff96994b6f0059 R12: ffffffff92e33a07 [ 1572.248179] R13: 0000000000000000 R14: 0000000000ffff0a R15: 0000000000000000 [ 1572.248183] FS: 00007ef911609740(0000) GS:ffff9699ad980000(0000) knlGS:0000000000000000 [ 1572.248186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.248188] CR2: 00007d5d665cfae0 CR3: 00000007cb6ec002 CR4: 00000000003606e0 [ 1572.248192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.248199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.248201] Call Trace: [ 1572.248203] vsnprintf+0x3e1/0x5b0 [ 1572.248207] scnprintf+0x4d/0x90 [ 1572.248212] msg_print_ext_header.constprop.0+0x7f/0xa0 [ 1572.248217] ? _cond_resched+0x10/0x20 [ 1572.248219] ? mutex_lock_interruptible+0x9/0x30 [ 1572.248225] devkmsg_read+0x17f/0x270 [ 1572.248227] vfs_read+0x98/0x150 [ 1572.248232] ksys_read+0x5d/0xe0 [ 1572.248234] do_syscall_64+0x50/0x170 [ 1572.248241] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1572.248245] RIP: 0033:0x7ef911723415 [ 1572.248251] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1572.248255] RSP: 002b:00007ffcf81cb498 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1572.248262] RAX: ffffffffffffffda RBX: 00005571c56a2440 RCX: 00007ef911723415 [ 1572.248267] RDX: 0000000000001fff RSI: 00005571c56a2440 RDI: 0000000000000003 [ 1572.248271] RBP: 00005571c569e7fa R08: 000000000000000a R09: 0000000000000000 [ 1572.248273] R10: 00007ffcf81cb2cc R11: 0000000000000246 R12: 000000000000001c [ 1572.248278] R13: 00005571c569d377 R14: 00005571c56a2440 R15: 00005571c56a2456 [ 1572.248285] NMI backtrace for cpu 9 [ 1572.248288] CPU: 9 PID: 400 Comm: systemd-journal Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.248292] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.248300] RIP: 0010:queued_spin_lock_slowpath+0x17b/0x1d0 [ 1572.248307] Code: e5 83 e0 03 48 c1 e0 04 48 05 c0 d7 05 00 c1 ee 12 ff ce 48 63 f6 48 03 04 f5 a0 15 ed 92 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 02 48 85 c0 74 8c 48 89 c6 0f 0d 08 eb [ 1572.248310] RSP: 0018:ffffbd38c34cbd50 EFLAGS: 00000046 [ 1572.248322] RAX: 0000000000000000 RBX: ffff9699a86f4370 RCX: 0000000000280000 [ 1572.248325] RDX: ffff9699adadd7c0 RSI: 0000000000000005 RDI: ffffffff935a02ac [ 1572.248333] RBP: ffffbd38c34cbda8 R08: 0000000000000009 R09: 0000000000000000 [ 1572.248336] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9699a4980000 [ 1572.248342] R13: 00007ffddcf63680 R14: ffffbd38c34cbe00 R15: ffff9699a86f4318 [ 1572.248346] FS: 00007c88db99c840(0000) GS:ffff9699ada80000(0000) knlGS:0000000000000000 [ 1572.248351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.248359] CR2: 00007c88cf9be3c0 CR3: 00000008286d4001 CR4: 00000000003606e0 [ 1572.248364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.248369] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.248375] Call Trace: [ 1572.248382] devkmsg_poll+0x3c/0x80 [ 1572.248388] ep_item_poll.isra.0+0x3a/0xb0 [ 1572.248391] ep_send_events_proc+0xec/0x230 [ 1572.248394] ? ep_read_events_proc+0xe0/0xe0 [ 1572.248403] ep_scan_ready_list.constprop.0+0x9b/0x1f0 [ 1572.248408] ep_poll+0x87/0x450 [ 1572.248412] do_epoll_wait+0xab/0xd0 [ 1572.248419] __x64_sys_epoll_wait+0x15/0x20 [ 1572.248424] do_syscall_64+0x50/0x170 [ 1572.248428] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1572.248435] RIP: 0033:0x7c88dd1b20b5 [ 1572.248439] Code: 89 55 f8 48 89 75 f0 89 7d fc e8 e6 18 f7 ff 41 89 c0 44 8b 55 ec 8b 55 f8 48 8b 75 f0 8b 7d fc b8 e8 00 00 00 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 25 89 45 fc 44 89 c7 e8 18 19 f7 ff 8b 45 fc [ 1572.248444] RSP: 002b:00007ffddcf63650 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8 [ 1572.248458] RAX: ffffffffffffffda RBX: 00006265e2ccc3d0 RCX: 00007c88dd1b20b5 [ 1572.248461] RDX: 0000000000000020 RSI: 00007ffddcf63680 RDI: 0000000000000007 [ 1572.248465] RBP: 00007ffddcf63670 R08: 0000000000000000 R09: 0000000000000020 [ 1572.248471] R10: 00000000dd84bf82 R11: 0000000000000293 R12: 00007ffddcf63680 [ 1572.248478] R13: 000007494e8c1088 R14: 0000000000000000 R15: 0000000000000001 [ 1572.248486] NMI backtrace for cpu 11 [ 1572.248490] CPU: 11 PID: 20040 Comm: dmesg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.248493] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.248502] RIP: 0033:0x7717bcea7cbb [ 1572.248509] Code: 48 83 ec 18 49 89 f5 4c 0f af ea 4d 85 ed 0f 84 f0 00 00 00 49 89 ff 49 89 f6 49 89 d4 48 89 cb 8b 11 81 e2 00 80 00 00 75 44 <64> 48 8b 04 25 10 00 00 00 48 89 45 c8 48 8b b9 88 00 00 00 48 39 [ 1572.248518] RSP: 002b:00007fff79d5f6a0 EFLAGS: 00000246 [ 1572.248531] RAX: 0000000000000001 RBX: 00007717bd00b500 RCX: 00007717bd00b500 [ 1572.248534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000062592cea0487 [ 1572.248538] RBP: 00007fff79d5f6e0 R08: 000062592cea0488 R09: 0000000000000000 [ 1572.248544] R10: 00007fff79d5f71c R11: 000062592cea0487 R12: 0000000000000001 [ 1572.248552] R13: 0000000000000001 R14: 0000000000000001 R15: 000062592cea0487 [ 1572.248557] FS: 00007717bce1c740 GS: 0000000000000000 [ 1572.248563] NMI backtrace for cpu 10 [ 1572.248567] CPU: 10 PID: 1260 Comm: grep Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #68 [ 1572.248570] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018 [ 1572.248577] RIP: 0010:ZSTD_decompressSequences+0x6be/0xb80 [ 1572.248586] Code: 8b 39 48 89 7c 24 68 84 c0 74 13 f3 0f bd c0 8d 48 e9 89 ce 49 83 fc ea 0f 86 2c fb ff ff 49 c7 c5 f2 ff ff ff e9 21 fa ff ff <89> d0 44 8b 34 85 00 f0 c7 92 44 8b 54 24 70 c4 e2 a9 f7 44 24 68 [ 1572.248589] RSP: 0018:ffffbd38c5a37838 EFLAGS: 00000202 [ 1572.248599] RAX: 0000000000000008 RBX: 0000000000000000 RCX: ffffbd38c6022048 [ 1572.248606] RDX: 0000000000000008 RSI: ffffbd38c6022d30 RDI: ffffbd38c60228a4 [ 1572.248611] RBP: 0000000000000000 R08: 0000000000000099 R09: 0000000000000001 [ 1572.248617] R10: 0000000000000007 R11: 0000000000000006 R12: 0000000000000001 [ 1572.248622] R13: 0000000000000006 R14: fffffffffffffffe R15: ffff969928146263 [ 1572.248628] FS: 000077c96a004740(0000) GS:ffff9699adb00000(0000) knlGS:0000000000000000 [ 1572.248635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1572.248642] CR2: 00005a61235ce000 CR3: 00000007cba90004 CR4: 00000000003606e0 [ 1572.248648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1572.248659] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1572.248666] Call Trace: [ 1572.248672] ZSTD_decompressMultiFrame+0x329/0x370 [ 1572.248677] ZSTD_decompressDCtx+0xc/0x10 [ 1572.248680] zstd_decompress+0x1f/0x40 [ 1572.248685] zcomp_decompress+0x2e/0x50 [ 1572.248690] zram_bvec_rw.isra.0+0x5e2/0x6a0 [ 1572.248699] zram_rw_page+0x7f/0xe0 [ 1572.248705] bdev_read_page+0x6e/0xa0 [ 1572.248710] swap_readpage+0xaf/0x1f0 [ 1572.248713] do_swap_page+0x72e/0x7e0 [ 1572.248718] __handle_mm_fault+0xa77/0x1310 [ 1572.248722] handle_mm_fault+0xa9/0x1d0 [ 1572.248725] __do_page_fault+0x237/0x480 [ 1572.248734] do_page_fault+0x1d/0x67 [ 1572.248741] ? reweight_entity+0x15a/0x1a0 [ 1572.248744] page_fault+0x1e/0x30 [ 1572.248749] RIP: 0010:copy_user_generic_unrolled+0x89/0xc0 [ 1572.248758] Code: 38 4c 89 47 20 4c 89 4f 28 4c 89 57 30 4c 89 5f 38 48 8d 76 40 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 4c 8b 06 <4c> 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10 89 d1 8a [ 1572.248766] RSP: 0018:ffffbd38c5a37d90 EFLAGS: 00050203 [ 1572.248780] RAX: 0000000000000016 RBX: ffff96972af7c000 RCX: 0000000000000001 [ 1572.248783] RDX: 0000000000000006 RSI: ffff96972af7c008 RDI: 00005a61235cdffe [ 1572.248789] RBP: 0000000000000016 R08: 363478302b6b6361 R09: 0000000000000000 [ 1572.248794] R10: 0000000000000003 R11: ffff9699a2261e00 R12: 0000000000000016 [ 1572.248800] R13: ffffbd38c5a37e50 R14: 0000000000000000 R15: 0000000000000000 [ 1572.248803] copyout+0x28/0x30 [ 1572.248806] copy_page_to_iter+0xbc/0x2f0 [ 1572.248809] pipe_read+0xad/0x2c0 [ 1572.248814] new_sync_read+0x10f/0x1a0 [ 1572.248821] vfs_read+0x98/0x150 [ 1572.248826] ksys_read+0x5d/0xe0 [ 1572.248832] do_syscall_64+0x50/0x170 [ 1572.248837] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1572.248840] RIP: 0033:0x77c96a140415 [ 1572.248849] Code: fe ff ff 55 48 89 e5 48 8d 3d 4f bf 0a 00 e8 22 25 02 00 66 90 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 c5 fc 77 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 55 48 89 e5 48 83 ec 20 48 89 55 [ 1572.248855] RSP: 002b:00007fff87565c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 1572.248869] RAX: ffffffffffffffda RBX: 000000000000e000 RCX: 000077c96a140415 [ 1572.248877] RDX: 000000000000e000 RSI: 00005a61235cdff6 RDI: 0000000000000000 [ 1572.248886] RBP: 000000000000e000 R08: 0000000000000000 R09: 0000000000019008 [ 1572.248890] R10: 00005a61235b85a0 R11: 0000000000000246 R12: 00005a61235cdff6 [ 1572.248893] R13: 0000000000000000 R14: 00005a61235c3100 R15: 0000000000000000 ``` Ignore the dmesg process, I've had two programs that keep querrying it and I've put process to 800Mhz max, to avoid high power usage during this. the swap is in zram /etc/fstab shows: /dev/zram0 none swap defaults,discard 0 0 $ zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram2 zstd 64G 4.7M 136.3K 248K 12 /var/tmp /dev/zram1 zstd 64G 320.3M 84.1M 88M 12 /tmp /dev/zram0 zstd 64G 1.6G 75.6M 230.6M 12 [SWAP] I can't find a reliable way to reproduce it, it seems to happen sort of random... ```
crash> bt -Tsx 21199
PID: 21199 TASK: ffff8fd7ef4e8000 CPU: 11 COMMAND: "stress"
[ffff9cb9aa1cf160] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf170] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf200] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf210] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf220] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf318] wake_all_kswapds+0x4f at ffffffff981d5e5f
[ffff9cb9aa1cf358] __alloc_pages_slowpath+0x334 at ffffffff981d9b14
[ffff9cb9aa1cf388] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf438] update_group_capacity+0x20 at ffffffff980d2530
[ffff9cb9aa1cf468] cpumask_next_and+0x18 at ffffffff987df928
[ffff9cb9aa1cf470] update_sd_lb_stats+0x159 at ffffffff980d2839
[ffff9cb9aa1cf560] update_load_avg+0x4b1 at ffffffff980cdae1
[ffff9cb9aa1cf568] account_entity_enqueue+0xbf at ffffffff980cc13f
[ffff9cb9aa1cf5b8] reweight_entity+0x15a at ffffffff980cd17a
[ffff9cb9aa1cf5e0] enqueue_task_fair+0x11f at ffffffff980cff0f
[ffff9cb9aa1cf608] check_preempt_wakeup+0xfd at ffffffff980cd39d
[ffff9cb9aa1cf650] check_preempt_curr+0x70 at ffffffff980c2ce0
[ffff9cb9aa1cf668] __update_load_avg_se+0x200 at ffffffff980e05d0
[ffff9cb9aa1cf6b8] update_load_avg+0x42b at ffffffff980cda5b
[ffff9cb9aa1cf6c0] account_entity_enqueue+0xbf at ffffffff980cc13f
[ffff9cb9aa1cf6f8] enqueue_entity+0xe4 at ffffffff980cea64
[ffff9cb9aa1cf700] record_times+0x10 at ffffffff980e8140
[ffff9cb9aa1cf750] check_preempt_wakeup+0x182 at ffffffff980cd422
[ffff9cb9aa1cf798] check_preempt_curr+0x70 at ffffffff980c2ce0
[ffff9cb9aa1cf7a8] ttwu_do_wakeup+0x12 at ffffffff980c2d02
[ffff9cb9aa1cf7c8] __update_load_avg_se+0x200 at ffffffff980e05d0
[ffff9cb9aa1cf810] update_load_avg+0x76 at ffffffff980cd6a6
[ffff9cb9aa1cf850] set_next_entity+0x89 at ffffffff980cdc29
[ffff9cb9aa1cf870] pick_next_task_fair+0x590 at ffffffff980d4a50
[ffff9cb9aa1cf890] apic_timer_interrupt+0xa at ffffffff9880146a
[ffff9cb9aa1cf8a0] apic_timer_interrupt+0xa at ffffffff9880146a
[ffff9cb9aa1cf928] node_page_state+0x15 at ffffffff981ac675
[ffff9cb9aa1cf950] isolate_migratepages_block+0x69 at ffffffff981b3539
[ffff9cb9aa1cfa10] compact_zone+0x577 at ffffffff981b4c07
[ffff9cb9aa1cfab8] compact_zone_order+0xde at ffffffff981b51de
[ffff9cb9aa1cfb78] try_to_compact_pages+0x187 at ffffffff981b5a17
[ffff9cb9aa1cfbd8] __alloc_pages_direct_compact+0x87 at ffffffff981d95a7
[ffff9cb9aa1cfc30] __alloc_pages_slowpath+0x20e at ffffffff981d99ee
[ffff9cb9aa1cfc48] try_charge+0x51a at ffffffff9821044a
[ffff9cb9aa1cfcd0] release_pages+0x348 at ffffffff9819aa08
[ffff9cb9aa1cfd00] __pagevec_lru_add_fn+0x189 at ffffffff9819ada9
[ffff9cb9aa1cfd40] __alloc_pages_nodemask+0x268 at ffffffff981da668
[ffff9cb9aa1cfda0] do_huge_pmd_anonymous_page+0x131 at ffffffff98202c01
[ffff9cb9aa1cfdf0] __handle_mm_fault+0xc0c at ffffffff981bfa4c
[ffff9cb9aa1cfea0] handle_mm_fault+0xa9 at ffffffff981c01f9
[ffff9cb9aa1cfec8] __do_page_fault+0x237 at ffffffff9803d5c7
[ffff9cb9aa1cff28] do_page_fault+0x1d at ffffffff9803d85d
[ffff9cb9aa1cff48] page_fault+0x8 at ffffffff98800de8
[ffff9cb9aa1cff50] page_fault+0x1e at ffffffff98800dfe
RIP: 00005879fc8c4c10 RSP: 00007ffd393aa0d0 RFLAGS: 00010206
RAX: 000000002438a000 RBX: 0000714104876010 RCX: 0000714358a5a3db
RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000714104876000
RBP: 00005879fc8c5a54 R8: 0000714104876010 R9: 0000000000000000
R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash> bt -Tsx 21199
PID: 21199 TASK: ffff8fd7ef4e8000 CPU: 1 COMMAND: "stress"
[ffff9cb9aa1cf160] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf170] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf200] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf210] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf220] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf318] wake_all_kswapds+0x4f at ffffffff981d5e5f
[ffff9cb9aa1cf358] __alloc_pages_slowpath+0x334 at ffffffff981d9b14
[ffff9cb9aa1cf388] get_page_from_freelist+0xe88 at ffffffff981d9078
[ffff9cb9aa1cf438] update_group_capacity+0x20 at ffffffff980d2530
[ffff9cb9aa1cf468] cpumask_next_and+0x18 at ffffffff987df928
[ffff9cb9aa1cf470] update_sd_lb_stats+0x159 at ffffffff980d2839
[ffff9cb9aa1cf560] update_load_avg+0x4b1 at ffffffff980cdae1
[ffff9cb9aa1cf568] account_entity_enqueue+0xbf at ffffffff980cc13f
[ffff9cb9aa1cf5b8] reweight_entity+0x15a at ffffffff980cd17a
[ffff9cb9aa1cf5e0] enqueue_task_fair+0x11f at ffffffff980cff0f
[ffff9cb9aa1cf608] check_preempt_wakeup+0xfd at ffffffff980cd39d
[ffff9cb9aa1cf650] check_preempt_curr+0x70 at ffffffff980c2ce0
[ffff9cb9aa1cf668] __update_load_avg_se+0x200 at ffffffff980e05d0
[ffff9cb9aa1cf6b8] update_load_avg+0x42b at ffffffff980cda5b
[ffff9cb9aa1cf6c0] account_entity_enqueue+0xbf at ffffffff980cc13f
[ffff9cb9aa1cf6f8] enqueue_entity+0xe4 at ffffffff980cea64
[ffff9cb9aa1cf700] record_times+0x10 at ffffffff980e8140
[ffff9cb9aa1cf750] check_preempt_wakeup+0x182 at ffffffff980cd422
[ffff9cb9aa1cf798] check_preempt_curr+0x70 at ffffffff980c2ce0
[ffff9cb9aa1cf7a8] ttwu_do_wakeup+0x12 at ffffffff980c2d02
[ffff9cb9aa1cf7c8] __update_load_avg_se+0x200 at ffffffff980e05d0
[ffff9cb9aa1cf808] __update_load_avg_cfs_rq+0x1df at ffffffff980e085f
[ffff9cb9aa1cf810] update_load_avg+0x76 at ffffffff980cd6a6
[ffff9cb9aa1cf850] set_next_entity+0x89 at ffffffff980cdc29
[ffff9cb9aa1cf870] pick_next_task_fair+0x590 at ffffffff980d4a50
[ffff9cb9aa1cf890] apic_timer_interrupt+0xa at ffffffff9880146a
[ffff9cb9aa1cf918] compact_unlock_should_abort+0x44 at ffffffff981b2834
[ffff9cb9aa1cf950] isolate_migratepages_block+0x2ed at ffffffff981b37bd
[ffff9cb9aa1cfa10] compact_zone+0x577 at ffffffff981b4c07
[ffff9cb9aa1cfab8] compact_zone_order+0xde at ffffffff981b51de
[ffff9cb9aa1cfb78] try_to_compact_pages+0x187 at ffffffff981b5a17
[ffff9cb9aa1cfbd8] __alloc_pages_direct_compact+0x87 at ffffffff981d95a7
[ffff9cb9aa1cfc30] __alloc_pages_slowpath+0x20e at ffffffff981d99ee
[ffff9cb9aa1cfc48] try_charge+0x51a at ffffffff9821044a
[ffff9cb9aa1cfcd0] release_pages+0x348 at ffffffff9819aa08
[ffff9cb9aa1cfd00] __pagevec_lru_add_fn+0x189 at ffffffff9819ada9
[ffff9cb9aa1cfd40] __alloc_pages_nodemask+0x268 at ffffffff981da668
[ffff9cb9aa1cfda0] do_huge_pmd_anonymous_page+0x131 at ffffffff98202c01
[ffff9cb9aa1cfdf0] __handle_mm_fault+0xc0c at ffffffff981bfa4c
[ffff9cb9aa1cfea0] handle_mm_fault+0xa9 at ffffffff981c01f9
[ffff9cb9aa1cfec8] __do_page_fault+0x237 at ffffffff9803d5c7
[ffff9cb9aa1cff28] do_page_fault+0x1d at ffffffff9803d85d
[ffff9cb9aa1cff48] page_fault+0x8 at ffffffff98800de8
[ffff9cb9aa1cff50] page_fault+0x1e at ffffffff98800dfe
RIP: 00005879fc8c4c10 RSP: 00007ffd393aa0d0 RFLAGS: 00010206
RAX: 000000002438a000 RBX: 0000714104876010 RCX: 0000714358a5a3db
RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000714104876000
RBP: 00005879fc8c5a54 R8: 0000714104876010 R9: 0000000000000000
R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff
R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
```
ok I just realized that the comments look broken only when cookies+scripts are allowed, because otherwise the markdown doesn't get interpreted so you just see the comments in their raw form! So the above comment has completely different contents when views via uMatrix extension as css+images allowed only, vs when viewed with everything allowed(cookies+scripts+css+images+everything else) On Tue, Jul 16, 2019 at 03:57:30AM +0000, howaboutsynergy@protonmail.com wrote: > > PID: 21120 TASK: ffff8fd7f10ddac0 CPU: 9 COMMAND: "stress" > > START: __schedule at ffffffff987f5a0c > > [ffff9cb9a9f57928] isolate_migratepages_block at ffffffff981b35c6 > > [ffff9cb9a9f57950] isolate_migratepages_block at ffffffff981b37bd > > [ffff9cb9a9f57a10] compact_zone at ffffffff981b4c07 > > [ffff9cb9a9f57ab8] compact_zone_order at ffffffff981b51de > > [ffff9cb9a9f57b78] try_to_compact_pages at ffffffff981b5a17 > > [ffff9cb9a9f57bd8] __alloc_pages_direct_compact at ffffffff981d95a7 > > [ffff9cb9a9f57c30] __alloc_pages_slowpath at ffffffff981d99ee > > [ffff9cb9a9f57cd0] release_pages at ffffffff9819aa08 > > [ffff9cb9a9f57d00] __pagevec_lru_add_fn at ffffffff9819ada9 > > [ffff9cb9a9f57d40] __alloc_pages_nodemask at ffffffff981da668 > > [ffff9cb9a9f57da0] do_huge_pmd_anonymous_page at ffffffff98202c01 > > [ffff9cb9a9f57df0] __handle_mm_fault at ffffffff981bfa4c > > [ffff9cb9a9f57ea0] handle_mm_fault at ffffffff981c01f9 > > [ffff9cb9a9f57ec8] __do_page_fault at ffffffff9803d5c7 > > [ffff9cb9a9f57f28] do_page_fault at ffffffff9803d85d > > [ffff9cb9a9f57f48] page_fault at ffffffff98800de8 > > [ffff9cb9a9f57f50] page_fault at ffffffff98800dfe > > RIP: 00005879fc8c4c10 RSP: 00007ffd393aa0d0 RFLAGS: 00010206 > > RAX: 0000000010b8a000 RBX: 0000714104876010 RCX: 0000714358a5a3db > > RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 0000714104876000 > > RBP: 00005879fc8c5a54 R8: 0000714104876010 R9: 0000000000000000 > > R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff > > R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 > > ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b > > crash> High CPU usage in this path is not something I've observed recently. When it happens and CPU usage is high, can you run the following commands please? trace-cmd record -e compaction:* sleep 10 trace-cmd report > trace.log and send me the resulting trace.log please? I tried reproducing this but after 300 attempts with various parameters and adding other workloads in the background, I was unable to reproduce the problem. Created attachment 283749 [details] publickey - howaboutsynergy@protonmail.com - 0x947B9B34.asc On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net> wrote: > I tried reproducing this but after 300 attempts with various parameters > and adding other workloads in the background, I was unable to reproduce > the problem. > The third time I ran this command `$ time stress -m 220 --vm-bytes 10000000000 --timeout 10`, got 10+ hung: PID %CPU COMMAND PR NI VIRT RES S USER 3785 94.5 stress 20 0 9769416 4 R user 3777 87.3 stress 20 0 9769416 4 R user 3923 85.5 stress 20 0 9769416 4 R user 3937 85.5 stress 20 0 9769416 4 R user 3943 81.8 stress 20 0 9769416 4 R user 3885 80.0 stress 20 0 9769416 4 R user 3970 80.0 stress 20 0 9769416 4 R user 3902 76.4 stress 20 0 9769416 4 R user 3954 72.7 stress 20 0 9769416 4 R user 3868 70.9 stress 20 0 9769416 4 R user 3893 69.1 stress 20 0 9769416 4 R user 3786 65.5 stress 20 0 9769416 4 R user 3783 60.0 stress 20 0 9769416 4 R user 3848 58.2 stress 20 0 9769416 4 R user 3863 58.2 stress 20 0 9769416 4 R user looked like this: ``` TERM='xterm-256color' 53.48 573.49 ----------- user@i87k 2019/07/16 20:36:47 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut53 !41866 1 0 5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019 /home/user $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [1744] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [1744] successful run completed in 19s real 0m19.036s user 0m0.794s sys 2m59.583s ----------- user@i87k 2019/07/16 20:37:12 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut79 !41867 2 0 5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019 /home/user $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [3520] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [3520] successful run completed in 18s real 0m18.657s user 0m0.901s sys 2m59.700s ----------- user@i87k 2019/07/16 20:42:28 -bash5.0.7 t:5 j:0 d:3 pp:1140 p:1623 ut394 !41868 3 0 5.2.1-g527a3db363a3 #71 SMP Tue Jul 16 19:41:12 CEST 2019 /home/user $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [3771] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd ``` (sure I waited a few minutes until I ran the 3rd one, I don't remember what trivial things I've been doing during the time - like making sure I got the right trace command(s)) I'm pretty sure you need swap in (ext4)zram for this to trigger (faster?), judging by the fuller stacktrace that I showed in prev. email(s) gotten via 'crash'-s `bt -Tsx pidhere` On Tuesday, July 16, 2019 9:11 AM, Mel Gorman <mgorman@techsingularity.net> wrote: > High CPU usage in this path is not something I've observed recently. > When it happens and CPU usage is high, can you run the following commands > please? > > trace-cmd record -e compaction:* sleep 10 > trace-cmd report > trace.log > > and send me the resulting trace.log please? Ok, getting trace.log as requested: ``` $ sudo trace-cmd record -e compaction:* sleep 10 [sudo] password for user: CPU 0: 12430 events lost CPU 1: 83959 events lost CPU 4: 13447 events lost CPU 6: 2825 events lost CPU 8: 791 events lost CPU 11: 8475 events lost CPU0 data recorded at offset=0x5bc000 114487296 bytes in size CPU1 data recorded at offset=0x72eb000 106885120 bytes in size CPU2 data recorded at offset=0xd8da000 125046784 bytes in size CPU3 data recorded at offset=0x1501b000 111022080 bytes in size CPU4 data recorded at offset=0x1b9fc000 120532992 bytes in size CPU5 data recorded at offset=0x22cef000 115990528 bytes in size CPU6 data recorded at offset=0x29b8d000 116109312 bytes in size CPU7 data recorded at offset=0x30a48000 73822208 bytes in size CPU8 data recorded at offset=0x350af000 98643968 bytes in size CPU9 data recorded at offset=0x3aec2000 96514048 bytes in size CPU10 data recorded at offset=0x40acd000 113967104 bytes in size CPU11 data recorded at offset=0x4777d000 127184896 bytes in size trace.dat is 1.3G -rw-r--r-- 1 root root 1326219264 16.07.2019 20:45 trace.dat $ LD_PRELOAD=/usr/lib/trace-cmd/python/ctracecmd.so trace-cmd report > trace.log trace-cmd: symbol lookup error: /usr/lib/trace-cmd/python/ctracecmd.so: undefined symbol: PyExc_SystemError $ trace-cmd report > trace.log could not load plugin '/usr/lib/trace-cmd/plugins/plugin_python.so' /usr/lib/trace-cmd/plugins/plugin_python.so: undefined symbol: PyString_FromString I guess we could ignore that? trace.log is like 4.3G -rw-r--r-- 1 user user 4370520245 16.07.2019 20:50 trace.log On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net> wrote: > I tried reproducing this but after 300 attempts with various parameters > and adding other workloads in the background, I was unable to reproduce > the problem. > As a reminder, here's how sysrq+l stacktraces look like (for two of the stress pids): ``` [ 1294.913508] NMI backtrace for cpu 5 [ 1294.913517] CPU: 5 PID: 3848 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #71 [ 1294.913522] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019 [ 1294.913526] RIP: 0010:ftrace_likely_update+0x1a/0x200 [ 1294.913533] Code: 0b eb bb 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 fb 41 89 d4 9c 41 5f 0f 01 ca <85> c9 0f 84 8b 00 00 00 48 ff 47 28 8b 15 34 72 47 01 85 d2 75 16 [ 1294.913537] RSP: 0000:ffffa3cd4c7ef848 EFLAGS: 00000286 [ 1294.913552] RAX: 0000000000000000 RBX: ffffffff97546180 RCX: 0000000000000000 [ 1294.913557] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff97546180 [ 1294.913562] RBP: 000000000017cc6f R08: 0000000000000000 R09: 0000000000000000 [ 1294.913567] R10: 0000000000000001 R11: 0000000000000004 R12: 0000000000000000 [ 1294.913571] R13: ffff93560dfdc000 R14: 000000000012f44c R15: 0000000000000286 [ 1294.913577] FS: 000072d6afdff740(0000) GS:ffff9355ed880000(0000) knlGS:0000000000000000 [ 1294.913582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1294.913590] CR2: 000070fbe8abdc88 CR3: 0000000827a46004 CR4: 00000000003606e0 [ 1294.913595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1294.913600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1294.913605] Call Trace: [ 1294.913609] _cond_resched+0x2d/0x50 [ 1294.913613] isolate_migratepages_block+0xf8/0xe40 [ 1294.913618] compact_zone+0x4f9/0xe10 [ 1294.913626] compact_zone_order+0xe3/0x120 [ 1294.913630] try_to_compact_pages+0xde/0x3b0 [ 1294.913635] __alloc_pages_direct_compact+0x8c/0x170 [ 1294.913639] __alloc_pages_slowpath+0x65c/0x1290 [ 1294.913644] __alloc_pages_nodemask+0x4cf/0x530 [ 1294.913648] do_huge_pmd_anonymous_page+0x17c/0x780 [ 1294.913653] __handle_mm_fault+0xeee/0x17d0 [ 1294.913661] handle_mm_fault+0x17b/0x330 [ 1294.913666] __do_page_fault+0x34e/0x800 [ 1294.913671] do_page_fault+0x57/0x1f9 [ 1294.913675] ? page_fault+0x8/0x30 [ 1294.913680] page_fault+0x1e/0x30 [ 1294.913684] RIP: 0033:0x5a46e611ac10 [ 1294.913690] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1294.913695] RSP: 002b:00007fffa51f0f60 EFLAGS: 00010206 [ 1294.913705] RAX: 00000000098c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db [ 1294.913710] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000 [ 1294.913715] RBP: 00005a46e611ba54 R08: 000072d45bd40010 R09: 0000000000000000 [ 1294.913720] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1294.913724] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1294.913730] NMI backtrace for cpu 11 [ 1294.913739] CPU: 11 PID: 3902 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #71 [ 1294.913744] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019 [ 1294.913748] RIP: 0010:isolate_migratepages_block+0x97b/0xe40 [ 1294.913754] Code: c6 43 79 01 4c 8b 74 24 08 4d 39 fe b9 00 00 00 00 ba 00 00 00 00 40 0f 92 c6 40 0f b6 f6 48 c7 c7 90 ac 56 97 e8 05 fe f5 ff <4d> 39 fe 0f 82 7a f9 ff ff 4c 39 7c 24 08 0f 84 6f f9 ff ff 0f 1f [ 1294.913758] RSP: 0000:ffffa3cd4c99f8b0 EFLAGS: 00000286 [ 1294.913768] RAX: 0000000000000000 RBX: ffffa3cd4c99fa50 RCX: 0000000000000000 [ 1294.913777] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9756ac90 [ 1294.913781] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1294.913786] R10: 0000000000000001 R11: 0000000000000004 R12: 0000000000000000 [ 1294.913790] R13: 0000000000000000 R14: 000000000070ee00 R15: 000000000070ece0 [ 1294.913795] FS: 000072d6afdff740(0000) GS:ffff9355edb80000(0000) knlGS:0000000000000000 [ 1294.913799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1294.913804] CR2: 00007d99c6200000 CR3: 00000007ecb1a003 CR4: 00000000003606e0 [ 1294.913813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1294.913817] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1294.913822] Call Trace: [ 1294.913826] compact_zone+0x4f9/0xe10 [ 1294.913830] compact_zone_order+0xe3/0x120 [ 1294.913835] try_to_compact_pages+0xde/0x3b0 [ 1294.913839] __alloc_pages_direct_compact+0x8c/0x170 [ 1294.913848] __alloc_pages_slowpath+0x65c/0x1290 [ 1294.913852] __alloc_pages_nodemask+0x4cf/0x530 [ 1294.913856] do_huge_pmd_anonymous_page+0x17c/0x780 [ 1294.913861] __handle_mm_fault+0xeee/0x17d0 [ 1294.913865] handle_mm_fault+0x17b/0x330 [ 1294.913869] __do_page_fault+0x34e/0x800 [ 1294.913874] do_page_fault+0x57/0x1f9 [ 1294.913882] ? page_fault+0x8/0x30 [ 1294.913886] page_fault+0x1e/0x30 [ 1294.913891] RIP: 0033:0x5a46e611ac10 [ 1294.913896] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1294.913901] RSP: 002b:00007fffa51f0f60 EFLAGS: 00010206 [ 1294.913907] RAX: 000000000d0c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db [ 1294.913916] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000 [ 1294.913921] RBP: 00005a46e611ba54 R08: 000072d45bd40010 R09: 0000000000000000 [ 1294.913925] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1294.913929] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 ``` and here's how they look via crash: ``` $ sudo crash "/usr/lib/modules/$(uname -r)/build/vmlinux" [sudo] password for user: crash 7.2.6 Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [336MB]: patching 121800 gdb minimal_symbol values KERNEL: /usr/lib/modules/5.2.1-g527a3db363a3/build/vmlinux DUMPFILE: /proc/kcore CPUS: 12 DATE: Tue Jul 16 20:59:48 2019 UPTIME: 00:23:54 LOAD AVERAGE: 26.83, 21.74, 16.35 TASKS: 508 NODENAME: i87k RELEASE: 5.2.1-g527a3db363a3 VERSION: #71 SMP Tue Jul 16 19:41:12 CEST 2019 MACHINE: x86_64 (3700 Mhz) MEMORY: 31.9 GB PID: 6417 COMMAND: "crash" TASK: ffff934f9c9a8000 [THREAD_INFO: ffff934f9c9a8000] CPU: 2 STATE: TASK_RUNNING (ACTIVE) crash> bt -Tsx 3848 PID: 3848 TASK: ffff934f83373d80 CPU: 5 COMMAND: "stress" [ffffa3cd4c7eefb8] get_page_from_freelist+0xaa9 at ffffffff96291799 [ffffa3cd4c7ef0e8] get_page_from_freelist+0xaa9 at ffffffff96291799 [ffffa3cd4c7ef108] __alloc_pages_slowpath+0x216 at ffffffff96292cb6 [ffffa3cd4c7ef130] get_page_from_freelist+0xb76 at ffffffff96291866 [ffffa3cd4c7ef178] get_page_from_freelist+0xaa9 at ffffffff96291799 [ffffa3cd4c7ef1a0] trace_hardirqs_on_caller+0x32 at ffffffff961bc4b2 [ffffa3cd4c7ef2e0] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b [ffffa3cd4c7ef388] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b [ffffa3cd4c7ef4b0] get_zspage_mapping+0x40 at ffffffff962e2220 [ffffa3cd4c7ef538] decay_load+0x6a at ffffffff96111d1a [ffffa3cd4c7ef5a0] __list_add_valid+0x90 at ffffffff965e3ff0 [ffffa3cd4c7ef650] check_preempt_wakeup+0x267 at ffffffff960f7097 [ffffa3cd4c7ef6b8] decay_load+0x6a at ffffffff96111d1a [ffffa3cd4c7ef700] trace_hardirqs_on_thunk+0x1a at ffffffff96001b02 [ffffa3cd4c7ef738] update_load_avg+0xca at ffffffff960f73aa [ffffa3cd4c7ef740] update_load_avg+0xca at ffffffff960f73aa [ffffa3cd4c7ef7a0] finish_task_switch+0xc9 at ffffffff960e6879 [ffffa3cd4c7ef7e8] finish_task_switch+0x17f at ffffffff960e692f [ffffa3cd4c7ef820] __schedule+0x552 at ffffffff969f4752 [ffffa3cd4c7ef8a8] isolate_migratepages_block+0x97b at ffffffff9626165b [ffffa3cd4c7ef918] isolate_migratepages_block+0xf at ffffffff96260cef [ffffa3cd4c7ef978] compact_zone+0x4f9 at ffffffff96263eb9 [ffffa3cd4c7efa38] compact_zone_order+0xe3 at ffffffff962648b3 [ffffa3cd4c7efaf0] try_to_compact_pages+0xde at ffffffff9626523e [ffffa3cd4c7efb60] __alloc_pages_direct_compact+0x8c at ffffffff9629285c [ffffa3cd4c7efbb8] __alloc_pages_slowpath+0x65c at ffffffff962930fc [ffffa3cd4c7efce8] __alloc_pages_nodemask+0x4cf at ffffffff962941ff [ffffa3cd4c7efd60] do_huge_pmd_anonymous_page+0x17c at ffffffff962c6fcc [ffffa3cd4c7efdb0] __handle_mm_fault+0xeee at ffffffff96272c3e [ffffa3cd4c7efe78] handle_mm_fault+0x17b at ffffffff9627369b [ffffa3cd4c7efeb0] __do_page_fault+0x34e at ffffffff9604f8ee [ffffa3cd4c7eff20] do_page_fault+0x57 at ffffffff9604fe27 [ffffa3cd4c7eff38] page_fault+0x8 at ffffffff96a00e78 [ffffa3cd4c7eff50] page_fault+0x1e at ffffffff96a00e8e RIP: 00005a46e611ac10 RSP: 00007fffa51f0f60 RFLAGS: 00010206 RAX: 00000000098c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000 RBP: 00005a46e611ba54 R8: 000072d45bd40010 R9: 0000000000000000 R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b crash> bt -l 3848 PID: 3848 TASK: ffff934f83373d80 CPU: 11 COMMAND: "stress" #0 [ffffa3cd4c7ef7f0] __schedule at ffffffff969f471a /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/kernel/sched/core.c: 2818 #1 [ffffa3cd4c7ef830] trace_hardirqs_on_thunk at ffffffff96001b02 /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/entry/thunk_64.S: 42 #2 [ffffa3cd4c7ef8a8] isolate_migratepages_block at ffffffff9626165b /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 1039 #3 [ffffa3cd4c7ef978] compact_zone at ffffffff96263eb9 /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 1817 #4 [ffffa3cd4c7efa38] compact_zone_order at ffffffff962648b3 /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 2313 #5 [ffffa3cd4c7efaf0] try_to_compact_pages at ffffffff9626523e /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/compaction.c: 2362 #6 [ffffa3cd4c7efb60] __alloc_pages_direct_compact at ffffffff9629285c /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 3831 #7 [ffffa3cd4c7efbb8] __alloc_pages_slowpath at ffffffff962930fc /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 4470 #8 [ffffa3cd4c7efce8] __alloc_pages_nodemask at ffffffff962941ff /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/page_alloc.c: 4678 #9 [ffffa3cd4c7efd60] do_huge_pmd_anonymous_page at ffffffff962c6fcc /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/./include/linux/topology.h: 73 #10 [ffffa3cd4c7efdb0] __handle_mm_fault at ffffffff96272c3e /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/memory.c: 3788 #11 [ffffa3cd4c7efe78] handle_mm_fault at ffffffff9627369b /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/mm/memory.c: 4058 #12 [ffffa3cd4c7efeb0] __do_page_fault at ffffffff9604f8ee /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/mm/fault.c: 1457 #13 [ffffa3cd4c7eff20] do_page_fault at ffffffff9604fe27 /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/./arch/x86/include/asm/jump_label.h: 23 #14 [ffffa3cd4c7eff50] page_fault at ffffffff96a00e8e /home/user/build/1packages/4used/kernel/linux-stable/makepkg_pacman/linux-stable/src/linux-stable/arch/x86/entry/entry_64.S: 1156 RIP: 00005a46e611ac10 RSP: 00007fffa51f0f60 RFLAGS: 00010206 RAX: 00000000098c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000 RBP: 00005a46e611ba54 R8: 000072d45bd40010 R9: 0000000000000000 R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b crash> bt -Tsx 3902 PID: 3902 TASK: ffff934f5264bd80 CPU: 1 COMMAND: "stress" [ffffa3cd4c99efb8] get_page_from_freelist+0xaa9 at ffffffff96291799 [ffffa3cd4c99f108] __alloc_pages_slowpath+0x216 at ffffffff96292cb6 [ffffa3cd4c99f130] get_page_from_freelist+0xb76 at ffffffff96291866 [ffffa3cd4c99f178] get_page_from_freelist+0xaa9 at ffffffff96291799 [ffffa3cd4c99f2e0] ZSTD_compressSequences_internal+0x8db at ffffffff9660a27b [ffffa3cd4c99f480] test_clear_page_writeback+0x12a at ffffffff9623ad8a [ffffa3cd4c99f4a8] trace_hardirqs_on+0x2c at ffffffff961bc30c [ffffa3cd4c99f4c8] test_clear_page_writeback+0x18c at ffffffff9623adec [ffffa3cd4c99f538] decay_load+0x6a at ffffffff96111d1a [ffffa3cd4c99f5a0] __list_add_valid+0x90 at ffffffff965e3ff0 [ffffa3cd4c99f650] check_preempt_wakeup+0x267 at ffffffff960f7097 [ffffa3cd4c99f6b8] decay_load+0x6a at ffffffff96111d1a [ffffa3cd4c99f6e0] __accumulate_pelt_segments+0x29 at ffffffff96111d89 [ffffa3cd4c99f700] __update_load_avg_se+0x1cb at ffffffff961120ab [ffffa3cd4c99f738] update_load_avg+0xca at ffffffff960f73aa [ffffa3cd4c99f740] update_load_avg+0xca at ffffffff960f73aa [ffffa3cd4c99f7e0] switch_mm_irqs_off+0x270 at ffffffff96057920 [ffffa3cd4c99f820] __schedule+0x51a at ffffffff969f471a [ffffa3cd4c99f880] preempt_schedule_common+0x15 at ffffffff969f4ed5 [ffffa3cd4c99f898] _cond_resched+0x3f at ffffffff969f4f3f [ffffa3cd4c99f8a8] isolate_migratepages_block+0xf8 at ffffffff96260dd8 [ffffa3cd4c99f978] compact_zone+0x4f9 at ffffffff96263eb9 [ffffa3cd4c99fa38] compact_zone_order+0xe3 at ffffffff962648b3 [ffffa3cd4c99faf0] try_to_compact_pages+0xde at ffffffff9626523e [ffffa3cd4c99fb60] __alloc_pages_direct_compact+0x8c at ffffffff9629285c [ffffa3cd4c99fbb8] __alloc_pages_slowpath+0x65c at ffffffff962930fc [ffffa3cd4c99fce8] __alloc_pages_nodemask+0x4cf at ffffffff962941ff [ffffa3cd4c99fd60] do_huge_pmd_anonymous_page+0x17c at ffffffff962c6fcc [ffffa3cd4c99fdb0] __handle_mm_fault+0xeee at ffffffff96272c3e [ffffa3cd4c99fe78] handle_mm_fault+0x17b at ffffffff9627369b [ffffa3cd4c99feb0] __do_page_fault+0x34e at ffffffff9604f8ee [ffffa3cd4c99ff20] do_page_fault+0x57 at ffffffff9604fe27 [ffffa3cd4c99ff38] page_fault+0x8 at ffffffff96a00e78 [ffffa3cd4c99ff50] page_fault+0x1e at ffffffff96a00e8e RIP: 00005a46e611ac10 RSP: 00007fffa51f0f60 RFLAGS: 00010206 RAX: 000000000d0c0000 RBX: 000072d45bd40010 RCX: 000072d6aff243db RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 000072d45bd40000 RBP: 00005a46e611ba54 R8: 000072d45bd40010 R9: 0000000000000000 R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b crash> bt -l 3902 PID: 3902 TASK: ffff934f5264bd80 CPU: 9 COMMAND: "stress" (active) crash> ``` As can be seen, it's doing stuff with(in) zstd, maybe that's why swap in zstd is needed to reproduce this (more easily? or at all?) trace.log is still being compressed(via xz) but those running 'stress' processes using 100% cpu are only making this slower :) I'll send it when it's ready? Created attachment 283751 [details]
signature.asc
On Tue, Jul 16, 2019 at 07:15:08PM +0000, howaboutsynergy@protonmail.com wrote: > On Tuesday, July 16, 2019 12:03 PM, Mel Gorman <mgorman@techsingularity.net> > wrote: > > I tried reproducing this but after 300 attempts with various parameters > > and adding other workloads in the background, I was unable to reproduce > > the problem. > > > > > The third time I ran this command `$ time stress -m 220 --vm-bytes > 10000000000 --timeout 10`, got 10+ hung: > > PID %CPU COMMAND > PR NI VIRT RES S USER > 3785 94.5 stress > 20 0 9769416 4 R user > 3777 87.3 stress > 20 0 9769416 4 R user > 3923 85.5 stress > 20 0 9769416 4 R user > 3937 85.5 stress > 20 0 9769416 4 R user > 3943 81.8 stress > 20 0 9769416 4 R user > 3885 80.0 stress > 20 0 9769416 4 R user > 3970 80.0 stress > 20 0 9769416 4 R user > > <SNIP> > > trace.dat is 1.3G > -rw-r--r-- 1 root root 1326219264 16.07.2019 20:45 trace.dat > Ok, great. From the trace, it was obvious that the scanner is making no progress. I don't think zswap is involved as such but it *may* be making it easier to trigger due to altering timing. At least, I see no reason why zswap would materially affect the termination conditions. From the path and your trace, I think what *might* be happening is that a fatal signal is pending which does not advance the scanner or look like a proper abort. I think it ends up looping in compaction instead of dying without either aborting or progressing the scanner. It might explain why stress-ng is hitting is as it is probably sending fatal signals on timeout (I didn't check the source). Can you try this (compile tested only) patch please? Note that the stress test might still take time to exit normally if it's stuck in a swap storm of some sort but I'm hoping the 100% compaction CPU usage goes away at least. diff --git a/mm/compaction.c b/mm/compaction.c index 9e1b9acb116b..952dc2fb24e5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -842,13 +842,15 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* * Periodically drop the lock (if held) regardless of its - * contention, to give chance to IRQs. Abort async compaction - * if contended. + * contention, to give chance to IRQs. Abort completely if + * a fatal signal is pending. */ if (!(low_pfn % SWAP_CLUSTER_MAX) && compact_unlock_should_abort(&pgdat->lru_lock, - flags, &locked, cc)) - break; + flags, &locked, cc)) { + low_pfn = 0; + goto fatal_pending; + } if (!pfn_valid_within(low_pfn)) goto isolate_fail; @@ -1060,6 +1062,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn, nr_scanned, nr_isolated); +fatal_pending: cc->total_migrate_scanned += nr_scanned; if (nr_isolated) count_compact_events(COMPACTISOLATED, nr_isolated); tl;dr: patch seems to work, thank you very much! ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, July 17, 2019 7:53 PM, Mel Gorman <mgorman@techsingularity.net> wrote: > Ok, great. From the trace, it was obvious that the scanner is making no > progress. I don't think zswap is involved as such but it may be making > it easier to trigger due to altering timing. At least, I see no reason > why zswap would materially affect the termination conditions. I don't know if it matters in this context, but I've been using the term `zswap`(somewhere else I think) to (wrongly)refer to swap in zram (and even sometimes called it ext4(in this bug report too) without realizing at the time that ext4 is only for /tmp and /var/tmp instead! they are ext4 in zram) but in fact this isn't zswap that I have been using (even though I have CONFIG_ZSWAP=y in .config) but it's just CONFIG_ZRAM=y with CONFIG_SWAP=y (and probably a bunch of others being needed too). > > From the path and your trace, I think what might be happening is that > a fatal signal is pending which does not advance the scanner or look like > a proper abort. I think it ends up looping in compaction instead of dying > without either aborting or progressing the scanner. It might explain why > stress-ng is hitting is as it is probably sending fatal signals on timeout > (I didn't check the source). Ah I didn't know there are multiple `stress` versions, here's what I used: /usr/bin/stress is owned by stress 1.0.4-5 $ pacman -Qs stress local/stress 1.0.4-5 A tool that stress tests your system (CPU, memory, I/O, disks) $ pacman -Qi stress Name : stress Version : 1.0.4-5 Description : A tool that stress tests your system (CPU, memory, I/O, disks) Architecture : x86_64 URL : http://people.seas.harvard.edu/~apw/stress/ Licenses : GPL Groups : None Provides : None Depends On : None Optional Deps : None Required By : None Optional For : None Conflicts With : None Replaces : None Installed Size : 25.00 KiB Packager : Florian Pritz <bluewind@xinu.at> Build Date : Thu 31 May 2018 01:46:16 PM CEST Install Date : Mon 10 Jun 2019 02:44:14 AM CEST Install Reason : Explicitly installed Install Script : No Validated By : Signature Note: not: $ yaourt -Ss stress-ng aur/stress-ng 0.09.59-1 (11) (0.10) stress-ng will stress test a computer system in various selectable ways (it probably doesn't matter anyway :) as I didn't look at the source either) > > Can you try this (compile tested only) patch please? Note that the stress > test might still take time to exit normally if it's stuck in a swap > storm of some sort but I'm hoping the 100% compaction CPU usage goes away > at least. > > diff --git a/mm/compaction.c b/mm/compaction.c > index 9e1b9acb116b..952dc2fb24e5 100644 > --- a/mm/compaction.c > +++ b/mm/compaction.c > @@ -842,13 +842,15 @@ isolate_migratepages_block(struct compact_control cc, > unsigned long low_pfn, > /* Periodically drop the lock (if held) regardless of its > > - * contention, to give chance to IRQs. Abort async compaction > > > - * if contended. > > > > - * contention, to give chance to IRQs. Abort completely if > > > - * a fatal signal is pending. > */ > if (!(low_pfn % SWAP_CLUSTER_MAX) > && compact_unlock_should_abort(&pgdat->lru_lock, > > > > - flags, &locked, cc)) > > > - break; > > > > - flags, &locked, cc)) { > > > - low_pfn = 0; > > > - goto fatal_pending; > > > - } > > > > if (!pfn_valid_within(low_pfn)) > goto isolate_fail; > @@ -1060,6 +1062,7 @@ isolate_migratepages_block(struct compact_control *cc, > unsigned long low_pfn, > trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn, > nr_scanned, nr_isolated); > > +fatal_pending: > cc->total_migrate_scanned += nr_scanned; > > if (nr_isolated) > count_compact_events(COMPACTISOLATED, nr_isolated); > > > -- > > Mel Gorman > SUSE Labs Now the "problem" is I can't tell if it would get stuck :D but it usually ends in no more than 17 sec: $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [7981] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: FAIL: [7981] (415) <-- worker 8202 got signal 9 stress: WARN: [7981] (417) now reaping child worker processes stress: FAIL: [7981] (415) <-- worker 8199 got signal 9 stress: WARN: [7981] (417) now reaping child worker processes stress: FAIL: [7981] (451) failed run completed in 18s real 0m17.397s user 0m1.069s sys 2m42.774s sometimes it's 14,15 or 16, but usually it's between 17 and 18(exclusive) seconds. Interestingly sometimes (rarely) it succeeds: $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [12629] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [12629] successful run completed in 15s real 0m14.571s user 0m1.122s sys 2m18.623s But it succeedes 100% of the time if I limit CPU max to 800Mhz (lowest): $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [8434] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [8434] successful run completed in 23s real 0m22.639s user 0m1.653s sys 3m18.693s sometimes it's in 7 sec less that the usual 22sec: $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [8689] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [8689] successful run completed in 15s real 0m15.585s user 0m1.576s sys 2m26.495s (probably irrelevant)Sometimes Xorg says it can't allocate any more memory but stacktrace looks like it's inside some zram i915 kernel stuff: [ 1416.842931] [drm] Atomic update on pipe (A) took 188 us, max time under evasion is 100 us [ 1425.416979] Xorg: page allocation failure: order:0, mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null),cpuset=/,mems_allowed=0 [ 1425.416984] CPU: 1 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #74 [ 1425.416985] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019 [ 1425.416986] Call Trace: [ 1425.416991] dump_stack+0x46/0x60 [ 1425.416993] warn_alloc.cold+0x7b/0xfb [ 1425.416996] __alloc_pages_slowpath+0xbe2/0xc20 [ 1425.416997] __alloc_pages_nodemask+0x268/0x2b0 [ 1425.416999] new_slab+0x3ff/0x610 [ 1425.417001] ? xas_alloc+0x9c/0xc0 [ 1425.417002] ___slab_alloc+0x33b/0x540 [ 1425.417003] ? xas_nomem+0x49/0x70 [ 1425.417005] ? xas_alloc+0x9c/0xc0 [ 1425.417006] ? xas_nomem+0x49/0x70 [ 1425.417007] __slab_alloc+0x9/0x10 [ 1425.417008] kmem_cache_alloc+0x118/0x150 [ 1425.417010] xas_nomem+0x49/0x70 [ 1425.417011] add_to_swap_cache+0x25c/0x310 [ 1425.417013] __read_swap_cache_async+0xef/0x1e0 [ 1425.417014] swap_cluster_readahead+0x1ac/0x2a0 [ 1425.417015] shmem_swapin+0x55/0xa0 [ 1425.417018] ? call_function_single_interrupt+0xa/0x20 [ 1425.417019] ? xas_init_marks+0x19/0x40 [ 1425.417020] ? xas_store+0x3e4/0x640 [ 1425.417022] shmem_swapin_page+0x418/0x590 [ 1425.417023] ? xas_load+0x5/0x70 [ 1425.417025] ? find_get_entry+0x55/0x120 [ 1425.417026] shmem_getpage_gfp.isra.0+0x37f/0x820 [ 1425.417028] shmem_read_mapping_page_gfp+0x3b/0x70 [ 1425.417051] i915_gem_object_get_pages_gtt+0x1f9/0x550 [i915] [ 1425.417070] ? gen8_ppgtt_alloc_4lvl+0x5a/0x150 [i915] [ 1425.417086] __i915_gem_object_get_pages+0x4e/0x60 [i915] [ 1425.417104] __i915_vma_do_pin+0x4bf/0x540 [i915] [ 1425.417120] eb_lookup_vmas+0x68b/0xb90 [i915] [ 1425.417136] ? i915_gem_do_execbuffer+0x351/0x1040 [i915] [ 1425.417152] ? i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915] [ 1425.417154] ? krealloc+0x21/0xa0 [ 1425.417169] i915_gem_do_execbuffer+0x502/0x1040 [i915] [ 1425.417172] ? ZSTD_decompressMultiFrame+0x329/0x370 [ 1425.417174] ? ZSTD_decompressDCtx+0xc/0x10 [ 1425.417175] ? ktime_get_with_offset+0x5c/0x130 [ 1425.417177] ? zram_bvec_rw.isra.0+0x154/0x6a0 [ 1425.417179] ? _cond_resched+0x10/0x20 [ 1425.417194] i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915] [ 1425.417196] ? zram_rw_page+0xba/0xe0 [ 1425.417211] ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915] [ 1425.417220] drm_ioctl_kernel+0xa5/0xf0 [drm] [ 1425.417228] drm_ioctl+0x201/0x380 [drm] [ 1425.417243] ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915] [ 1425.417245] do_vfs_ioctl+0x3f4/0x660 [ 1425.417246] ? handle_mm_fault+0xa9/0x1d0 [ 1425.417248] ksys_ioctl+0x59/0x90 [ 1425.417249] __x64_sys_ioctl+0x11/0x20 [ 1425.417251] do_syscall_64+0x50/0x170 [ 1425.417253] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1425.417255] RIP: 0033:0x78521f1b1c2b [ 1425.417256] Code: 0f 1e fa 48 8b 05 65 d2 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 35 d2 0c 00 f7 d8 64 89 01 48 [ 1425.417258] RSP: 002b:00007ffcf8fd0d08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 1425.417259] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 000078521f1b1c2b [ 1425.417260] RDX: 00007ffcf8fd0d40 RSI: 0000000040406469 RDI: 000000000000000a [ 1425.417261] RBP: 00005a6a541acad0 R08: 0000000000000000 R09: 00007852196c2a60 [ 1425.417262] R10: 0000785213f3e000 R11: 0000000000000246 R12: 0000000000002000 [ 1425.417263] R13: 000078521af98000 R14: 00007ffcf8fd0d40 R15: 00007ffcf8fd0d40 [ 1425.417264] Mem-Info: [ 1425.417267] active_anon:6998108 inactive_anon:442511 isolated_anon:9465 active_file:2251 inactive_file:2329 isolated_file:259 unevictable:16735 dirty:20 writeback:0 unstable:0 slab_reclaimable:28873 slab_unreclaimable:37272 mapped:3501 shmem:39527 pagetables:32186 bounce:0 free:87238 free_pcp:2904 free_cma:0 [ 1425.417269] Node 0 active_anon:27992432kB inactive_anon:1770044kB active_file:9004kB inactive_file:9316kB unevictable:66940kB isolated(anon):37860kB isolated(file):1036kB mapped:14004kB dirty:80kB writeback:0kB shmem:158108kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 22118400kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1425.417271] Node 0 DMA free:15892kB min:100kB low:124kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15892kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1425.417273] lowmem_reserve[]: 0 2346 31714 31714 [ 1425.417275] Node 0 DMA32 free:132920kB min:15608kB low:19508kB high:23408kB active_anon:2259000kB inactive_anon:1412kB active_file:0kB inactive_file:0kB unevictable:1296kB writepending:0kB present:2744176kB managed:2416496kB mlocked:0kB kernel_stack:16kB pagetables:3420kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB [ 1425.417277] lowmem_reserve[]: 0 0 29368 29368 [ 1425.417279] Node 0 Normal free:200140kB min:198384kB low:246956kB high:295528kB active_anon:25733140kB inactive_anon:1769340kB active_file:10288kB inactive_file:10456kB unevictable:65560kB writepending:80kB present:30654464kB managed:30077908kB mlocked:32kB kernel_stack:9408kB pagetables:125324kB bounce:0kB free_pcp:11740kB local_pcp:1420kB free_cma:0kB [ 1425.417281] lowmem_reserve[]: 0 0 0 0 [ 1425.417282] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB [ 1425.417287] Node 0 DMA32: 248*4kB (UME) 256*8kB (UME) 172*16kB (UME) 138*32kB (UME) 87*64kB (UME) 68*128kB (UME) 38*256kB (UE) 22*512kB (UME) 11*1024kB (E) 0*2048kB 19*4096kB (M) = 134560kB [ 1425.417292] Node 0 Normal: 17*4kB (UM) 20*8kB (UM) 16*16kB (UM) 1721*32kB (UM) 733*64kB (UME) 140*128kB (UM) 1*256kB (E) 1*512kB (M) 2*1024kB (ME) 37*2048kB (M) 0*4096kB = 198980kB [ 1425.417297] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 1425.417298] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1425.417299] 50365 total pagecache pages [ 1425.417306] 6377 pages in swap cache [ 1425.417307] Swap cache stats: add 728656996, delete 728204250, find 6351630/8862186 [ 1425.417308] Free swap = 35959128kB [ 1425.417308] Total swap = 67108860kB [ 1425.417309] 8353655 pages RAM [ 1425.417310] 0 pages HighMem/MovableOnly [ 1425.417310] 226081 pages reserved [ 1425.417311] 0 pages cma reserved [ 1425.417312] SLUB: Unable to allocate memory on node -1, gfp=0xc0(__GFP_IO|__GFP_FS) [ 1425.417313] cache: radix_tree_node, object size: 576, buffer size: 584, default order: 2, min order: 0 [ 1425.417314] node 0: slabs: 5357, objs: 149702, free: 0 [ 1426.236297] Xorg: page allocation failure: order:0, mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null),cpuset=/,mems_allowed=0 [ 1426.236306] CPU: 9 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #74 [ 1426.236307] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019 [ 1426.236308] Call Trace: [ 1426.236318] dump_stack+0x46/0x60 [ 1426.236322] warn_alloc.cold+0x7b/0xfb [ 1426.236326] __alloc_pages_slowpath+0xbe2/0xc20 [ 1426.236328] __alloc_pages_nodemask+0x268/0x2b0 [ 1426.236331] new_slab+0x3ff/0x610 [ 1426.236335] ? xas_alloc+0x9c/0xc0 [ 1426.236337] ___slab_alloc+0x33b/0x540 [ 1426.236339] ? xas_nomem+0x49/0x70 [ 1426.236340] ? xas_alloc+0x9c/0xc0 [ 1426.236341] ? xas_nomem+0x49/0x70 [ 1426.236342] __slab_alloc+0x9/0x10 [ 1426.236343] kmem_cache_alloc+0x118/0x150 [ 1426.236345] xas_nomem+0x49/0x70 [ 1426.236347] add_to_swap_cache+0x25c/0x310 [ 1426.236349] __read_swap_cache_async+0xef/0x1e0 [ 1426.236351] read_swap_cache_async+0x24/0x60 [ 1426.236352] swap_cluster_readahead+0x237/0x2a0 [ 1426.236354] ? get_page_from_freelist+0xe88/0x1330 [ 1426.236356] ? ktime_get+0x49/0x110 [ 1426.236358] shmem_swapin+0x55/0xa0 [ 1426.236361] shmem_swapin_page+0x418/0x590 [ 1426.236362] ? xas_load+0x5/0x70 [ 1426.236366] ? find_get_entry+0x55/0x120 [ 1426.236367] shmem_getpage_gfp.isra.0+0x37f/0x820 [ 1426.236368] shmem_read_mapping_page_gfp+0x3b/0x70 [ 1426.236438] i915_gem_object_get_pages_gtt+0x1f9/0x550 [i915] [ 1426.236445] ? set_next_entity+0x89/0x150 [ 1426.236447] ? pick_next_task_fair+0x590/0x760 [ 1426.236482] __i915_gem_object_get_pages+0x4e/0x60 [i915] [ 1426.236501] __i915_vma_do_pin+0x4bf/0x540 [i915] [ 1426.236518] eb_lookup_vmas+0x68b/0xb90 [i915] [ 1426.236534] i915_gem_do_execbuffer+0x502/0x1040 [i915] [ 1426.236537] ? xas_init_marks+0x19/0x40 [ 1426.236538] ? xas_store+0x3e4/0x640 [ 1426.236540] ? __delete_from_swap_cache+0x15a/0x190 [ 1426.236541] ? _swap_info_get+0xa/0x30 [ 1426.236543] ? free_swap_slot+0x43/0xc0 [ 1426.236544] ? __swap_entry_free.constprop.0+0x92/0xa0 [ 1426.236545] ? shmem_swapin_page+0x4fe/0x590 [ 1426.236547] ? xas_load+0x5/0x70 [ 1426.236549] ? find_get_entry+0x55/0x120 [ 1426.236550] ? _cond_resched+0x10/0x20 [ 1426.236566] i915_gem_execbuffer2_ioctl+0x1b7/0x380 [i915] [ 1426.236577] ? __mod_lruvec_state+0x3a/0xe0 [ 1426.236593] ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915] [ 1426.236618] drm_ioctl_kernel+0xa5/0xf0 [drm] [ 1426.236627] drm_ioctl+0x201/0x380 [drm] [ 1426.236645] ? i915_gem_execbuffer_ioctl+0x2a0/0x2a0 [i915] [ 1426.236649] ? __set_current_blocked+0x38/0x50 [ 1426.236650] ? signal_setup_done+0x8b/0xa0 [ 1426.236654] do_vfs_ioctl+0x3f4/0x660 [ 1426.236656] ? recalc_sigpending+0xe/0x40 [ 1426.236659] ? _copy_from_user+0x37/0x60 [ 1426.236661] ksys_ioctl+0x59/0x90 [ 1426.236662] __x64_sys_ioctl+0x11/0x20 [ 1426.236664] do_syscall_64+0x50/0x170 [ 1426.236669] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1426.236671] RIP: 0033:0x78521f1b1c2b [ 1426.236673] Code: 0f 1e fa 48 8b 05 65 d2 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 35 d2 0c 00 f7 d8 64 89 01 48 [ 1426.236674] RSP: 002b:00007ffcf8fcee18 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 1426.236676] RAX: ffffffffffffffda RBX: 000078521dd60d40 RCX: 000078521f1b1c2b [ 1426.236677] RDX: 00007ffcf8fcee80 RSI: 0000000040406469 RDI: 000000000000000a [ 1426.236677] RBP: 00007ffcf8fcee80 R08: 0000000000000000 R09: 00007852196c3d60 [ 1426.236678] R10: 00007852184ea000 R11: 0000000000000246 R12: 0000000040406469 [ 1426.236679] R13: 000000000000000a R14: 00007ffcf8fcee80 R15: 00007ffcf8fcee80 [ 1426.236682] SLUB: Unable to allocate memory on node -1, gfp=0xc0(__GFP_IO|__GFP_FS) [ 1426.236683] cache: radix_tree_node, object size: 576, buffer size: 584, default order: 2, min order: 0 [ 1426.236684] node 0: slabs: 1195, objs: 33082, free: 0 [ 1466.913788] stress invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0 [ 1466.913792] CPU: 3 PID: 15698 Comm: stress Kdump: loaded Tainted: G U 5.2.1-g527a3db363a3 #74 [ 1466.913793] Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 2201 05/27/2019 [ 1466.913794] Call Trace: [ 1466.913798] dump_stack+0x46/0x60 [ 1466.913800] dump_header+0x4f/0x2fd [ 1466.913801] ? oom_unkillable_task+0x95/0xc0 [ 1466.913802] ? find_lock_task_mm+0x2e/0x70 [ 1466.913803] oom_kill_process.cold+0xb/0x10 [ 1466.913805] out_of_memory+0x1bc/0x460 [ 1466.913806] __alloc_pages_slowpath+0xb19/0xc20 [ 1466.913808] __alloc_pages_nodemask+0x268/0x2b0 [ 1466.913810] alloc_pages_vma+0x74/0x1c0 [ 1466.913812] __handle_mm_fault+0xe1f/0x1310 [ 1466.913814] handle_mm_fault+0xa9/0x1d0 [ 1466.913816] __do_page_fault+0x237/0x480 [ 1466.913817] do_page_fault+0x1d/0x67 [ 1466.913819] ? page_fault+0x8/0x30 [ 1466.913821] page_fault+0x1e/0x30 [ 1466.913822] RIP: 0033:0x56f413018c10 [ 1466.913824] Code: c0 0f 84 53 02 00 00 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f fa 00 00 00 31 c0 4d 85 ff 7e 10 0f 1f 40 00 <c6> 04 03 5a 4c 01 f0 49 39 c7 7f f4 4d 85 e4 0f 84 f4 01 00 00 7e [ 1466.913825] RSP: 002b:00007ffd17a85910 EFLAGS: 00010206 [ 1466.913827] RAX: 00000000187e3000 RBX: 00007a864406e010 RCX: 00007a88982523db [ 1466.913827] RDX: 0000000000000000 RSI: 00000002540bf000 RDI: 00007a864406e000 [ 1466.913828] RBP: 000056f413019a54 R08: 00007a864406e010 R09: 0000000000000000 [ 1466.913829] R10: 0000000000000022 R11: 00000002540be400 R12: ffffffffffffffff [ 1466.913830] R13: 0000000000000002 R14: 0000000000001000 R15: 00000002540be400 [ 1466.913831] Mem-Info: [ 1466.913834] active_anon:6889086 inactive_anon:437196 isolated_anon:44189 active_file:210 inactive_file:248 isolated_file:32 unevictable:16068 dirty:0 writeback:0 unstable:0 slab_reclaimable:10669 slab_unreclaimable:55287 mapped:492 shmem:16679 pagetables:49030 bounce:0 free:85711 free_pcp:1702 free_cma:0 [ 1466.913837] Node 0 active_anon:27556344kB inactive_anon:1748784kB active_file:840kB inactive_file:992kB unevictable:64272kB isolated(anon):176756kB isolated(file):128kB mapped:1968kB dirty:0kB writeback:0kB shmem:66716kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 18806784kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1466.913838] Node 0 DMA free:15892kB min:100kB low:124kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15892kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1466.913841] lowmem_reserve[]: 0 2346 31714 31714 [ 1466.913842] Node 0 DMA32 free:133064kB min:15608kB low:19508kB high:23408kB active_anon:2246984kB inactive_anon:516kB active_file:32kB inactive_file:20kB unevictable:56kB writepending:0kB present:2744176kB managed:2416496kB mlocked:0kB kernel_stack:16kB pagetables:6120kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB [ 1466.913845] lowmem_reserve[]: 0 0 29368 29368 [ 1466.913847] Node 0 Normal free:193888kB min:194288kB low:242860kB high:291432kB active_anon:25307420kB inactive_anon:1748072kB active_file:2148kB inactive_file:2432kB unevictable:62728kB writepending:0kB present:30654464kB managed:30077908kB mlocked:32kB kernel_stack:9392kB pagetables:190000kB bounce:0kB free_pcp:6804kB local_pcp:344kB free_cma:0kB [ 1466.913849] lowmem_reserve[]: 0 0 0 0 [ 1466.913851] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB [ 1466.913856] Node 0 DMA32: 117*4kB (UME) 133*8kB (UME) 95*16kB (UME) 98*32kB (UME) 127*64kB (UME) 80*128kB (UME) 40*256kB (UME) 27*512kB (UME) 18*1024kB (ME) 2*2048kB (ME) 15*4096kB (ME) = 132588kB [ 1466.913861] Node 0 Normal: 482*4kB (UME) 478*8kB (UME) 648*16kB (ME) 849*32kB (E) 661*64kB (UME) 345*128kB (UE) 132*256kB (UE) 49*512kB (UME) 4*1024kB (ME) 0*2048kB 0*4096kB = 192728kB [ 1466.913867] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 1466.913868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1466.913869] 64205 total pagecache pages [ 1466.913875] 47161 pages in swap cache [ 1466.913876] Swap cache stats: add 776634640, delete 776102875, find 6734206/9399597 [ 1466.913877] Free swap = 0kB [ 1466.913877] Total swap = 67108860kB [ 1466.913878] 8353655 pages RAM [ 1466.913879] 0 pages HighMem/MovableOnly [ 1466.913879] 226081 pages reserved [ 1466.913880] 0 pages cma reserved [ 1466.913880] Tasks state (memory values in pages): [ 1466.913881] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 1466.913888] [ 399] 0 399 21409 0 212992 330 0 systemd-journal [ 1466.913889] [ 413] 0 413 9265 0 94208 420 -1000 systemd-udevd [ 1466.913891] [ 604] 81 604 2773 0 65536 225 -900 dbus-daemon [ 1466.913893] [ 609] 0 609 649 0 45056 45 0 gpm [ 1466.913894] [ 610] 0 610 6600 0 90112 269 0 systemd-logind [ 1466.913896] [ 695] 0 695 11319 0 131072 157 0 login [ 1466.913898] [ 799] 1000 799 7472 1 102400 377 0 systemd [ 1466.913899] [ 800] 1000 800 16164 131 159744 463 0 (sd-pam) [ 1466.913900] [ 812] 1000 812 3407 1 69632 1626 0 bash [ 1466.913902] [ 1001] 1000 1001 1895 1 57344 143 0 startx [ 1466.913903] [ 1023] 1000 1023 1006 0 49152 51 0 xinit [ 1466.913905] [ 1024] 1000 1024 179163 512 634880 3292 0 Xorg [ 1466.913906] [ 1033] 1000 1033 92987 0 217088 1934 0 xfce4-session [ 1466.913908] [ 1037] 1000 1037 2758 0 61440 241 0 dbus-daemon [ 1466.913909] [ 1040] 1000 1040 76635 1 98304 232 0 at-spi-bus-laun [ 1466.913911] [ 1044] 1000 1044 57827 0 81920 290 0 xfconfd [ 1466.913912] [ 1051] 1000 1051 2716 1 65536 165 0 dbus-daemon [ 1466.913914] [ 1054] 1000 1054 40960 1 81920 256 0 at-spi2-registr [ 1466.913915] [ 1057] 102 1057 483104 0 233472 1331 0 polkitd [ 1466.913916] [ 1075] 1000 1075 1498 0 49152 120 0 ssh-agent [ 1466.913918] [ 1080] 1000 1080 39233 18 69632 57 0 gpg-agent [ 1466.913919] [ 1082] 1000 1082 90685 109 274432 4696 0 xfwm4 [ 1466.913921] [ 1088] 1000 1088 58565 1 176128 1375 0 xfsettingsd [ 1466.913922] [ 1089] 1000 1089 69802 0 184320 1784 0 xfce4-panel [ 1466.913924] [ 1092] 0 1092 65176 0 126976 419 0 upowerd [ 1466.913925] [ 1097] 1000 1097 87154 0 184320 1268 0 Thunar [ 1466.913926] [ 1118] 1000 1118 78926 0 258048 10182 0 xfdesktop [ 1466.913928] [ 1123] 1000 1123 69044 0 167936 1478 0 panel-16-whiske [ 1466.913929] [ 1126] 1000 1126 60071 0 196608 1436 0 panel-22-screen [ 1466.913931] [ 1127] 1000 1127 49794 0 163840 1227 0 panel-12-system [ 1466.913932] [ 1128] 1000 1128 49428 0 159744 1239 0 panel-24-cpugra [ 1466.913933] [ 1129] 1000 1129 50483 0 167936 1603 0 panel-20-cpufre [ 1466.913935] [ 1130] 1000 1130 49795 0 155648 1230 0 panel-13-netloa [ 1466.913936] [ 1131] 1000 1131 49794 0 159744 1267 0 panel-10-diskpe [ 1466.913938] [ 1132] 1000 1132 49795 0 163840 1225 0 panel-4-diskper [ 1466.913939] [ 1133] 1000 1133 49928 0 159744 1242 0 panel-8-xfce4-s [ 1466.913940] [ 1134] 1000 1134 59673 0 192512 1491 0 panel-3-power-m [ 1466.913942] [ 1137] 1000 1137 49443 0 159744 1225 0 panel-6-systray [ 1466.913943] [ 1140] 1000 1140 59653 0 192512 1497 0 panel-11-notifi [ 1466.913944] [ 1141] 1000 1141 58452 4 176128 695 0 panel-17-xfce4- [ 1466.913946] [ 1142] 1000 1142 49851 0 159744 1243 0 panel-2-actions [ 1466.913947] [ 1145] 1000 1145 48311 160 143360 350 0 panel-25-xfce4- [ 1466.913949] [ 1174] 1000 1174 95427 0 212992 2255 0 xfce4-terminal [ 1466.913950] [ 1176] 1000 1176 116553 0 258048 4697 0 xfce4-terminal [ 1466.913951] [ 1189] 1000 1189 48086 0 139264 467 0 polkit-gnome-au [ 1466.913953] [ 1197] 1000 1197 50286 0 159744 1402 0 xfce4-power-man [ 1466.913954] [ 1200] 1000 1200 42772 1 114688 874 0 pulseaudio [ 1466.913956] [ 1206] 1000 1206 67174 0 155648 541 0 xfce4-notifyd [ 1466.913957] [ 1216] 1000 1216 1895 1 53248 121 0 showdns [ 1466.913958] [ 1223] 1000 1223 1862 0 57344 107 0 logw [ 1466.913960] [ 1224] 1000 1224 1895 0 53248 124 0 showdns [ 1466.913961] [ 1225] 1000 1225 1895 1 53248 121 0 showdns_success [ 1466.913963] [ 1226] 1000 1226 1895 1 57344 120 0 showdns_fails [ 1466.913964] [ 1227] 1000 1227 1862 1 53248 120 0 showdns_from_fi [ 1466.913965] [ 1228] 1000 1228 1862 1 57344 120 0 showdns_from_ch [ 1466.913967] [ 1231] 1000 1231 1528 0 53248 82 0 dmesg [ 1466.913968] [ 1254] 1000 1254 1528 0 53248 77 0 dmesg [ 1466.913970] [ 1255] 1000 1255 1862 0 53248 108 0 stripcolors [ 1466.913971] [ 1256] 1000 1256 1625 0 53248 121 0 grep [ 1466.913972] [ 1257] 1000 1257 2479 1 53248 78 0 sed [ 1466.913974] [ 1260] 1000 1260 1528 0 53248 74 0 dmesg [ 1466.913975] [ 1261] 1000 1261 1862 0 57344 107 0 stripcolors [ 1466.913977] [ 1262] 1000 1262 1528 0 53248 87 0 dmesg [ 1466.913978] [ 1263] 1000 1263 1625 0 53248 119 0 grep [ 1466.913979] [ 1264] 1000 1264 1862 0 53248 108 0 stripcolors [ 1466.913981] [ 1265] 1000 1265 1635 0 53248 108 0 grep [ 1466.913982] [ 1266] 1000 1266 1624 0 53248 169 0 grep [ 1466.913984] [ 1267] 1000 1267 2545 0 61440 213 0 sed [ 1466.913985] [ 1268] 1000 1268 1623 0 53248 103 0 grep [ 1466.913986] [ 1269] 1000 1269 1528 0 53248 93 0 dmesg [ 1466.913988] [ 1270] 1000 1270 1862 0 49152 106 0 stripcolors [ 1466.913989] [ 1271] 1000 1271 1627 0 53248 143 0 grep [ 1466.913991] [ 1272] 1000 1272 2512 0 61440 154 0 sed [ 1466.913992] [ 1273] 1000 1273 2479 0 57344 101 0 sed [ 1466.913993] [ 1274] 1000 1274 2479 0 61440 94 0 sed [ 1466.913995] [ 1275] 1000 1275 2479 0 57344 82 0 sed [ 1466.913996] [ 1276] 1000 1276 2479 0 57344 114 0 sed [ 1466.913998] [ 1325] 133 1325 38725 0 73728 106 0 rtkit-daemon [ 1466.913999] [ 1336] 1000 1336 2649 0 69632 99 0 dbus-daemon [ 1466.914000] [ 1364] 1000 1364 61444 0 106496 288 0 gsettings-helpe [ 1466.914002] [ 1369] 1000 1369 57721 0 81920 137 0 xfconfd [ 1466.914003] [ 1765] 1000 1765 39295 0 69632 188 0 dconf-service [ 1466.914005] [ 3405] 1000 3405 3410 1 69632 1632 0 bash [ 1466.914007] [ 3472] 1000 3472 3487 0 61440 277 0 top [ 1466.914008] [ 3824] 1000 3824 3411 1 69632 1656 0 bash [ 1466.914010] [ 5947] 1000 5947 3410 1 73728 1633 0 bash [ 1466.914012] [ 11271] 1000 11271 3410 1 73728 1669 0 bash [ 1466.914014] [ 11355] 1000 11355 1862 1 53248 113 0 v [ 1466.914015] [ 11363] 1000 11363 16261 0 167936 1709 0 vim [ 1466.914017] [ 14553] 1000 14553 3501 1 73728 1714 0 bash [ 1466.914019] [ 14624] 1000 14624 1646 131 65536 82 0 watch [ 1466.914020] [ 15658] 1000 15658 947 1 49152 26 0 stress [ 1466.914022] [ 15659] 1000 15659 2442354 35472 733184 49380 0 stress [ 1466.914023] [ 15660] 1000 15660 2442354 17461 712704 64448 0 stress [ 1466.914025] [ 15661] 1000 15661 2442354 31447 1048576 92880 0 stress [ 1466.914026] [ 15662] 1000 15662 2442354 24856 692224 54878 0 stress [ 1466.914027] [ 15663] 1000 15663 2442354 78517 1007616 40479 0 stress [ 1466.914029] [ 15664] 1000 15664 2442354 28535 946176 82962 0 stress [ 1466.914030] [ 15665] 1000 15665 2442354 763 458752 49668 0 stress [ 1466.914032] [ 15666] 1000 15666 2442354 33217 905216 73150 0 stress [ 1466.914033] [ 15667] 1000 15667 2442354 27000 1216512 118297 0 stress [ 1466.914034] [ 15668] 1000 15668 2442354 30129 868352 71648 0 stress [ 1466.914036] [ 15669] 1000 15669 2442354 58911 1298432 96600 0 stress [ 1466.914037] [ 15670] 1000 15670 2442354 34067 1028096 87659 0 stress [ 1466.914038] [ 15671] 1000 15671 2442354 37665 1290240 116863 0 stress [ 1466.914040] [ 15672] 1000 15672 2442354 27605 1064960 98712 0 stress [ 1466.914041] [ 15673] 1000 15673 2442354 72245 1294336 82778 0 stress [ 1466.914042] [ 15674] 1000 15674 2442354 32309 1101824 98646 0 stress [ 1466.914044] [ 15675] 1000 15675 2442354 16205 659456 59473 0 stress [ 1466.914045] [ 15676] 1000 15676 2442354 75163 1179648 65513 0 stress [ 1466.914047] [ 15677] 1000 15677 2442354 1 577536 65327 0 stress [ 1466.914048] [ 15678] 1000 15678 2442354 87 643072 73506 0 stress [ 1466.914050] [ 15679] 1000 15679 2442354 21381 598016 46576 0 stress [ 1466.914051] [ 15680] 1000 15680 2442354 27317 925696 81287 0 stress [ 1466.914052] [ 15681] 1000 15681 2442354 53527 1007616 65673 0 stress [ 1466.914054] [ 15682] 1000 15682 2442354 33620 790528 58401 0 stress [ 1466.914055] [ 15683] 1000 15683 2442354 22478 688128 56765 0 stress [ 1466.914056] [ 15684] 1000 15684 2442354 45031 774144 44974 0 stress [ 1466.914058] [ 15685] 1000 15685 2442354 167 466944 51456 0 stress [ 1466.914059] [ 15686] 1000 15686 2442354 30506 684032 48204 0 stress [ 1466.914060] [ 15687] 1000 15687 2442354 76 360448 38174 0 stress [ 1466.914062] [ 15688] 1000 15688 2442354 27924 716800 54795 0 stress [ 1466.914063] [ 15689] 1000 15689 2442354 24105 823296 72068 0 stress [ 1466.914064] [ 15690] 1000 15690 2442354 13187 651264 60993 0 stress [ 1466.914066] [ 15691] 1000 15691 2442354 20202 675840 57520 0 stress [ 1466.914067] [ 15692] 1000 15692 2442354 37068 1036288 85690 0 stress [ 1466.914069] [ 15693] 1000 15693 2442354 34960 753664 52047 0 stress [ 1466.914070] [ 15694] 1000 15694 2442354 33364 1003520 85280 0 stress [ 1466.914071] [ 15695] 1000 15695 2442354 37500 1122304 95993 0 stress [ 1466.914073] [ 15696] 1000 15696 2442354 40010 774144 49648 0 stress [ 1466.914074] [ 15697] 1000 15697 2442354 25200 733184 59669 0 stress [ 1466.914075] [ 15698] 1000 15698 2442354 33928 860160 66367 0 stress [ 1466.914077] [ 15699] 1000 15699 2442354 28883 696320 51372 0 stress [ 1466.914078] [ 15700] 1000 15700 2442354 63688 1032192 58535 0 stress [ 1466.914079] [ 15701] 1000 15701 2442354 206 462848 50676 0 stress [ 1466.914081] [ 15702] 1000 15702 2442354 37586 978944 78007 0 stress [ 1466.914082] [ 15703] 1000 15703 2442354 57038 950272 54697 0 stress [ 1466.914083] [ 15704] 1000 15704 2442354 23691 688128 55570 0 stress [ 1466.914085] [ 15705] 1000 15705 2442354 24519 704512 56769 0 stress [ 1466.914086] [ 15706] 1000 15706 2442354 41070 831488 56065 0 stress [ 1466.914087] [ 15707] 1000 15707 2442354 16687 950272 95308 0 stress [ 1466.914089] [ 15708] 1000 15708 2442354 72969 1179648 67689 0 stress [ 1466.914090] [ 15709] 1000 15709 2442354 46107 962560 67440 0 stress [ 1466.914091] [ 15710] 1000 15710 2442354 30480 733184 54368 0 stress [ 1466.914093] [ 15711] 1000 15711 2442354 28799 757760 58943 0 stress [ 1466.914094] [ 15712] 1000 15712 2442354 24325 806912 69759 0 stress [ 1466.914095] [ 15713] 1000 15713 2442354 17610 737280 67758 0 stress [ 1466.914097] [ 15714] 1000 15714 2442354 26403 917504 81508 0 stress [ 1466.914098] [ 15715] 1000 15715 2442354 1099 577536 64016 0 stress [ 1466.914099] [ 15716] 1000 15716 2442354 36219 880640 67074 0 stress [ 1466.914101] [ 15717] 1000 15717 2442354 47198 1003520 71486 0 stress [ 1466.914102] [ 15718] 1000 15718 2442354 22209 798720 70805 0 stress [ 1466.914103] [ 15719] 1000 15719 2442354 36188 729088 48179 0 stress [ 1466.914105] [ 15720] 1000 15720 2442354 48222 1081344 80144 0 stress [ 1466.914106] [ 15721] 1000 15721 2442354 114843 1372160 49891 0 stress [ 1466.914108] [ 15722] 1000 15722 2442354 162 557056 62352 0 stress [ 1466.914109] [ 15723] 1000 15723 2442354 34078 843776 64635 0 stress [ 1466.914110] [ 15724] 1000 15724 2442354 1 516096 57478 0 stress [ 1466.914112] [ 15725] 1000 15725 2442354 33713 684032 45031 0 stress [ 1466.914113] [ 15726] 1000 15726 2442354 70880 1007616 48303 0 stress [ 1466.914114] [ 15727] 1000 15727 2442354 33070 794624 59294 0 stress [ 1466.914116] [ 15728] 1000 15728 2442354 49921 884736 53910 0 stress [ 1466.914117] [ 15729] 1000 15729 2442354 37100 786432 54438 0 stress [ 1466.914118] [ 15730] 1000 15730 2442354 772 745472 85673 0 stress [ 1466.914120] [ 15731] 1000 15731 2442354 20798 647168 53336 0 stress [ 1466.914121] [ 15732] 1000 15732 2442354 66860 1286144 87132 0 stress [ 1466.914122] [ 15733] 1000 15733 2442354 58022 1003520 60648 0 stress [ 1466.914124] [ 15734] 1000 15734 2442354 40229 794624 52337 0 stress [ 1466.914125] [ 15735] 1000 15735 2442354 52416 831488 44743 0 stress [ 1466.914127] [ 15736] 1000 15736 2442354 22784 1220608 123039 0 stress [ 1466.914128] [ 15737] 1000 15737 2442354 47752 1228800 99082 0 stress [ 1466.914130] [ 15738] 1000 15738 2442354 5388 598016 62219 0 stress [ 1466.914131] [ 15739] 1000 15739 2442354 42512 991232 74612 0 stress [ 1466.914132] [ 15740] 1000 15740 2442354 30409 843776 68268 0 stress [ 1466.914134] [ 15741] 1000 15741 2442354 61 385024 41076 0 stress [ 1466.914135] [ 15742] 1000 15742 2442354 22049 688128 57225 0 stress [ 1466.914136] [ 15743] 1000 15743 2442354 11593 1163264 127053 0 stress [ 1466.914138] [ 15744] 1000 15744 2442354 12709 700416 68066 0 stress [ 1466.914139] [ 15745] 1000 15745 2442354 58506 966656 55522 0 stress [ 1466.914140] [ 15746] 1000 15746 2442354 37704 978944 77912 0 stress [ 1466.914142] [ 15747] 1000 15747 2442354 557 430080 46244 0 stress [ 1466.914143] [ 15748] 1000 15748 2442354 47961 929792 61466 0 stress [ 1466.914144] [ 15749] 1000 15749 2442354 35600 880640 67733 0 stress [ 1466.914146] [ 15750] 1000 15750 2442354 36018 897024 69355 0 stress [ 1466.914147] [ 15751] 1000 15751 2442354 397 552960 61659 0 stress [ 1466.914149] [ 15752] 1000 15752 2442354 9336 745472 77088 0 stress [ 1466.914150] [ 15753] 1000 15753 2442354 57814 1249280 91607 0 stress [ 1466.914151] [ 15754] 1000 15754 2442354 46202 1044480 77582 0 stress [ 1466.914153] [ 15755] 1000 15755 2442354 63801 1069056 63083 0 stress [ 1466.914154] [ 15756] 1000 15756 2442354 21609 765952 67300 0 stress [ 1466.914155] [ 15757] 1000 15757 2442354 59274 892928 45599 0 stress [ 1466.914157] [ 15758] 1000 15758 2442354 1 479232 52862 0 stress [ 1466.914158] [ 15759] 1000 15759 2442354 76444 1163264 62175 0 stress [ 1466.914160] [ 15760] 1000 15760 2442354 31653 933888 78283 0 stress [ 1466.914161] [ 15761] 1000 15761 2442354 30021 790528 62026 0 stress [ 1466.914163] [ 15762] 1000 15762 2442354 45007 901120 60875 0 stress [ 1466.914164] [ 15763] 1000 15763 2442354 21275 659456 54316 0 stress [ 1466.914165] [ 15764] 1000 15764 2442354 29773 778240 60715 0 stress [ 1466.914167] [ 15765] 1000 15765 2442354 34685 815104 60423 0 stress [ 1466.914168] [ 15766] 1000 15766 2442354 19320 741376 66595 0 stress [ 1466.914169] [ 15767] 1000 15767 2442354 26619 630784 45137 0 stress [ 1466.914171] [ 15768] 1000 15768 2442354 65238 1044480 58545 0 stress [ 1466.914172] [ 15769] 1000 15769 2442354 34694 851968 65018 0 stress [ 1466.914174] [ 15770] 1000 15770 2442354 7212 589824 59319 0 stress [ 1466.914175] [ 15771] 1000 15771 2442354 35594 729088 48796 0 stress [ 1466.914177] [ 15772] 1000 15772 2442354 17326 811008 77258 0 stress [ 1466.914178] [ 15773] 1000 15773 2442354 32841 1118208 100160 0 stress [ 1466.914180] [ 15774] 1000 15774 2442354 202 847872 98614 0 stress [ 1466.914181] [ 15775] 1000 15775 2442354 31 651264 74185 0 stress [ 1466.914182] [ 15776] 1000 15776 2442354 44425 1032192 77815 0 stress [ 1466.914184] [ 15777] 1000 15777 2442354 23280 675840 54459 0 stress [ 1466.914185] [ 15778] 1000 15778 2442354 20222 839680 77633 0 stress [ 1466.914187] [ 15779] 1000 15779 2442354 34766 1187840 106667 0 stress [ 1466.914188] [ 15780] 1000 15780 2442354 18561 540672 42251 0 stress [ 1466.914189] [ 15781] 1000 15781 2442354 157 360448 37959 0 stress [ 1466.914191] [ 15782] 1000 15782 2442354 31124 733184 53724 0 stress [ 1466.914209] [ 15783] 1000 15783 2442354 25112 638976 48009 0 stress [ 1466.914210] [ 15784] 1000 15784 2442354 9932 495616 45085 0 stress [ 1466.914212] [ 15785] 1000 15785 2442354 33793 724992 50071 0 stress [ 1466.914213] [ 15786] 1000 15786 2442354 20149 860160 80493 0 stress [ 1466.914214] [ 15787] 1000 15787 2442354 49386 1040384 73876 0 stress [ 1466.914216] [ 15788] 1000 15788 2442354 26792 794624 65738 0 stress [ 1466.914217] [ 15789] 1000 15789 2442354 463 647168 73278 0 stress [ 1466.914219] [ 15790] 1000 15790 2442354 27350 651264 47280 0 stress [ 1466.914220] [ 15791] 1000 15791 2442354 27893 974848 87221 0 stress [ 1466.914221] [ 15792] 1000 15792 2442354 51549 917504 56361 0 stress [ 1466.914223] [ 15793] 1000 15793 2442354 31852 860160 68872 0 stress [ 1466.914224] [ 15794] 1000 15794 2442354 28996 827392 67626 0 stress [ 1466.914226] [ 15795] 1000 15795 2442354 49525 806912 44556 0 stress [ 1466.914227] [ 15796] 1000 15796 2442354 59806 819200 35827 0 stress [ 1466.914228] [ 15797] 1000 15797 2442354 42778 966656 71305 0 stress [ 1466.914230] [ 15798] 1000 15798 2442354 29384 745472 57056 0 stress [ 1466.914231] [ 15799] 1000 15799 2442354 29609 655360 45554 0 stress [ 1466.914232] [ 15800] 1000 15800 2442354 19677 602112 48838 0 stress [ 1466.914234] [ 15801] 1000 15801 2442354 91631 1261568 59202 0 stress [ 1466.914235] [ 15802] 1000 15802 2442354 37784 708608 44013 0 stress [ 1466.914236] [ 15803] 1000 15803 2442354 38862 843776 59834 0 stress [ 1466.914238] [ 15804] 1000 15804 2442354 53364 1110016 78641 0 stress [ 1466.914239] [ 15805] 1000 15805 2442354 93 380928 40778 0 stress [ 1466.914241] [ 15806] 1000 15806 2442354 38511 1003520 80159 0 stress [ 1466.914242] [ 15807] 1000 15807 2442354 73352 1150976 63780 0 stress [ 1466.914243] [ 15808] 1000 15808 2442354 41336 745472 45096 0 stress [ 1466.914245] [ 15809] 1000 15809 2442354 22708 831488 74460 0 stress [ 1466.914246] [ 15810] 1000 15810 2442354 30775 716800 52072 0 stress [ 1466.914247] [ 15811] 1000 15811 2442354 28548 1024000 92686 0 stress [ 1466.914249] [ 15812] 1000 15812 2442354 45063 1224704 101243 0 stress [ 1466.914250] [ 15813] 1000 15813 2442354 41298 786432 50210 0 stress [ 1466.914252] [ 15814] 1000 15814 2442354 16514 557056 45980 0 stress [ 1466.914253] [ 15815] 1000 15815 2442354 38228 950272 73777 0 stress [ 1466.914255] [ 15816] 1000 15816 2442354 3546 1073152 123763 0 stress [ 1466.914256] [ 15817] 1000 15817 2442354 228 774144 89437 0 stress [ 1466.914258] [ 15818] 1000 15818 2442354 147848 2117632 109587 0 stress [ 1466.914259] [ 15819] 1000 15819 2442354 30535 1105920 100966 0 stress [ 1466.914260] [ 15820] 1000 15820 2442354 35952 1097728 94515 0 stress [ 1466.914262] [ 15821] 1000 15821 2442354 1 716800 82464 0 stress [ 1466.914263] [ 15822] 1000 15822 2442354 24288 1056768 101008 0 stress [ 1466.914265] [ 15823] 1000 15823 2442354 82 753664 87027 0 stress [ 1466.914266] [ 15824] 1000 15824 2442354 46572 1122304 86919 0 stress [ 1466.914268] [ 15825] 1000 15825 2442354 27250 1073152 100103 0 stress [ 1466.914269] [ 15826] 1000 15826 2442354 16521 851968 83211 0 stress [ 1466.914270] [ 15827] 1000 15827 2442354 65108 1241088 83270 0 stress [ 1466.914272] [ 15828] 1000 15828 2442354 42946 995328 74688 0 stress [ 1466.914273] [ 15829] 1000 15829 2442354 29091 1073152 97972 0 stress [ 1466.914275] [ 15830] 1000 15830 2442354 43149 1134592 91934 0 stress [ 1466.914276] [ 15831] 1000 15831 2442354 20040 962560 93516 0 stress [ 1466.914277] [ 15832] 1000 15832 2442354 39625 1032192 82643 0 stress [ 1466.914279] [ 15833] 1000 15833 2442354 28676 802816 64882 0 stress [ 1466.914280] [ 15834] 1000 15834 2442354 32089 1122304 101453 0 stress [ 1466.914281] [ 15835] 1000 15835 2442354 53649 1032192 68572 0 stress [ 1466.914283] [ 15836] 1000 15836 2442354 176 585728 66213 0 stress [ 1466.914284] [ 15837] 1000 15837 2442354 16 487424 53694 0 stress [ 1466.914286] [ 15838] 1000 15838 2442354 34092 753664 53328 0 stress [ 1466.914287] [ 15839] 1000 15839 2442354 24923 733184 59956 0 stress [ 1466.914288] [ 15840] 1000 15840 2442354 75766 1343488 85385 0 stress [ 1466.914290] [ 15841] 1000 15841 2442354 24663 819200 70933 0 stress [ 1466.914291] [ 15842] 1000 15842 2442354 53107 966656 60930 0 stress [ 1466.914292] [ 15843] 1000 15843 2442354 38419 925696 70548 0 stress [ 1466.914294] [ 15844] 1000 15844 2442354 34410 823296 61763 0 stress [ 1466.914295] [ 15845] 1000 15845 2442354 18088 724992 65749 0 stress [ 1466.914297] [ 15846] 1000 15846 2442354 48202 1228800 98646 0 stress [ 1466.914298] [ 15847] 1000 15847 2442354 82782 1212416 61974 0 stress [ 1466.914299] [ 15848] 1000 15848 2442354 19695 872448 82357 0 stress [ 1466.914301] [ 15849] 1000 15849 2442354 34530 860160 66229 0 stress [ 1466.914302] [ 15850] 1000 15850 2442354 40215 819200 55441 0 stress [ 1466.914303] [ 15851] 1000 15851 2442354 33624 716800 49227 0 stress [ 1466.914305] [ 15852] 1000 15852 2442354 55436 995328 62192 0 stress [ 1466.914306] [ 15853] 1000 15853 2442354 21205 929792 88066 0 stress [ 1466.914308] [ 15854] 1000 15854 2442354 52815 1171456 86823 0 stress [ 1466.914309] [ 15855] 1000 15855 2442354 16708 667648 59546 0 stress [ 1466.914310] [ 15856] 1000 15856 2442354 33506 925696 75409 0 stress [ 1466.914312] [ 15857] 1000 15857 2442354 4436 417792 41037 0 stress [ 1466.914313] [ 15858] 1000 15858 2442354 41905 839680 56283 0 stress [ 1466.914314] [ 15859] 1000 15859 2442354 16448 802816 77136 0 stress [ 1466.914316] [ 15860] 1000 15860 2442354 48905 950272 63128 0 stress [ 1466.914317] [ 15861] 1000 15861 2442354 14641 704512 66276 0 stress [ 1466.914319] [ 15862] 1000 15862 2442354 244 638976 72828 0 stress [ 1466.914320] [ 15863] 1000 15863 2442354 37726 978944 77901 0 stress [ 1466.914321] [ 15864] 1000 15864 2442354 69044 1224704 77302 0 stress [ 1466.914323] [ 15865] 1000 15865 2442354 50818 892928 54057 0 stress [ 1466.914324] [ 15866] 1000 15866 2442354 149 815104 94511 0 stress [ 1466.914326] [ 15867] 1000 15867 2442354 21797 753664 65579 0 stress [ 1466.914327] [ 15868] 1000 15868 2442354 86029 3780608 379238 0 stress [ 1466.914329] [ 15869] 1000 15869 2442354 35631 1433600 136771 0 stress [ 1466.914330] [ 15870] 1000 15870 2442354 73646 1695744 131540 0 stress [ 1466.914332] [ 15871] 1000 15871 2442354 70948 1466368 105576 0 stress [ 1466.914333] [ 15872] 1000 15872 2442354 627 2543616 309624 0 stress [ 1466.914335] [ 15873] 1000 15873 2442354 592 802816 93005 0 stress [ 1466.914336] [ 15874] 1000 15874 2442354 50241 3207168 343395 0 stress [ 1466.914337] [ 15875] 1000 15875 2442354 33324 1392640 133978 0 stress [ 1466.914339] [ 15876] 1000 15876 2442354 75920 1761280 137479 0 stress [ 1466.914340] [ 15877] 1000 15877 2442354 36961 3280896 365416 0 stress [ 1466.914341] [ 15878] 1000 15878 2442354 33187 1441792 140260 0 stress [ 1466.914342] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=stress,pid=15868,uid=1000 [ 1466.914349] Out of memory: Killed process 15868 (stress) total-vm:9769416kB, anon-rss:344112kB, file-rss:4kB, shmem-rss:0kB [ 1467.129760] oom_reaper: reaped process 15868 (stress), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB I got one at 19.2sec $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 stress: info: [21374] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: FAIL: [21374] (415) <-- worker 21585 got signal 9 stress: WARN: [21374] (417) now reaping child worker processes stress: FAIL: [21374] (451) failed run completed in 19s real 0m19.223s user 0m1.077s sys 2m53.496s I did 45 tries without tracers. Recompiled kernel just like the last time(when I was able to trigger it on the third try) aka with tracers: I did 15 tries and they were all successfull(at full CPU speed) with durations between 17-22 sec. I guess all that tracing code in kernel introduced some minor lag which is enough for this to never trigger OOM-killer, just like when I limit cpu max to 800Mhz. I even tried running two in parallel, no OOM-killer. Or this: $ time stress -m 220 --vm-bytes 10000000000 --timeout 30 stress: info: [12381] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: info: [12381] successful run completed in 56s real 0m55.957s user 0m1.908s sys 9m20.969s Longer timeout I could see 65000MiB of swap were used at some point! (in 'top') still nothing triggered OOM-killer :D ok, a timeout of 60 did it(triggered OOM): $ time stress -m 220 --vm-bytes 10000000000 --timeout 60 stress: info: [13225] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd stress: FAIL: [13225] (415) <-- worker 13445 got signal 9 stress: WARN: [13225] (417) now reaping child worker processes stress: FAIL: [13225] (451) failed run completed in 59s real 0m58.812s user 0m2.312s sys 9m53.327s But anyway, since last time I was able to trigger it with the normal(timeout 10) command on the third try, I've decided to keep trying that: after 170 more tries via `$ while true; do time stress -m 220 --vm-bytes 10000000000 --timeout 10; done` I saw no hangs or any runs taking more time than the usual 16-26 sec. So the patch must be working as intended. Thanks very much. Let me know if you want me to do anything else. Cheers! On Wed, Jul 17, 2019 at 10:00:18PM +0000, howaboutsynergy@protonmail.com wrote: > tl;dr: patch seems to work, thank you very much! > \o/ > ????????????????????? Original Message ????????????????????? > On Wednesday, July 17, 2019 7:53 PM, Mel Gorman <mgorman@techsingularity.net> > wrote: > > > Ok, great. From the trace, it was obvious that the scanner is making no > > progress. I don't think zswap is involved as such but it may be making > > it easier to trigger due to altering timing. At least, I see no reason > > why zswap would materially affect the termination conditions. > > I don't know if it matters in this context, but I've been using the term > `zswap`(somewhere else I think) to (wrongly)refer to swap in zram (and > even sometimes called it ext4(in this bug report too) without realizing > at the time that ext4 is only for /tmp and /var/tmp instead! they are > ext4 in zram) but in fact this isn't zswap that I have been using (even > though I have CONFIG_ZSWAP=y in .config) but it's just CONFIG_ZRAM=y with > CONFIG_SWAP=y (and probably a bunch of others being needed too). For other bugs, it would matter a lot. In this specific case, all that was altered was the timing. With enough effort I was able to reproduce the problem reliably within 30 minutes. With the patch, it ran without hitting the problem for hours. I had tracing patches applied to log when the specific problem occurred. Without the patch, when the problem occurred, it hit 197009 times in a tight loop driving up CPU usage. With the patch, the same condition was hit 4 times and in each case the task exited immediately as expected so I think we're good. > > a proper abort. I think it ends up looping in compaction instead of dying > > without either aborting or progressing the scanner. It might explain why > > stress-ng is hitting is as it is probably sending fatal signals on timeout > > (I didn't check the source). > Ah I didn't know there are multiple `stress` versions, here's what I used: > > /usr/bin/stress is owned by stress 1.0.4-5 > Fortunately, they were all equivalent. The key was getting a task to receive a fatal signal while compacting and while scanning a PFN that was not aligned to SWAP_CLUSTER_MAX. Tricky to hit but fortunately your test case was exactly what we needed. > > <SNIP> > > Now the "problem" is I can't tell if it would get stuck :D but it usually > ends in no more than 17 sec: > $ time stress -m 220 --vm-bytes 10000000000 --timeout 10 > stress: info: [7981] dispatching hogs: 0 cpu, 0 io, 220 vm, 0 hdd > stress: FAIL: [7981] (415) <-- worker 8202 got signal 9 > stress: WARN: [7981] (417) now reaping child worker processes > stress: FAIL: [7981] (415) <-- worker 8199 got signal 9 > stress: WARN: [7981] (417) now reaping child worker processes > stress: FAIL: [7981] (451) failed run completed in 18s > That's fine. Fortunately my own debug tracing added based on your trace gives me enough confidence. > <SNIP> > > (probably irrelevant)Sometimes Xorg says it can't allocate any more memory > but stacktrace looks like it's inside some zram i915 kernel stuff: > > [ 1416.842931] [drm] Atomic update on pipe (A) took 188 us, max time under > evasion is 100 us > [ 1425.416979] Xorg: page allocation failure: order:0, > mode:0x400d0(__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_RECLAIMABLE), > nodemask=(null),cpuset=/,mems_allowed=0 > [ 1425.416984] CPU: 1 PID: 1024 Comm: Xorg Kdump: loaded Tainted: G U > 5.2.1-g527a3db363a3 #74 > [ 1425.416985] Hardware name: System manufacturer System Product Name/PRIME > Z370-A, BIOS 2201 05/27/2019 > [ 1425.416986] Call Trace: So this looks like the system is still under a lot of stress trying to swap. It's unfortunate but unrelated and relatively benign given the level of stress the system is under. > <SNIP> > > But anyway, since last time I was able to trigger it with the normal(timeout > 10) command on the third try, I've decided to keep trying that: > after 170 more tries via `$ while true; do time stress -m 220 --vm-bytes > 10000000000 --timeout 10; done` > I saw no hangs or any runs taking more time than the usual 16-26 sec. > So the patch must be working as intended. Thanks very much. Let me know if > you want me to do anything else. > Perfect. Thanks a million for your patience, testing and tracing (the tracing pinpointed exactly where I needed to look -- of 3 potential candidates, only one really made sense). I'll put a proper changelog on this and send it out where it should get picked up for 5.3 and -stable. For completeness this is what I've had to change in kernel .config to get trace-cmd to work: seen here https://bugzilla.kernel.org/show_bug.cgi?id=204231#c3 Probably less CONFIG_* settings needed to be added than that, but I just made sure and added a bunch(if not all) so that trace-cmd didn't complain. |