Bug 210177

Summary: mlx5_core crashed with AMD-Vi: Event logged [IO_PAGE_FAULT...
Product: Drivers Reporter: Christian Theune (ct)
Component: NetworkAssignee: drivers_network (drivers_network)
Status: NEW ---    
Severity: high    
Priority: P1    
Hardware: x86-64   
OS: Linux   
See Also: https://bugzilla.kernel.org/show_bug.cgi?id=210153
Kernel Version: 4.19.157 Subsystem:
Regression: No Bisected commit-id:

Description Christian Theune 2020-11-12 17:09:21 UTC 
This is with a Mellanox Connect-X4:

Nov 12 16:48:03 cartman21 kernel: [245468.944890] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d3dbd8ac flags=0x0000]
Nov 12 16:48:03 cartman21 kernel: [245468.949584] mlx5_core 0000:42:00.0 ethstb: Error cqe on cqn 0xa6, ci 0x190, sqn 0x1a6, syndrome 0x4, vendor syndrome 0x51
Nov 12 16:48:03 cartman21 kernel: [245468.949592] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Nov 12 16:48:03 cartman21 kernel: [245468.949593] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Nov 12 16:48:03 cartman21 kernel: [245468.949594] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Nov 12 16:48:03 cartman21 kernel: [245468.949595] 00000030: 00 00 00 00 04 00 51 04 0a 00 01 a6 7e 64 d6 d2
Nov 12 16:48:04 cartman21 kernel: [245469.138476] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e2c33000 flags=0x0020]
Nov 12 16:48:04 cartman21 kernel: [245469.628296] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e4443000 flags=0x0020]
Nov 12 16:48:04 cartman21 kernel: [245469.642000] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ea350000 flags=0x0020]
Nov 12 16:48:04 cartman21 kernel: [245469.642016] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ea350700 flags=0x0020]
Nov 12 16:48:05 cartman21 kernel: [245470.340177] ethstb: hw csum failure
Nov 12 16:48:05 cartman21 kernel: [245470.340182] CPU: 20 PID: 587365 Comm: tp_osd_tp Not tainted 4.19.85 #1
Nov 12 16:48:05 cartman21 kernel: [245470.340183] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:05 cartman21 kernel: [245470.340184] Call Trace:
Nov 12 16:48:05 cartman21 kernel: [245470.340188]  <IRQ>
Nov 12 16:48:05 cartman21 kernel: [245470.340197]  dump_stack+0x50/0x70
Nov 12 16:48:05 cartman21 kernel: [245470.340203]  __skb_gro_checksum_complete.cold.168+0x1f/0x24
Nov 12 16:48:05 cartman21 kernel: [245470.340209]  tcp4_gro_receive+0xc6/0x1a0
Nov 12 16:48:05 cartman21 kernel: [245470.340213]  inet_gro_receive+0x20f/0x290
Nov 12 16:48:05 cartman21 kernel: [245470.340217]  dev_gro_receive+0x2ad/0x620
Nov 12 16:48:05 cartman21 kernel: [245470.340220] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f0d5a000 flags=0x0070]
Nov 12 16:48:05 cartman21 kernel: [245470.340242]  ? mlx5e_build_rx_skb+0x2c2/0xa80 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340258]  ? mlx5e_skb_from_cqe_nonlinear+0x5c/0x250 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340261]  napi_gro_receive+0x33/0xe0
Nov 12 16:48:05 cartman21 kernel: [245470.340277]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340294]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340312]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340318]  ? __note_gp_changes+0x14c/0x230
Nov 12 16:48:05 cartman21 kernel: [245470.340321]  net_rx_action+0x12f/0x350
Nov 12 16:48:05 cartman21 kernel: [245470.340325]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:05 cartman21 kernel: [245470.340330]  irq_exit+0xb4/0xc0
Nov 12 16:48:05 cartman21 kernel: [245470.340333]  do_IRQ+0x7f/0xe0
Nov 12 16:48:05 cartman21 kernel: [245470.340336]  common_interrupt+0xf/0xf
Nov 12 16:48:05 cartman21 kernel: [245470.340338]  </IRQ>
Nov 12 16:48:05 cartman21 kernel: [245470.340340] RIP: 0033:0x7f3541055dd9
Nov 12 16:48:05 cartman21 kernel: [245470.340343] Code: 48 8b 05 62 f5 20 00 48 8b 00 49 81 fc 00 00 04 00 0f 87 a8 00 00 00 48 8b 15 23 fc 20 00 48 63 4d cc 0f b6 b4 0a 58 01 00 00 <4c> 8b bc f2 d8 09 00 00 48 85 c0 7e 14 48 8b 43 20 49 39 c7 0f 87
Nov 12 16:48:05 cartman21 kernel: [245470.340345] RSP: 002b:00007f351c2f5ad0 EFLAGS: 00000287 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:05 cartman21 kernel: [245470.340349] RAX: 0000000000000000 RBX: 000055f64ac88ac0 RCX: 0000000000000004
Nov 12 16:48:05 cartman21 kernel: [245470.340350] RDX: 00007f3541286120 RSI: 0000000000000003 RDI: 000000000000001b
Nov 12 16:48:05 cartman21 kernel: [245470.340352] RBP: 00007f351c2f5b10 R08: 00007f351c2f5a80 R09: 0000000000000020
Nov 12 16:48:05 cartman21 kernel: [245470.340353] R10: 00007f353e8ca400 R11: 0000000000000000 R12: 000000000000001b
Nov 12 16:48:05 cartman21 kernel: [245470.340354] R13: 0000000000000001 R14: 0000000000002000 R15: 0000000000000000
Nov 12 16:48:05 cartman21 kernel: [245470.340360] ethstb: hw csum failure
Nov 12 16:48:05 cartman21 kernel: [245470.340363] CPU: 20 PID: 587365 Comm: tp_osd_tp Not tainted 4.19.85 #1
Nov 12 16:48:05 cartman21 kernel: [245470.340364] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:05 cartman21 kernel: [245470.340364] Call Trace:
Nov 12 16:48:05 cartman21 kernel: [245470.340366]  <IRQ>
Nov 12 16:48:05 cartman21 kernel: [245470.340369]  dump_stack+0x50/0x70
Nov 12 16:48:05 cartman21 kernel: [245470.340373]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:05 cartman21 kernel: [245470.340386]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:05 cartman21 kernel: [245470.340397]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:05 cartman21 kernel: [245470.340403]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:05 cartman21 kernel: [245470.340407]  ip_rcv+0x91/0xd0
Nov 12 16:48:05 cartman21 kernel: [245470.340411]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:05 cartman21 kernel: [245470.340414]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:05 cartman21 kernel: [245470.340417]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:05 cartman21 kernel: [245470.340420]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:05 cartman21 kernel: [245470.340437]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340453]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340469]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:05 cartman21 kernel: [245470.340472]  ? __note_gp_changes+0x14c/0x230
Nov 12 16:48:05 cartman21 kernel: [245470.340475]  net_rx_action+0x12f/0x350
Nov 12 16:48:05 cartman21 kernel: [245470.340477]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:05 cartman21 kernel: [245470.340480]  irq_exit+0xb4/0xc0
Nov 12 16:48:05 cartman21 kernel: [245470.340482]  do_IRQ+0x7f/0xe0
Nov 12 16:48:05 cartman21 kernel: [245470.340484]  common_interrupt+0xf/0xf
Nov 12 16:48:05 cartman21 kernel: [245470.340484]  </IRQ>
Nov 12 16:48:05 cartman21 kernel: [245470.340486] RIP: 0033:0x7f3541055dd9
Nov 12 16:48:05 cartman21 kernel: [245470.340487] Code: 48 8b 05 62 f5 20 00 48 8b 00 49 81 fc 00 00 04 00 0f 87 a8 00 00 00 48 8b 15 23 fc 20 00 48 63 4d cc 0f b6 b4 0a 58 01 00 00 <4c> 8b bc f2 d8 09 00 00 48 85 c0 7e 14 48 8b 43 20 49 39 c7 0f 87
Nov 12 16:48:05 cartman21 kernel: [245470.340488] RSP: 002b:00007f351c2f5ad0 EFLAGS: 00000287 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:05 cartman21 kernel: [245470.340490] RAX: 0000000000000000 RBX: 000055f64ac88ac0 RCX: 0000000000000004
Nov 12 16:48:05 cartman21 kernel: [245470.340490] RDX: 00007f3541286120 RSI: 0000000000000003 RDI: 000000000000001b
Nov 12 16:48:05 cartman21 kernel: [245470.340491] RBP: 00007f351c2f5b10 R08: 00007f351c2f5a80 R09: 0000000000000020
Nov 12 16:48:05 cartman21 kernel: [245470.340492] R10: 00007f353e8ca400 R11: 0000000000000000 R12: 000000000000001b
Nov 12 16:48:05 cartman21 kernel: [245470.340493] R13: 0000000000000001 R14: 0000000000002000 R15: 0000000000000000
Nov 12 16:48:06 cartman21 kernel: [245471.791731] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000aee42000 flags=0x0020]
Nov 12 16:48:06 cartman21 kernel: [245471.870338] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d34b1000 flags=0x0020]
Nov 12 16:48:06 cartman21 kernel: [245471.870354] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d34b1f00 flags=0x0020]
Nov 12 16:48:06 cartman21 kernel: [245471.870369] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c6f80000 flags=0x0020]
Nov 12 16:48:06 cartman21 kernel: [245471.870384] amd_iommu_report_page_fault: 7 callbacks suppressed
Nov 12 16:48:06 cartman21 kernel: [245471.870386] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000e7eb5000 flags=0x0020]
Nov 12 16:48:06 cartman21 kernel: [245471.870400] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000d9e30000 flags=0x0020]
Nov 12 16:48:07 cartman21 kernel: [245472.494434] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000c6265000 flags=0x0020]
Nov 12 16:48:07 cartman21 kernel: [245472.741923] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000dd9a8000 flags=0x0020]
Nov 12 16:48:08 cartman21 kernel: [245473.633548] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000b29a5000 flags=0x0020]
Nov 12 16:48:08 cartman21 kernel: [245473.934186] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000f75a0000 flags=0x0020]
Nov 12 16:48:09 cartman21 kernel: [245474.952035] amd_iommu_report_page_fault: 6 callbacks suppressed
Nov 12 16:48:09 cartman21 kernel: [245474.952040] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f9082000 flags=0x0020]
Nov 12 16:48:10 cartman21 kernel: [245475.949986] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000de1fa000 flags=0x0020]
Nov 12 16:48:11 cartman21 kernel: [245476.071010] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000cd5e5000 flags=0x0020]
Nov 12 16:48:11 cartman21 kernel: [245476.300543] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d7847000 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.940059] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e2863000 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.940075] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e2863a00 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.943662] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e41a0000 flags=0x0070]
Nov 12 16:48:13 cartman21 kernel: [245478.943755] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e85e3000 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.943821] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6ebc000 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.943835] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d6ebcf00 flags=0x0020]
Nov 12 16:48:13 cartman21 kernel: [245478.944046] ethstb: hw csum failure
Nov 12 16:48:13 cartman21 kernel: [245478.944050] CPU: 30 PID: 0 Comm: swapper/30 Not tainted 4.19.85 #1
Nov 12 16:48:13 cartman21 kernel: [245478.944051] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:13 cartman21 kernel: [245478.944052] Call Trace:
Nov 12 16:48:13 cartman21 kernel: [245478.944055]  <IRQ>
Nov 12 16:48:13 cartman21 kernel: [245478.944063]  dump_stack+0x50/0x70
Nov 12 16:48:13 cartman21 kernel: [245478.944068]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:13 cartman21 kernel: [245478.944080]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:13 cartman21 kernel: [245478.944089]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:13 cartman21 kernel: [245478.944091]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:13 cartman21 kernel: [245478.944095]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:13 cartman21 kernel: [245478.944099]  ip_rcv+0x91/0xd0
Nov 12 16:48:13 cartman21 kernel: [245478.944102]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:13 cartman21 kernel: [245478.944106]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:13 cartman21 kernel: [245478.944109]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:13 cartman21 kernel: [245478.944111]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:13 cartman21 kernel: [245478.944129]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:13 cartman21 kernel: [245478.944147]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:13 cartman21 kernel: [245478.944163]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:13 cartman21 kernel: [245478.944167]  ? enqueue_task_fair+0x42a/0x1190
Nov 12 16:48:13 cartman21 kernel: [245478.944170]  net_rx_action+0x12f/0x350
Nov 12 16:48:13 cartman21 kernel: [245478.944174]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:13 cartman21 kernel: [245478.944178]  irq_exit+0xb4/0xc0
Nov 12 16:48:13 cartman21 kernel: [245478.944180]  do_IRQ+0x7f/0xe0
Nov 12 16:48:13 cartman21 kernel: [245478.944182]  common_interrupt+0xf/0xf
Nov 12 16:48:13 cartman21 kernel: [245478.944183]  </IRQ>
Nov 12 16:48:13 cartman21 kernel: [245478.944187] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:13 cartman21 kernel: [245478.944189] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:13 cartman21 kernel: [245478.944190] RSP: 0018:ffff97f38660be90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:13 cartman21 kernel: [245478.944194] RAX: 000001f3ffffffff RBX: 0000df430541f69d RCX: 000000000000001f
Nov 12 16:48:13 cartman21 kernel: [245478.944195] RDX: 0000df430541f69d RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:13 cartman21 kernel: [245478.944196] RBP: ffff9639ffa99800 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:13 cartman21 kernel: [245478.944197] R10: 00029e35386ccd12 R11: ffff963a0ed9ebe8 R12: 0000000000000001
Nov 12 16:48:13 cartman21 kernel: [245478.944198] R13: ffffffffb16f5d98 R14: 0000000000000001 R15: 0000000000000000
Nov 12 16:48:13 cartman21 kernel: [245478.944202]  do_idle+0x1e3/0x210
Nov 12 16:48:13 cartman21 kernel: [245478.944204]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:13 cartman21 kernel: [245478.944208]  start_secondary+0x183/0x1b0
Nov 12 16:48:13 cartman21 kernel: [245478.944212]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:15 cartman21 kernel: [245480.274399] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c46df000 flags=0x0020]
Nov 12 16:48:15 cartman21 kernel: [245480.281754] ethstb: hw csum failure
Nov 12 16:48:15 cartman21 kernel: [245480.281759] CPU: 20 PID: 0 Comm: swapper/20 Not tainted 4.19.85 #1
Nov 12 16:48:15 cartman21 kernel: [245480.281760] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:15 cartman21 kernel: [245480.281761] Call Trace:
Nov 12 16:48:15 cartman21 kernel: [245480.281765]  <IRQ>
Nov 12 16:48:15 cartman21 kernel: [245480.281774]  dump_stack+0x50/0x70
Nov 12 16:48:15 cartman21 kernel: [245480.281780]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:15 cartman21 kernel: [245480.281782] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ed990000 flags=0x0070]
Nov 12 16:48:15 cartman21 kernel: [245480.281798]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:15 cartman21 kernel: [245480.281808]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:15 cartman21 kernel: [245480.281811]  ? __skb_checksum+0x66/0x2b0
Nov 12 16:48:15 cartman21 kernel: [245480.281817]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:15 cartman21 kernel: [245480.281821]  ip_rcv+0x91/0xd0
Nov 12 16:48:15 cartman21 kernel: [245480.281824]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:15 cartman21 kernel: [245480.281829]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:15 cartman21 kernel: [245480.281832]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:15 cartman21 kernel: [245480.281835]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:15 cartman21 kernel: [245480.281865]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:15 cartman21 kernel: [245480.281883]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:15 cartman21 kernel: [245480.281899]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:15 cartman21 kernel: [245480.281902]  ? __tcp_transmit_skb+0x51b/0xac0
Nov 12 16:48:15 cartman21 kernel: [245480.281904]  net_rx_action+0x12f/0x350
Nov 12 16:48:15 cartman21 kernel: [245480.281909]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:15 cartman21 kernel: [245480.281914]  irq_exit+0xb4/0xc0
Nov 12 16:48:15 cartman21 kernel: [245480.281916]  do_IRQ+0x7f/0xe0
Nov 12 16:48:15 cartman21 kernel: [245480.281919]  common_interrupt+0xf/0xf
Nov 12 16:48:15 cartman21 kernel: [245480.281920]  </IRQ>
Nov 12 16:48:15 cartman21 kernel: [245480.281924] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:15 cartman21 kernel: [245480.281926] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:15 cartman21 kernel: [245480.281928] RSP: 0018:ffff97f38032fe90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:15 cartman21 kernel: [245480.281931] RAX: 000001f3ffffffff RBX: 0000df4354fe2df0 RCX: 000000000000001f
Nov 12 16:48:15 cartman21 kernel: [245480.281933] RDX: 0000df4354fe2df0 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:15 cartman21 kernel: [245480.281934] RBP: ffff9639ffa91c00 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:15 cartman21 kernel: [245480.281935] R10: 00029e3627a16dd4 R11: ffff963a0eb1ebe8 R12: 0000000000000001
Nov 12 16:48:15 cartman21 kernel: [245480.281935] R13: ffffffffb16f5d98 R14: 0000000000000001 R15: 0000000000000000
Nov 12 16:48:15 cartman21 kernel: [245480.281941]  do_idle+0x1e3/0x210
Nov 12 16:48:15 cartman21 kernel: [245480.281944]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:15 cartman21 kernel: [245480.281947]  start_secondary+0x183/0x1b0
Nov 12 16:48:15 cartman21 kernel: [245480.281952]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:15 cartman21 kernel: [245480.300342] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e2c33000 flags=0x0070]
Nov 12 16:48:15 cartman21 kernel: [245480.370238] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c8802000 flags=0x0020]
Nov 12 16:48:16 cartman21 kernel: [245481.484917] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000bcbe8000 flags=0x0020]
Nov 12 16:48:16 cartman21 kernel: [245481.485059] ethstb: hw csum failure
Nov 12 16:48:16 cartman21 kernel: [245481.485074] CPU: 30 PID: 0 Comm: swapper/30 Not tainted 4.19.85 #1
Nov 12 16:48:16 cartman21 kernel: [245481.485075] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:16 cartman21 kernel: [245481.485076] Call Trace:
Nov 12 16:48:16 cartman21 kernel: [245481.485080]  <IRQ>
Nov 12 16:48:16 cartman21 kernel: [245481.485088]  dump_stack+0x50/0x70
Nov 12 16:48:16 cartman21 kernel: [245481.485093]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.485108]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:16 cartman21 kernel: [245481.485120]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:16 cartman21 kernel: [245481.485123]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:16 cartman21 kernel: [245481.485128]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.485132]  ip_rcv+0x91/0xd0
Nov 12 16:48:16 cartman21 kernel: [245481.485136]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:16 cartman21 kernel: [245481.485140]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:16 cartman21 kernel: [245481.485144]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.485147]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.485170]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.485192]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.485216]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.485222]  ? enqueue_task_fair+0x42a/0x1190
Nov 12 16:48:16 cartman21 kernel: [245481.485228]  ? __note_gp_changes+0x14c/0x230
Nov 12 16:48:16 cartman21 kernel: [245481.485232]  net_rx_action+0x12f/0x350
Nov 12 16:48:16 cartman21 kernel: [245481.485238]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:16 cartman21 kernel: [245481.485244]  irq_exit+0xb4/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.485247]  do_IRQ+0x7f/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.485252]  common_interrupt+0xf/0xf
Nov 12 16:48:16 cartman21 kernel: [245481.485254]  </IRQ>
Nov 12 16:48:16 cartman21 kernel: [245481.485258] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:16 cartman21 kernel: [245481.485261] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:16 cartman21 kernel: [245481.485263] RSP: 0018:ffff97f38660be90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:16 cartman21 kernel: [245481.485267] RAX: 000001f3ffffffff RBX: 0000df439cb70023 RCX: 000000000000001f
Nov 12 16:48:16 cartman21 kernel: [245481.485268] RDX: 0000df439cb70023 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:16 cartman21 kernel: [245481.485269] RBP: ffff9639ffa99800 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:16 cartman21 kernel: [245481.485271] R10: 00029e36fecbdfba R11: ffff963a0ed9ebe8 R12: 0000000000000002
Nov 12 16:48:16 cartman21 kernel: [245481.485272] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:16 cartman21 kernel: [245481.485277]  do_idle+0x1e3/0x210
Nov 12 16:48:16 cartman21 kernel: [245481.485280]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:16 cartman21 kernel: [245481.485284]  start_secondary+0x183/0x1b0
Nov 12 16:48:16 cartman21 kernel: [245481.485288]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:16 cartman21 kernel: [245481.653226] ethstb: hw csum failure
Nov 12 16:48:16 cartman21 kernel: [245481.653230] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ec87b000 flags=0x0020]
Nov 12 16:48:16 cartman21 kernel: [245481.653235] CPU: 30 PID: 0 Comm: swapper/30 Not tainted 4.19.85 #1
Nov 12 16:48:16 cartman21 kernel: [245481.653237] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:16 cartman21 kernel: [245481.653240] Call Trace:
Nov 12 16:48:16 cartman21 kernel: [245481.653244]  <IRQ>
Nov 12 16:48:16 cartman21 kernel: [245481.653248]  dump_stack+0x50/0x70
Nov 12 16:48:16 cartman21 kernel: [245481.653251] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ee56c000 flags=0x0020]
Nov 12 16:48:16 cartman21 kernel: [245481.653255]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.653267]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:16 cartman21 kernel: [245481.653270] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ee56cf00 flags=0x0020]
Nov 12 16:48:16 cartman21 kernel: [245481.653281]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:16 cartman21 kernel: [245481.653284]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:16 cartman21 kernel: [245481.653287]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.653291]  ip_rcv+0x91/0xd0
Nov 12 16:48:16 cartman21 kernel: [245481.653295]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:16 cartman21 kernel: [245481.653300]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:16 cartman21 kernel: [245481.653304]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.653308]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.653332]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.653356]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.653377]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:16 cartman21 kernel: [245481.653381]  ? enqueue_task_fair+0x42a/0x1190
Nov 12 16:48:16 cartman21 kernel: [245481.653384]  net_rx_action+0x12f/0x350
Nov 12 16:48:16 cartman21 kernel: [245481.653387]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:16 cartman21 kernel: [245481.653391]  irq_exit+0xb4/0xc0
Nov 12 16:48:16 cartman21 kernel: [245481.653393]  do_IRQ+0x7f/0xe0
Nov 12 16:48:16 cartman21 kernel: [245481.653395]  common_interrupt+0xf/0xf
Nov 12 16:48:16 cartman21 kernel: [245481.653397]  </IRQ>
Nov 12 16:48:16 cartman21 kernel: [245481.653400] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:16 cartman21 kernel: [245481.653403] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:16 cartman21 kernel: [245481.653405] RSP: 0018:ffff97f38660be90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:16 cartman21 kernel: [245481.653408] RAX: 000001f3ffffffff RBX: 0000df43a6bcfb0f RCX: 000000000000001f
Nov 12 16:48:16 cartman21 kernel: [245481.653410] RDX: 0000df43a6bcfb0f RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:16 cartman21 kernel: [245481.653411] RBP: ffff9639ffa99800 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:16 cartman21 kernel: [245481.653412] R10: 00029e371cddcfd6 R11: ffff963a0ed9ebe8 R12: 0000000000000002
Nov 12 16:48:16 cartman21 kernel: [245481.653413] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:16 cartman21 kernel: [245481.653418]  do_idle+0x1e3/0x210
Nov 12 16:48:16 cartman21 kernel: [245481.653420]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:16 cartman21 kernel: [245481.653423]  start_secondary+0x183/0x1b0
Nov 12 16:48:16 cartman21 kernel: [245481.653427]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:17 cartman21 kernel: [245482.497289] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000cd48d000 flags=0x0020]
Nov 12 16:48:17 cartman21 kernel: [245482.497300] ethstb: hw csum failure
Nov 12 16:48:17 cartman21 kernel: [245482.497304] CPU: 30 PID: 161 Comm: ksoftirqd/30 Not tainted 4.19.85 #1
Nov 12 16:48:17 cartman21 kernel: [245482.497305] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:17 cartman21 kernel: [245482.497306] Call Trace:
Nov 12 16:48:17 cartman21 kernel: [245482.497314]  dump_stack+0x50/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497318]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:17 cartman21 kernel: [245482.497330]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:17 cartman21 kernel: [245482.497334]  ? tcp_v4_do_rcv+0x12a/0x1e0
Nov 12 16:48:17 cartman21 kernel: [245482.497342]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:17 cartman21 kernel: [245482.497344]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:17 cartman21 kernel: [245482.497348]  ? find_busiest_group+0x73b/0xac0
Nov 12 16:48:17 cartman21 kernel: [245482.497351]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:17 cartman21 kernel: [245482.497354]  ip_rcv+0x91/0xd0
Nov 12 16:48:17 cartman21 kernel: [245482.497358]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:17 cartman21 kernel: [245482.497361]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497364]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:17 cartman21 kernel: [245482.497367]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:17 cartman21 kernel: [245482.497385]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:17 cartman21 kernel: [245482.497403]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:17 cartman21 kernel: [245482.497404]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497421]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:17 cartman21 kernel: [245482.497422]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497423]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497424]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497425]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497427]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497428]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497429]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497431]  net_rx_action+0x12f/0x350
Nov 12 16:48:17 cartman21 kernel: [245482.497434]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:17 cartman21 kernel: [245482.497438]  ? sort_range+0x20/0x20
Nov 12 16:48:17 cartman21 kernel: [245482.497441]  run_ksoftirqd+0x1a/0x20
Nov 12 16:48:17 cartman21 kernel: [245482.497443]  smpboot_thread_fn+0xc5/0x160
Nov 12 16:48:17 cartman21 kernel: [245482.497446]  kthread+0x112/0x130
Nov 12 16:48:17 cartman21 kernel: [245482.497449]  ? kthread_create_worker_on_cpu+0x70/0x70
Nov 12 16:48:17 cartman21 kernel: [245482.497450]  ret_from_fork+0x22/0x40
Nov 12 16:48:19 cartman21 kernel: [245483.988118] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ca695000 flags=0x0020]
Nov 12 16:48:19 cartman21 kernel: [245483.988131] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000ef889000 flags=0x0020]
Nov 12 16:48:19 cartman21 kernel: [245484.460798] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000e4443000 flags=0x0020]
Nov 12 16:48:19 cartman21 kernel: [245484.463337] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000f2a18000 flags=0x0020]
Nov 12 16:48:19 cartman21 kernel: [245484.463732] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000d0285000 flags=0x0070]
Nov 12 16:48:19 cartman21 kernel: [245484.850316] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000ce6ab000 flags=0x0020]
Nov 12 16:48:21 cartman21 kernel: [245486.526774] amd_iommu_report_page_fault: 5 callbacks suppressed
Nov 12 16:48:21 cartman21 kernel: [245486.526779] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d212c000 flags=0x0020]
Nov 12 16:48:21 cartman21 kernel: [245486.681840] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c8e87000 flags=0x0020]
Nov 12 16:48:21 cartman21 kernel: [245486.681857] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f6e21000 flags=0x0020]
Nov 12 16:48:21 cartman21 kernel: [245486.681873] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d76b5000 flags=0x0020]
Nov 12 16:48:21 cartman21 kernel: [245486.681890] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000db40c000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008903] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ddfec000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008914] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ca0cc000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008923] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000ca0ccf00 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008935] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000aee42000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008944] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000aee42f00 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008954] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000d2c08000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.008970] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000d2c08600 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.009021] ethstb: hw csum failure
Nov 12 16:48:24 cartman21 kernel: [245489.009024] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.19.85 #1
Nov 12 16:48:24 cartman21 kernel: [245489.009025] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:24 cartman21 kernel: [245489.009025] Call Trace:
Nov 12 16:48:24 cartman21 kernel: [245489.009028]  <IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.009033]  dump_stack+0x50/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.009037]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009047]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.009055]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.009061]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:24 cartman21 kernel: [245489.009064]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009067]  ip_rcv+0x91/0xd0
Nov 12 16:48:24 cartman21 kernel: [245489.009070]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.009077]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.009082]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009084]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009098]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009107]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009116]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009118]  net_rx_action+0x12f/0x350
Nov 12 16:48:24 cartman21 kernel: [245489.009121]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:24 cartman21 kernel: [245489.009124]  irq_exit+0xb4/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009126]  do_IRQ+0x7f/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009128]  common_interrupt+0xf/0xf
Nov 12 16:48:24 cartman21 kernel: [245489.009129]  </IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.009132] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.009134] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:24 cartman21 kernel: [245489.009135] RSP: 0018:ffff97f380237e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:24 cartman21 kernel: [245489.009137] RAX: 000001f3ffffffff RBX: 0000df455d2d3b6e RCX: 000000000000001f
Nov 12 16:48:24 cartman21 kernel: [245489.009138] RDX: 0000df455d2d3b6e RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.009138] RBP: ffff963a06a3ac00 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:24 cartman21 kernel: [245489.009139] R10: 00029e3c402e7440 R11: ffff963a0e79ebe8 R12: 0000000000000002
Nov 12 16:48:24 cartman21 kernel: [245489.009140] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.009143]  do_idle+0x1e3/0x210
Nov 12 16:48:24 cartman21 kernel: [245489.009145]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:24 cartman21 kernel: [245489.009148]  start_secondary+0x183/0x1b0
Nov 12 16:48:24 cartman21 kernel: [245489.009150]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:24 cartman21 kernel: [245489.009151] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000d55fd000 flags=0x0020]
Nov 12 16:48:24 cartman21 kernel: [245489.009193] ethstb: hw csum failure
Nov 12 16:48:24 cartman21 kernel: [245489.009197] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.19.85 #1
Nov 12 16:48:24 cartman21 kernel: [245489.009200] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:24 cartman21 kernel: [245489.009202] Call Trace:
Nov 12 16:48:24 cartman21 kernel: [245489.009206]  <IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.009211]  dump_stack+0x50/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.009215]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009222]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.009229]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.009234]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:24 cartman21 kernel: [245489.009238]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009243]  ip_rcv+0x91/0xd0
Nov 12 16:48:24 cartman21 kernel: [245489.009247]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.009251]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.009256]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009260]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009271]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009282]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009294]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.009299]  net_rx_action+0x12f/0x350
Nov 12 16:48:24 cartman21 kernel: [245489.009301]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:24 cartman21 kernel: [245489.009302]  irq_exit+0xb4/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.009303]  do_IRQ+0x7f/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.009304]  common_interrupt+0xf/0xf
Nov 12 16:48:24 cartman21 kernel: [245489.009304]  </IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.009306] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.009307] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:24 cartman21 kernel: [245489.009307] RSP: 0018:ffff97f380237e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:24 cartman21 kernel: [245489.009308] RAX: 000001f3ffffffff RBX: 0000df455d2d3b6e RCX: 000000000000001f
Nov 12 16:48:24 cartman21 kernel: [245489.009309] RDX: 0000df455d2d3b6e RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.009310] RBP: ffff963a06a3ac00 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:24 cartman21 kernel: [245489.009310] R10: 00029e3c402e7440 R11: ffff963a0e79ebe8 R12: 0000000000000002
Nov 12 16:48:24 cartman21 kernel: [245489.009311] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.009313]  do_idle+0x1e3/0x210
Nov 12 16:48:24 cartman21 kernel: [245489.009314]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:24 cartman21 kernel: [245489.009315]  start_secondary+0x183/0x1b0
Nov 12 16:48:24 cartman21 kernel: [245489.009316]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:24 cartman21 kernel: [245489.227984] ethstb: hw csum failure
Nov 12 16:48:24 cartman21 kernel: [245489.227990] CPU: 30 PID: 0 Comm: swapper/30 Not tainted 4.19.85 #1
Nov 12 16:48:24 cartman21 kernel: [245489.227990] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:24 cartman21 kernel: [245489.227991] Call Trace:
Nov 12 16:48:24 cartman21 kernel: [245489.227994]  <IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.228000]  dump_stack+0x50/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.228005]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.228013]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.228019]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:24 cartman21 kernel: [245489.228024]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:24 cartman21 kernel: [245489.228029] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000e7dda000 flags=0x0070]
Nov 12 16:48:24 cartman21 kernel: [245489.228035]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.228040]  ip_rcv+0x91/0xd0
Nov 12 16:48:24 cartman21 kernel: [245489.228044]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.228048] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000e7ddaf00 flags=0x0070]
Nov 12 16:48:24 cartman21 kernel: [245489.228053]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:24 cartman21 kernel: [245489.228055]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.228057]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.228067]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.228076]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.228084]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.228091]  ? mlx5e_napi_poll+0x256/0xca0 [mlx5_core]
Nov 12 16:48:24 cartman21 kernel: [245489.228092]  net_rx_action+0x12f/0x350
Nov 12 16:48:24 cartman21 kernel: [245489.228096]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:24 cartman21 kernel: [245489.228100]  irq_exit+0xb4/0xc0
Nov 12 16:48:24 cartman21 kernel: [245489.228101]  do_IRQ+0x7f/0xe0
Nov 12 16:48:24 cartman21 kernel: [245489.228103]  common_interrupt+0xf/0xf
Nov 12 16:48:24 cartman21 kernel: [245489.228104]  </IRQ>
Nov 12 16:48:24 cartman21 kernel: [245489.228107] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:24 cartman21 kernel: [245489.228111] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:24 cartman21 kernel: [245489.228114] RSP: 0018:ffff97f38660be90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:24 cartman21 kernel: [245489.228119] RAX: 000001f3ffffffff RBX: 0000df456a3ac969 RCX: 000000000000001f
Nov 12 16:48:24 cartman21 kernel: [245489.228122] RDX: 0000df456a3ac969 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.228124] RBP: ffff9639ffa99800 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:24 cartman21 kernel: [245489.228127] R10: 00029e3c67571d56 R11: ffff963a0ed9ebe8 R12: 0000000000000002
Nov 12 16:48:24 cartman21 kernel: [245489.228129] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:24 cartman21 kernel: [245489.228135]  do_idle+0x1e3/0x210
Nov 12 16:48:24 cartman21 kernel: [245489.228137]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:24 cartman21 kernel: [245489.228140]  start_secondary+0x183/0x1b0
Nov 12 16:48:24 cartman21 kernel: [245489.228143]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:26 cartman21 kernel: [245491.094094] ethstb: hw csum failure
Nov 12 16:48:26 cartman21 kernel: [245491.094097] CPU: 30 PID: 161 Comm: ksoftirqd/30 Not tainted 4.19.85 #1
Nov 12 16:48:26 cartman21 kernel: [245491.094098] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:26 cartman21 kernel: [245491.094098] Call Trace:
Nov 12 16:48:26 cartman21 kernel: [245491.094106]  dump_stack+0x50/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094109]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:26 cartman21 kernel: [245491.094119]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:26 cartman21 kernel: [245491.094123]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:26 cartman21 kernel: [245491.094125]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:26 cartman21 kernel: [245491.094128]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:26 cartman21 kernel: [245491.094131]  ip_rcv+0x91/0xd0
Nov 12 16:48:26 cartman21 kernel: [245491.094133]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:26 cartman21 kernel: [245491.094137] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000c3d8a000 flags=0x0020]
Nov 12 16:48:26 cartman21 kernel: [245491.094140]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094142]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:26 cartman21 kernel: [245491.094143]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:26 cartman21 kernel: [245491.094159]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:26 cartman21 kernel: [245491.094162] AMD-Vi: Event logged [IO_PAGE_FAULT device=42:00.0 domain=0x0000 address=0x00000000c3d8af00 flags=0x0020]
Nov 12 16:48:26 cartman21 kernel: [245491.094171]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:26 cartman21 kernel: [245491.094179]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:26 cartman21 kernel: [245491.094180]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094181]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094182]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094182]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094182]  ? __switch_to_asm+0x35/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094183]  ? __switch_to_asm+0x41/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094184]  net_rx_action+0x12f/0x350
Nov 12 16:48:26 cartman21 kernel: [245491.094186]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:26 cartman21 kernel: [245491.094189]  ? sort_range+0x20/0x20
Nov 12 16:48:26 cartman21 kernel: [245491.094191]  run_ksoftirqd+0x1a/0x20
Nov 12 16:48:26 cartman21 kernel: [245491.094192]  smpboot_thread_fn+0xc5/0x160
Nov 12 16:48:26 cartman21 kernel: [245491.094194]  kthread+0x112/0x130
Nov 12 16:48:26 cartman21 kernel: [245491.094195]  ? kthread_create_worker_on_cpu+0x70/0x70
Nov 12 16:48:26 cartman21 kernel: [245491.094196]  ret_from_fork+0x22/0x40
Nov 12 16:48:27 cartman21 kernel: [245492.413716] amd_iommu_report_page_fault: 7 callbacks suppressed
Nov 12 16:48:27 cartman21 kernel: [245492.413721] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d2452000 flags=0x0020]
Nov 12 16:48:27 cartman21 kernel: [245492.803265] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e054d000 flags=0x0020]
Nov 12 16:48:27 cartman21 kernel: [245492.849182] ethsto: hw csum failure
Nov 12 16:48:27 cartman21 kernel: [245492.849186] CPU: 22 PID: 0 Comm: swapper/22 Not tainted 4.19.85 #1
Nov 12 16:48:27 cartman21 kernel: [245492.849187] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:27 cartman21 kernel: [245492.849188] Call Trace:
Nov 12 16:48:27 cartman21 kernel: [245492.849192]  <IRQ>
Nov 12 16:48:27 cartman21 kernel: [245492.849199]  dump_stack+0x50/0x70
Nov 12 16:48:27 cartman21 kernel: [245492.849204]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:27 cartman21 kernel: [245492.849216]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:27 cartman21 kernel: [245492.849225]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:27 cartman21 kernel: [245492.849228]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:27 cartman21 kernel: [245492.849232]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:27 cartman21 kernel: [245492.849236]  ip_rcv+0x91/0xd0
Nov 12 16:48:27 cartman21 kernel: [245492.849239]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:27 cartman21 kernel: [245492.849243]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:27 cartman21 kernel: [245492.849246]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:27 cartman21 kernel: [245492.849248]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:27 cartman21 kernel: [245492.849266]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:27 cartman21 kernel: [245492.849283]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:27 cartman21 kernel: [245492.849299]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:27 cartman21 kernel: [245492.849303]  net_rx_action+0x12f/0x350
Nov 12 16:48:27 cartman21 kernel: [245492.849306]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:27 cartman21 kernel: [245492.849311]  irq_exit+0xb4/0xc0
Nov 12 16:48:27 cartman21 kernel: [245492.849313]  do_IRQ+0x7f/0xe0
Nov 12 16:48:27 cartman21 kernel: [245492.849315]  common_interrupt+0xf/0xf
Nov 12 16:48:27 cartman21 kernel: [245492.849317]  </IRQ>
Nov 12 16:48:27 cartman21 kernel: [245492.849320] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:27 cartman21 kernel: [245492.849322] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:27 cartman21 kernel: [245492.849324] RSP: 0018:ffff97f38033fe90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd2
Nov 12 16:48:27 cartman21 kernel: [245492.849326] RAX: 000001f3ffffffff RBX: 0000df464211d472 RCX: 000000000000001f
Nov 12 16:48:27 cartman21 kernel: [245492.849327] RDX: 0000df464211d472 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:27 cartman21 kernel: [245492.849328] RBP: ffff9639ffa93400 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:27 cartman21 kernel: [245492.849329] R10: 00029e3eeedc3052 R11: ffff963a0eb9ebe8 R12: 0000000000000001
Nov 12 16:48:27 cartman21 kernel: [245492.849330] R13: ffffffffb16f5d98 R14: 0000000000000001 R15: 0000000000000000
Nov 12 16:48:27 cartman21 kernel: [245492.849336]  do_idle+0x1e3/0x210
Nov 12 16:48:27 cartman21 kernel: [245492.849338]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:27 cartman21 kernel: [245492.849342]  start_secondary+0x183/0x1b0
Nov 12 16:48:27 cartman21 kernel: [245492.849346]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:32 cartman21 kernel: [245497.193544] ethstb: hw csum failure
Nov 12 16:48:32 cartman21 kernel: [245497.193553] CPU: 20 PID: 0 Comm: swapper/20 Not tainted 4.19.85 #1
Nov 12 16:48:32 cartman21 kernel: [245497.193555] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:32 cartman21 kernel: [245497.193562] Call Trace:
Nov 12 16:48:32 cartman21 kernel: [245497.193570]  <IRQ>
Nov 12 16:48:32 cartman21 kernel: [245497.193578]  dump_stack+0x50/0x70
Nov 12 16:48:32 cartman21 kernel: [245497.193581] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f0d5a000 flags=0x0070]
Nov 12 16:48:32 cartman21 kernel: [245497.193586]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:32 cartman21 kernel: [245497.193602]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:32 cartman21 kernel: [245497.193606] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000f0d5af00 flags=0x0070]
Nov 12 16:48:32 cartman21 kernel: [245497.193620]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:32 cartman21 kernel: [245497.193626]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:32 cartman21 kernel: [245497.193631]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:32 cartman21 kernel: [245497.193637]  ip_rcv+0x91/0xd0
Nov 12 16:48:32 cartman21 kernel: [245497.193646]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:32 cartman21 kernel: [245497.193653]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:32 cartman21 kernel: [245497.193657]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:32 cartman21 kernel: [245497.193661]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:32 cartman21 kernel: [245497.193684]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:32 cartman21 kernel: [245497.193706]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:32 cartman21 kernel: [245497.193723]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:32 cartman21 kernel: [245497.193726]  net_rx_action+0x12f/0x350
Nov 12 16:48:32 cartman21 kernel: [245497.193730]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:32 cartman21 kernel: [245497.193735]  irq_exit+0xb4/0xc0
Nov 12 16:48:32 cartman21 kernel: [245497.193737]  do_IRQ+0x7f/0xe0
Nov 12 16:48:32 cartman21 kernel: [245497.193740]  common_interrupt+0xf/0xf
Nov 12 16:48:32 cartman21 kernel: [245497.193741]  </IRQ>
Nov 12 16:48:32 cartman21 kernel: [245497.193745] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:32 cartman21 kernel: [245497.193747] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:32 cartman21 kernel: [245497.193748] RSP: 0018:ffff97f38032fe90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:32 cartman21 kernel: [245497.193752] RAX: 000001f3ffffffff RBX: 0000df47450364a3 RCX: 000000000000001f
Nov 12 16:48:32 cartman21 kernel: [245497.193753] RDX: 0000df47450364a3 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:32 cartman21 kernel: [245497.193754] RBP: ffff9639ffa91c00 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:32 cartman21 kernel: [245497.193755] R10: 00029e41f7b0cff0 R11: 0000000000000000 R12: 0000000000000002
Nov 12 16:48:32 cartman21 kernel: [245497.193756] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:32 cartman21 kernel: [245497.193761]  do_idle+0x1e3/0x210
Nov 12 16:48:32 cartman21 kernel: [245497.193764]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:32 cartman21 kernel: [245497.193767]  start_secondary+0x183/0x1b0
Nov 12 16:48:32 cartman21 kernel: [245497.193772]  secondary_startup_64+0xa4/0xb0
Nov 12 16:48:35 cartman21 kernel: [245500.265815] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d34b1000 flags=0x0020]
Nov 12 16:48:35 cartman21 kernel: [245500.265832] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d34b1f00 flags=0x0020]
Nov 12 16:48:35 cartman21 kernel: [245500.265846] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c6f80000 flags=0x0020]
Nov 12 16:48:45 cartman21 kernel: [245510.624379] mlx5_core 0000:42:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e7dda000 flags=0x0070]
Nov 12 16:48:45 cartman21 kernel: [245510.624396] mlx5_core 0000:42:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000e7ddaf00 flags=0x0070]
Nov 12 16:48:55 cartman21 kernel: [245520.301657] ethstb: hw csum failure
Nov 12 16:48:55 cartman21 kernel: [245520.301665] CPU: 16 PID: 0 Comm: swapper/16 Not tainted 4.19.85 #1
Nov 12 16:48:55 cartman21 kernel: [245520.301666] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:48:55 cartman21 kernel: [245520.301666] Call Trace:
Nov 12 16:48:55 cartman21 kernel: [245520.301669]  <IRQ>
Nov 12 16:48:55 cartman21 kernel: [245520.301674]  dump_stack+0x50/0x70
Nov 12 16:48:55 cartman21 kernel: [245520.301678]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:48:55 cartman21 kernel: [245520.301685]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:48:55 cartman21 kernel: [245520.301689]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:48:55 cartman21 kernel: [245520.301691]  ? __skb_checksum+0x184/0x2b0
Nov 12 16:48:55 cartman21 kernel: [245520.301694]  nf_hook_slow+0x44/0xc0
Nov 12 16:48:55 cartman21 kernel: [245520.301696]  ip_rcv+0x91/0xd0
Nov 12 16:48:55 cartman21 kernel: [245520.301698]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:48:55 cartman21 kernel: [245520.301701]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:48:55 cartman21 kernel: [245520.301702]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:48:55 cartman21 kernel: [245520.301704]  napi_gro_receive+0xba/0xe0
Nov 12 16:48:55 cartman21 kernel: [245520.301713]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:48:55 cartman21 kernel: [245520.301717] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d576c000 flags=0x0020]
Nov 12 16:48:55 cartman21 kernel: [245520.301727]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:48:55 cartman21 kernel: [245520.301735]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:48:55 cartman21 kernel: [245520.301739]  ? enqueue_task_fair+0x42a/0x1190
Nov 12 16:48:55 cartman21 kernel: [245520.301740]  net_rx_action+0x12f/0x350
Nov 12 16:48:55 cartman21 kernel: [245520.301743]  __do_softirq+0xd8/0x2cc
Nov 12 16:48:55 cartman21 kernel: [245520.301745]  irq_exit+0xb4/0xc0
Nov 12 16:48:55 cartman21 kernel: [245520.301747]  do_IRQ+0x7f/0xe0
Nov 12 16:48:55 cartman21 kernel: [245520.301748]  common_interrupt+0xf/0xf
Nov 12 16:48:55 cartman21 kernel: [245520.301749]  </IRQ>
Nov 12 16:48:55 cartman21 kernel: [245520.301751] RIP: 0010:cpuidle_enter_state+0xb8/0x300
Nov 12 16:48:55 cartman21 kernel: [245520.301753] Code: ff e8 dc f7 8e ff 80 7c 24 0b 00 74 12 9c 58 f6 c4 02 0f 85 31 02 00 00 31 ff e8 13 8e 94 ff fb 48 b8 ff ff ff ff f3 01 00 00 <48> 2b 1c 24 ba ff ff ff 7f 48 39 c3 7f 1a 48 ba cf f7 53 e3 a5 9b
Nov 12 16:48:55 cartman21 kernel: [245520.301754] RSP: 0018:ffff97f38030fe90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd1
Nov 12 16:48:55 cartman21 kernel: [245520.301756] RAX: 000001f3ffffffff RBX: 0000df4ca65d6b66 RCX: 000000000000001f
Nov 12 16:48:55 cartman21 kernel: [245520.301756] RDX: 0000df4ca65d6b66 RSI: 000000002aaaab99 RDI: 0000000000000000
Nov 12 16:48:55 cartman21 kernel: [245520.301757] RBP: ffff9639ffa8ec00 R08: 0000000000000002 R09: 000000000001f380
Nov 12 16:48:55 cartman21 kernel: [245520.301757] R10: 00029e521bbe8a12 R11: ffff963a0ea1ebe8 R12: 0000000000000002
Nov 12 16:48:55 cartman21 kernel: [245520.301757] R13: ffffffffb16f5df8 R14: 0000000000000002 R15: 0000000000000000
Nov 12 16:48:55 cartman21 kernel: [245520.301760]  do_idle+0x1e3/0x210
Nov 12 16:48:55 cartman21 kernel: [245520.301761]  cpu_startup_entry+0x6f/0x80
Nov 12 16:48:55 cartman21 kernel: [245520.301764]  start_secondary+0x183/0x1b0
Nov 12 16:48:55 cartman21 kernel: [245520.301766]  secondary_startup_64+0xa4/0xb0
Nov 12 16:49:10 cartman21 kernel: [245535.754048] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000c8a24000 flags=0x0020]
Nov 12 16:53:17 cartman21 kernel: [245782.381917] ethstb: hw csum failure
Nov 12 16:53:17 cartman21 kernel: [245782.381920] CPU: 30 PID: 161 Comm: ksoftirqd/30 Not tainted 4.19.85 #1
Nov 12 16:53:17 cartman21 kernel: [245782.381920] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:53:17 cartman21 kernel: [245782.381921] Call Trace:
Nov 12 16:53:17 cartman21 kernel: [245782.381929]  dump_stack+0x50/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381933]  __skb_gro_checksum_complete.cold.168+0x1f/0x24
Nov 12 16:53:17 cartman21 kernel: [245782.381937]  tcp4_gro_receive+0xc6/0x1a0
Nov 12 16:53:17 cartman21 kernel: [245782.381941] mlx5_core 0000:42:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x00000000d2cb7000 flags=0x0020]
Nov 12 16:53:17 cartman21 kernel: [245782.381944]  inet_gro_receive+0x20f/0x290
Nov 12 16:53:17 cartman21 kernel: [245782.381946]  dev_gro_receive+0x2ad/0x620
Nov 12 16:53:17 cartman21 kernel: [245782.381963]  ? mlx5e_build_rx_skb+0x2c2/0xa80 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.381971]  ? mlx5e_skb_from_cqe_nonlinear+0x5c/0x250 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.381972]  napi_gro_receive+0x33/0xe0
Nov 12 16:53:17 cartman21 kernel: [245782.381980]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.381988]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.381995]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.381997]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381997]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381998]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381998]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381999]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.381999]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382001]  net_rx_action+0x12f/0x350
Nov 12 16:53:17 cartman21 kernel: [245782.382003]  __do_softirq+0xd8/0x2cc
Nov 12 16:53:17 cartman21 kernel: [245782.382006]  ? sort_range+0x20/0x20
Nov 12 16:53:17 cartman21 kernel: [245782.382008]  run_ksoftirqd+0x1a/0x20
Nov 12 16:53:17 cartman21 kernel: [245782.382009]  smpboot_thread_fn+0xc5/0x160
Nov 12 16:53:17 cartman21 kernel: [245782.382011]  kthread+0x112/0x130
Nov 12 16:53:17 cartman21 kernel: [245782.382013]  ? kthread_create_worker_on_cpu+0x70/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382013]  ret_from_fork+0x22/0x40
Nov 12 16:53:17 cartman21 kernel: [245782.382017] ethstb: hw csum failure
Nov 12 16:53:17 cartman21 kernel: [245782.382018] CPU: 30 PID: 161 Comm: ksoftirqd/30 Not tainted 4.19.85 #1
Nov 12 16:53:17 cartman21 kernel: [245782.382018] Hardware name: Supermicro Super Server/H11SSL-i, BIOS 2.1 02/21/2020
Nov 12 16:53:17 cartman21 kernel: [245782.382018] Call Trace:
Nov 12 16:53:17 cartman21 kernel: [245782.382020]  dump_stack+0x50/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382022]  __skb_checksum_complete+0xa9/0xc0
Nov 12 16:53:17 cartman21 kernel: [245782.382027]  tcp_error+0x12a/0x140 [nf_conntrack]
Nov 12 16:53:17 cartman21 kernel: [245782.382032]  nf_conntrack_in+0xd8/0x350 [nf_conntrack]
Nov 12 16:53:17 cartman21 kernel: [245782.382035]  nf_hook_slow+0x44/0xc0
Nov 12 16:53:17 cartman21 kernel: [245782.382037]  ip_rcv+0x91/0xd0
Nov 12 16:53:17 cartman21 kernel: [245782.382038]  ? ip_rcv_finish_core.isra.18+0x300/0x300
Nov 12 16:53:17 cartman21 kernel: [245782.382040]  __netif_receive_skb_one_core+0x52/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382041]  netif_receive_skb_internal+0x34/0xe0
Nov 12 16:53:17 cartman21 kernel: [245782.382042]  napi_gro_receive+0xba/0xe0
Nov 12 16:53:17 cartman21 kernel: [245782.382050]  mlx5e_handle_rx_cqe+0x83/0xd0 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.382057]  mlx5e_poll_rx_cq+0xc7/0x960 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.382064]  mlx5e_napi_poll+0xb0/0xca0 [mlx5_core]
Nov 12 16:53:17 cartman21 kernel: [245782.382065]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382066]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382066]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382067]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382067]  ? __switch_to_asm+0x35/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382068]  ? __switch_to_asm+0x41/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382069]  net_rx_action+0x12f/0x350
Nov 12 16:53:17 cartman21 kernel: [245782.382070]  __do_softirq+0xd8/0x2cc
Nov 12 16:53:17 cartman21 kernel: [245782.382071]  ? sort_range+0x20/0x20
Nov 12 16:53:17 cartman21 kernel: [245782.382072]  run_ksoftirqd+0x1a/0x20
Nov 12 16:53:17 cartman21 kernel: [245782.382073]  smpboot_thread_fn+0xc5/0x160
Nov 12 16:53:17 cartman21 kernel: [245782.382073]  kthread+0x112/0x130
Nov 12 16:53:17 cartman21 kernel: [245782.382074]  ? kthread_create_worker_on_cpu+0x70/0x70
Nov 12 16:53:17 cartman21 kernel: [245782.382075]  ret_from_fork+0x22/0x40
Comment 1 Christian Theune 2020-11-12 22:00:53 UTC 
So, this lead us to investigate the IOMMU as potential issue and I disabled the IOMMU through BIOS. This might be related to https://bugzilla.kernel.org/show_bug.cgi?id=210153 as that seems to have subsided after disabling the IOMMU, too.