Bug 91661
| Summary: | General protection fault may occur when removing TPROXY rule | ||
|---|---|---|---|
| Product: | Networking | Reporter: | James Oakley (jfunk) |
| Component: | Netfilter/Iptables | Assignee: | networking_netfilter-iptables (networking_netfilter-iptables) |
| Status: | NEW --- | ||
| Severity: | normal | ||
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 3.8.0 - 3.19.0-rc5+ | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
I have tracked this issue down further. It appears that the sk returned from __inet_lookup_established is a time wait socket, set on the skb by tcp_v4_early_demux. In tun_net_xmit, it attempts to set the timestamp, which appends to sk->sk_error_queue, which is uninitialized memory. If I attempt to drop the skb, it crashes later in sock_wfree: [ 265.528530] BUG: unable to handle kernel paging request at ffff880030f8ed28 [ 265.535552] IP: [<ffff880030f8ed28>] 0xffff880030f8ed28 [ 265.540818] PGD 1c72067 PUD 1c73067 PMD 30c20063 PTE 8000000030f8e163 [ 265.547402] Oops: 0011 [#1] SMP [ 265.550675] Modules linked in: xt_TPROXY xt_socket nf_defrag_ipv6 xt_REDIRECT nf_nat_redirect xt_multiport sch_htb tun xt_CLASSIFY xt_dscp xt_TCPMSS xt_mark xt_tcpudp tcp_yeah tcp_westwood tcp_veno tcp_vegas tcp_scalable tcp_lp tcp_illinois tcp_hybla tcp_htcp tcp_highspeed tcp_diag inet_diag tcp_bic xt_nat iptable_mangle iptable_nat nf_nat_ipv4 nf_nat 8021q garp mrp stp llc ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment arptable_filter arp_tables iptable_filter ip_tables x_tables nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc joydev hid_generic usbhid hid loop x86_pkg_temp_thermal intel_powerclamp intel_rapl iosf_mbi coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ast aes_x86_64 ttm lrw drm_kms_helper gf128mul evdev glue_helper snd_pcm iTCO_wdt iTCO_vendor_support drm snd_timer snd soundcore tpm_tis ipmi_si lpc_ich ablk_helper tpm ipmi_msghandler cryptd mfd_core pcspkr shpchp i2c_i801 battery video acpi_pad mei_me xhci_pci xhci_hcd mei button processor ext4 crc16 mbcache jbd2 sg sd_mod ahci libahci igb libata i2c_algo_bit i2c_core dca scsi_mod ehci_pci ehci_hcd crc32c_intel e1000e thermal usbcore fan ptp thermal_sys pps_core usb_common [ 265.663340] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.19.0-rc4+ #51 [ 265.669800] Hardware name: Supermicro X10SLM-F/X10SLM-F, BIOS 2.0 04/24/2014 [ 265.676867] task: ffff88040d5e72f0 ti: ffff88040d5f0000 task.ti: ffff88040d5f0000 [ 265.684369] RIP: 0010:[<ffff880030f8ed28>] [<ffff880030f8ed28>] 0xffff880030f8ed28 [ 265.692070] RSP: 0018:ffff88041fcc3820 EFLAGS: 00010202 [ 265.697396] RAX: 00000000000008ff RBX: ffff8803f77a11c0 RCX: 0000000000000000 [ 265.704551] RDX: ffff8800d3fb0040 RSI: ffff88040c494000 RDI: ffff8803f77a11c0 [ 265.711705] RBP: ffff88041fcc3838 R08: ffff8803f83b10a0 R09: 0000000000000001 [ 265.718859] R10: 0000000000000000 R11: 0000000000000570 R12: ffff8803f8765a00 [ 265.726012] R13: 0000000000000001 R14: ffff8803f8710c00 R15: ffff8803f8765a00 [ 265.733161] FS: 0000000000000000(0000) GS:ffff88041fcc0000(0000) knlGS:0000000000000000 [ 265.741266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 265.747875] CR2: ffff880030f8ed28 CR3: 0000000001a14000 CR4: 00000000001407e0 [ 265.755878] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 265.763881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 265.771876] Stack: [ 265.774752] ffffffff813f2e82 0000000000000096 ffff8803f8765a00 ffff88041fcc3858 [ 265.783104] ffffffff813f7f25 0000000000000071 ffff8803f8765a00 ffff88041fcc3878 [ 265.791471] ffffffff813f8f81 ffff8803f74d6150 ffff8803f8765a00 ffff88041fcc3898 [ 265.799906] Call Trace: [ 265.803233] <IRQ> [ 265.805165] [<ffffffff813f2e82>] ? sock_wfree+0x32/0x60 [ 265.812458] [<ffffffff813f7f25>] skb_release_head_state+0x75/0xe0 [ 265.819525] [<ffffffff813f8f81>] skb_release_all+0x11/0x30 [ 265.825989] [<ffffffff813f8fe1>] __kfree_skb+0x11/0x80 [ 265.832111] [<ffffffff813f9085>] kfree_skb+0x35/0x40 [ 265.838064] [<ffffffffa06d6010>] tun_net_xmit+0x350/0x3a0 [tun] [ 265.844974] [<ffffffff81409f45>] dev_hard_start_xmit+0x275/0x330 [ 265.851989] [<ffffffff81409a0e>] ? validate_xmit_skb.isra.85+0x2e/0x2f0 [ 265.859596] [<ffffffff81429654>] sch_direct_xmit+0xa4/0x1d0 [ 265.866151] [<ffffffff8140a2af>] __dev_queue_xmit+0x2af/0x510 [ 265.872890] [<ffffffff8140a52b>] dev_queue_xmit+0xb/0x10 [ 265.879204] [<ffffffff8141206c>] neigh_direct_output+0xc/0x10 [ 265.885927] [<ffffffff8144459c>] ip_finish_output2+0x27c/0x310 [ 265.892725] [<ffffffff814441db>] ip_fragment+0x72b/0x870 [ 265.898997] [<ffffffff81438b01>] ? netfilter_net_init+0x11/0x60 [ 265.905865] [<ffffffff81444320>] ? ip_fragment+0x870/0x870 [ 265.912291] [<ffffffff81444adf>] ip_finish_output+0x4af/0x800 [ 265.918962] [<ffffffff81445731>] ip_output+0x51/0xa0 [ 265.924825] [<ffffffff814412e7>] ip_forward_finish+0x77/0x80 [ 265.931356] [<ffffffff8144160f>] ip_forward+0x31f/0x470 [ 265.937429] [<ffffffff8143f67c>] ip_rcv_finish+0x2ec/0x340 [ 265.943741] [<ffffffff8143fe16>] ip_rcv+0x336/0x3c0 [ 265.949416] [<ffffffff814690b8>] ? tcp4_gro_receive+0x178/0x1b0 [ 265.956111] [<ffffffff81407f71>] __netif_receive_skb_core+0x661/0x720 [ 265.963309] [<ffffffff8143f390>] ? inet_add_protocol+0x50/0x50 [ 265.969883] [<ffffffff81408087>] __netif_receive_skb+0x57/0x80 [ 265.976431] [<ffffffff8140826e>] netif_receive_skb_internal+0x5e/0xa0 [ 265.983569] [<ffffffff81408398>] napi_gro_complete+0xd8/0xf0 [ 265.989917] [<ffffffff81408792>] napi_gro_flush+0x72/0x90 [ 265.995998] [<ffffffff8140880a>] napi_complete_done+0x5a/0xc0 [ 266.002426] [<ffffffffa01e8931>] igb_poll+0x451/0x710 [igb] [ 266.008662] [<ffffffff814089a1>] net_rx_action+0x131/0x2f0 [ 266.014831] [<ffffffff81065d0a>] __do_softirq+0x10a/0x210 [ 266.020882] [<ffffffff81065fbd>] irq_exit+0x3d/ [ 266.026408] [<ffffffff8100faa5>] do_IRQ+0xc5/0xf0 [ 266.031754] [<ffffffff8150506d>] common_interrupt+0x6d/0x6d [ 266.037962] <EOI> [ 266.039902] [<ffffffff813d5e0c>] ? cpuidle_enter_state+0x5c/0xd0 [ 266.047309] [<ffffffff813d5e02>] ? cpuidle_enter_state+0x52/0xd0 [ 266.053727] [<ffffffff813d5f52>] cpuidle_enter+0x12/0x20 [ 266.059426] [<ffffffff81099149>] cpu_startup_entry+0x229/0x300 [ 266.065645] [<ffffffff8103f094>] start_secondary+0x1b4/0x1c0 [ 266.071709] Code: 00 00 00 40 dd a8 81 ff ff ff ff 79 e2 00 00 03 88 ff ff 01 00 00 00 00 00 ad de 00 00 00 00 00 00 00 00 d8 dd b0 05 00 c9 ff ff <58> f0 c0 ff 03 88 ff ff 00 00 00 00 00 00 00 00 84 0e fd f6 07 [ 266.092288] RIP [<ffff880030f8ed28>] 0xffff880030f8ed28 [ 266.097971] RSP <ffff88041fcc3820> [ 266.101808] CR2: ffff880030f8ed28 [ 266.105464] ---[ end trace 753c193a4a34f813 ]--- I suspect the skb probably shouldn't be getting this far, but I am not familiar enough with the code to know where or how this should be handled. |
I have been experiencing GPFs when removing TPROXY rules in modern kernels for a while. This occurs on many different bare-metal x86 machines, plus KVM. The setup to reproduce is a bit specific: 1. TUN tunnel to remote server, with default route through tunnel (I have no idea whether this is important, but I haven't reproduced without it) 2. Transparent TCP proxy, using IP_TRANSPARENT to preserve src and dst addresses. (Basically a TCP accelerator) 3. TPROXY setup with the following rules: iptables -t nat -A OUTPUT -o tun10 -p tcp -m multiport --ports 80,443,8080 -j REDIRECT --to-ports 5080 iptables -t mangle -A bridge_existing -j MARK --set-xmark 0xf0/0xffffffff iptables -t mangle -A bridge_existing -j ACCEPT iptables -t mangle -A tcp_proxy -d 10.123.3.0/24 -p tcp -m socket -j bridge_existing iptables -t mangle -A tcp_proxy -d 10.123.3.1/32 -p tcp -j RETURN iptables -t mangle -A tcp_proxy -d 10.123.3.0/24 -p tcp -m multiport --dports 80,443,8080 -j TPROXY --on-port 5080 --on-ip 127.0.0.1 --tproxy-mark 0xf0/0xffffffff 4. As per the TPROXY documentation, the marked packets are sent to a separate routing table: local default dev lo scope host 5. Regular traffic through the box from outside. (I use 40 browser tabs accessing random pages every 10 seconds) Sometimes, when these rules are removed, the GPF occurs. To reproduce, I add and remove the rules every 3 seconds. It usually occurs within 10 minutes. Here is the info with a normal kernel (bare-metal): [ 262.017241] general protection fault: 0000 [#1] SMP [ 262.017436] Modules linked in: nf_conntrack_netlink nfnetlink netconsole configfs sch_sfq xt_connbytes xt_hashlimit xt_TPROXY xt_socket xt_length nf_defrag_ipv6 xt_REDIRECT nf_nat_redirect xt_multiport sch_htb xt_TCPMSS xt_CLASSIFY xt_dscp xt_mark xt_nat ipt_REJECT nf_reject_ipv4 xt_state xt_comment veth xt_CHECKSUM xt_tcpudp iptable_mangle iptable_filter xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables arptable_filter arp_tables x_tables tun nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc 8021q garp mrp joydev hid_generic usbhid hid bridge stp llc x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_pcm ast ghash_clmulni_intel snd_timer ttm aesni_intel iTCO_wdt evdev iTCO_vendor_support aes_x86_64 snd drm_kms_helper lrw gf128mul glue_helper soundcore ablk_helper drm cryptd i2c_i801 pcspkr tpm_tis battery tpm acpi_pad video button xhci_pci mei_me xhci_hcd mei processor lpc_ich shpchp mfd_core ipmi_watchdog ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod sg sd_mod crc32c_intel ahci libahci libata scsi_mod igb i2c_algo_bit ehci_pci i2c_core ehci_hcd dca ptp pps_core usbcore usb_common fan thermal thermal_sys [ 262.022561] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.19.0-rc4+ #2 [ 262.023149] Hardware name: Supermicro X10SLM+-LN4F/X10SLM+-LN4F, BIOS 2.00 04/24/2014 [ 262.023744] task: ffff8807fbecea00 ti: ffff8807fbee0000 task.ti: ffff8807fbee0000 [ 262.024370] RIP: 0010:[<ffffffff8140f553>] [<ffffffff8140f553>] __sk_free+0x13/0x130 [ 262.025027] RSP: 0018:ffff88081fd03b28 EFLAGS: 00010202 [ 262.025676] RAX: 3fa6000500003a98 RBX: ffff8806f2a80280 RCX: ffffffffa052c780 [ 262.026414] RDX: 6c6c616d560a026b RSI: ffff8807c8acc000 RDI: ffff8806f2a80280 [ 262.027133] RBP: ffff88078b5abc00 R08: ffff88078b45709c R09: 0000000000000001 [ 262.027838] R10: ffff88077c6e1c00 R11: 0000000000000001 R12: ffff880036c1bec0 [ 262.028559] R13: ffff880036fcd000 R14: ffff8807c8acc000 R15: 0000000000000000 [ 262.029314] FS: 0000000000000000(0000) GS:ffff88081fd00000(0000) knlGS:0000000000000000 [ 262.030096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 262.030887] CR2: ffffffffff600400 CR3: 0000000001816000 CR4: 00000000001407e0 [ 262.031677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 262.032495] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 262.033310] Stack: [ 262.034118] ffff8806f2a80280 ffff88078b5abc00 ffff880036c1bec0 ffffffffa0529113 [ 262.034993] ffff88078b5abc00 ffff8807c8acc000 ffff88078b5abc00 ffffffff818cf4d8 [ 262.035847] 00000000000005ac ffffffff8142630a 0000000000000000 ffff8807bc43dc00 [ 262.036752] Call Trace: [ 262.037637] <IRQ> [ 262.037705] [<ffffffffa0529113>] ? tun_net_xmit+0x143/0x390 [tun] [ 262.039532] [<ffffffff8142630a>] ? dev_hard_start_xmit+0x16a/0x3a0 [ 262.040440] [<ffffffff81425ec0>] ? validate_xmit_skb.isra.93.part.94+0x10/0x2f0 [ 262.041349] [<ffffffff81445da7>] ? sch_direct_xmit+0xc7/0x1d0 [ 262.042294] [<ffffffff81445f3e>] ? __qdisc_run+0x8e/0x1c0 [ 262.043271] [<ffffffff814267ca>] ? __dev_queue_xmit+0x28a/0x520 [ 262.044228] [<ffffffff81460477>] ? ip_finish_output2+0x137/0x3c0 [ 262.045203] [<ffffffff8145efff>] ? ip_fragment+0x2df/0xa70 [ 262.046170] [<ffffffff81460340>] ? ip_append_data.part.46+0xe0/0xe0 [ 262.047156] [<ffffffff81460770>] ? skb_set_owner_w+0x70/0x70 [ 262.048155] [<ffffffff81460bcf>] ? ip_finish_output+0x45f/0x850 [ 262.049147] [<ffffffff81424402>] ? __netif_receive_skb_core+0x552/0x7d0 [ 262.050159] [<ffffffff8101bca6>] ? native_sched_clock+0x26/0x90 [ 262.051140] [<ffffffff8101bd15>] ? sched_clock+0x5/0x10 [ 262.052140] [<ffffffff814253d2>] ? process_backlog+0xa2/0x130 [ 262.053157] [<ffffffff81424be1>] ? net_rx_action+0x201/0x340 [ 262.054148] [<ffffffff8106aecc>] ? __do_softirq+0x10c/0x280 [ 262.055132] [<ffffffff8106b195>] ? irq_exit+0x95/0xa0 [ 262.056076] [<ffffffff81014cda>] ? do_IRQ+0x4a/0xd0 [ 262.056984] [<ffffffff815178ed>] ? common_interrupt+0x6d/0x6d [ 262.057925] <EOI> [ 262.057936] [<ffffffff813f3a0c>] ? cpuidle_enter_state+0x5c/0x150 [ 262.059617] [<ffffffff813f39f9>] ? cpuidle_enter_state+0x49/0x150 [ 262.060425] [<ffffffff810a142d>] ? cpu_startup_entry+0x2fd/0x3a0 [ 262.061204] [<ffffffff810439a5>] ? start_secondary+0x155/0x180 [ 262.061961] Code: f0 29 9d 20 01 00 00 eb e8 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 41 54 55 53 48 8b 87 b0 02 00 00 48 89 fb 48 85 c0 74 02 <ff> d0 48 8b b3 e0 00 00 00 48 85 f6 74 13 48 89 df e8 f7 a2 02 [ 262.063621] RIP [<ffffffff8140f553>] __sk_free+0x13/0x130 [ 262.064359] RSP <ffff88081fd03b28> [ 262.065164] ---[ end trace 79ce0371f0ffc393 ]--- [ 262.761655] Kernel panic - not syncing: Fatal exception in interrupt [ 262.763647] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 262.764073] drm_kms_helper: panic occurred, switching back to text console [ 263.366508] ---[ end Kernel panic - not syncing: Fatal exception in interrupt If I enable lock debugging, I get a similar, but different result: [ 1080.464457] general protection fault: 0000 [#1] SMP [ 1080.469469] Modules linked in: xt_TPROXY xt_socket nf_defrag_ipv6 xt_REDIRECT nf_nat_redirect xt_multiport sch_htb tun xt_CLASSIFY xt_dscp xt_TCPMSS xt_mark xt_tcpudp tcp_yeah tcp_westwood tcp_veno tcp_vegas tcp_scalable tcp_lp tcp_illinois tcp_hybla tcp_htcp tcp_highspeed tcp_diag inet_diag tcp_bic xt_nat iptable_mangle iptable_nat nf_nat_ipv4 nf_nat 8021q garp mrp stp llc ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment arptable_filter arp_tables iptable_filter ip_tables x_tables nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc joydev hid_generic usbhid hid loop x86_pkg_temp_thermal intel_powerclamp intel_rapl iosf_mbi coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel evdev ast ttm drm_kms_helper snd_pcm aesni_intel aes_x86_64 drm lrw snd_timer gf128mul glue_helper snd ablk_helper cryptd soundcore i2c_i801 pcspkr ipmi_si battery tpm_tis ipmi_msghandler tpm video shpchp acpi_pad lpc_ich xhci_pci mfd_core mei_me xhci_hcd mei processor button ext4 crc16 mbcache jbd2 sg sd_mod crc32c_intel igb i2c_algo_bit ahci libahci i2c_core dca ehci_pci ehci_hcd libata fan thermal thermal_sys e1000e scsi_mod usbcore usb_common ptp pps_core [ 1080.581534] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0-rc5+ #19 [ 1080.587978] Hardware name: Supermicro X10SLM-F/X10SLM-F, BIOS 2.0 04/24/2014 [ 1080.595028] task: ffffffff81a1b520 ti: ffffffff81a00000 task.ti: ffffffff81a00000 [ 1080.602521] RIP: 0010:[<ffffffff810c01f0>] [<ffffffff810c01f0>] __lock_acquire+0x660/0x1ca0 [ 1080.610982] RSP: 0018:ffff88041fc03668 EFLAGS: 00010002 [ 1080.616302] RAX: 0000000000000000 RBX: 3ca096b028d7976b RCX: 0000000000000000 [ 1080.623447] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8804071f3220 [ 1080.630594] RBP: ffff88041fc03758 R08: 0000000000000001 R09: 0000000000000001 [ 1080.637740] R10: ffffffff81a1b520 R11: 0000000000000001 R12: 0000000000000000 [ 1080.644885] R13: ffff8804071f3220 R14: 0000000000000001 R15: 0000000000000000 [ 1080.652031] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [ 1080.660130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1080.665882] CR2: 00007fc91a21f000 CR3: 0000000001a14000 CR4: 00000000001407f0 [ 1080.673016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1080.680154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1080.687292] Stack: [ 1080.689812] ffff8800d34ba390 ffffffff81076fe4 00000000f7167000 ffff8803f71090a0 [ 1080.697816] 000000001fc036a8 ffffffff81a1bda8 0000000000000007 ffffffff81a1bd58 [ 1080.705819] 0000000000000005 0000000000000001 000000000000002e 000000000000002e [ 1080.713823] Call Trace: [ 1080.716784] <IRQ> [ 1080.718715] [<ffffffff81076fe4>] ? __local_bh_enable_ip+0xa4/0xf0 [ 1080.726125] [<ffffffff810c1e35>] lock_acquire+0xe5/0x140 [ 1080.732040] [<ffffffff814ba99b>] ? skb_queue_tail+0x2b/0x60 [ 1080.738215] [<ffffffff815f679e>] _raw_spin_lock_irqsave+0x4e/0x70 [ 1080.744918] [<ffffffff814ba99b>] ? skb_queue_tail+0x2b/0x60 [ 1080.751102] [<ffffffff810c28a7>] ? trace_hardirqs_on_caller+0x1d7/0x210 [ 1080.758327] [<ffffffff814ba99b>] skb_queue_tail+0x2b/0x60 [ 1080.764347] [<ffffffff814bc6c8>] sock_queue_err_skb+0xe8/0x150 [ 1080.770802] [<ffffffff814ba70e>] ? __skb_clone+0x2e/0x140 [ 1080.776820] [<ffffffff814bcc03>] __skb_complete_tx_timestamp+0xe3/0x100 [ 1080.784060] [<ffffffff814bcc98>] __skb_tstamp_tx+0x78/0x90 [ 1080.790168] [<ffffffff814bccc4>] skb_tstamp_tx+0x14/0x20 [ 1080.796100] [<ffffffffa07d89a4>] tun_net_xmit+0x304/0x4c0 [tun] [ 1080.802641] [<ffffffffa07d86a5>] ? tun_net_xmit+0x5/0x4c0 [tun] [ 1080.809171] [<ffffffff814d0852>] dev_hard_start_xmit+0x3a2/0x4f0 [ 1080.815790] [<ffffffff814f64b9>] sch_direct_xmit+0xa9/0x1d0 [ 1080.821972] [<ffffffff814d0de4>] __dev_queue_xmit+0x444/0x7a0 [ 1080.828312] [<ffffffff814d0a00>] ? __dev_queue_xmit+0x60/0x7a0 [ 1080.834721] [<ffffffff814d1160>] dev_queue_xmit+0x10/0x20 [ 1080.840692] [<ffffffff814d8ed1>] neigh_direct_output+0x11/0x20 [ 1080.847073] [<ffffffff81516ad4>] ip_finish_output2+0x494/0x600 [ 1080.853449] [<ffffffff815167d8>] ? ip_finish_output2+0x198/0x600 [ 1080.859981] [<ffffffff810bd6ff>] ? __lock_is_held+0x4f/0x80 [ 1080.866068] [<ffffffff81517508>] ip_finish_output+0x8c8/0xa00 [ 1080.872313] [<ffffffff81518238>] ip_output+0x88/0xe0 [ 1080.877763] [<ffffffff815130a9>] ip_forward_finish+0xe9/0x150 [ 1080.883981] [<ffffffff8151348c>] ip_forward+0x37c/0x550 [ 1080.889663] [<ffffffff8151131e>] ip_rcv_finish+0x46e/0x580 [ 1080.895606] [<ffffffff815119ce>] ip_rcv+0x33e/0x3d0 [ 1080.900932] [<ffffffff814cccfe>] __netif_receive_skb_core+0x83e/0x950 [ 1080.907819] [<ffffffff814cc569>] ? __netif_receive_skb_core+0xa9/0x950 [ 1080.914784] [<ffffffff810edc75>] ? ktime_get_with_offset+0xb5/0x150 [ 1080.921485] [<ffffffff814cce67>] __netif_receive_skb+0x57/0x80 [ 1080.927748] [<ffffffff814ce218>] netif_receive_skb_internal+0x168/0x1e0 [ 1080.934798] [<ffffffff814cf080>] napi_gro_receive+0x70/0xf0 [ 1080.940817] [<ffffffffa01b0b19>] igb_poll+0xa89/0xe10 [igb] [ 1080.946830] [<ffffffff814cec10>] net_rx_action+0x140/0x340 [ 1080.952747] [<ffffffff814268da>] ? add_interrupt_randomness+0x3a/0x1e0 [ 1080.959700] [<ffffffff81076cd7>] __do_softirq+0x167/0x2f0 [ 1080.965532] [<ffffffff810770e7>] irq_exit+0x47/0xb0 [ 1080.970840] [<ffffffff815f9cdd>] do_IRQ+0xcd/0xf0 [ 1080.975975] [<ffffffff815f7932>] common_interrupt+0x72/0x72 [ 1080.981969] <EOI> [ 1080.983900] [<ffffffff8149598b>] ? cpuidle_enter_state+0xbb/0x190 [ 1080.990961] [<ffffffff81495984>] ? cpuidle_enter_state+0xb4/0x190 [ 1080.997465] [<ffffffff81495b37>] cpuidle_enter+0x17/0x20 [ 1081.003192] [<ffffffff810b78f6>] cpu_startup_entry+0x2c6/0x400 [ 1081.009438] [<ffffffff815e143d>] rest_init+0x12d/0x140 [ 1081.014990] [<ffffffff815e1315>] ? rest_init+0x5/0x140 [ 1081.020541] [<ffffffff81b4ccc3>] ? ftrace_init+0xc6/0x159 [ 1081.026352] [<ffffffff81b27129>] start_kernel+0x4b2/0x4bf [ 1081.032156] [<ffffffff81b269d7>] ? set_init_arg+0x57/0x57 [ 1081.037959] [<ffffffff81b26117>] ? early_idt_handlers+0x117/0x120 [ 1081.044457] [<ffffffff81b265f0>] x86_64_start_reservations+0x2a/0x2c [ 1081.051212] [<ffffffff81b26738>] x86_64_start_kernel+0x146/0x155 [ 1081.057619] Code: 81 48 c7 c2 b6 5e 7f 81 be 3d 03 00 00 48 c7 c7 7c 96 7f 81 31 c0 e8 90 24 fb ff e9 03 05 00 00 48 85 db 0f 84 fa 04 00 00 66 90 <3e> ff 83 98 01 00 00 8b 05 9b 6d 87 01 45 8b a2 68 07 00 00 85 [ 1081.078236] RIP [<ffffffff810c01f0>] __lock_acquire+0x660/0x1ca0 [ 1081.084695] RSP <ffff88041fc03668> [ 1081.088535] ---[ end trace 1aa64404a291b379 ]--- [ 1081.095085] Kernel panic - not syncing: Fatal exception in interrupt [ 1081.101800] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 1081.112336] drm_kms_helper: panic occurred, switching back to text console git bisect points to this changeset as the first bad one: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=fda55eca5a33f33ffcd4192c6b2d75179714a52c The previous changeset does not appear to exhibit this behaviour. I ran it a number of times, including overnight, with no issues. Obviously, that change can't be the true cause, but it does appear to trigger it.