Bug 217826
Summary: | Since 6.1: flow_dissector.c __skb_flow_dissect+0xa91/0x1cd0 raises WARNING in specific circumstances | ||
---|---|---|---|
Product: | Networking | Reporter: | stephane.poignant |
Component: | Other | Assignee: | Stephen Hemminger (stephen) |
Status: | NEW --- | ||
Severity: | low | CC: | bagasdotme, fw |
Priority: | P3 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | Subsystem: | ||
Regression: | No | Bisected commit-id: | |
Attachments: | pass netns pointer to flow dissector to avoid warn splat |
Description
stephane.poignant
2023-08-26 12:35:09 UTC
(In reply to stephane.poignant from comment #0) > ## Overall description > > After migration from kernel 5.10 to 6.1, i started to encounter the > following warning although they do not seem to have any significant > functional impact. > Following further investigation, it appears that the warning appears when > both of the following conditions are: > - At least one policy based routing rules with L4 selectors (among others, > L3 matchers and fwmark DO NOT trigger the issue) > - A nftable or iptable filtering policy with at least one action rejecting > traffic with TCP reset (drop action, or reject with ICMP unreachable DO NOT > trigger) > > The first time a packet matches the reject policy, and that a RST segment is > forged and sent, a kernel WARNING as follows will show up in the dmesg: > ``` > WARNING: CPU: 0 PID: 0 at net/core/flow_dissector.c:1016 > __skb_flow_dissect+0xa91/0x1cd0 > ``` > > Traffic does not appear to be affected, including the RST segment that is > reliably delivered in spite of the WARNING. > > > ## Expected behaviour > > The warning should not appear, in this particular setup the namespace of the > RST packet should be resolved to the default one. > > > ## Reproducing steps > > The following minimalistic configuration will reproduce the issue: > > - Running last 6.1 "vanilla" kernel at the time of writing: > ``` > root@v-debian12:~# uname -a > Linux v-debian12 6.1.47 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 13:23:01 CEST 2023 > x86_64 GNU/Linux > ``` > NB: Issue won't happen on different 5.10 kernels that i tried > > - A routing policy with a rule that uses L4 selectors: > ``` > root@v-debian12:~# cat /etc/iproute2/rt_tables.d/wanalt.conf > 100 wanalt > > root@v-debian12:~# ip rule ls > 0: from all lookup local > 100: from all ipproto udp dport 27000-27999 lookup wanalt > 32766: from all lookup main > 32767: from all lookup default > > root@v-debian12:~# ip route ls > default via 192.168.56.1 dev enp0s8 > 192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.3 > > root@v-debian12:~# ip route ls table wanalt > default via 192.168.56.2 dev enp0s8 > ``` > > - A nftables or iptables rules that rejects some traffic with tcp reset: > ``` > root@v-debian12:~# nft list ruleset > table inet filter { > chain input { > type filter hook input priority filter; policy accept; > tcp dport 12345 log reject with tcp reset > } > > chain output { > type filter hook output priority filter; policy accept; > tcp sport 12345 tcp flags rst / fin,syn,rst log > } > } > ``` > (NB: the rule in thhhe output chain is only for debugging purpose, it let us > see what packet caused the warning in the dmesg) > > - from another VM, we attempt to connect to port 12345 on the reproducing > host: > ``` > root@v-debian11:~# nc 192.168.56.3 12345 > ``` > > - the following appears in the dmesg of the reproducing host: > ``` > # the packet that matched the reject rule > 2023-08-26T14:04:35.619764+02:00 v-debian12 kernel: [ 3335.550250] IN=enp0s8 > OUT= MAC=08:00:27:76:64:62:08:00:27:69:c9:e5:08:00 SRC=192.168.56.4 > DST=192.168.56.3 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30150 DF PROTO=TCP > SPT=48576 DPT=12345 WINDOW=64240 RES=0x00 SYN URGP=0 > > # the warning showing up > 2023-08-26T14:04:35.619791+02:00 v-debian12 kernel: [ 3335.550267] > ------------[ cut here ]------------ > 2023-08-26T14:04:35.619793+02:00 v-debian12 kernel: [ 3335.550269] WARNING: > CPU: 0 PID: 0 at net/core/flow_dissector.c:1016 > __skb_flow_dissect+0xa91/0x1cd0 > 2023-08-26T14:04:35.619795+02:00 v-debian12 kernel: [ 3335.550277] Modules > linked in: nf_log_syslog(E) nft_reject_inet(E) nf_reject_ipv4(E) > nf_reject_ipv6(E) nft_reject(E) nft_log(E) nf_tables(E) libcrc32c(E) > nfnetlink(E) intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core(E) > intel_powerclamp(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha512_generic(E) > vmwgfx(E) aesni_intel(E) drm_ttm_helper(E) snd_pcm(E) crypto_simd(E) > iTCO_wdt(E) cryptd(E) intel_pmc_bxt(E) ttm(E) iTCO_vendor_support(E) rapl(E) > watchdog(E) snd_timer(E) drm_kms_helper(E) snd(E) joydev(E) soundcore(E) > vboxguest(E) pcspkr(E) rng_core(E) ac(E) button(E) evdev(E) serio_raw(E) > sg(E) binfmt_misc(E) drm(E) fuse(E) dm_mod(E) loop(E) efi_pstore(E) > configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) > crc16(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) crc64_rocksoft(E) crc64(E) > crc_t10dif(E) sr_mod(E) cdrom(E) crct10dif_generic(E) ata_generic(E) ahci(E) > libahci(E) ata_piix(E) crct10dif_pclmul(E) crct10dif_common(E) > crc32_pclmul(E) crc32c_intel(E) psmouse(E) > 2023-08-26T14:04:35.619796+02:00 v-debian12 kernel: [ 3335.550336] > libata(E) scsi_mod(E) scsi_common(E) lpc_ich(E) i2c_piix4(E) e1000(E) > battery(E) video(E) wmi(E) > 2023-08-26T14:04:35.619797+02:00 v-debian12 kernel: [ 3335.550345] CPU: 0 > PID: 0 Comm: swapper/0 Tainted: G E 6.1.47 #1 > 2023-08-26T14:04:35.619797+02:00 v-debian12 kernel: [ 3335.550348] Hardware > name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > 2023-08-26T14:04:35.619798+02:00 v-debian12 kernel: [ 3335.550350] RIP: > 0010:__skb_flow_dissect+0xa91/0x1cd0 > 2023-08-26T14:04:35.619799+02:00 v-debian12 kernel: [ 3335.550354] Code: 00 > bf 02 00 00 00 49 89 0c 12 66 89 78 02 41 8b 55 00 e9 e7 f6 ff ff 49 8b 44 > 24 18 48 85 c0 74 09 4c 8b 78 30 e9 30 f7 ff ff <0f> 0b e9 f6 f7 ff ff f6 c2 > 20 0f 84 e4 f6 ff ff 41 0f b7 45 0e 4c > 2023-08-26T14:04:35.619809+02:00 v-debian12 kernel: [ 3335.550356] RSP: > 0018:ffffc130800036b0 EFLAGS: 00010246 > 2023-08-26T14:04:35.619810+02:00 v-debian12 kernel: [ 3335.550358] RAX: > 0000000000000000 RBX: 0000000000000008 RCX: ffffc13080003810 > 2023-08-26T14:04:35.619811+02:00 v-debian12 kernel: [ 3335.550360] RDX: > ffffffff94240e40 RSI: ffff9b53c770ee00 RDI: 0000000000000000 > 2023-08-26T14:04:35.619812+02:00 v-debian12 kernel: [ 3335.550361] RBP: > ffffc13080003798 R08: 0000000000000000 R09: 0000000000000000 > 2023-08-26T14:04:35.619813+02:00 v-debian12 kernel: [ 3335.550363] R10: > ffffc13080003810 R11: ffff9b53e812ba80 R12: ffff9b53c770ee00 > 2023-08-26T14:04:35.619814+02:00 v-debian12 kernel: [ 3335.550364] R13: > ffffffff94240e40 R14: ffff9b53c6456080 R15: 0000000000000000 > 2023-08-26T14:04:35.619814+02:00 v-debian12 kernel: [ 3335.550366] FS: > 0000000000000000(0000) GS:ffff9b53fdc00000(0000) knlGS:0000000000000000 > 2023-08-26T14:04:35.619815+02:00 v-debian12 kernel: [ 3335.550368] CS: 0010 > DS: 0000 ES: 0000 CR0: 0000000080050033 > 2023-08-26T14:04:35.619816+02:00 v-debian12 kernel: [ 3335.550369] CR2: > 00007ffdda939440 CR3: 000000004d326005 CR4: 00000000000706f0 > 2023-08-26T14:04:35.619817+02:00 v-debian12 kernel: [ 3335.550371] Call > Trace: > 2023-08-26T14:04:35.619817+02:00 v-debian12 kernel: [ 3335.550374] <IRQ> > 2023-08-26T14:04:35.619818+02:00 v-debian12 kernel: [ 3335.550378] ? > __warn+0x7d/0xc0 > 2023-08-26T14:04:35.619819+02:00 v-debian12 kernel: [ 3335.550382] ? > __skb_flow_dissect+0xa91/0x1cd0 > 2023-08-26T14:04:35.619820+02:00 v-debian12 kernel: [ 3335.550385] ? > report_bug+0xe6/0x170 > 2023-08-26T14:04:35.619821+02:00 v-debian12 kernel: [ 3335.550388] ? > handle_bug+0x41/0x70 > 2023-08-26T14:04:35.619821+02:00 v-debian12 kernel: [ 3335.550391] ? > exc_invalid_op+0x13/0x60 > 2023-08-26T14:04:35.619822+02:00 v-debian12 kernel: [ 3335.550393] ? > asm_exc_invalid_op+0x16/0x20 > 2023-08-26T14:04:35.619832+02:00 v-debian12 kernel: [ 3335.550397] ? > __skb_flow_dissect+0xa91/0x1cd0 > 2023-08-26T14:04:35.619833+02:00 v-debian12 kernel: [ 3335.550401] ? > __inet_dev_addr_type+0xe8/0x180 > 2023-08-26T14:04:35.619834+02:00 v-debian12 kernel: [ 3335.550405] > ip_route_me_harder+0x143/0x330 > 2023-08-26T14:04:35.619835+02:00 v-debian12 kernel: [ 3335.550411] > nf_send_reset+0x17c/0x2d0 [nf_reject_ipv4] > 2023-08-26T14:04:35.619835+02:00 v-debian12 kernel: [ 3335.550417] > nft_reject_inet_eval+0xa9/0xf2 [nft_reject_inet] > 2023-08-26T14:04:35.619836+02:00 v-debian12 kernel: [ 3335.550422] > nft_do_chain+0x198/0x5d0 [nf_tables] > 2023-08-26T14:04:35.619837+02:00 v-debian12 kernel: [ 3335.550439] > nft_do_chain_inet+0xa4/0x110 [nf_tables] > 2023-08-26T14:04:35.619838+02:00 v-debian12 kernel: [ 3335.550454] > nf_hook_slow+0x41/0xc0 > 2023-08-26T14:04:35.619838+02:00 v-debian12 kernel: [ 3335.550458] > ip_local_deliver+0xce/0x110 > 2023-08-26T14:04:35.619839+02:00 v-debian12 kernel: [ 3335.550461] ? > ip_protocol_deliver_rcu+0x230/0x230 > 2023-08-26T14:04:35.619840+02:00 v-debian12 kernel: [ 3335.550464] > ip_sublist_rcv_finish+0x7e/0x90 > 2023-08-26T14:04:35.619840+02:00 v-debian12 kernel: [ 3335.550466] > ip_sublist_rcv+0x183/0x230 > 2023-08-26T14:04:35.619841+02:00 v-debian12 kernel: [ 3335.550468] ? > __netif_receive_skb_core.constprop.0+0xaf/0xf00 > 2023-08-26T14:04:35.619842+02:00 v-debian12 kernel: [ 3335.550472] > ip_list_rcv+0x139/0x170 > 2023-08-26T14:04:35.619843+02:00 v-debian12 kernel: [ 3335.550475] > __netif_receive_skb_list_core+0x29e/0x2c0 > 2023-08-26T14:04:35.619843+02:00 v-debian12 kernel: [ 3335.550479] > netif_receive_skb_list_internal+0x1cd/0x300 > 2023-08-26T14:04:35.619844+02:00 v-debian12 kernel: [ 3335.550483] ? > e1000_clean_rx_irq+0x369/0x500 [e1000] > 2023-08-26T14:04:35.619845+02:00 v-debian12 kernel: [ 3335.550492] > napi_complete_done+0x6d/0x1a0 > 2023-08-26T14:04:35.619845+02:00 v-debian12 kernel: [ 3335.550495] > e1000_clean+0x296/0x6a0 [e1000] > 2023-08-26T14:04:35.619846+02:00 v-debian12 kernel: [ 3335.550505] ? > vbg_heartbeat_host_config+0x90/0x90 [vboxguest] > 2023-08-26T14:04:35.619847+02:00 v-debian12 kernel: [ 3335.550512] > __napi_poll+0x2b/0x160 > 2023-08-26T14:04:35.619848+02:00 v-debian12 kernel: [ 3335.550515] > net_rx_action+0x2a5/0x360 > 2023-08-26T14:04:35.619848+02:00 v-debian12 kernel: [ 3335.550518] ? > __napi_schedule+0x8e/0xb0 > 2023-08-26T14:04:35.619849+02:00 v-debian12 kernel: [ 3335.550521] > __do_softirq+0xf0/0x2fe > 2023-08-26T14:04:35.619850+02:00 v-debian12 kernel: [ 3335.550525] > __irq_exit_rcu+0xc7/0x130 > 2023-08-26T14:04:35.619850+02:00 v-debian12 kernel: [ 3335.550528] > common_interrupt+0xb9/0xd0 > 2023-08-26T14:04:35.619851+02:00 v-debian12 kernel: [ 3335.550531] </IRQ> > 2023-08-26T14:04:35.619852+02:00 v-debian12 kernel: [ 3335.550532] <TASK> > 2023-08-26T14:04:35.619852+02:00 v-debian12 kernel: [ 3335.550533] > asm_common_interrupt+0x22/0x40 > 2023-08-26T14:04:35.619853+02:00 v-debian12 kernel: [ 3335.550535] RIP: > 0010:mwait_idle+0x54/0x80 > 2023-08-26T14:04:35.619854+02:00 v-debian12 kernel: [ 3335.550539] Code: 31 > d2 48 89 d1 65 48 8b 04 25 c0 fb 01 00 0f 01 c8 48 8b 00 a8 08 75 14 66 90 > 0f 00 2d 05 f5 5c 00 31 c0 48 89 c1 fb 0f 01 c9 <eb> 06 fb 0f 1f 44 00 00 65 > 48 8b 04 25 c0 fb 01 00 3e 80 60 02 df > 2023-08-26T14:04:35.619855+02:00 v-debian12 kernel: [ 3335.550540] RSP: > 0018:ffffffff94003e98 EFLAGS: 00000246 > 2023-08-26T14:04:35.619855+02:00 v-debian12 kernel: [ 3335.550542] RAX: > 0000000000000000 RBX: ffffffff9401aa40 RCX: 0000000000000000 > 2023-08-26T14:04:35.619856+02:00 v-debian12 kernel: [ 3335.550544] RDX: > 0000000000000000 RSI: ffffffff9399cba6 RDI: ffffffff939761d1 > 2023-08-26T14:04:35.619857+02:00 v-debian12 kernel: [ 3335.550545] RBP: > 0000000000000000 R08: 000003a6bb03287e R09: 0000000000000000 > 2023-08-26T14:04:35.619857+02:00 v-debian12 kernel: [ 3335.550546] R10: > 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 > 2023-08-26T14:04:35.619858+02:00 v-debian12 kernel: [ 3335.550547] R13: > 0000000000000000 R14: ffffffff9401a118 R15: 0000000000000000 > 2023-08-26T14:04:35.619859+02:00 v-debian12 kernel: [ 3335.550550] ? > default_idle+0x10/0x10 > 2023-08-26T14:04:35.619859+02:00 v-debian12 kernel: [ 3335.550553] > default_idle_call+0x36/0xf0 > 2023-08-26T14:04:35.619860+02:00 v-debian12 kernel: [ 3335.550556] > do_idle+0x225/0x2b0 > 2023-08-26T14:04:35.619861+02:00 v-debian12 kernel: [ 3335.550560] > cpu_startup_entry+0x19/0x20 > 2023-08-26T14:04:35.619862+02:00 v-debian12 kernel: [ 3335.550562] > rest_init+0xcb/0xd0 > 2023-08-26T14:04:35.619862+02:00 v-debian12 kernel: [ 3335.550565] > arch_call_rest_init+0xa/0x14 > 2023-08-26T14:04:35.619863+02:00 v-debian12 kernel: [ 3335.550572] > start_kernel+0x6fe/0x727 > 2023-08-26T14:04:35.619873+02:00 v-debian12 kernel: [ 3335.550575] > secondary_startup_64_no_verify+0xe5/0xeb > 2023-08-26T14:04:35.619875+02:00 v-debian12 kernel: [ 3335.550580] </TASK> > 2023-08-26T14:04:35.619875+02:00 v-debian12 kernel: [ 3335.550581] ---[ end > trace 0000000000000000 ]--- > > # the RST packet that caused the warning > 2023-08-26T14:04:35.619876+02:00 v-debian12 kernel: [ 3335.550593] IN= > OUT=enp0s8 SRC=192.168.56.3 DST=192.168.56.4 LEN=40 TOS=0x00 PREC=0x00 > TTL=64 ID=0 DF PROTO=TCP SPT=12345 DPT=48576 WINDOW=0 RES=0x00 ACK RST URGP=0 > ``` > > > ## Known workarounds > > The issue is benign and does not prevent the traffic from being sent > correctly so there is no strict necessity for a workaround. > On my affected setup i could prevent the warning from showing up by marking > traffic for the alternate uplink using nftables and rewriting the routing > policy to only use L3 and fwmark selectors. Can you bisect between v5.10 and v6.1? > Can you bisect between v5.10 and v6.1?
You're not joking, right?
On 28/08/2023 14:50, bugzilla-daemon@kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=217826 > > --- Comment #2 from Artem S. Tashkinov (aros@gmx.com) --- >> Can you bisect between v5.10 and v6.1? > > You're not joking, right? > Oops, I thought it was regression. Hi, thanks for the replies. I did a few tests with other kernel versions (using the Debian packages). Both the older 6.1.12 and 6.4.11 are reproducing. Here is the output obtained with 6.4.11: ``` 2023-08-28T21:25:09.311710+02:00 v-debian kernel: [ 40.586788] IN=ens33 OUT= MAC=00:0c:29:ef:12:dc:00:0c:29:6a:e2:37:08:00 SRC=192.168.8.81 DST=192.168.8.88 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=61393 DF PROTO=TCP SPT=55436 DPT=12345 WINDOW=64106 RES=0x00 SYN URGP=0 2023-08-28T21:25:09.311865+02:00 v-debian kernel: [ 40.586975] ------------[ cut here ]------------ 2023-08-28T21:25:09.311867+02:00 v-debian kernel: [ 40.587004] WARNING: CPU: 1 PID: 0 at net/core/flow_dissector.c:1018 __skb_flow_dissect+0x7ad/0x1b60 2023-08-28T21:25:09.312315+02:00 v-debian kernel: [ 40.587016] Modules linked in: nf_log_syslog nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_log nf_tables libcrc32c nfnetlink binfmt_misc intel_rapl_msr vsock_loopback intel_rapl_common vmw_vsock_virtio_transport_common ghash_clmulni_intel sha512_ssse3 sha512_generic vmw_vsock_vmci_transport vsock aesni_intel crypto_simd cryptd rapl vmwgfx vmw_balloon drm_ttm_helper ttm pcspkr drm_kms_helper vmw_vmci button ac joydev sg evdev serio_raw drm loop fuse dm_mod efi_pstore configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif_generic hid_generic usbhid hid sr_mod cdrom ata_generic ata_piix crct10dif_pclmul crct10dif_common mptspi mptscsih crc32_pclmul mptbase libata ehci_pci uhci_hcd crc32c_intel ehci_hcd scsi_transport_spi psmouse i2c_piix4 usbcore scsi_mod e1000 usb_common scsi_common 2023-08-28T21:25:09.312318+02:00 v-debian kernel: [ 40.587462] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.4.0-3-amd64 #1 Debian 6.4.11-1 2023-08-28T21:25:09.312323+02:00 v-debian kernel: [ 40.587470] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 2023-08-28T21:25:09.312326+02:00 v-debian kernel: [ 40.587477] RIP: 0010:__skb_flow_dissect+0x7ad/0x1b60 2023-08-28T21:25:09.312724+02:00 v-debian kernel: [ 40.587485] Code: 00 00 8b 45 18 2b 45 10 83 f8 07 0f 8e 5a 0b 00 00 48 63 45 10 4c 01 f8 74 09 80 38 11 0f 84 1c 04 00 00 31 c9 e9 11 ff ff ff <0f> 0b e9 ad fa ff ff 66 41 81 fd 89 2f 0f 84 84 01 00 00 66 41 81 2023-08-28T21:25:09.312727+02:00 v-debian kernel: [ 40.587494] RSP: 0018:ffffaa10804e87f0 EFLAGS: 00010246 2023-08-28T21:25:09.312728+02:00 v-debian kernel: [ 40.587505] RAX: 0000000000000000 RBX: ffff9ddb796c3800 RCX: ffffaa10804e8950 2023-08-28T21:25:09.312728+02:00 v-debian kernel: [ 40.587512] RDX: 0000000000000028 RSI: ffff9ddb796c3800 RDI: 0000000000000000 2023-08-28T21:25:09.312729+02:00 v-debian kernel: [ 40.587518] RBP: ffffaa10804e88d8 R08: 0000000000000000 R09: 0000000000000000 2023-08-28T21:25:09.312729+02:00 v-debian kernel: [ 40.587525] R10: 0000000000000000 R11: ffff9ddb79473a80 R12: ffffffffa6843c20 2023-08-28T21:25:09.312730+02:00 v-debian kernel: [ 40.587531] R13: 0000000000000008 R14: ffffaa10804e8950 R15: ffff9ddb45349200 2023-08-28T21:25:09.312730+02:00 v-debian kernel: [ 40.587538] FS: 0000000000000000(0000) GS:ffff9ddc75e40000(0000) knlGS:0000000000000000 2023-08-28T21:25:09.312731+02:00 v-debian kernel: [ 40.587545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2023-08-28T21:25:09.312731+02:00 v-debian kernel: [ 40.587552] CR2: 00007f12a2b8bbbc CR3: 0000000138f3a001 CR4: 00000000001706e0 2023-08-28T21:25:09.312732+02:00 v-debian kernel: [ 40.587560] Call Trace: 2023-08-28T21:25:09.312732+02:00 v-debian kernel: [ 40.587566] <IRQ> 2023-08-28T21:25:09.312732+02:00 v-debian kernel: [ 40.587572] ? __skb_flow_dissect+0x7ad/0x1b60 2023-08-28T21:25:09.312733+02:00 v-debian kernel: [ 40.587579] ? __warn+0x81/0x130 2023-08-28T21:25:09.312733+02:00 v-debian kernel: [ 40.587591] ? __skb_flow_dissect+0x7ad/0x1b60 2023-08-28T21:25:09.312734+02:00 v-debian kernel: [ 40.587600] ? report_bug+0x191/0x1c0 2023-08-28T21:25:09.312734+02:00 v-debian kernel: [ 40.587612] ? handle_bug+0x3c/0x80 2023-08-28T21:25:09.312734+02:00 v-debian kernel: [ 40.587621] ? exc_invalid_op+0x17/0x70 2023-08-28T21:25:09.312735+02:00 v-debian kernel: [ 40.587631] ? asm_exc_invalid_op+0x1a/0x20 2023-08-28T21:25:09.312735+02:00 v-debian kernel: [ 40.587643] ? __skb_flow_dissect+0x7ad/0x1b60 2023-08-28T21:25:09.312736+02:00 v-debian kernel: [ 40.587655] ? __inet_dev_addr_type+0xe8/0x180 2023-08-28T21:25:09.312736+02:00 v-debian kernel: [ 40.587669] ip_route_me_harder+0x147/0x340 2023-08-28T21:25:09.312736+02:00 v-debian kernel: [ 40.587686] nf_send_reset+0x183/0x2e0 [nf_reject_ipv4] 2023-08-28T21:25:09.312737+02:00 v-debian kernel: [ 40.587703] nft_reject_inet_eval+0xd5/0x100 [nft_reject_inet] 2023-08-28T21:25:09.312742+02:00 v-debian kernel: [ 40.587716] nft_do_chain+0x101/0x600 [nf_tables] 2023-08-28T21:25:09.313930+02:00 v-debian kernel: [ 40.587895] nft_do_chain_inet+0x54/0x130 [nf_tables] 2023-08-28T21:25:09.313933+02:00 v-debian kernel: [ 40.588100] nf_hook_slow+0x42/0xc0 2023-08-28T21:25:09.313934+02:00 v-debian kernel: [ 40.588113] ip_local_deliver+0xd0/0x120 2023-08-28T21:25:09.313934+02:00 v-debian kernel: [ 40.588124] ? __pfx_ip_local_deliver_finish+0x10/0x10 2023-08-28T21:25:09.313935+02:00 v-debian kernel: [ 40.588136] ip_sublist_rcv_finish+0x7e/0x90 2023-08-28T21:25:09.313935+02:00 v-debian kernel: [ 40.588147] ip_sublist_rcv+0x186/0x230 2023-08-28T21:25:09.313936+02:00 v-debian kernel: [ 40.588158] ? __netif_receive_skb_core.constprop.0+0x316/0xf40 2023-08-28T21:25:09.313936+02:00 v-debian kernel: [ 40.588168] ip_list_rcv+0x13d/0x170 2023-08-28T21:25:09.313936+02:00 v-debian kernel: [ 40.588181] __netif_receive_skb_list_core+0x29d/0x2c0 2023-08-28T21:25:09.313936+02:00 v-debian kernel: [ 40.588193] netif_receive_skb_list_internal+0x1d1/0x310 2023-08-28T21:25:09.313937+02:00 v-debian kernel: [ 40.588202] ? e1000_clean_rx_irq+0x356/0x4f0 [e1000] 2023-08-28T21:25:09.313937+02:00 v-debian kernel: [ 40.588239] napi_complete_done+0x72/0x1a0 2023-08-28T21:25:09.313938+02:00 v-debian kernel: [ 40.588249] e1000_clean+0x291/0x940 [e1000] 2023-08-28T21:25:09.313938+02:00 v-debian kernel: [ 40.588311] ? check_preempt_curr+0x37/0x70 2023-08-28T21:25:09.313938+02:00 v-debian kernel: [ 40.588323] ? ttwu_do_activate+0x5d/0x1f0 2023-08-28T21:25:09.313939+02:00 v-debian kernel: [ 40.588335] __napi_poll+0x28/0x1b0 2023-08-28T21:25:09.313939+02:00 v-debian kernel: [ 40.588344] net_rx_action+0x2ab/0x390 2023-08-28T21:25:09.313939+02:00 v-debian kernel: [ 40.588356] __do_softirq+0xf1/0x301 2023-08-28T21:25:09.313939+02:00 v-debian kernel: [ 40.588367] __irq_exit_rcu+0xb5/0x130 2023-08-28T21:25:09.313940+02:00 v-debian kernel: [ 40.588378] common_interrupt+0xbd/0xe0 2023-08-28T21:25:09.313940+02:00 v-debian kernel: [ 40.588388] </IRQ> 2023-08-28T21:25:09.313940+02:00 v-debian kernel: [ 40.588447] <TASK> 2023-08-28T21:25:09.313941+02:00 v-debian kernel: [ 40.588454] asm_common_interrupt+0x26/0x40 2023-08-28T21:25:09.313941+02:00 v-debian kernel: [ 40.588463] RIP: 0010:pv_native_safe_halt+0xf/0x20 2023-08-28T21:25:09.313941+02:00 v-debian kernel: [ 40.588472] Code: 0b 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d e5 f9 26 00 fb f4 <c3> cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 2023-08-28T21:25:09.313942+02:00 v-debian kernel: [ 40.588479] RSP: 0018:ffffaa10800bbe58 EFLAGS: 00000246 2023-08-28T21:25:09.313942+02:00 v-debian kernel: [ 40.588490] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 0000000000000020 2023-08-28T21:25:09.313943+02:00 v-debian kernel: [ 40.588497] RDX: ffff9ddc75e40000 RSI: ffff9ddb4123f800 RDI: ffff9ddb4123f864 2023-08-28T21:25:09.313943+02:00 v-debian kernel: [ 40.588529] RBP: ffff9ddb42386800 R08: ffff9ddc75e64560 R09: 0000000000000007 2023-08-28T21:25:09.313943+02:00 v-debian kernel: [ 40.588535] R10: ffff9ddc75e71d84 R11: 0000000000008601 R12: ffffffffa67a2e00 2023-08-28T21:25:09.313944+02:00 v-debian kernel: [ 40.588542] R13: ffff9ddb4123f864 R14: 0000000000000001 R15: 0000000000000000 2023-08-28T21:25:09.313944+02:00 v-debian kernel: [ 40.588555] acpi_safe_halt+0x15/0x30 2023-08-28T21:25:09.313944+02:00 v-debian kernel: [ 40.588565] acpi_idle_enter+0xa3/0x100 2023-08-28T21:25:09.313945+02:00 v-debian kernel: [ 40.588576] cpuidle_enter_state+0x81/0x440 2023-08-28T21:25:09.313945+02:00 v-debian kernel: [ 40.588586] cpuidle_enter+0x2d/0x40 2023-08-28T21:25:09.313945+02:00 v-debian kernel: [ 40.588598] do_idle+0x217/0x270 2023-08-28T21:25:09.313945+02:00 v-debian kernel: [ 40.588609] cpu_startup_entry+0x1d/0x20 2023-08-28T21:25:09.313946+02:00 v-debian kernel: [ 40.588617] start_secondary+0x134/0x160 2023-08-28T21:25:09.313946+02:00 v-debian kernel: [ 40.588628] secondary_startup_64_no_verify+0x10b/0x10b 2023-08-28T21:25:09.313946+02:00 v-debian kernel: [ 40.588643] </TASK> 2023-08-28T21:25:09.313947+02:00 v-debian kernel: [ 40.588649] ---[ end trace 0000000000000000 ]--- 2023-08-28T21:25:09.313947+02:00 v-debian kernel: [ 40.588671] IN= OUT=ens33 SRC=192.168.8.88 DST=192.168.8.81 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=12345 DPT=55436 WINDOW=0 RES=0x00 ACK RST URGP=0 ``` Also the issue only happens with packets generated by nftables or iptables reject. Sending the same RST segment using hping3, or simply trying to connect to a closed port, does not trigger the issue. Some additional observations in case this may help. It does not look like it is an ebpf issue, especially the relevant parts of __skb_flow_dissect haven't changed between reproducing and not reproducing versions. It appears that __skb_flow_dissect is called with net being NULL and the namespace cannot be deduced from the skb->dev or skb->sk. The issue does not happens when rejecting with ICMP unreachable (default) because contrary to nf_send_reset, nf_send_unreach does not call ip_route_me_harder. One change between both version is that 5.10 would pass nskb->sk to ip_route_me_harder (the sk_buff allocated inside nf_send_reset to hold the RST packet information), whereas in 6.1 and 6.4 we pass the original sk (in my understanding, from the packet that was rejected). 5.10.x, not reproducing: ``` void nf_send_reset(struct net *net, struct sk_buff *oldskb, int hook) { struct net_device *br_indev __maybe_unused; struct sk_buff *nskb; ... nskb = alloc_skb(sizeof(struct iphdr) + sizeof(struct tcphdr) + LL_MAX_HEADER, GFP_ATOMIC); ... if (ip_route_me_harder(net, nskb->sk, nskb, RTN_UNSPEC)) goto free_nskb; ``` 6.1.x, reproducing: ``` void nf_send_reset(struct net *net, struct sock *sk, struct sk_buff *oldskb, int hook) { struct net_device *br_indev __maybe_unused; struct sk_buff *nskb; ... nskb = alloc_skb(sizeof(struct iphdr) + sizeof(struct tcphdr) + LL_MAX_HEADER, GFP_ATOMIC); ... if (ip_route_me_harder(net, sk, nskb, RTN_UNSPEC)) goto free_nskb; ``` Is there a reason for this? It seems to me we are passing the wrong packet information, but i don't have the background on this to be sure. Best regards, Created attachment 304984 [details]
pass netns pointer to flow dissector to avoid warn splat
Could you please test the attached patch? Thanks.
Sure. I just tested and i'm not able to reproduce anymore on 6.1.47 with the patch applied. TCP reset is still sent reliably and i haven't seen any other issue. |