Bug 219766
| Summary: | Garbage Ethernet Frames | ||
|---|---|---|---|
| Product: | Networking | Reporter: | Florian Meister (fmei) |
| Component: | Other | Assignee: | Stephen Hemminger (stephen) |
| Status: | NEW --- | ||
| Severity: | normal | CC: | fmei |
| Priority: | P3 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | https://github.com/siderolabs/talos/issues/9837 | ||
| Kernel Version: | 6.6.60 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
kernel log of one affacted machine
kernel log of one affacted machine with absolute timestamps |
||
What is kernel log? Are you using any other out of tree drivers? I found nothing relevant in the kernel log, no network interfaces created at time of the garbage packets, no suspicious messages about drivers or network stack. Nope, no out of tree drivers. Of course I have _not_ saved the kernel logs :/ - I'll try to reproduce it - should reappear when upgrading the kernel version - and paste the logs here. Anything else which could help to trace it down? It will take a day or two 'til the problem will appear again. Created attachment 307621 [details]
kernel log of one affacted machine
Created attachment 307622 [details]
kernel log of one affacted machine with absolute timestamps
As announced, I've upgraded the kernel to 6.6.60 again, same problem appeared.
I've uploaded the kernel log of the machine which produced the following tcpdump output.
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp6s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
07:50:02.759638 20:00:40:11:b8:7f (oui Unknown) > 45:00:00:44:02:f6 (oui Unknown), ethertype Unknown (0xb23f), length 68:
0x0000: 0a53 5863 8a3e ca6c ca6c 0098 caea 0400 .SXc.>.l.l......
0x0010: 0000 87ff 1e4f 1700 0000 0000 0000 5890 .....O........X.
0x0020: aaa4 32f6 b8d8 c10c ca32 d4f7 722d 2acc ..2......2..r-*.
0x0030: e6b9 17e0 82c7 ......
07:50:02.759655 20:06:40:11:b8:79 (oui Unknown) > 45:00:00:44:02:f6 (oui Unknown), ethertype Unknown (0xb23f), length 68:
0x0000: 0a53 5863 8a3e f277 3061 a192 f2dc 1f6b .SXc.>.w0a.....k
0x0010: 411c 5af3 833b 8327 775f 4c09 5d9a 8c74 A.Z..;.'w_L.]..t
0x0020: 3b9e 1c45 e3c8 82ff cf39 e80a 931a 56a6 ;..E.....9....V.
0x0030: 4ddb fcb2 daf9 M.....
07:50:02.759657 20:0c:40:11:b8:73 (oui Unknown) > 45:00:00:44:02:f6 (oui Unknown), ethertype Unknown (0xb23f), length 68:
0x0000: 0a53 5863 8a3e 79fa bddf c248 c36a e59a .SXc.>y....H.j..
0x0010: 0f17 4753 d8b1 f9b5 7632 7e72 cd29 fe34 ..GS....v2~r.).4
0x0020: 1709 3d16 790d bde2 15e8 1ff2 1128 6179 ..=.y........(ay
0x0030: 299d 071a b41c ).....
07:50:02.759659 00:12:40:11:d8:95 (oui Unknown) > 45:00:00:1c:02:f6 (oui Unknown), ethertype Unknown (0xb23f), length 28:
0x0000: 0a53 5863 8a3e 0b7b 07b8 1dc3 4fb5 .SXc.>.{....O.
|
I am currently troubleshooting a very strange problem which appears when upgrading Kernel 6.6.58 to 6.6.60. The kernel version change is part of a change of talos linux (www.talos.dev) from 1.8.2 to 1.8.3. We are running this machines at hetzner - a company which is providing server hosting. they complain that we are using mac addresses which are not allowed (are not the mac addresses of the physical nic) In the investigation of the problem I did tcpdumps on the physical adapters and captured this suspicious ethernet frames. The frames do neither have a known ethertype, nor do they have a mac address of a known vendor or a known virtual mac address range. They seem garbage to me. Below an example. More can be found in the github issue. This frames are not emitted very often and the systems are operating normally. If I would not be informed by the hosting provider I would not have noticed it at all. I also tried to track it down to a specific hardware (r8169), but we have the same problem with e1000e. I checked the changelogs of the two kernel versions (6.6.59 & 6.6.60) and noticed there were some changes which could be the problem, but I simply do not have the experience for it. Can anybody check the changelog of the 2 versions and see if there is a change which might cause the problem? Can anybody give me a hint how to track it down further? tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on enp9s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 22:07:02.329668 20:00:40:11:18:fb > 45:00:00:44:f4:94, ethertype Unknown (0x58c6), length 68: 0x0000: 8dda 74ca f1ae ca6c ca6c 0098 969c 0400 ..t....l.l...... 0x0010: 0000 4730 3f18 6800 0000 0000 0000 9971 ..G0?.h........q 0x0020: c4c9 9055 a157 0a70 9ead bf83 38ca ab38 ...U.W.p....8..8 0x0030: 8add ab96 e052 .....R Issue with more information: https://github.com/siderolabs/talos/issues/9837