Bug 216327
| Summary: | 5.18.15 breaks ax88179_178a USB NIC: transmit queue 0 timed out | ||
|---|---|---|---|
| Product: | Drivers | Reporter: | Jay Mann (jmandawg) |
| Component: | Network | Assignee: | drivers_network (drivers_network) |
| Status: | CLOSED CODE_FIX | ||
| Severity: | normal | CC: | adam, claudio, dwrobel, jared, jayguerette, joalonsof, matteo.fortini, mihamina, msylwester, pupilla, rc.poison, tedtmsadler, vimregisters |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 5.18.15 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: |
log from .18.16
lsusb -vvv startech-dual-ethernet.jpg lsusb-vv-jaymann log from 5.18.14 |
||
|
Description
Jay Mann
2022-08-05 01:08:19 UTC
After reverting that commit do you see any packet loss with the ping command?
(if connect to USB2)
ping -s 462 -4 {destination}
(if connected to USB3)
ping -s 974 -4 {destination}
I'm not seeing anything, but my setup is kinda strange because i'm passing the adapter into a VM as a macvtap passthrough: root@OpenWrt:~# ping -s 974 -4 192.168.16.9 PING 192.168.16.9 (192.168.16.9): 974 data bytes 982 bytes from 192.168.16.9: seq=0 ttl=64 time=0.526 ms 982 bytes from 192.168.16.9: seq=1 ttl=64 time=0.711 ms 982 bytes from 192.168.16.9: seq=2 ttl=64 time=0.665 ms 982 bytes from 192.168.16.9: seq=3 ttl=64 time=0.775 ms 982 bytes from 192.168.16.9: seq=4 ttl=64 time=0.821 ms 982 bytes from 192.168.16.9: seq=5 ttl=64 time=0.642 ms 982 bytes from 192.168.16.9: seq=6 ttl=64 time=0.592 ms 982 bytes from 192.168.16.9: seq=7 ttl=64 time=0.649 ms 982 bytes from 192.168.16.9: seq=8 ttl=64 time=0.600 ms 982 bytes from 192.168.16.9: seq=9 ttl=64 time=0.847 ms 982 bytes from 192.168.16.9: seq=10 ttl=64 time=0.561 ms 982 bytes from 192.168.16.9: seq=11 ttl=64 time=0.612 ms 982 bytes from 192.168.16.9: seq=12 ttl=64 time=0.604 ms 982 bytes from 192.168.16.9: seq=13 ttl=64 time=0.588 ms 982 bytes from 192.168.16.9: seq=14 ttl=64 time=0.653 ms 982 bytes from 192.168.16.9: seq=15 ttl=64 time=0.502 ms 982 bytes from 192.168.16.9: seq=16 ttl=64 time=0.695 ms ^C --- 192.168.16.9 ping statistics --- 17 packets transmitted, 17 packets received, 0% packet loss round-trip min/avg/max = 0.502/0.649/0.847 ms Created attachment 301523 [details]
log from .18.16
Created attachment 301524 [details]
lsusb -vvv
I do not see any packet loss on 5.18.16 with commit f8bfce2177cdd64357dd57b8f43154f236d25f51 reverted: $ ping -s 974 -4 192.168.0.1 ... --- 192.168.0.1 ping statistics --- 382 packets transmitted, 382 received, 0% packet loss, time 381523ms rtt min/avg/max/mdev = 1.574/1.957/2.567/0.161 ms Created attachment 301525 [details]
startech-dual-ethernet.jpg
Interesting Andreas, i noticed you have 2 ax88179 adapters just like me, are you using the same dual port ethernet adapter as me?
Mine is a "StarTech.com USB 3.0 to Dual Port Gigabit Ethernet Adapter w/USB Port - 10/100/100 - USB Gigabit LAN Network NIC Adapter (USB32000SPT)" (pic attached)
Created attachment 301526 [details]
lsusb-vv-jaymann
No, using 2 Amazon Basics adapters. Since I experienced more network related issues on 5.18.16 I've downgraded to 5.18.14 and unfortunately, I've encountered the issue there too. I can't quite explain why I didn't encounter this issue before. Created attachment 301538 [details]
log from 5.18.14
Try going back to 5.18.9 which also removes this commit: commit 1ba8568436046ad2dd9f8d6fcc2a192c611d6f26 Author: Jose Alonso <joalonsof@gmail.com> Date: Tue Jun 28 12:13:02 2022 -0300 net: usb: ax88179_178a: Fix packet receiving commit f8ebb3ac881b17712e1d5967c97ab1806b16d3d6 upstream. This patch corrects packet receiving in ax88179_rx_fixup. This patch was made using only the dumpfile information and tested with only one device: 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet Really strange we are the only 2 people experiencing this issue. My system was running very stable before these last few commits. (In reply to Jay Mann from comment #11) > Try going back to 5.18.9 Currently running an experiment, will try after that. I had both adapters on a USB hub. After the issue first occurred 5.18.15, I thought that maybe the USB hub is causing issues and moved the adapters around. When downgrading to 5.18.14 again, I noticed only the adapter connected directly without the hub was locking up. Also, only the adapter directly connected had the log entries: ax88179_178a 4-4:1.0 netUSBInet0: Failed to read reg index 0x0040: -32 I really don't know if this is useful information at all, but after 24h+ the connection didn't drop on 5.18.14 with both adapters on a hub and also no 'Failed to read reg index' message. > Really strange we are the only 2 people experiencing this issue. I don't know how the behavior is at your end, but most of the time I have to wait several hours after the link going down for the error message to pop up. I wouldn't be surprised if few catch that. I've ordered 4 different USB NICs for testing and 3 of them are AX88179 so that chipset seems to be prevalent ATM. I could reproduce the issue with all of the AX88179 ones so probably firmware version isn't the issue? I could use some help in order to provide meaningful debug information for this. I also think using a usb 2.0 port vs 3.0 port might come into play. As when i used a usb 2.0 port it seemed more stable than using a usb 3.0 port. But going back to 5.18.9 has been rock solid for the past day. I see one commit was reverted: https://lore.kernel.org/netdev/20220808113504.128907-1-joalonsof@gmail.com/T/ but i'm still having issues with the adapter randomly stop working with this commit as well, most likely both need reverted to run stable. commit 1ba8568436046ad2dd9f8d6fcc2a192c611d6f26 Author: Jose Alonso <joalonsof@gmail.com> Date: Tue Jun 28 12:13:02 2022 -0300 net: usb: ax88179_178a: Fix packet receiving commit f8ebb3ac881b17712e1d5967c97ab1806b16d3d6 upstream. This patch corrects packet receiving in ax88179_rx_fixup. This patch was made using only the dumpfile information and tested with only one device: 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet Jay Mann,
I need a tcpdump with two messages to/from the usb device.
First do a lsusb to see the Bus and Device.
lsusb
Bus 002 Device 008: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
002 -> usbmon2
008 -> device 2:8:end_point
end_point=2 -> IN end_point=3 -> OUT
sudo tcpdump -i usbmon2 -X >dumpfile.txt
(The dump can be large depending of activity on the bus)
(do some activity with the device)
(^C to kill tcpdump)
call editor and extract from dumfile.txt two complete messages:
One from the device: 2:8:2 (Bus:device:IN)
10:24:14.614144 USB BULK COMPLETE from 2:8:2
0x0000: eeee f8e4 3b09 8700 94de 80f3 daaa 0800 ....;...........
...
0x0830: 2008 3402 0000 0080 0000 0000 0400 2808 ..4...........(.
One to the device: 2:8:3 (Bus:device:OUT)
10:24:14.613772 USB BULK SUBMIT to 2:8:3
0x0000: 3202 0000 0000 0000 94de 80f3 daaa f8e4 2...............
...
0x0230: c6c7 c8c9 cacb cccd cecf ..........
Hopefully this is what your asking for.
Current kernel is running without
f8bfce2177cdd
But does include:
1ba85684360
Bus 002 Device 007: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
20:25:53.720255 USB BULK COMPLETE from 2:7:2¬
»·0x0000: eeee 000a cd34 4e1e 5448 10e3 3c86 0800 .....4N.TH..<...¬
»·0x0010: 4500 0179 30ae 4000 3511 a279 1752 8949 E..y0.@.5..y.R.I¬
»·0x0020: c0a8 1009 01bb a42a 0165 0e18 0400 0000 .......*.e......¬
»·0x0030: b1ea f1c0 1d00 0000 0000 0000 2f40 2eef ............/@..¬
»·0x0040: 9c5a 690c 9be5 3436 56a4 c80d 5135 cd9f .Zi...46V...Q5..¬
»·0x0050: d72f bf2e c53b a541 37f7 e2ec 53a4 84dc ./...;.A7...S...¬
»·0x0060: a6a0 da35 652c 8b43 17ec 7bc0 3170 154e ...5e,.C..{.1p.N¬
»·0x0070: 589f e35c 9e8d e414 8456 c9ab 2bfd bdda X..\.....V..+...¬
»·0x0080: e3d2 1db5 2183 2bc9 3871 df69 704e 2e99 ....!.+.8q.ipN..¬
»·0x0090: cefe 72e2 8768 1558 ba44 9884 86e7 d905 ..r..h.X.D......¬
»·0x00a0: fdaa beb1 e85e ecc2 944b 27dd b5c9 19b5 .....^...K'.....¬
»·0x00b0: d97d 73d4 0834 0ae6 12b3 bec1 9fcc c333 .}s..4.........3¬
»·0x00c0: 5b57 8a4d 6881 0285 048a c1d7 0680 8bfe [W.Mh...........¬
»·0x00d0: 4e8b 0884 b6e9 ae38 813d a14e 08c0 c236 N......8.=.N...6¬
»·0x00e0: e443 d2cd 40c0 dcfb 22b0 36e2 da57 7eb3 .C..@...".6..W~.¬
»·0x00f0: dff4 f382 5fb0 64d7 ac68 b762 5ef2 3638 ...._.d..h.b^.68¬
»·0x0100: fce9 5843 7b97 5f9c 7d38 d29e 9c71 fca7 ..XC{._.}8...q..¬
»·0x0110: afdb bb5e 563d 4d33 bdd9 04f7 90c2 3908 ...^V=M3......9.¬
»·0x0120: 06c3 c44b f4d8 eb45 9e95 9b3c 8b11 018b ...K...E...<....¬
»·0x0130: c418 d236 3d19 286e d35d c59f b695 540c ...6=.(n.]....T.¬
»·0x0140: 7d7e a699 21ff 8d95 d6d0 100f 307b e2aa }~..!.......0{..¬
»·0x0150: 565c d804 dbdf 623f cb60 d9d9 3bc7 28ad V\....b?.`..;.(.¬
»·0x0160: 7aad 5b51 6c1a 369f 8b2d 4485 2803 f3c4 z.[Ql.6..-D.(...¬
»·0x0170: 7e57 be3e c410 1e0d 1d39 b2f1 afda 829d ~W.>.....9......¬
»·0x0180: fc72 b153 4a65 f0ab 3f7f 5b69 5300 f0ab .r.SJe..?.[iS...¬
»·0x0190: 2408 8901 0100 9001 $.......¬
20:25:53.720277 USB BULK SUBMIT to 2:7:2¬
20:25:53.459393 USB BULK SUBMIT to 2:7:3¬
»·0x0000: ca00 0000 0000 0000 5448 10e3 3c86 000a ........TH..<...¬
»·0x0010: cd34 4e1e 0800 4500 00bc b366 0000 3f11 .4N...E....f..?.¬
»·0x0020: 567e c0a8 1009 1752 8949 a42a 01bb 00a8 V~.....R.I.*....¬
»·0x0030: 7206 0400 0000 6974 0600 2600 0000 0000 r.....it..&.....¬
»·0x0040: 0000 1078 ba6c ce45 e972 203c 327b efe9 ...x.l.E.r.<2{..¬
»·0x0050: 5697 1118 803c 7139 f03d 93bd 10d3 1252 V....<q9.=.....R¬
»·0x0060: 636e 822b f5ef c176 58e7 4e63 e666 e5b3 cn.+...vX.Nc.f..¬
»·0x0070: 3338 dd5a a094 999b ced8 7fe2 63c4 7bf0 38.Z........c.{.¬
»·0x0080: c97e 8b15 4ee8 9a16 2419 9464 8e73 5070 .~..N...$..d.sPp¬
»·0x0090: 80b7 9391 32b7 b2a2 c2a2 4f08 c348 0ad9 ....2.....O..H..¬
»·0x00a0: 9423 678c ea80 d46d 8501 f06f eee5 c77a .#g....m...o...z¬
»·0x00b0: 30b8 c4c6 a1a9 ebd2 7a04 58be b672 6fb4 0.......z.X..ro.¬
»·0x00c0: 07e5 1f5d 25b9 e163 0055 6779 48a8 cd40 ...]%..c.UgyH..@¬
»·0x00d0: 63da c.¬
20:25:53.459438 USB BULK COMPLETE from 2:7:3¬
Just a quick update, it's been running fine for a couple days with only the FLAG_SEND_ZLP 36a15e1cb13 commit reverted. Thanks. Thanks for dumps and logs.
The chip ax88179 have two versions currently in use (site: asix.com.tw):
AX88179 USB 3.0 to 10/100/1000M Gigabit Ethernet Controller [NOT FOR NEW DESIGN!!]
AX88179A USB 3.2 Gen 1 to Gigabit Ethernet Controller
One way to identify the version is:
sudo lsusb -v -d 0b95:1790 | grep iProduct
iProduct 2 AX88179
or:
iProduct 2 AX88179A
The kernel log also prints:
grep "Product: AX" klog-5.18.14.txt
Aug 08 14:42:26 firestarter kernel: usb 4-4: Product: AX88179A
Aug 08 14:42:27 firestarter kernel: usb 4-3.4: Product: AX88179
Andreas have the two versions.
Now for the kernel versions 5.18.x:
<= 5.18.14 only the AX88179 works
>= 5.18.15 only the AX88179A works
The commit with FLAG_SEND_ZLP is reverting and
a new patch is necessary for support AX88179A perhaps
using iProduct=="AX88179A".
Here's a reliable way to reproduce the issue without having to wait for it to happen randomly. 1. Open up a Google Meet call https://meet.google.com/ 2. Share video (either through webcam or screenshare) 3. Wait. The internet should cut out within a minute (In reply to Jose Alonso from comment #18) > The commit with FLAG_SEND_ZLP is reverting and > a new patch is necessary for support AX88179A perhaps > using iProduct=="AX88179A". I built the kernel-5.18.17-200.fc36.1.src.rpm in copr[1] with 36a15e1cb134c0395261ba1940762703f778438c [2] being reverted and it's working fine for me. [1] https://copr.fedorainfracloud.org/coprs/dwrobel/kernel-kbug-216327/ [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=36a15e1cb134c0395261ba1940762703f778438c Fixed in 5.19.2 Runing 5.19.7 but still having the issue on USB3 port. On USB < 3 it is stable. (In reply to rakotomandimby from comment #22) > Runing 5.19.7 but still having the issue on USB3 port. > On USB < 3 it is stable. Same here....Also running it on USB 3 but on newer kernel (Fedora 36 kernel: 5.19.9-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC): [104033.973649] ------------[ cut here ]------------ [104033.973665] NETDEV WATCHDOG: enp58s0u1 (ax88179_178a): transmit queue 0 timed out [104033.973710] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x1f3/0x200 [104033.973730] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_compat nf_nat_tftp nf_conntrack_tftp bridge stp llc uinput rfcomm snd_seq_dummy snd_hrtimer bonding tls nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set overlay nf_tables nfnetlink qrtr bnep btusb uvcvideo btrtl btbcm btintel videobuf2_vmalloc btmtk videobuf2_memops videobuf2_v4l2 videobuf2_common bluetooth videodev mc sunrpc binfmt_misc snd_sof_pci_intel_cnl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda vfat snd_sof_pci snd_sof_xtensa_dsp fat snd_sof snd_sof_utils soundwire_bus snd_hda_codec_hdmi snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_sst_ipc snd_soc_sst_dsp snd_ctl_led snd_soc_acpi_intel_match snd_soc_acpi iwlmvm snd_soc_core [104033.974089] snd_hda_codec_realtek snd_compress intel_tcc_cooling ac97_bus x86_pkg_temp_thermal snd_pcm_dmaengine snd_hda_codec_generic intel_powerclamp snd_hda_intel coretemp snd_intel_dspcfg mac80211 snd_intel_sdw_acpi kvm_intel iTCO_wdt libarc4 mei_hdcp mei_pxp ee1004 intel_pmc_bxt intel_rapl_msr iTCO_vendor_support kvm snd_hda_codec dell_laptop iwlwifi irqbypass snd_hda_core rapl intel_cstate snd_hwdep dell_wmi intel_uncore snd_seq processor_thermal_device_pci_legacy ledtrig_audio snd_seq_device pcspkr dell_smbios processor_thermal_device cfg80211 snd_pcm dcdbas processor_thermal_rfim snd_timer mei_me processor_thermal_mbox i2c_i801 processor_thermal_rapl dell_wmi_descriptor intel_wmi_thunderbolt wmi_bmof mxm_wmi snd i2c_smbus intel_rapl_common mei joydev idma64 soundcore rfkill intel_pch_thermal intel_soc_dts_iosf int3403_thermal int340x_thermal_zone intel_hid int3400_thermal acpi_thermal_rel sparse_keymap acpi_pad zram dm_thin_pool dm_persistent_data dm_bio_prison [104033.974421] hid_logitech_hidpp typec_displayport cdc_mbim cdc_wdm cdc_ncm hid_logitech_dj cdc_ether i915 drm_buddy drm_display_helper alx cec nvme hid_multitouch ucsi_acpi crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel nvme_core typec_ucsi serio_raw mdio ttm typec i2c_hid_acpi i2c_hid wmi video pinctrl_cannonlake ax88179_178a usbnet mii ip6_tables ip_tables fuse [104033.974581] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [104033.974957] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.19.9-200.fc36.x86_64 #1 [104033.974979] Hardware name: Dell Inc. G7 7588/0FDMYT, BIOS 1.20.0 03/18/2022 [104033.974985] RIP: 0010:dev_watchdog+0x1f3/0x200 [104033.974999] Code: 00 e9 40 ff ff ff 48 89 df c6 05 70 26 54 01 01 e8 92 9e f9 ff 44 89 e9 48 89 de 48 c7 c7 68 d2 72 98 48 89 c2 e8 a2 05 19 00 <0f> 0b e9 22 ff ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 55 53 48 89 [104033.975007] RSP: 0018:ffffaf12c01c8eb0 EFLAGS: 00010282 [104033.975018] RAX: 0000000000000045 RBX: ffff93c244831000 RCX: 0000000000000000 [104033.975026] RDX: 0000000000000103 RSI: 00000000000000f6 RDI: 00000000ffffffff [104033.975033] RBP: ffff93c2448314c8 R08: 0000000000000000 R09: ffffaf12c01c8d68 [104033.975040] R10: 0000000000000003 R11: ffffffff98f45d88 R12: ffff93c24483141c [104033.975047] R13: 0000000000000000 R14: ffffffff97b7a2a0 R15: ffff93c2448314c8 [104033.975053] FS: 0000000000000000(0000) GS:ffff93c5af480000(0000) knlGS:0000000000000000 [104033.975061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [104033.975068] CR2: 00007f02f94f8020 CR3: 0000000453e10001 CR4: 00000000003706e0 [104033.975076] Call Trace: [104033.975083] <IRQ> [104033.975093] ? pfifo_fast_reset+0x130/0x130 [104033.975104] call_timer_fn+0x24/0x130 [104033.975118] __run_timers.part.0+0x1b3/0x240 [104033.975130] ? _raw_spin_lock_irq+0x19/0x40 [104033.975140] ? __hrtimer_run_queues+0x129/0x2a0 [104033.975152] ? sched_clock_cpu+0xb/0xc0 [104033.975166] run_timer_softirq+0x31/0x70 [104033.975176] __do_softirq+0xf8/0x30f [104033.975191] __irq_exit_rcu+0xc5/0x140 [104033.975204] sysvec_apic_timer_interrupt+0x9e/0xc0 [104033.975218] </IRQ> [104033.975223] <TASK> [104033.975229] asm_sysvec_apic_timer_interrupt+0x16/0x20 [104033.975239] RIP: 0010:cpuidle_enter_state+0xd8/0x380 [104033.975250] Code: 00 00 31 ff e8 39 18 6a ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 92 02 00 00 31 ff e8 fe 6d 70 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 ce 48 8d 04 49 48 8d 04 81 49 8d [104033.975258] RSP: 0018:ffffaf12c011fea0 EFLAGS: 00000246 [104033.975269] RAX: ffff93c5af480000 RBX: 0000000000000007 RCX: 0000000000000000 [104033.975276] RDX: 00005e9e4c0afb5f RSI: ffffffff986811a0 RDI: ffffffff98641f45 [104033.975282] RBP: ffffcf12bfa80270 R08: 0000000000000002 R09: 0000000039f89ebb [104033.975288] R10: 0000000000000018 R11: 0000000000003904 R12: ffffffff9902c9e0 [104033.975295] R13: 00005e9e4c0afb5f R14: 0000000000000007 R15: 0000000000000000 [104033.975314] cpuidle_enter+0x29/0x40 [104033.975323] do_idle+0x1b6/0x230 [104033.975337] cpu_startup_entry+0x19/0x20 [104033.975348] start_secondary+0x102/0x120 [104033.975360] secondary_startup_64_no_verify+0xe5/0xeb [104033.975380] </TASK> [104033.975385] ---[ end trace 0000000000000000 ]--- Still present on Linux 5.19.6 (Debian) ``` [64488.716565] ------------[ cut here ]------------ [64488.716570] NETDEV WATCHDOG: eth0 (ax88179_178a): transmit queue 0 timed out [64488.716582] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x1f9/0x200 [64488.716590] Modules linked in: snd_usb_audio snd_usbmidi_lib uhid exfat tls nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter tun nvme_fabrics ctr ccm snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device binder_linux xt_CHECKSUM iptable_mangle xt_MASQUERADE xt_comment iptable_nat bridge stp llc iptable_filter typec_displayport snd_hda_codec_hdmi nfnetlink_queue nft_chain_nat xt_nat nf_nat xt_NFQUEUE xt_mark xt_connmark ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog xt_multiport nft_limit xt_limit rfcomm qrtr xt_addrtype xt_tcpudp cmac algif_hash algif_skcipher af_alg xt_conntrack bnep overlay nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables binfmt_misc nfnetlink nls_ascii nls_cp437 snd_sof_pci_intel_icl vfat fat snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp iwlmvm snd_sof snd_sof_utils [64488.716659] btusb snd_soc_hdac_hda ext4 snd_hda_ext_core btrtl snd_soc_acpi_intel_match btbcm mbcache snd_soc_acpi x86_pkg_temp_thermal intel_powerclamp jbd2 btintel mac80211 btmtk coretemp snd_soc_core snd_ctl_led snd_compress bluetooth mei_hdcp intel_rapl_msr libarc4 snd_hda_codec_realtek kvm_intel soundwire_bus snd_hda_codec_generic iwlwifi jitterentropy_rng snd_hda_intel dell_laptop ledtrig_audio kvm snd_intel_dspcfg snd_intel_sdw_acpi irqbypass sha512_ssse3 sha512_generic rapl cfg80211 dell_smm_hwmon snd_hda_codec dell_wmi drbg intel_cstate ansi_cprng dell_smbios ecdh_generic intel_uncore serio_raw efi_pstore pcspkr dcdbas snd_hda_core dell_wmi_sysman firmware_attributes_class snd_hwdep dell_wmi_descriptor snd_pcm_oss wmi_bmof intel_wmi_thunderbolt uvcvideo snd_mixer_oss iTCO_wdt snd_pcm videobuf2_vmalloc intel_pmc_bxt videobuf2_memops iTCO_vendor_support videobuf2_v4l2 watchdog hid_sensor_als processor_thermal_device_pci_legacy videobuf2_common snd_timer processor_thermal_device [64488.716732] hid_sensor_trigger videodev snd processor_thermal_rfim hid_sensor_iio_common ucsi_acpi soundcore processor_thermal_mbox industrialio_triggered_buffer mei_me typec_ucsi kfifo_buf processor_thermal_rapl intel_rapl_common ecc roles int3403_thermal mei hid_multitouch mc joydev crc16 industrialio rfkill int340x_thermal_zone typec intel_soc_dts_iosf evdev squashfs sg loop int3400_thermal intel_pmc_core intel_hid acpi_pad acpi_thermal_rel ac acpi_tad sparse_keymap msr parport_pc ppdev lp parport fuse configfs efivarfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq zstd_compress libcrc32c crc32c_generic sd_mod dm_crypt dm_mod uas usb_storage scsi_mod scsi_common ax88179_178a usbnet mii mmc_block hid_sensor_hub intel_ishtp_hid wacom crc32_pclmul crc32c_intel usbhid hid_generic rtsx_pci_sdmmc xhci_pci ghash_clmulni_intel i915 mmc_core xhci_hcd nvme drm_buddy i2c_algo_bit drm_display_helper nvme_core aesni_intel cec rc_core crypto_simd t10_pi ttm cryptd i2c_hid_acpi [64488.716794] usbcore crc64_rocksoft_generic psmouse drm_kms_helper crc64_rocksoft crc_t10dif i2c_hid crct10dif_generic i2c_i801 intel_lpss_pci intel_ish_ipc crct10dif_pclmul intel_lpss crc64 drm thunderbolt rtsx_pci crct10dif_common i2c_smbus intel_ishtp idma64 usb_common hid wmi battery button video [64488.716825] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G U 5.19.0-1-amd64 #1 Debian 5.19.6-1 [64488.716829] Hardware name: Dell Inc. XPS 13 9300/077Y9N, BIOS 1.14.0 07/12/2022 [64488.716830] RIP: 0010:dev_watchdog+0x1f9/0x200 [64488.716835] Code: 00 e9 40 ff ff ff 48 89 ef c6 05 26 e7 1f 01 01 e8 7c 91 f9 ff 44 89 e9 48 89 ee 48 c7 c7 b0 81 bd b2 48 89 c2 e8 8b d7 17 00 <0f> 0b e9 22 ff ff ff 0f 1f 44 00 00 55 53 48 89 fb 48 8b 6f 18 0f [64488.716838] RSP: 0018:ffffb215001d4ea8 EFLAGS: 00010286 [64488.716840] RAX: 0000000000000000 RBX: ffff912402ccf4c8 RCX: 0000000000000000 [64488.716842] RDX: 0000000000000103 RSI: ffffffffb2b6ed69 RDI: 00000000ffffffff [64488.716843] RBP: ffff912402ccf000 R08: 0000000000000000 R09: 00000000fffff0bf [64488.716844] R10: ffffb215001d4d48 R11: ffffffffb32bb108 R12: ffff912402ccf41c [64488.716845] R13: 0000000000000000 R14: ffffffffb21ef9e0 R15: ffff91277f8e0ac0 [64488.716846] FS: 0000000000000000(0000) GS:ffff91277f8c0000(0000) knlGS:0000000000000000 [64488.716847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [64488.716848] CR2: 00007fe4a7ffde20 CR3: 0000000122c6e002 CR4: 0000000000770ee0 [64488.716850] PKRU: 55555554 [64488.716851] Call Trace: [64488.716852] <IRQ> [64488.716855] ? pfifo_fast_reset+0x130/0x130 [64488.716858] call_timer_fn+0x24/0x130 [64488.716861] __run_timers.part.0+0x1d2/0x270 [64488.716863] ? enqueue_hrtimer+0x2f/0x80 [64488.716865] ? __hrtimer_run_queues+0x159/0x2c0 [64488.716867] run_timer_softirq+0x31/0x70 [64488.716868] __do_softirq+0xeb/0x2ec [64488.716872] __irq_exit_rcu+0xc9/0x140 [64488.716875] sysvec_apic_timer_interrupt+0xa2/0xd0 [64488.716890] </IRQ> [64488.716890] <TASK> [64488.716891] asm_sysvec_apic_timer_interrupt+0x16/0x20 [64488.716894] RIP: 0010:cpuidle_enter_state+0xd8/0x370 [64488.716896] Code: 00 00 31 ff e8 f9 f3 9a ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 7a 02 00 00 31 ff e8 2e 81 a1 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 15 01 00 00 49 63 d6 4c 2b 2c 24 48 8d 04 52 48 8d [64488.716897] RSP: 0018:ffffb21500127e90 EFLAGS: 00000246 [64488.716899] RAX: ffff91277f8c0000 RBX: 0000000000000001 RCX: 000000000000001f [64488.716900] RDX: 0000000000000000 RSI: ffffffffb2b6ed69 RDI: ffffffffb2b49940 [64488.716900] RBP: ffff91277f8fc200 R08: 00003aa6f30aead3 R09: 0000000000000006 [64488.716901] R10: 000000000000002e R11: 0000000000000016 R12: ffffffffb338da60 [64488.716902] R13: 00003aa6f30aead3 R14: 0000000000000001 R15: 0000000000000000 [64488.716905] cpuidle_enter+0x29/0x40 [64488.716907] do_idle+0x1db/0x260 [64488.716909] cpu_startup_entry+0x19/0x20 [64488.716911] start_secondary+0x112/0x130 [64488.716914] secondary_startup_64_no_verify+0xd3/0xdb [64488.716918] </TASK> [64488.716919] ---[ end trace 0000000000000000 ]--- ``` Hello everyone, happens with 6.1-rc2 https://bugzilla.kernel.org/show_bug.cgi?id=216631 Marco Hi all, Also been getting this for a while (6 months+) at one of our sites with Linux running on a mini PC (ThinkCentre M93p) as the router. Kernel version 6.2.10. Unfortunately while the machine's been doing a great job over the years as a router it only has a single 1GBPS Ethernet port so we've been using (you guessed it) a USB Ethernet adapter [0b95:1790] "ASIX Electronics Corp. AX88179 Gigabit Ethernet" for the LAN side. Every so often, maybe once a week, or once in a month - Every device on the LAN will lose connectivity to this router via its AX88179 USB3 Ethernet dongle. The link status on the router itself for this USB3 interface is UP and has its IP. But cannot reach anybody on its network suffix. During this time, everyone on the VLAN for this network cannot reach the router's IP either, despite the IP being set and everything being in motion. For some interesting detail - If I tcpdump the USB3 Ethernet interface on the router while it's having this moment - I can see the entire LAN sending out ARPs for the gateway IP - the gateway simply isn't (or "cannot") respond to those ARP requests. This lasts at least 10 minutes at a time, I often toggle the interface down then up and eventually the bridge it's part of will re-enter a FORWARDING state. I understand this is closed but wanted to track this experience. Won't flood us with more `(ax88179_178a): transmit queue 0 timed out` logs and have scheduled an upgrade for this host. Hi all, Also been getting this for a while (6 months+) at one of our sites with Linux running on a mini PC (ThinkCentre M93p) as the router. Kernel version 6.2.10. Unfortunately while the machine's been doing a great job over the years as a router it only has a single 1GBPS Ethernet port so we've been using (you guessed it) a USB Ethernet adapter [0b95:1790] "ASIX Electronics Corp. AX88179 Gigabit Ethernet" for the LAN side. Every so often, maybe once a week, or once in a month - Every device on the LAN will lose connectivity to this router via its AX88179 USB3 Ethernet dongle. The link status on the router itself for this USB3 interface is UP and has its IP. But cannot reach anybody on its network suffix. During this time, everyone on the VLAN for this network cannot reach the router's IP either, despite the IP being set and everything being in motion. For some interesting detail - If I tcpdump the USB3 Ethernet interface on the router while it's having this moment - I can see the entire LAN sending out ARPs for the gateway IP - the gateway simply isn't (or "cannot") respond to those ARP requests during this panic. This lasts at least 10 minutes at a time, I often toggle the interface down then up and eventually the bridge it's part of will re-enter a FORWARDING state. I understand this is closed but wanted to track this experience. Won't flood us with more `(ax88179_178a): transmit queue 0 timed out` logs and have scheduled an upgrade for this host. (Apologies for double and now triple posting just now, didn't expect the bugzilla reauth to also post the comment afterwards.) Still occurring on 6.2.13. Happened at 3:40AM, repaired now at 6:20PM. The gateway could only send packets on enp0s20u3 again after the interface were downed and upped again (ip link set enp0s20u3 down ; ip link set enp0s20u3 up), then packet transmission resumed to that network. ``` May 24 03:42:16 gateway kernel: ------------[ cut here ]------------ May 24 03:42:16 gateway kernel: NETDEV WATCHDOG: enp0s20u3 (ax88179_178a): transmit queue 0 timed out May 24 03:42:16 gateway kernel: WARNING: CPU: 7 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x20b/0x220 May 24 03:42:16 gateway kernel: Modules linked in: ipt_REJECT nf_reject_ipv4 tun tls 8021q garp mrp bridge stp llc rfkill iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_generic xt_multiport xt_pkttype xt_comment xt_tcpudp iptable_filter hwmon_vid snd_hda_codec_realtek intel_rapl_msr snd_hda_codec_generic intel_rapl_common snd_hda_codec_hdmi ledtrig_audio x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_intel crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni snd_intel_dspcfg polyval_generic snd_intel_sdw_acpi gf128mul spi_nor ghash_clmulni_intel snd_hda_codec sha512_ssse3 snd_hda_core aesni_intel crypto_simd mtd cryptd snd_hwdep ax88179_178a spi_intel_platform snd_pcm iTCO_wdt mei_pxp mei_wdt rapl mei_hdcp usbnet intel_pmc_bxt vfat intel_cstate snd_timer fat iTCO_vendor_support mei_me spi_intel snd think_lmi i2c_i801 intel_uncore mii wmi_bmof e1000e firmware_attributes_class pcspkr lpc_ich soundcore i2c_smbus mei May 24 03:42:16 gateway kernel: mac_hid dm_multipath crypto_user dm_mod fuse loop bpf_preload ip_tables x_tables zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) i915 icp(POE) zcommon(POE) znvpair(POE) spl(OE) drm_buddy intel_gtt drm_display_helper cec xhci_pci video ttm xhci_pci_renesas wmi May 24 03:42:16 gateway kernel: CPU: 7 PID: 0 Comm: swapper/7 Tainted: P OE 6.2.13-arch1-1 #1 fa8d27bf98b6325495c4c46c21163862ead484d0 May 24 03:42:16 gateway kernel: Hardware name: LENOVO 10AAS3U300/SHARKBAY, BIOS FBKTA7AUS 03/11/2015 May 24 03:42:16 gateway kernel: RIP: 0010:dev_watchdog+0x20b/0x220 May 24 03:42:16 gateway kernel: Code: ff e9 40 ff ff ff 48 89 df c6 05 a3 35 46 01 01 e8 da ce f8 ff 44 89 e9 48 89 de 48 c7 c7 58 c1 aa 85 48 89 c2 e8 55 1c 55 ff <0f> 0b e9 22 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 May 24 03:42:16 gateway kernel: RSP: 0018:ffff97b3002b8e88 EFLAGS: 00010286 May 24 03:42:16 gateway kernel: RAX: 0000000000000000 RBX: ffff8936021e3000 RCX: 0000000000000000 May 24 03:42:16 gateway kernel: RDX: ffff893716bee280 RSI: ffff893716be1640 RDI: 0000000000000300 May 24 03:42:16 gateway kernel: RBP: ffff8936021e34c8 R08: 0000000000000000 R09: ffff97b3002b8d18 May 24 03:42:16 gateway kernel: R10: 0000000000000003 R11: ffffffff862b51b8 R12: ffff8936021e341c May 24 03:42:16 gateway kernel: R13: 0000000000000000 R14: ffff97b3002b8f00 R15: ffff893716be1f00 May 24 03:42:16 gateway kernel: FS: 0000000000000000(0000) GS:ffff893716bc0000(0000) knlGS:0000000000000000 May 24 03:42:16 gateway kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 24 03:42:16 gateway kernel: CR2: 00007fcb3e56eb3c CR3: 000000017d210002 CR4: 00000000001706e0 May 24 03:42:16 gateway kernel: Call Trace: May 24 03:42:16 gateway kernel: <IRQ> May 24 03:42:16 gateway kernel: ? __pfx_dev_watchdog+0x10/0x10 May 24 03:42:16 gateway kernel: call_timer_fn+0x27/0x130 May 24 03:42:16 gateway kernel: ? __pfx_dev_watchdog+0x10/0x10 May 24 03:42:16 gateway kernel: __run_timers+0x222/0x2c0 May 24 03:42:16 gateway kernel: run_timer_softirq+0x1d/0x40 May 24 03:42:16 gateway kernel: __do_softirq+0xd4/0x2c8 May 24 03:42:16 gateway kernel: ? sched_clock_cpu+0xd/0xb0 May 24 03:42:16 gateway kernel: __irq_exit_rcu+0xb7/0xe0 May 24 03:42:16 gateway kernel: sysvec_apic_timer_interrupt+0x72/0x90 May 24 03:42:16 gateway kernel: </IRQ> May 24 03:42:16 gateway kernel: <TASK> May 24 03:42:16 gateway kernel: asm_sysvec_apic_timer_interrupt+0x1a/0x20 May 24 03:42:16 gateway kernel: RIP: 0010:cpuidle_enter_state+0xe2/0x420 May 24 03:42:16 gateway kernel: Code: 00 00 31 ff e8 bf 06 6a ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 25 03 00 00 31 ff e8 64 0b 72 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d May 24 03:42:16 gateway kernel: RSP: 0018:ffff97b30014be90 EFLAGS: 00000246 May 24 03:42:16 gateway kernel: RAX: ffff893716bf3100 RBX: ffff893716bfe300 RCX: 0000000000000000 May 24 03:42:16 gateway kernel: RDX: 0000000000000007 RSI: fffffffe7c12321c RDI: 0000000000000000 May 24 03:42:16 gateway kernel: RBP: 0000000000000005 R08: 0000000000000004 R09: 000000004025f162 May 24 03:42:16 gateway kernel: R10: 0000000000000018 R11: 000000000000343a R12: ffffffff86343ca0 May 24 03:42:16 gateway kernel: R13: 0000ad759098253d R14: 0000000000000005 R15: 0000000000000000 May 24 03:42:16 gateway kernel: cpuidle_enter+0x2d/0x40 May 24 03:42:16 gateway kernel: do_idle+0x1ed/0x270 May 24 03:42:16 gateway kernel: cpu_startup_entry+0x1d/0x20 May 24 03:42:16 gateway kernel: start_secondary+0x115/0x140 May 24 03:42:16 gateway kernel: secondary_startup_64_no_verify+0xe5/0xeb May 24 03:42:16 gateway kernel: </TASK> May 24 03:42:16 gateway kernel: ---[ end trace 0000000000000000 ]--- ``` |