After I upgraded a Fedora (FC20) router to Fedora 26, I noticed issues in the networking. The transfer speed for files sent to the box via Samba or vsftpd was horribly slow at about 5 to 10mb/s while the other direction works perfectly. ifconfig shows a large amount of packet drops (RX dropped), both on the Internet adapter enp1s0 as on the network connected adapter enp2s0 hence packet routing to the network when surfing is enough to cause packet drops. To rule out a firewall issue, I set all default rules to allow and flushed the tables. This had no effect. I also changed the ACPI level in BIOS, but that did not have an impact either. Because I had a couple of system backups (done trough dd if=/dev/...) I restored the oldest backup I had, which was running Fedora 20 Kernel 3.17.4-200.fc20.x86_64. There I saw that although there where also packets dropped, the amount of dropped packets was much lower (like 10 or more times) and the speed sending files to the box was much faster (20 to 30 Mb/s), why I eventually didn't notice. The latest fedora 20 backup before installing FC26 was 3.19.8-100.fc20.x86_64 and behaved almost like the 3.17 kernel. One interesting thing I noticed is that when I run tcpdump, then the amount of doped RX packets drops significantly, sometime to no drops and the speed measured when moving files to the box goes from 5 to 10 mb/s to about 20 mb/s. Eventually showing an issue with the RX queue management?! Tests I have done so far: - Disabled one of the two adapters - Directly connected to the box without switch - Changed the Cat6a cable and directly connected - Started/Stopped/Disabled other services (VPN, named, dhcp, ...) - Tested the write speed to the share drive directly on the box (was 110mb/s so no bottleneck) - Ping: no losses until the network load rises, then some ICMPs are lost. - Checked link speed and duplex mode (all of which are 1000TX and Full duplex, so OK. none of which gave any noticeable results. Running dropwatch under network load I get a lot of "drops at net_tx_action+0" which points to dev.c.. By the way, VIA advertises the Network chip on their web page as VT6130G for the EPIA-3002 board. Not sure if it is the same as the usually found VT6130 chip.
Output of DROPWATCH: -------------------- # dropwatch -l kas Initalizing kallsyms db dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 2 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at __netif_receive_skb_core+6f0 (0xffffffffa973e630) 4 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at __netif_receive_skb_core+6f0 (0xffffffffa973e630) 412 drops at net_tx_action+0 (0xffffffffa973a110) ##### Start of SMB Traffic! 1 drops at __per_cpu_start+0 ((nil)) 1068 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at __netif_receive_skb_core+6f0 (0xffffffffa973e630) 556 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at __per_cpu_start+0 ((nil)) 511 drops at net_tx_action+0 (0xffffffffa973a110) 1 drops at __netif_receive_skb_core+6f0 (0xffffffffa973e630) 333 drops at net_tx_action+0 (0xffffffffa973a110) IFCONFIG: --------- enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.13.2 netmask 255.255.255.0 broadcast 192.168.13.255 ether 00:1f:f2:07:0c:1e txqueuelen 1000 (Ethernet) RX packets 7204 bytes 2630161 (2.5 MiB) RX errors 0 dropped 1746 overruns 0 frame 0 TX packets 5696 bytes 639613 (624.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 28 enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.80.13.1 netmask 255.255.255.0 broadcast 10.80.13.255 ether 00:1f:f2:07:0c:1f txqueuelen 1000 (Ethernet) RX packets 42992 bytes 52544297 (50.1 MiB) RX errors 0 dropped 16 overruns 0 frame 0 TX packets 8010 bytes 2509167 (2.3 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 29 LSWH: ----- # lshw -c network *-network description: Ethernet interface product: VT6120/VT6121/VT6122 Gigabit Ethernet Adapter vendor: VIA Technologies, Inc. physical id: 0 bus info: pci@0000:01:00.0 logical name: enp1s0 version: 82 serial: 00:1f:f2:07:0c:1e size: 100Mbit/s capacity: 1Gbit/s width: 64 bits clock: 33MHz capabilities: pm pciexpress msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=via-velocity driverversion=1.15 duplex=full ip=192.168.13.2 latency=0 link=yes multicast=yes port=twisted pair speed=100Mbit/s resources: irq:28 ioport:c800(size=256) memory:feaffc00-feaffcff *-network description: Ethernet interface product: VT6120/VT6121/VT6122 Gigabit Ethernet Adapter vendor: VIA Technologies, Inc. physical id: 0 bus info: pci@0000:02:00.0 logical name: enp2s0 version: 82 serial: 00:1f:f2:07:0c:1f size: 1Gbit/s capacity: 1Gbit/s width: 64 bits clock: 33MHz capabilities: pm pciexpress msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=via-velocity driverversion=1.15 duplex=full ip=10.80.13.1 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s resources: irq:29 ioport:d800(size=256) memory:febffc00-febffcff LSPCI: ------ # lspci 00:00.0 Host bridge: VIA Technologies, Inc. VX900 Host Bridge: Host Control (rev 80) 00:00.1 Host bridge: VIA Technologies, Inc. VX900 Error Reporting 00:00.2 Host bridge: VIA Technologies, Inc. VX900 CPU Bus Controller 00:00.3 Host bridge: VIA Technologies, Inc. VX900 DRAM Bus Control 00:00.4 Host bridge: VIA Technologies, Inc. VX900 Power Management and Chip Testing Control 00:00.5 Host bridge: VIA Technologies, Inc. VX900 APIC and Central Traffic Control 00:00.6 Host bridge: VIA Technologies, Inc. VX900 Scratch Registers 00:00.7 Host bridge: VIA Technologies, Inc. VX900 North-South Module Interface Control 00:01.0 VGA compatible controller: VIA Technologies, Inc. VX900 Graphics [Chrome9 HD] 00:01.1 Audio device: VIA Technologies, Inc. Device 9170 00:03.0 PCI bridge: VIA Technologies, Inc. VX900 PCI Express Root Port 0 00:03.1 PCI bridge: VIA Technologies, Inc. VX900 PCI Express Root Port 1 00:03.2 PCI bridge: VIA Technologies, Inc. VX900 PCI Express Root Port 2 00:03.3 PCI bridge: VIA Technologies, Inc. VX900 PCI Express Root Port 3 00:03.4 Host bridge: VIA Technologies, Inc. VX900 PCI Express Physical Layer Electrical Sub-block 00:0f.0 IDE interface: VIA Technologies, Inc. VX900 Serial ATA Controller 00:10.0 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev a0) 00:10.1 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev a0) 00:10.2 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev a0) 00:10.3 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev a0) 00:10.4 USB controller: VIA Technologies, Inc. USB 2.0 (rev 90) 00:11.0 ISA bridge: VIA Technologies, Inc. VX900 Bus Control and Power Management 00:11.7 Host bridge: VIA Technologies, Inc. VX8xx South-North Module Interface Control 00:13.0 PCI bridge: VIA Technologies, Inc. VX855/VX875/VX900 PCI to PCI Bridge 00:14.0 Audio device: VIA Technologies, Inc. VT8237A/VT8251 HDA Controller (rev 20) 01:00.0 Ethernet controller: VIA Technologies, Inc. VT6120/VT6121/VT6122 Gigabit Ethernet Adapter (rev 82) 02:00.0 Ethernet controller: VIA Technologies, Inc. VT6120/VT6121/VT6122 Gigabit Ethernet Adapter (rev 82) DMESG: ------ [ 0.000000] DMI: VIA technology Inc. VX900/EITX-3002, BIOS I2100101 02/14/2012 [ 0.000000] ACPI: HPET 0x00000000BFFCA4B0 000038 (v01 021412 VIA HPET 20120214 MSFT 00000097) [ 0.137000] smpboot: CPU0: Centaur VIA Eden X2 U4200 @ 1.0+ GHz (family: 0x6, model: 0xf, stepping: 0xc) [ 0.445073] pci 0000:00:03.0: disabling DAC on VIA PCI bridge [ 7.097988] VIA Networking Velocity Family Gigabit Ethernet Adapter Driver Ver. 1.15 [ 7.097993] Copyright (c) 2002, 2003 VIA Networking Technologies, Inc. [ 7.099434] eth0: VIA Networking Velocity Family Gigabit Ethernet Adapter [ 7.116004] eth1: VIA Networking Velocity Family Gigabit Ethernet Adapter MODINFO: -------- # modinfo via-velocity filename: /lib/modules/4.11.10-300.fc26.x86_64/kernel/drivers/net/ethernet/via/via-velocity.ko.xz description: VIA Networking Velocity Family Gigabit Ethernet Adapter Driver license: GPL author: VIA Networking Technologies, Inc. alias: pci:v00001106d00003119sv*sd*bc*sc*i* alias: of:N*T*Cvia,velocity-vt6110C* alias: of:N*T*Cvia,velocity-vt6110 depends: intree: Y vermagic: 4.11.10-300.fc26.x86_64 SMP mod_unload signat: PKCS#7 signer: sig_key: sig_hashalgo: md4 parm: RxDescriptors:Number of receive descriptors (array of int) parm: TxDescriptors:Number of transmit descriptors (array of int) parm: rx_thresh:Receive fifo threshold (array of int) parm: DMA_length:DMA length (array of int) parm: IP_byte_align:Enable IP header dword aligned (array of int) parm: flow_control:Enable flow control ability (array of int) parm: speed_duplex:Setting the speed and duplex mode (array of int) parm: wol_opts:Wake On Lan options (array of int) parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int) NETSTAT: -------- # netstat -s Ip: Forwarding: 1 45420 total packets received 6209 forwarded 16 with unknown protocol 0 incoming packets discarded 38264 incoming packets delivered 13808 requests sent out 4 outgoing packets dropped 133 reassemblies required 8 packets reassembled ok 8 fragments received ok 133 fragments created Icmp: 28 ICMP messages received 0 input ICMP message failed ICMP input histogram: destination unreachable: 8 echo requests: 20 208 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 188 echo replies: 20 IcmpMsg: InType3: 8 InType8: 20 OutType0: 20 OutType3: 188 Tcp: 246 active connection openings 3 passive connection openings 0 failed connection attempts 0 connection resets received 3 connections established 36249 segments received 5529 segments sent out 25 segments retransmitted 0 bad segments received 105 resets sent Udp: 1927 packets received 155 packets to unknown port received 0 packet receive errors 1837 packets sent 0 receive buffer errors 0 send buffer errors UdpLite: TcpExt: 212 TCP sockets finished time wait in fast timer 9 delayed acks sent Quick ack mode was activated 2 times 1 packets directly queued to recvmsg prequeue TCPDirectCopyFromPrequeue: 1 32777 packet headers predicted 773 acknowledgments not containing data payload received 2 predicted acknowledgments TCPTimeouts: 3 TCPLossProbes: 7 TCPLossProbeRecovery: 3 TCPDSACKOldSent: 4 TCPDSACKRecv: 4 2 connections aborted due to timeout TCPDSACKIgnoredNoUndo: 3 TCPSackShiftFallback: 1 TCPRcvCoalesce: 31832 TCPOFOQueue: 40 TCPAutoCorking: 2 TCPFromZeroWindowAdv: 80 TCPToZeroWindowAdv: 80 TCPWantZeroWindowAdv: 206 TCPOrigDataSent: 2365 IpExt: InMcastPkts: 29 InBcastPkts: 96 OutBcastPkts: 67 InOctets: 54112877 OutOctets: 5238505 InMcastOctets: 1044 InBcastOctets: 14962 OutBcastOctets: 9980 InNoECTPkts: 45427 InCEPkts: 1
If it is useful, I could try to install older drivers of via-velocity to check when the issue occurred or run debug driver etc., but I am not sure on how to compile and install them, so I could need some help on that.
Thanks for the report. You may use Fedora 26 then rpm force install of kernel-3.19.8-100.fc20.x86_64.rpm on top of it to check that you recover the Fedora 20 behavior. It would help to have an environment where nothing changes but the kernel. You may revert 6c606fa32c4272894248c1abbc15257fc04250be ("via-velocity: unconditionally drop frames with bad l2 length") in 4.11. It isn't a fix - actually it's quite the opposite - but it should bring the rx errors back to the usual values and I'd rather check that it really has no unexpected effect on throughput before going any further. Btw, do you use standard MTU (no jumbo) ? -- Ueimor