When assigninig a simple Tocket Bucket Filter (TBF) classless trafic shaping policy to the eth0 device (provided by forcedeth), the device's upstream speed drops to ~12kbyte/s, regardless of the traffic limit I set in the shaping rule. In fact, the traffic limit values could be larger than the connection's throughput and it'll still limit to mere 12kbyte/s. Once I revert the rule to pfifo_fast, the rate resumes to the connection's real one (64kbyte/s). If I set the TBF lower than 96kbit on eth0, it indeed applies; in other words, I can make the connection slower than 12kbyte/s but not faster. If I assign the same rule to ppp0, it works just as intended, for rules higher than 12kbyte/s, leading me to believe forcedeth has something to do with it. (Is there any module parameter I can try tweaking?)
Forgot to mention: this is an x86-64 kernel.
On Fri, 6 Oct 2006 17:18:13 -0700 bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=7278 > > Summary: forcedeth slowed down by traffic shaping > Kernel Version: 2.6.16 > Status: NEW > Severity: normal > Owner: jgarzik@pobox.com > Submitter: kernel-bugzilla@future.shiny.co.il > > > When assigninig a simple Tocket Bucket Filter (TBF) classless trafic shaping > policy to the eth0 device (provided by forcedeth), the device's upstream speed > drops to ~12kbyte/s, regardless of the traffic limit I set in the shaping rule. > In fact, the traffic limit values could be larger than the connection's > throughput and it'll still limit to mere 12kbyte/s. Once I revert the rule to > pfifo_fast, the rate resumes to the connection's real one (64kbyte/s). > > If I set the TBF lower than 96kbit on eth0, it indeed applies; in other words, I > can make the connection slower than 12kbyte/s but not faster. > > If I assign the same rule to ppp0, it works just as intended, for rules higher > than 12kbyte/s, leading me to believe forcedeth has something to do with it. > > (Is there any module parameter I can try tweaking?) >
On 07-10-2006 02:22, Andrew Morton wrote: > On Fri, 6 Oct 2006 17:18:13 -0700 > bugme-daemon@bugzilla.kernel.org wrote: > >> http://bugzilla.kernel.org/show_bug.cgi?id=7278 >> >> Summary: forcedeth slowed down by traffic shaping >> Kernel Version: 2.6.16 >> Status: NEW >> Severity: normal >> Owner: jgarzik@pobox.com >> Submitter: kernel-bugzilla@future.shiny.co.il >> >> >> When assigninig a simple Tocket Bucket Filter (TBF) classless trafic shaping >> policy to the eth0 device (provided by forcedeth), the device's upstream speed >> drops to ~12kbyte/s, regardless of the traffic limit I set in the shaping rule. >> In fact, the traffic limit values could be larger than the connection's >> throughput and it'll still limit to mere 12kbyte/s. Once I revert the rule to >> pfifo_fast, the rate resumes to the connection's real one (64kbyte/s). >> >> If I set the TBF lower than 96kbit on eth0, it indeed applies; in other words, I >> can make the connection slower than 12kbyte/s but not faster. >> >> If I assign the same rule to ppp0, it works just as intended, for rules higher >> than 12kbyte/s, leading me to believe forcedeth has something to do with it. >> >> (Is there any module parameter I can try tweaking?) Isn't it to unpolite to expect from the submitter some details like this tbf rule or testing with other kernel version? Jarek P.
Jarek Poplawski wrote: > On 07-10-2006 02:22, Andrew Morton wrote: > >>On Fri, 6 Oct 2006 17:18:13 -0700 >>bugme-daemon@bugzilla.kernel.org wrote: >> >> >>>http://bugzilla.kernel.org/show_bug.cgi?id=7278 >>> >>> Summary: forcedeth slowed down by traffic shaping >>> Kernel Version: 2.6.16 >>> Status: NEW >>> Severity: normal >>> Owner: jgarzik@pobox.com >>> Submitter: kernel-bugzilla@future.shiny.co.il >>> >>> >>>When assigninig a simple Tocket Bucket Filter (TBF) classless trafic shaping >>>policy to the eth0 device (provided by forcedeth), the device's upstream speed >>>drops to ~12kbyte/s, regardless of the traffic limit I set in the shaping rule. >>>In fact, the traffic limit values could be larger than the connection's >>>throughput and it'll still limit to mere 12kbyte/s. Once I revert the rule to >>>pfifo_fast, the rate resumes to the connection's real one (64kbyte/s). >>> >>>If I set the TBF lower than 96kbit on eth0, it indeed applies; in other words, I >>>can make the connection slower than 12kbyte/s but not faster. >>> >>>If I assign the same rule to ppp0, it works just as intended, for rules higher >>>than 12kbyte/s, leading me to believe forcedeth has something to do with it. >>> >>>(Is there any module parameter I can try tweaking?) > > > Isn't it to unpolite to expect from the submitter some details > like this tbf rule or testing with other kernel version? The TBF command would be a good start to make sure its not a configuration error.
The TBF rule which dropped eth0 to 12kb/s was applied with: tc qdisc replace dev $IFACE root tbf rate 512kbit burst 10kb latency 50ms where $IFACE is eth0 or ppp0. I did not test on other kernel versions. I might upgrade to 2.6.17 soon. If you are aware of a specific change made in 2.16.17 or later which might assist in solving this problem, do tell.
Created attachment 9220 [details] [PKT_SCHED]: Restore TBF change semantic Please try this patch, if you also used "tc qdisc _change_" for changing the rate it should fix the problem (already included in 2.6.17).
I did not use "tc qdisc change" ever -- just "tc qdisc replace". 1. Am I wrong to do so? 2. Does this patch only fix cases when the limit is changed later on, or does it also help in my "replace" case? 3. Do you have a reason to assume this compensates for a difference between the forcedeth and ppp devices? (As I said, I'm fully able to shape my ppp0 interface with the 'tc' command I've given.)
It should also help with "replace", replacing a qdisc by the same type with different parameters is equivalent to change. You specify the queue length as latency, so it a function of the rate. The patch fixes tbf ignoring the new queue length, which _could_ be responsible in case you directly replaced the TBF qdisc during testing instead of doing TBF -> pfifo_fast -> TBF. BTW, what is the MTU on that device, are you using TSO and what clock source are you using (CONFIG_NET_SCH_CLK_*)?
I'll try the patch once I get home and could test it locally. I made a habit of doing 'qdisc del dev eth0 root' in-between changes, thus returning to pfifo_fast. So, from what you described about the patch, it's not likely to solve the issue. I'll try it anyway. The MTU on eth0 is 1500. (Adding it as an 'mtu 1500' to the tbf parameters didn't make any difference.) As to CONFIG_NET_SCH_CLK_*: # CONFIG_NET_SCH_CLK_JIFFIES is not set CONFIG_NET_SCH_CLK_GETTIMEOFDAY=y # CONFIG_NET_SCH_CLK_CPU is not set
What about TSO (ethtool -k eth0)?
# ethtool -k eth0 Offload parameters for eth0: Cannot get device rx csum settings: Operation not supported Cannot get device tx csum settings: Operation not supported Cannot get device scatter-gather settings: Operation not supported Cannot get device tcp segmentation offload settings: Operation not supported no offload info available (Hey, it's forcedeth, a cheap on-board chipset!)
Thanks. What does "tc -s -d qdisc show dev eth0" show?
With pfifo_fast: qdisc pfifo_fast 0: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 505170 bytes 392 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 With tbf (after some 10 seconds of uploading which could potentially reach 64Kbyte/s but only reaches 12Kbyte/s): qdisc tbf 800e: rate 512000bit burst 10Kb/8 mpu 0b lat 50.0ms Sent 173677 bytes 369 pkt (dropped 51, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0
I tested locally and it works fine here. Unless your iproute version is buggy the drops must come from exceeding the inner qdiscs limits, which is a bit strange, it should be about 13k with your settings. What version of iproute are you using? One thing that is notable is that the average packet size is a lot smaller with TBF in your case (~400byte). What kind of traffic are you using for testing and how is it produced?
# tc -V tc utility, iproute2-ss060323 I'm testing with an Apache 2 HTTP server running on the problematic machine and wget on another machine downloading from the problematic machine. (There's some SSH traffic to control the remote machine but it's probably neglectible.) Specifically, I'm transferring an MP3 file.
Please attach a tcpdump starting at the beginning of the download, maybe this will show whats going on.
Created attachment 9236 [details] tcpdump of a "slow" upload A dump of an upload made with the TBF (512000bit burst 10Kb lat 50.0ms) which capped at approx. 12kb. The dump is filtered to only include the two involved hosts.
Looking at the driver, it does support TSO and uses it when HW checksumming is available. HW checksumming is enabled in your case (as can be seen in the dump), so TSO must be enabled as well. Looking at the changelog, the forcedeth driver only got support for querying TSO settings after 2.6.16. This explains the drops, the packet sizes exceed the expected maximum. Try setting the tbf mtu to something like 16k and see if it helps.
Indeed, setting the tbf mtu up to 16384 allows me to reach the peak speed. Values below 16384 give me suboptimal speed. Should this bug become a dup of "Add TSO status reporting to forcedeth" bug, or is it already solved in recent kernels?
Its not a bug, its a configuration issue. TBF needs to know the size of the largest possible packet to calculate its rate tables. With TSO its larger than the interface MTU.
Yes, but apparently, except for this Bugzilla item, there's no documentation of the TSO buffer size in the forcedeth card. The bug isn't in the traffic shaping; the bug is in the forcedeth card for not reporting its TSO MTU size and therefore making it impossible for the user to know the right MTU to use. Are newer (>2.6.16) kernels really report TSO size for forcedeth?
It allows to query the presence of TSO. The maximum possible packet size is always 64k with TSO. But again, its not a bug, neither the documentation is wrong (there is none), nor does the driver do anything its not supposed to do. TBF (and other schedulers that depend on correct setting of the MTU) could do better at reporting MTU problems though, maybe through a new statistic or simply by printing a warning .. Anyway, I guess this report can be resolved.
From the tc-tbf(8) manpage, I've got the impression that the "mtu" option is only needed for fine-tuning: "For perfect accuracy, should be set to the MTU of the interface." The manpage doesn't specify the MTU default value when its omitted. Does the in-kernel tbf code decide it in this case, and if so, could it apply a better judgment by querying the Ethernet driver for TSO and using 64K in this case?
To clarify myself, there's indeed no kernel bug, but I'm looking for a way we can improve the kernel + iproute2 combo so that it'll be friendlier and nobody else would have to come across the problem I presented here. Do keep in mind that TSO is not something a casual user keeps in the back of his head, and unless you suggested it, I'd never guess this factor comes into play.
The kernel code can't do anything about it but complain, the MTU is used to calculate the rate-tables and this is done by userspace. A patch to the TBF manpage (and HTB, SFQ) makes sense of course. A new statistic (oversized) would be the best solution for the kernel IMO, but its quite ugly because of compatibility issues and I'm not convinced its worth the trouble.
I'll make an effort to patch the manpage. Does the iproute2 code also require a patch to help it determine the *real* MTU properly when TSO is enabled, or does it already do so?
iproute currently doesn't try to get the MTU at all, if no MTU is given it uses 2047. TSO settings are not available through rtnetlink, so you would have to add support for that before iproute can determine the proper MTU to use.