Bug 72561 - missing some icmp redirects
Summary: missing some icmp redirects
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-20 14:45 UTC by Per Jessen
Modified: 2014-06-10 18:07 UTC (History)
1 user (show)

See Also:
Kernel Version: 3.11.6-4-default
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Per Jessen 2014-03-20 14:45:21 UTC
Background:  I have recently upgraded our firewall to openSUSE 13.1. It used to
be 11.x I think. We have long had a transparent webcache setup, but after
upgrading I disabled this as there was something odd with it, and I didn't have
time to investigate initially. 

Setup:  the webcache works by redirecting port 80 and 443 traffic to a system
running squid.  

Problem:  complex websites, e.g. an on-line newspaper, typically pull in
content from several sources. When a page is loaded, I often see e.g. 8-10
different sources (IP-addresses) of content.  With the webcaching enabled,
loading e.g. http://www.tagesanzeiger.ch leads to a 2-3 minute wait, usually
for one of those IPs/resources.  I have traced the ICMP redirects and noticed
that usually only one of the IP-addresses is not redirected. This is the one
the browser waits for. After about 2 minutes, the connection-attempt times out
and is retried, and then properly redirected.  This is reproducable with a
variety of websites. 
Relatively "simple" web pages using content from only one webserver, e.g. a
wikipedia page or the google start page show no problems.  

Some sort of race condition in the ip stack?

Reproducible: Always

Steps to Reproduce:
in a transparent webcache setup:
1. load http://www.tagesanzeiger.ch/ 
2. observe how the browser will end up waiting for one source.
3. check the address of this resource
4. ip route get <ipaddr> will show it was not redirected
5. after 2 minutes, the page is shown.
6. ip route get <ipaddr> will show it was now redirected
Comment 1 Alan 2014-04-08 10:47:50 UTC
This isn't a support forum., just used for bug tracking

Your best bet after capturing traces is netdev@vger.kernel.org

ICMP error reporting is ratelimited however, so it seems a dumb way to implement a web cache
Comment 2 Per Jessen 2014-06-10 18:07:04 UTC
(In reply to Alan from comment #1)
> This isn't a support forum., just used for bug tracking
> 
> Your best bet after capturing traces is netdev@vger.kernel.org
>
> ICMP error reporting is ratelimited however, so it seems a dumb way to
> implement a web cache

It used to work fine until very recently. 

I have devised a way to reproduce the problem. I have a test setup of three machines: 

“client”, “firewall” and “server”. All on the same network.

Client:
Set up default route via “firewall”.

Server:
Assign 10.232.1.1-2-3-4-...-15/24 to an interface.
Run a tcp echo service (port 7).

Firewall:
Create routing table “test99”.  /etc/iproute2/rt_tables.
ip route add default via <server> dev eth0 table test99
ip rule add fwmark 5 table test99
iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 7 -j MARK --set-mark 5

(this setup is what will produce the ICMP redirects).

On “firewall”, run tcpdump to document (missing) redirects:
tcpdump -n -i eth0 proto \\icmp 

On “client”, create some test input:

cat <<XXX >test.input
klop
alpha
nothing
tagi
line1
line2
line3
line4
XXX

Create a script
cat <<XXX >doit
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    telnet 10.232.1.$i 7 <test.input &
done

When you run “sh doit”, all of the telnet requests to 10.232.1.x should be redirected, but the tcpdump running on "firewall" will only show some of them.

Note You need to log in before you can comment on or make changes to this bug.