Bug 10790

David, can you look at this bug (email answers to this email are fine)?


----- Forwarded message from bugme-daemon@bugzilla.kernel.org -----

Date: Sun, 25 May 2008 04:10:04 -0700 (PDT)
From: bugme-daemon@bugzilla.kernel.org
To: bunk@kernel.org
Subject: [Bug 10790] New: driver "sunhme" experiences corrupt packets if
	machine has more than 2GB of memory

http://bugzilla.kernel.org/show_bug.cgi?id=10790

           Summary: driver "sunhme" experiences corrupt packets if machine
                    has more than 2GB of memory
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.22.17-0.1
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: jgarzik@pobox.com
        ReportedBy: f0h3a-kernel@yahoo.com
                CC: kkeil@suse.de


Latest working kernel version: unknown
Earliest failing kernel version: unknown
Distribution: OpenSUSE 10.3 (also tested on vanilla kernel) - filed on Novell
bugzilla at https://bugzilla.novell.com/show_bug.cgi?id=376831


Hardware Environment:
System 1: Q6600 cpu, 4GB mem, Asus P5N32-E SLI Plus, Sun QFE card
System 2: Q6600 cpu, 4GB mem, MSI P35 NEO-F (P35), Sun QFE card

Software Environment: OpenSUSE 10.3, both SUSE & vanilla kernels

Problem Description:

Network interfaces on the QFE card experience corrupted packets when the
machine has more than 2GB of memory. The NIC can successfully obtain a DHCP
address from a server, but attempts to ping or ssh to the address give
unexpected results


hazard@mustang:~/hme-debug> ping 192.168.2.90
PING 192.168.2.90 (192.168.2.90) 56(84) bytes of data.
64 bytes from 192.168.2.90: icmp_seq=1 ttl=64 time=2.20 ms
wrong data byte #54 should be 0x36 but was 0xba
#16     10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
28 29 2a 2b 2c 2d 2e 2f 
#48     30 31 32 33 34 35 ba cc 
64 bytes from 192.168.2.90: icmp_seq=2 ttl=64 time=0.223 ms
wrong data byte #54 should be 0x36 but was 0x64
#16     10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
28 29 2a 2b 2c 2d 2e 2f 
#48     30 31 32 33 34 35 64 d8 


hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Corrupted MAC on input.
hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.


Steps to reproduce:

Plumb an interface on a Sun QFE card in a machine with > 2GB memory. Ping or
ssh to the IP of the interface.

Reply-To: akpm@linux-foundation.org

On Sun, 25 May 2008 04:10:04 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=10790
> 
>            Summary: driver "sunhme" experiences corrupt packets if machine
>                     has more than 2GB of memory
>            Product: Drivers
>            Version: 2.5
>      KernelVersion: 2.6.22.17-0.1
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Network
>         AssignedTo: jgarzik@pobox.com
>         ReportedBy: f0h3a-kernel@yahoo.com
>                 CC: kkeil@suse.de
> 
> 
> Latest working kernel version: unknown
> Earliest failing kernel version: unknown
> Distribution: OpenSUSE 10.3 (also tested on vanilla kernel) - filed on Novell
> bugzilla at https://bugzilla.novell.com/show_bug.cgi?id=376831
> 
> 
> Hardware Environment:
> System 1: Q6600 cpu, 4GB mem, Asus P5N32-E SLI Plus, Sun QFE card
> System 2: Q6600 cpu, 4GB mem, MSI P35 NEO-F (P35), Sun QFE card
> 
> Software Environment: OpenSUSE 10.3, both SUSE & vanilla kernels
> 
> Problem Description:
> 
> Network interfaces on the QFE card experience corrupted packets when the
> machine has more than 2GB of memory. The NIC can successfully obtain a DHCP
> address from a server, but attempts to ping or ssh to the address give
> unexpected results
> 
> 
> hazard@mustang:~/hme-debug> ping 192.168.2.90
> PING 192.168.2.90 (192.168.2.90) 56(84) bytes of data.
> 64 bytes from 192.168.2.90: icmp_seq=1 ttl=64 time=2.20 ms
> wrong data byte #54 should be 0x36 but was 0xba
> #16     10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26
> 27
> 28 29 2a 2b 2c 2d 2e 2f 
> #48     30 31 32 33 34 35 ba cc 
> 64 bytes from 192.168.2.90: icmp_seq=2 ttl=64 time=0.223 ms
> wrong data byte #54 should be 0x36 but was 0x64
> #16     10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26
> 27
> 28 29 2a 2b 2c 2d 2e 2f 
> #48     30 31 32 33 34 35 64 d8 
> 
> 
> hazard@mustang:~/hme-debug> ssh 192.168.2.90
> Received disconnect from 192.168.2.90: 2: Corrupted MAC on input.
> hazard@mustang:~/hme-debug> ssh 192.168.2.90
> Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
> hazard@mustang:~/hme-debug> ssh 192.168.2.90
> Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
> hazard@mustang:~/hme-debug> ssh 192.168.2.90
> Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
> 
> 
> Steps to reproduce:
> 
> Plumb an interface on a Sun QFE card in a machine with > 2GB memory. Ping or
> ssh to the IP of the interface.
> 

Note that this patch from Karsten:
https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.

Stoopid question: if it fails at 2GB, should we try DMA_31BIT_MASK?

hi all, just to clarify.

driver works a-ok 100% if I take out 2GB memory (leaving 2GB installed). I tested this again just now, to make sure I wasn't imagining things. 2GB memory = working driver.

with 4GB memory installed, bug appears...

From: Adrian Bunk <bunk@kernel.org>
Date: Sun, 25 May 2008 20:30:08 +0300

> David, can you look at this bug (email answers to this email are fine)?

I've been looking on and off at this bug and haven't forgotten it.

I've checked the obvious things like whether the PCI front-end supports
full 32-bit addresses (all documentation states that it does) or whether
there is some signedness issue with how the driver handles DMA
addresses (I cannot find any such problems).

I'll reply again if I make any more progress on this bug.

I do have a QFE card somewhere and an x86 system with 4GB of ram
so I might try to reproduce this problem locally in the end.

From: Andrew Morton <akpm@linux-foundation.org>
Date: Sun, 25 May 2008 11:45:48 -0700

> Note that this patch from Karsten:
> https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.

That patch is unlikely to be the issue, right.

> Stoopid question: if it fails at 2GB, should we try DMA_31BIT_MASK?

I've scoured all of the docs and the chip itself and the PCI front-end
it is connected to support full 32-bit PCI addressing.

The bug report says platform "All" but the report specifically
mentions Q6600 cpus, is this x86_64 by chance?  I wonder if the
problem is that we use GFP_ATOMIC/GFP_KERNEL for allocations
of SKB data, and we end up with buffers above 4GB but the IOMMU
mapping operation doesn't cope with that correctly.

In the arch/x86/kernel/pci-nommu.c case, this would result in
kernel log messages from check_addr():

		if (*hwdev->dma_mask >= DMA_32BIT_MASK)
			printk(KERN_ERR
			    "nommu_%s: overflow %Lx+%zu of device mask %Lx\n",
				name, (long long)bus, size,
				(long long)*hwdev->dma_mask);

On Tue, Jun 03, 2008 at 04:33:17PM -0700, David Miller wrote:
> From: Andrew Morton <akpm@linux-foundation.org>
> Date: Sun, 25 May 2008 11:45:48 -0700
> 
> > Note that this patch from Karsten:
> > https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.
> 
> That patch is unlikely to be the issue, right.
> 

No, this patch was a experiment to fix the issue, it is not in our kernel
by default.

> > Stoopid question: if it fails at 2GB, should we try DMA_31BIT_MASK?
> 
> I've scoured all of the docs and the chip itself and the PCI front-end
> it is connected to support full 32-bit PCI addressing.
> 
> The bug report says platform "All" but the report specifically
> mentions Q6600 cpus, is this x86_64 by chance?  I wonder if the
> problem is that we use GFP_ATOMIC/GFP_KERNEL for allocations
> of SKB data, and we end up with buffers above 4GB but the IOMMU
> mapping operation doesn't cope with that correctly.

Yes it is x86_64 and I think, that if you put more as 2G in this machine,
some physical addresses are behind the 4 GB border.


> 
> In the arch/x86/kernel/pci-nommu.c case, this would result in
> kernel log messages from check_addr():
> 
>               if (*hwdev->dma_mask >= DMA_32BIT_MASK)
>                       printk(KERN_ERR
>                           "nommu_%s: overflow %Lx+%zu of device mask %Lx\n",
>                               name, (long long)bus, size,
>                               (long long)*hwdev->dma_mask);

From: Karsten Keil <kkeil@suse.de>
Date: Wed, 4 Jun 2008 10:21:21 +0200

> Yes it is x86_64 and I think, that if you put more as 2G in this machine,
> some physical addresses are behind the 4 GB border.

If that is the case then I think there might be some bug in the
dma_sync_single_for_cpu() implementation on whatever IOMMU
implementation is being used on this x86_64 system.

sunhme has two cases:

1) If the packet is greater than 256 bytes the original packet
   is unmapped.  So, if soft-iommu uses a bounce buffer,
   for example, this should do the necessary copy from
   the bounce buffer to the actual SKB data area.

2) If the packet is less than or equal to 256 bytes, the
   packet is DMA sync'd using pci_dma_sync_single_for_cpu(),
   which should cause soft-iommu or similar to copy from any
   bounce buffer to the real buffer, and then we copy the
   SKB data into a new freshly allocated SKB and reuse the original
   one after calling pci_dma_sync_single_for_device().

I've validated that the DMA address and lengths used by those
calls in the sunmhe driver are correct, so really my top suspect
would be the IOMMU implementation being used on this system for
these specific kinds of cases.

I also have this problem.  I tested a X86_64 debian kernel 2.6.26, and an X86_64 stock 2.6.27.8 from kernel.org.  Both exhibit this problem.  The system has 6 Gig of ram.  If I boot a 32bit 2.6.26 debian kernel without PAE I driver works fine.  This problem appears to be unique to the X86_64 kernels.  I haven't tried with only 2Gig of ram.

I have a number of these sun cards in use and was hoping to move to the X86_64 arch.  Any thoughts on a solution for this.

I also have these problems, but not always:
I am running Centos 5, with it's standard kernel: 
Linux nest.pheasant 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux. this works fine. I have never seen the issues on this kernel. When I however boot the Centos Xen kernel, the problem appears. It normally takes a few hours (on a very lightly loaded system) to show up.
This makes running Xen on this kernel impossible as packet corruption occurs in roughly 30% of the packages. If you need more information, please let me now.

I'm also experiencing the problem with the sunhme driver and > 2GB of ram.

Core 2 Quad Q9400 @ 2.6GHz
6GB Ram
Asus P5Q SE2
Ubuntu 8.10 amd_64

Linux gslse 2.6.27-7-server #1 SMP Fri Oct 24 07:20:47 UTC 2008 x86_64 GNU/Linux

I've set the system up for remote access to a functioning NIC, as well as to all 4 sunhme interfaces.  Plus, there are 4 additional interfaces on SPAN ports for monitoring incoming traffic to the sunhme card, as depicted here: http://216.93.244.149/images/sunhme.gif

I'd be happy to provide remote access to this system if lack of a development platform is holding up investigation into the issue.

sunhmebug@marget.com

Did this (sunhme packets on x86_64 > 2G ram) ever get fixed ?

To my knowledge this has not been fixed.  If I get some time, I may try and test again with a 2.6.30 kernel.  It would be really nice to have a fix for this.

Adding the __GFP_DMA flag to the skb requests in sunhme.c successfully hides the problem. I have a x86_64 (Q6600) 8GB system running with 2 QFE boards (8 ports) with this patch for the last month. No doubt i've broken something else with this sledgehammer...

--- sunhme.c.dist       2010-04-17 15:33:35.000000000 +0100
+++ sunhme.c    2010-04-18 01:14:55.000000000 +0100
@@ -1274,7 +1274,7 @@ static void happy_meal_init_rings(struct
        for (i = 0; i < RX_RING_SIZE; i++) {
                struct sk_buff *skb;

-               skb = happy_meal_alloc_skb(RX_BUF_ALLOC_SIZE, GFP_ATOMIC);
+               skb = happy_meal_alloc_skb(RX_BUF_ALLOC_SIZE, GFP_ATOMIC||__GFP_DMA);
                if (!skb) {
                        hme_write_rxd(hp, &hb->happy_meal_rxd[i], 0, 0);
                        continue;
@@ -2033,7 +2033,7 @@ static void happy_meal_rx(struct happy_m
                        struct sk_buff *new_skb;

                        /* Now refill the entry, if we can. */
-                       new_skb = happy_meal_alloc_skb(RX_BUF_ALLOC_SIZE, GFP_ATOMIC);
+                       new_skb = happy_meal_alloc_skb(RX_BUF_ALLOC_SIZE, GFP_ATOMIC||__GFP_DMA);
                        if (new_skb == NULL) {
                                drops++;
                                goto drop_it;

stephen -- I wonder, does this patch still work for you without problems or side-effects up until now?
If so, maybe this patch can be considered for inclusion to the mainline as a fix for this bug? While SUN QFE boards are arguably not common hardware, more than 2 GB of RAM and x86-64 are very common nowadays, especially on machines that require 4 network ports per expansion slot - so it would be nice to have this fixed.

I use Sun Quad 501-4366 hme nics on a few 86_64 machines with > 2gb ram

The hme nics caused exactly this problem after upgrading to 3.6.11
And the patch fixed it.

Similarly to Roman: I'm equally sure these are not the worlds most popular cards but there are plenty still on market; you can get a 10/100 4 port nic for $low.
And who doesnt have a 64bit mc with > 2gb ram?

So if its possible to get patch included I for one will be grateful
please, pretty please

sorry, just read my post and should clarify:
I suspect it wasnt the upgrade to 3.6.11 that caused problem, but the simultaneous hardware upgrades.