Latest working kernel version: 2.6.22.9 Earliest failing kernel version: 2.6.23 Distribution: Gentoo Hardware Environment: Sparc64 (Sun Blade 100) Software Environment: Problem Description: I have two nearly identical Sun Blade 100 systems. Up to and including kernel version 2.6.22.9, both were working fine. However, with the 2.6.23- and 2.6.24- series I cannot get the Sun GEM network device on one of the two Blades to work. The other one strangely does not have this problem at all. The network device detects fine and the sungem module loads without problems. Mii-diag reports that a 100FDX connection exists and there is link beat. However, packets are neither being sent nor received, all counters in ifconfig remain at zero. Attempts to use the network eventually result in the following message in dmesg: eth0: Sun GEM (PCI) 10/100/1000BaseT Ethernet 00:03:ba:08:61:7c eth0: Found Generic MII PHY eth0: Link is up at 100 Mbps, full-duplex. eth0: Link is up at 100 Mbps, full-duplex. eth0: Pause is disabled NETDEV WATCHDOG: eth0: transmit timed out eth0: transmit timed out, resetting eth0: TX_STATE[003ffc05:00000001:0000001f] eth0: RX_STATE[0100c805:00000001:00000021] eth0: Link is up at 100 Mbps, full-duplex. eth0: Pause is disabled The difference between the non-working system and the working one are as follows: - The non-working system has more memory (1G opposed to 512M) - A dual SCSI controller is present in this system. Connected to it are two external 72G disks and an Exabyte Mammoth tapedrive. 01:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c875 (rev 14) 01:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c875 (rev 14) - The following kernel modules are loaded on this system and not on the working one: sym53c8xx, sd_mod, st, nfsd, md_mod and raid1 plus of course their respective dependencies. The problem is exactly the same with both Gentoo-patched kernels and vanilla kernels, that's why I am reporting it here. Steps to reproduce: Just boot an affected kernel and try to use eth0 in any way.
Reply-To: akpm@linux-foundation.org On Mon, 17 Mar 2008 14:25:50 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=10273 > > Summary: Sun GEM (PCI) - network device doesn't work > Product: Drivers > Version: 2.5 > KernelVersion: 2.6.23 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Network > AssignedTo: jgarzik@pobox.com > ReportedBy: seraph@xs4all.nl > > > Latest working kernel version: 2.6.22.9 > Earliest failing kernel version: 2.6.23 A regression in 2.6.23. > Distribution: Gentoo > Hardware Environment: Sparc64 (Sun Blade 100) > Software Environment: > Problem Description: > > I have two nearly identical Sun Blade 100 systems. Up to and including kernel > version 2.6.22.9, both were working fine. However, with the 2.6.23- and > 2.6.24- > series I cannot get the Sun GEM network device on one of the two Blades to > work. The other one strangely does not have this problem at all. > > The network device detects fine and the sungem module loads without problems. > Mii-diag reports that a 100FDX connection exists and there is link beat. > However, packets are neither being sent nor received, all counters in > ifconfig > remain at zero. > > Attempts to use the network eventually result in the following message in > dmesg: > > eth0: Sun GEM (PCI) 10/100/1000BaseT Ethernet 00:03:ba:08:61:7c > eth0: Found Generic MII PHY > eth0: Link is up at 100 Mbps, full-duplex. > eth0: Link is up at 100 Mbps, full-duplex. > eth0: Pause is disabled > NETDEV WATCHDOG: eth0: transmit timed out > eth0: transmit timed out, resetting > eth0: TX_STATE[003ffc05:00000001:0000001f] > eth0: RX_STATE[0100c805:00000001:00000021] > eth0: Link is up at 100 Mbps, full-duplex. > eth0: Pause is disabled > > > The difference between the non-working system and the working one are as > follows: > - The non-working system has more memory (1G opposed to 512M) > - A dual SCSI controller is present in this system. Connected to it are two > external 72G disks and an Exabyte Mammoth tapedrive. > 01:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c875 (rev 14) > 01:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c875 (rev 14) > - The following kernel modules are loaded on this system and not on the > working > one: sym53c8xx, sd_mod, st, nfsd, md_mod and raid1 plus of course their > respective dependencies. > > The problem is exactly the same with both Gentoo-patched kernels and vanilla > kernels, that's why I am reporting it here. > > > Steps to reproduce: > > Just boot an affected kernel and try to use eth0 in any way. >
Hi Jos, Could you add to this report some more information, like: dmesg, lspci -vvv, cat /proc/interrupts, .config from problematic box with working (2.6.22.9) and not working kernels, and from the other box with newer kernel? Thanks, Jarek P.
With pleasure. I'll work on it right away. The non-working box is called 'seraphim'. It currently runs a vanilla 2.6.22.9 kernel, the last one that did work. The working box is called 'succubus'. It currently runs a kernel with Gentoo patches, 2.6.24-gentoo-r3. As it is a lot of output, I'll just tarball the info and create attachments. I'll also compile vanilla 2.6.24.4 for both boxen, but that takes a while so it may be 2-3 hours before I can get the info from that.
Created attachment 15512 [details] Working box - 2.6.24-gentoo-r3 kernel This is the output from the working system, "succubus", running a kernel with Gentoo patches, 2.6.24-gentoo-r3.
Created attachment 15513 [details] Non-working box, with last working kernel 2.6.22.9 This is the output from the non-working system, "succubus", with a vanilla 2.6.22.9 kernel, the last kernel that did work.
Created attachment 15514 [details] Non-working box, with broken kernel 2.6..24-gentoo-r3 This is the output from the non-working box, "seraphim", with a gentoo-patched kernel in which the problem occurs, 2.6.24-gentoo-r3.
> As it is a lot of output, I'll just tarball the info and create attachments. > I'll also compile vanilla 2.6.24.4 for both boxen, but that takes a while so > it > may be 2-3 hours before I can get the info from that. Since it waited "a little bit" probably no need to hurry... If there is some private information in dmesg, mask it with some visible pattern, but don't cut too much, especially around some warnings. Jarek P.
> As it is a lot of output, I'll just tarball the info and create attachments. > I'll also compile vanilla 2.6.24.4 for both boxen, but that takes a while so > it > may be 2-3 hours before I can get the info from that. I'm not sure how much such tests could be a problem, e.g. if it's a production box, then let me know. For now I can't see anything wrong, but I wonder if there could be less changes in seraphim's .config after upgrade? Anyway, one of the biggest changes from 2.6.23 was a scheduler, and it would be better to exclude it at the beginning, so could you try to test network with as little processes/services on as possible, and after setting this: echo 1 > /proc/sys/kernel/sched_compat_yield (I mean reloading sungem modules, ifconfig and some ping tests.)
It's a personal machine. Testing is a bit of a hassle and takes the wiki and image gallery on my person site down as the MySQL database is on this box, but nothing serious. Besides, I'm eager to get this one fixed. :-) I just rebooted the system with a vanilla 2.6.24.4 kernel. The problem stays the same, the device shows up in ifconfig, but all counters remain at zero and no network packets can be send or recieved over this interface. I should perhaps note that IP over FireWire still works. Also, after booting into 2.6.24.4, I did the following: # ifconfig eth0 down # rmmod sungem # echo 1 >/proc/sys/kernel/sched_compat_yield # modprobe sungem Problem remains the same. I'll upload the info for the 2.6.24.4 kernel in a few.
Created attachment 15519 [details] Working machine, vanilla 2.6.24.4 kernel This is the info from the working system, "succubus", running a vanilla 2.6.24.4 kernel.
Created attachment 15520 [details] Non-working machine, vanilla 2.6.24.4 kernel This is the information from the non-working machine running a vanilla 2.6.24.4 kernel. The bug is present here.
I presume there are no warnings or error messages, except this WATCHDOG time out, in log files either? Maybe you could try something like this?: to boot it with a kernel as similar to succubus's as possible, so without this additional hardware modules compiled and services started, and before such a timeout try something like: strace -o strace.log ping -n -c3 some.ip and add the log here? Another alternative will be probably a git bisect or something similar: so trying kerenel version between working and not working, eg. 2.6.23-rc1 etc.
I've scanned the message file, there is one persistent error from ntpd, generated about once every 15 minutes: Mar 31 12:16:41 seraphim ntpd[4522]: frequency error 512 PPM exceeds tolerance 5 00 PPM Mar 31 12:29:35 seraphim ntpd[4522]: synchronized to 192.168.0.2, stratum 3 Mar 31 12:31:44 seraphim ntpd[4522]: time reset +0.565469 s The other, working machine does not (currently at least) generate the frequency errors, although it does have the time resets. This is a hardware issue that I have known for some time: the Blade 100 has an internal clock that runs a little too fast, ~300 PPM, causing it to advance ~1.5 seconds every hour. http://people.spacelabs.nl/~admar/usb100faq/76.html While disabling services can be done, that is a major hassle which I prefer to avoid unless there really are no other options left. I will first try the bisecting route. Compiling a 2.6.23-rc1 kernel now, I should have some results soon.
> While disabling services can be done, that is a major hassle which I prefer > to > avoid unless there really are no other options left. I will first try the > bisecting route. Compiling a 2.6.23-rc1 kernel now, I should have some > results > soon. If bisecting isn't a problem the git version should be prefered as it leads to detailed patch which could be responsible. Anyway, I have some doubts about your config: it should be as little changed between versions as possible during make oldconfig. But I see eg. auth_rpcgss module on seraphim after upgrade, and not elsewhere. Such details could matter. BTW, could you check it without nfsd present at all?
Mm, the .config for the 2.6.23 and 2.6.24 kernels was made by just using 'make oldconfig' and answering the questions to the best of my ability (the default answer in most cases). I did not manually add or remove anything. Also, I did once try just copying the .config from the working system and just adding the modules I needed, as the hardware is the same except for that dual SCSI controller (sym53c8xx) and the attached disks and tape drive. It didn't work in that case either. However, now that I think about it, I did have a 'working' 2.6.23 kernel once. A bug in the new IEEE1394 stack caused the kernel to oops and not load some modules during boot. I did have a working network then, although pretty much everything else did not work. :-) Could it be that some other module is in the way? The sym53c8xx module perhaps? As for testing with nfsd removed, that I can do. I'll work on it right away.
> This is a hardware issue that I have known for some time: the Blade 100 has > an > internal clock that runs a little too fast, ~300 PPM, causing it to advance > ~1.5 seconds every hour. And if we are talking about time, probably setting off config options like these for one test could also help to simplify this problem a little: CONFIG_NO_HZ CONFIG_HIGH_RES_TIMERS CONFIG_CPU_FREQ
> ------- Comment #15 from seraph@xs4all.nl 2008-03-31 04:43 ------- > Mm, the .config for the 2.6.23 and 2.6.24 kernels was made by just using > 'make > oldconfig' and answering the questions to the best of my ability (the default > answer in most cases). I did not manually add or remove anything. Yes, make oldconfig isn't enough: it can add some stupid defaults or remove even basic components, so it needs some checking... > > Also, I did once try just copying the .config from the working system and > just > adding the modules I needed, as the hardware is the same except for that dual > SCSI controller (sym53c8xx) and the attached disks and tape drive. It didn't > work in that case either. I try to use make oldconfig after any kernel change (up or down), anyway. > > However, now that I think about it, I did have a 'working' 2.6.23 kernel > once. > A bug in the new IEEE1394 stack caused the kernel to oops and not load some > modules during boot. I did have a working network then, although pretty much > everything else did not work. :-) Could it be that some other module is in > the > way? The sym53c8xx module perhaps? Sure: it can even lockup all kernel so affects all drivers. It can do this in many ways (interrupts, PCI bus, wrong timer or workqueue). So it's allways better to test such things with some minimal config. > As for testing with nfsd removed, that I can do. I'll work on it right away. Since you are using this box for some useful things, and there is a lot of guessing, it would be probably better to gather a few ideas and check them together to limit the number of breaks (I don't hurry you with answering.)
Well, it looks like sym53c8xx is the culprit. I managed to get a very minimal 2.6.24.4 kernel running (make allnoconfig, then just enable the bare minimum needed to make the machine tick). Network device was working fine. Then I compiled modules for sym53c8xx, sd_mod and st, making no other changes than just that. Reboot, and lo and behold, eth0 is unresponsive. I also noticed that after loading sym53c8xx, udev halted for an unusually long time (several seconds), but no messages appeared and it did finally go through.
> ------- Comment #18 from seraph@xs4all.nl 2008-03-31 12:45 ------- > Well, it looks like sym53c8xx is the culprit. > > I managed to get a very minimal 2.6.24.4 kernel running (make allnoconfig, > then > just enable the bare minimum needed to make the machine tick). Network device > was working fine. > > Then I compiled modules for sym53c8xx, sd_mod and st, making no other changes > than just that. Reboot, and lo and behold, eth0 is unresponsive. > > I also noticed that after loading sym53c8xx, udev halted for an unusually > long > time (several seconds), but no messages appeared and it did finally go > through. Congratulations! BTW, I don't know these devices, but I see a change in the option (WRT 2.6.22): CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE. So, it looks like you should try some options around this yet, and if it doesn't help probably a new bug report should be opened: since sungem driver didn't change between 2.6.22 and 2.6.23, then probably sym53c8xx would be the subject of this regression?
Mm, as Alice said, things keep getting weirder and weirder. If I allow udev to load sym53c8xx, the network won't work no matter what I try to do. However, if I blacklist it in /etc/modprobe.d/blacklist, let the machine boot properly and then manually load it afterwards when everything else is running, there are no problems. I guess I will re-file this as a bug in sym53c8xx then.
> Mm, as Alice said, things keep getting weirder and weirder. > > If I allow udev to load sym53c8xx, the network won't work no matter what I > try > to do. However, if I blacklist it in /etc/modprobe.d/blacklist, let the > machine > boot properly and then manually load it afterwards when everything else is > running, there are no problems. > > I guess I will re-file this as a bug in sym53c8xx then. Does this mean there is no problem if sym53c8xx is loaded after sungem, but in 2.6.22 the order didn't matter? I think, it would be nice to recheck this, and if possible to try something more current like 2.6.25-rc7. If it's true then it seems sym53c8xx can do some check while loading and sungem doesn't. If it's true you can alternatively try to mail maintainers first (53c8xx according to MAINTAINERS file, plus probably David Miller, Jeff Garzik and of course linux-kernel@ and netdev@ lists) pointing to this bugzilla report.
I have already filed the new bug (bugid #10374) and was contacted by Andrew Morton, asking me for dmesg output from both the good and the bad kernels. He suspects IRQ routing. But answering your question, udev loads sungem as one of its first modules and sym53c8xx as one of the last, which is the same order that would happen if I do things manually. Anyway, I guess we'll take it from there and see what happens.
You are right, the order does matter. I experimented a little with blacklisting one or both modules in /etc/modprobe.d/blacklist and found the following: The bug is triggered if sym53c8xx is loaded before sungem is. If I load sungem before sym53c8xx, there is no problem at all.
I did bisection. The bad commit is: 5a606b72a4309a656cd1a19ad137dc5557c4b8ea is first bad commit commit 5a606b72a4309a656cd1a19ad137dc5557c4b8ea Author: David S. Miller <davem@sunset.davemloft.net> Date: Mon Jul 9 22:40:36 2007 -0700 [SPARC64]: Do not ACK an INO if it is disabled or inprogress.
Nice work Jos! I forward it to the author and the lists. Jarek P. Andrew Morton wrote, On 03/17/2008 10:36 PM: > On Mon, 17 Mar 2008 14:25:50 -0700 (PDT) > bugme-daemon@bugzilla.kernel.org wrote: > >> http://bugzilla.kernel.org/show_bug.cgi?id=10273 >> >> Summary: Sun GEM (PCI) - network device doesn't work ----- Forwarded message from bugme-daemon@bugzilla.kernel.org ----- > ------- Comment #24 from seraph@xs4all.nl 2008-04-02 07:24 ------- > I did bisection. The bad commit is: > > 5a606b72a4309a656cd1a19ad137dc5557c4b8ea is first bad commit > commit 5a606b72a4309a656cd1a19ad137dc5557c4b8ea > Author: David S. Miller <davem@sunset.davemloft.net> > Date: Mon Jul 9 22:40:36 2007 -0700 > > [SPARC64]: Do not ACK an INO if it is disabled or inprogress. > > > -- > Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email > ------- You are receiving this mail because: ------- > You are on the CC list for the bug, or are watching someone who is. ----- End forwarded message -----
From: Jarek Poplawski <jarkao2@gmail.com> Date: Wed, 2 Apr 2008 20:26:55 +0200 > Nice work Jos! I forward it to the author and the lists. Thanks for tracking this down. I'm very busy currently but I promise I'll try to resolve this somehow.
From: Jarek Poplawski <jarkao2@gmail.com> Date: Wed, 2 Apr 2008 20:26:55 +0200 > Nice work Jos! I forward it to the author and the lists. I haven't forgotten about this bug report, in fact I did some investigation and thinking about it. I think the case being triggered in the new code is IRQ_DISABLED. If this is the problem, I suspect that what needs to happen is that when we re-enable the interrupt we have to forcefully hit the clear register to put it back into transmit state. Otherwise it can get stuck. Please give this patch a try (this is against 2.6.23, which is the version you reported the bug against, let me know if another version is more convenient). Thanks! diff --git a/arch/sparc64/kernel/irq.c b/arch/sparc64/kernel/irq.c index 2395609..98b68d2 100644 --- a/arch/sparc64/kernel/irq.c +++ b/arch/sparc64/kernel/irq.c @@ -313,6 +313,8 @@ static void sun4u_irq_enable(unsigned int virt_irq) IMAP_AID_SAFARI | IMAP_NID_SAFARI); val |= tid | IMAP_VALID; upa_writeq(val, imap); + + upa_writeq(ICLR_IDLE, data->iclr); } }
> Please give this patch a try (this is against 2.6.23, which is the > version you reported the bug against, let me know if another version > is more convenient). I applied the patch against 2.6.24-gentoo-r4, the current kernel on this machine. It worked fine, with only an offset of 12 lines. Nothing unusual during compile either. The first results are hopeful: I am no longer able to trigger this bug by loading sym53c8xx before sungem, nor by having them both loaded by udev. Nothing odd in dmesg and the machine is completely functional. I don't have much time for testing today, but if desired I will gladly do some more testing tomorrow or Sunday.
From: Jos van der Ende <seraph@xs4all.nl> Date: Fri, 25 Apr 2008 21:59:05 +0200 > > Please give this patch a try (this is against 2.6.23, which is the > > version you reported the bug against, let me know if another version > > is more convenient). > > I applied the patch against 2.6.24-gentoo-r4, the current kernel on this > machine. It worked fine, with only an offset of 12 lines. Nothing unusual > during compile either. > > The first results are hopeful: I am no longer able to trigger this bug by > loading sym53c8xx before sungem, nor by having them both loaded by udev. > Nothing odd in dmesg and the machine is completely functional. > > > I don't have much time for testing today, but if desired I will gladly do > some more testing tomorrow or Sunday. Thanks for testing the change, I'll do some of my own testing and push the bug fix upstream. Thanks again!
fixed by commit 227c3311786dbe64cb221e63d53817f98240e587
I had the same bug on Debian Lenny with kernel 2.6.24-1 on Sun Blade 150. I solved it by blacklisting the sunhme module (I've recently removed the hme card so I don't need it) and regenerating initramfs.