Checked (failing) kernel: 2.6.29-rc1 Latest working kernel version: unknown Earliest failing kernel version: not checked but probably any. RHEL5 kernels are also affected. Distribution: Enterprise Linux Enterprise Linux Server release 5.1 (Carthage) Hardware Environment: lspci: 15:00.0 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit Mezzanine Adapter (rev 06) 15:00.1 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit Mezzanine Adapter (rev 06) 15:00.0 0200: 8086:10da (rev 06) Subsystem: 103c:1717 Flags: bus master, fast devsel, latency 0, IRQ 154 Memory at fdde0000 (32-bit, non-prefetchable) [size=128K] Memory at fdd00000 (32-bit, non-prefetchable) [size=512K] I/O ports at 6000 [size=32] [virtual] Expansion ROM at d1300000 [disabled] [size=512K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Capabilities: [e0] Express Endpoint IRQ 0 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 15:00.1 0200: 8086:10da (rev 06) Subsystem: 103c:1717 Flags: bus master, fast devsel, latency 0, IRQ 162 Memory at fdce0000 (32-bit, non-prefetchable) [size=128K] Memory at fdc00000 (32-bit, non-prefetchable) [size=512K] I/O ports at 6020 [size=32] [virtual] Expansion ROM at d1380000 [disabled] [size=512K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Capabilities: [e0] Express Endpoint IRQ 0 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 Problem Description: Bonding does not work over NICs supported by e1000e: if you brake/restore physical links of bonding slaves one by one - network won't work anymore. Steps to reproduce: 2 NICs supported by e1000e put into bond device (Bonding Mode: fault-tolerance (active-backup)). * ping to the outside node is ok * physically brake the link of active bond slave (1) * bond detects the failure, makes another slave (2) active. * ping works fine * restore the connection of (1) * ping works fine * brake the link of (2) * bond detects it, reports that it makes active (1), but * ping _does not_ work anymore Logs: /var/log/messages: Jan 27 11:53:29 host kernel: 0000:15:00.0: eth2: Link is Down Jan 27 11:53:29 host kernel: bonding: bond1: link status definitely down for interface eth2, disabling it Jan 27 11:53:29 host kernel: bonding: bond1: making interface eth3 the new active one. Jan 27 11:56:37 host kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jan 27 11:56:37 host kernel: bonding: bond1: link status definitely up for interface eth2. Jan 27 11:57:39 host kernel: 0000:15:00.1: eth3: Link is Down Jan 27 11:57:39 host kernel: bonding: bond1: link status definitely down for interface eth3, disabling it Jan 27 11:57:39 host kernel: bonding: bond1: making interface eth2 the new active one. What was done + dumps of /proc/net/bonding/bond1: ## 11:52:42 ##cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1e ## 11:53:05 shutdown eth2 uplink on the virtual connect bay5 ##cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1e ## 11:56:01 turn on eth2 uplink on the virtual connect bay5 ##cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1e ## 11:57:22 turn off eth3 uplink on the virtual connect bay5 ##cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1e
Reply-To: akpm@linux-foundation.org (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Thu, 29 Jan 2009 03:12:01 -0800 (PST) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=12570 > > Summary: Bonding does not work over e1000e. > Product: Drivers > Version: 2.5 > KernelVersion: 2.6.29-rc1 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Network > AssignedTo: jgarzik@pobox.com > ReportedBy: khorenko@parallels.com > > > Checked (failing) kernel: 2.6.29-rc1 > Latest working kernel version: unknown > Earliest failing kernel version: not checked but probably any. RHEL5 kernels > are also affected. > > Distribution: Enterprise Linux Enterprise Linux Server release 5.1 (Carthage) > > Hardware Environment: > lspci: > 15:00.0 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit > Mezzanine Adapter (rev 06) > 15:00.1 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit > Mezzanine Adapter (rev 06) > > 15:00.0 0200: 8086:10da (rev 06) > Subsystem: 103c:1717 > Flags: bus master, fast devsel, latency 0, IRQ 154 > Memory at fdde0000 (32-bit, non-prefetchable) [size=128K] > Memory at fdd00000 (32-bit, non-prefetchable) [size=512K] > I/O ports at 6000 [size=32] > [virtual] Expansion ROM at d1300000 [disabled] [size=512K] > Capabilities: [c8] Power Management version 2 > Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 > Enable+ > Capabilities: [e0] Express Endpoint IRQ 0 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 > > 15:00.1 0200: 8086:10da (rev 06) > Subsystem: 103c:1717 > Flags: bus master, fast devsel, latency 0, IRQ 162 > Memory at fdce0000 (32-bit, non-prefetchable) [size=128K] > Memory at fdc00000 (32-bit, non-prefetchable) [size=512K] > I/O ports at 6020 [size=32] > [virtual] Expansion ROM at d1380000 [disabled] [size=512K] > Capabilities: [c8] Power Management version 2 > Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 > Enable+ > Capabilities: [e0] Express Endpoint IRQ 0 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 > > Problem Description: Bonding does not work over NICs supported by e1000e: if > you brake/restore physical links of bonding slaves one by one - network won't > work anymore. > > Steps to reproduce: > 2 NICs supported by e1000e put into bond device (Bonding Mode: > fault-tolerance > (active-backup)). > * ping to the outside node is ok > * physically brake the link of active bond slave (1) > * bond detects the failure, makes another slave (2) active. > * ping works fine > * restore the connection of (1) > * ping works fine > * brake the link of (2) > * bond detects it, reports that it makes active (1), but > * ping _does not_ work anymore > > Logs: > /var/log/messages: > Jan 27 11:53:29 host kernel: 0000:15:00.0: eth2: Link is Down > Jan 27 11:53:29 host kernel: bonding: bond1: link status definitely down for > interface eth2, disabling it > Jan 27 11:53:29 host kernel: bonding: bond1: making interface eth3 the new > active one. > Jan 27 11:56:37 host kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps Full > Duplex, Flow Control: RX/TX > Jan 27 11:56:37 host kernel: bonding: bond1: link status definitely up for > interface eth2. > Jan 27 11:57:39 host kernel: 0000:15:00.1: eth3: Link is Down > Jan 27 11:57:39 host kernel: bonding: bond1: link status definitely down for > interface eth3, disabling it > Jan 27 11:57:39 host kernel: bonding: bond1: making interface eth2 the new > active one. > > What was done + dumps of /proc/net/bonding/bond1: > ## 11:52:42 > ##cat /proc/net/bonding/bond1 > Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth2 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 0 > Down Delay (ms): 0 > > Slave Interface: eth2 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:77:00:1c > > Slave Interface: eth3 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:77:00:1e > > ## 11:53:05 shutdown eth2 uplink on the virtual connect bay5 > ##cat /proc/net/bonding/bond1 > Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth3 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 0 > Down Delay (ms): 0 > > Slave Interface: eth2 > MII Status: down > Link Failure Count: 1 > Permanent HW addr: 00:17:a4:77:00:1c > > Slave Interface: eth3 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:77:00:1e > > ## 11:56:01 turn on eth2 uplink on the virtual connect bay5 > ##cat /proc/net/bonding/bond1 > Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth3 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 0 > Down Delay (ms): 0 > > Slave Interface: eth2 > MII Status: down > Link Failure Count: 1 > Permanent HW addr: 00:17:a4:77:00:1c > > Slave Interface: eth3 > MII Status: up > Link Failure Count: 0 > Permanent HW addr: 00:17:a4:77:00:1e > > ## 11:57:22 turn off eth3 uplink on the virtual connect bay5 > ##cat /proc/net/bonding/bond1 > Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) > > Bonding Mode: fault-tolerance (active-backup) > Primary Slave: None > Currently Active Slave: eth2 > MII Status: up > MII Polling Interval (ms): 100 > Up Delay (ms): 0 > Down Delay (ms): 0 > > Slave Interface: eth2 > MII Status: up > Link Failure Count: 1 > Permanent HW addr: 00:17:a4:77:00:1c > > Slave Interface: eth3 > MII Status: down > Link Failure Count: 1 > Permanent HW addr: 00:17:a4:77:00:1e >
Reply-To: david.graham@intel.com Hi Konstantin. I have been trying but so far been failed to reproduce the reported problem. I have a few questions. 1) While I can't repro your problem, I can see something very similar if I don't load with module load parameter miimon=100. From your /proc/net/bonding/bond1 dumps it looks like you do the right thing, but would you please confirm for me by listing exactly which bonding module params you load with. 2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 still shows MII status *down*, which would be incorrect. Can you confirm that this snippet of the file really is in the correct place in the reported sequence - that is, there is already a problem at this step, and that's where we should look for the problem. 3) Could you send me your network scripts for the two slaves and the bonding interface itself (on RH systems, I think that's /etc/sysconfig/networking-scripts/ifcfg-*. They should be modeled on the sample info in <kernel>Documentation/networking/bonding.txt, and I'd like to check them. 4) I'm probably not controlling the slave link state in the same way that you are, because in the NEC bladeserver that I'm using, I am bringing the e1000e link-partern ports up & down by using an admins console, using SW I don't understand. I (so far) have not been able to physically disconnect one of the (serdes) links connecting the 82571 without also disconnecting the other, as I have to pull an entire switch module to make the disconnect. Can you give me mmore information on what your system is, and how you can physically disconnect on link at a time. Then I might be able to get hold of a similar setup and see the problem. 5) I have been testing on 2.6.29-rc3 and on 2.6.28 kernels, not 2.6.29-rc1 which is what you reported the problem on. I think its unlikely that the problem is only on the 2.6.29-rc1 build, but would like to know if you've had a chance to try any other build, and what the results were. Also please let me know if you have tested with any other non-INTEL 1GB interfaces, and if you have *ever* seen bonding work properly on the system you are testing. 6) While I can't repro your issue yet, I have made some changes very recently to the serdes link detect logic in the e1000e driver. They were written to address a separate issue, and are actually NOT in the kernel that you have been testing. I also can't see how fixing that problem might fix your problem. However, because the fixes do concern serdes link detection, and so does yours, it's probably worth a (long) shot. If you are comfortable trying them out, I have attached them to this email. They are also being queued for upstream, but only after some further local testing. Thanks Dave >-----Original Message----- >From: Andrew Morton [mailto:akpm@linux-foundation.org] >Sent: Thursday, January 29, 2009 9:53 AM >To: netdev@vger.kernel.org; e1000-devel@lists.sourceforge.net; bonding- >devel@lists.sourceforge.net >Cc: khorenko@parallels.com; bugme-daemon@bugzilla.kernel.org >Subject: Re: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not >work over e1000e. > > >(switched to email. Please respond via emailed reply-to-all, not via the >bugzilla web interface). > >On Thu, 29 Jan 2009 03:12:01 -0800 (PST) bugme-daemon@bugzilla.kernel.org >wrote: > >> http://bugzilla.kernel.org/show_bug.cgi?id=12570 >> >> Summary: Bonding does not work over e1000e. >> Product: Drivers >> Version: 2.5 >> KernelVersion: 2.6.29-rc1 >> Platform: All >> OS/Version: Linux >> Tree: Mainline >> Status: NEW >> Severity: normal >> Priority: P1 >> Component: Network >> AssignedTo: jgarzik@pobox.com >> ReportedBy: khorenko@parallels.com >> >> >> Checked (failing) kernel: 2.6.29-rc1 >> Latest working kernel version: unknown >> Earliest failing kernel version: not checked but probably any. RHEL5 >kernels >> are also affected. >> >> Distribution: Enterprise Linux Enterprise Linux Server release 5.1 >(Carthage) >> >> Hardware Environment: >> lspci: >> 15:00.0 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit >> Mezzanine Adapter (rev 06) >> 15:00.1 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit >> Mezzanine Adapter (rev 06) >> >> 15:00.0 0200: 8086:10da (rev 06) >> Subsystem: 103c:1717 >> Flags: bus master, fast devsel, latency 0, IRQ 154 >> Memory at fdde0000 (32-bit, non-prefetchable) [size=128K] >> Memory at fdd00000 (32-bit, non-prefetchable) [size=512K] >> I/O ports at 6000 [size=32] >> [virtual] Expansion ROM at d1300000 [disabled] [size=512K] >> Capabilities: [c8] Power Management version 2 >> Capabilities: [d0] Message Signalled Interrupts: 64bit+ >Queue=0/0 >> Enable+ >> Capabilities: [e0] Express Endpoint IRQ 0 >> Capabilities: [100] Advanced Error Reporting >> Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 >> >> 15:00.1 0200: 8086:10da (rev 06) >> Subsystem: 103c:1717 >> Flags: bus master, fast devsel, latency 0, IRQ 162 >> Memory at fdce0000 (32-bit, non-prefetchable) [size=128K] >> Memory at fdc00000 (32-bit, non-prefetchable) [size=512K] >> I/O ports at 6020 [size=32] >> [virtual] Expansion ROM at d1380000 [disabled] [size=512K] >> Capabilities: [c8] Power Management version 2 >> Capabilities: [d0] Message Signalled Interrupts: 64bit+ >Queue=0/0 >> Enable+ >> Capabilities: [e0] Express Endpoint IRQ 0 >> Capabilities: [100] Advanced Error Reporting >> Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00 >> >> Problem Description: Bonding does not work over NICs supported by >e1000e: if >> you brake/restore physical links of bonding slaves one by one - network >won't >> work anymore. >> >> Steps to reproduce: >> 2 NICs supported by e1000e put into bond device (Bonding Mode: fault- >tolerance >> (active-backup)). >> * ping to the outside node is ok >> * physically brake the link of active bond slave (1) >> * bond detects the failure, makes another slave (2) active. >> * ping works fine >> * restore the connection of (1) >> * ping works fine >> * brake the link of (2) >> * bond detects it, reports that it makes active (1), but >> * ping _does not_ work anymore >> >> Logs: >> /var/log/messages: >> Jan 27 11:53:29 host kernel: 0000:15:00.0: eth2: Link is Down >> Jan 27 11:53:29 host kernel: bonding: bond1: link status definitely down >for >> interface eth2, disabling it >> Jan 27 11:53:29 host kernel: bonding: bond1: making interface eth3 the >new >> active one. >> Jan 27 11:56:37 host kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps >Full >> Duplex, Flow Control: RX/TX >> Jan 27 11:56:37 host kernel: bonding: bond1: link status definitely up >for >> interface eth2. >> Jan 27 11:57:39 host kernel: 0000:15:00.1: eth3: Link is Down >> Jan 27 11:57:39 host kernel: bonding: bond1: link status definitely down >for >> interface eth3, disabling it >> Jan 27 11:57:39 host kernel: bonding: bond1: making interface eth2 the >new >> active one. >> >> What was done + dumps of /proc/net/bonding/bond1: >> ## 11:52:42 >> ##cat /proc/net/bonding/bond1 >> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) >> >> Bonding Mode: fault-tolerance (active-backup) >> Primary Slave: None >> Currently Active Slave: eth2 >> MII Status: up >> MII Polling Interval (ms): 100 >> Up Delay (ms): 0 >> Down Delay (ms): 0 >> >> Slave Interface: eth2 >> MII Status: up >> Link Failure Count: 0 >> Permanent HW addr: 00:17:a4:77:00:1c >> >> Slave Interface: eth3 >> MII Status: up >> Link Failure Count: 0 >> Permanent HW addr: 00:17:a4:77:00:1e >> >> ## 11:53:05 shutdown eth2 uplink on the virtual connect bay5 >> ##cat /proc/net/bonding/bond1 >> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) >> >> Bonding Mode: fault-tolerance (active-backup) >> Primary Slave: None >> Currently Active Slave: eth3 >> MII Status: up >> MII Polling Interval (ms): 100 >> Up Delay (ms): 0 >> Down Delay (ms): 0 >> >> Slave Interface: eth2 >> MII Status: down >> Link Failure Count: 1 >> Permanent HW addr: 00:17:a4:77:00:1c >> >> Slave Interface: eth3 >> MII Status: up >> Link Failure Count: 0 >> Permanent HW addr: 00:17:a4:77:00:1e >> >> ## 11:56:01 turn on eth2 uplink on the virtual connect bay5 >> ##cat /proc/net/bonding/bond1 >> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) >> >> Bonding Mode: fault-tolerance (active-backup) >> Primary Slave: None >> Currently Active Slave: eth3 >> MII Status: up >> MII Polling Interval (ms): 100 >> Up Delay (ms): 0 >> Down Delay (ms): 0 >> >> Slave Interface: eth2 >> MII Status: down >> Link Failure Count: 1 >> Permanent HW addr: 00:17:a4:77:00:1c >> >> Slave Interface: eth3 >> MII Status: up >> Link Failure Count: 0 >> Permanent HW addr: 00:17:a4:77:00:1e >> >> ## 11:57:22 turn off eth3 uplink on the virtual connect bay5 >> ##cat /proc/net/bonding/bond1 >> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) >> >> Bonding Mode: fault-tolerance (active-backup) >> Primary Slave: None >> Currently Active Slave: eth2 >> MII Status: up >> MII Polling Interval (ms): 100 >> Up Delay (ms): 0 >> Down Delay (ms): 0 >> >> Slave Interface: eth2 >> MII Status: up >> Link Failure Count: 1 >> Permanent HW addr: 00:17:a4:77:00:1c >> >> Slave Interface: eth3 >> MII Status: down >> Link Failure Count: 1 >> Permanent HW addr: 00:17:a4:77:00:1e >> > > >-------------------------------------------------------------------------- >---- >This SF.net email is sponsored by: >SourcForge Community >SourceForge wants to tell your story. >http://p.sf.net/sfu/sf-spreadtheword >_______________________________________________ >E1000-devel mailing list >E1000-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/e1000-devel
Hello Dave, thank you for investigating this, i'll try to answer your questions: > 1) ... if I don't load with module load parameter miimon=100. ... would you > please confirm for me by listing exactly which bonding module params you load > with. Yes, surely, /etc/modprobe.conf contains "miimon=100" parameter for all bonding devices on the node, and you are right, if i remove "miimon=100", /proc/net/bonding/bondX will report "MII Polling Interval (ms): 0". # cat etc/modprobe.conf alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000e alias eth3 e1000e alias scsi_hostadapter cciss alias scsi_hostadapter1 lpfc alias scsi_hostadapter2 usb-storage alias bond0 bonding options bond0 miimon=100 mode=1 max_bonds=2 alias bond1 bonding options bond1 miimon=100 mode=1 # Disable IPV6 alias net-pf-10 off alias ipv6 off options ip_conntrack ip_conntrack_disable_ve0=1 ###################################################################### > 2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect > bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 > still shows MII status *down*, which would be incorrect. Can you confirm that > this snippet of the file really is in the correct place in the reported > sequence - that is, there is already a problem at this step, and that's where > we should look for the problem. Well, yes, the order is correct and you noticed right, but: /var/log/messages were saved on other server so there might be a bit mistiming, but estimate it: Here is the mix of messages from console and taken from /var/log/messages: ---------- ## 11:53:05 shutdown eth2 uplink on the virtual connect bay5 Jan 27 11:53:29 sdencltst01blade12 kernel: 0000:15:00.0: eth2: Link is Down ## 11:56:01 turn on eth2 uplink on the virtual connect bay5 Jan 27 11:56:37 sdencltst01blade12 kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX ## 11:57:22 turn off eth3 uplink on the virtual connect bay5 Jan 27 11:57:39 sdencltst01blade12 kernel: 0000:15:00.1: eth3: Link is Down ---------- Deltas are: 24 sec, 36 sec and 17 secs. (i mean the reaction that we can see in /var/log/messages on action from console). The thus timing is different in no more than 17 secs, then we have: 7 secs, 19 secs and 0 secs deltas. Well, may be 7 and 19 secs for link detection is too much, but i cannot blame for that, it works at least. But surely if you think this is a problem, let's also focus on it. (BTW, RHEL5 kernel does not provide correct link status at all, e.g. /proc/net/bonding/bond1 still shows eth2's mii status is UP if we unplug eth2 link.) > 3) .../etc/sysconfig/networking-scripts/ifcfg-*.... ifcfg-eth2: DEVICE=eth2 BOOTPROTO=none ONBOOT=yes MASTER=bond1 SLAVE=yes USERCTR=no HWADDR=00:1E:0B:93:D2:E2 ifcfg-bond1: DEVICE=bond1 BOOTPROTO=none ONBOOT=yes USERCTL=no IPADDR=xxx NETMASK=xxx GATEWAY=xxx 4) physical links managing HP C7000 blade system with the Virtual Connect switch modules for the server's uplink is used. The Onboard Administrator for the C7000 is used to power off the switch to simulate the uplink failure for this test. Yes, the powering off switches off all links connected to the switch, but network cards are connected to different switches - thus we can disconnect links one by one. > 5) ... if you have *ever* seen bonding work properly on the system you are > testing. Yes, we've seen. The node has 2 Broadcom network cards (eth0 and eth1) which are in bond0 the same node - bonding works fine there. Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 0200: 14e4:16ac (rev 12) Subsystem: 103c:703b 6) test kernel with additional patches. Thank you for the patches, i'll let you know as soon as the kernel with them will be tested. Thank you! -- Best regards, Konstantin Khorenko, Virtuozzo/OpenVZ developer, Parallels On 02/03/2009 04:42 AM, Graham, David wrote: > Hi Konstantin. > > I have been trying but so far been failed to reproduce the reported problem. > I have a few questions. > > 1) While I can't repro your problem, I can see something very similar if I > don't load with module load parameter miimon=100. From your > /proc/net/bonding/bond1 dumps it looks like you do the right thing, but would > you please confirm for me by listing exactly which bonding module params you > load with. > > 2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect > bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 > still shows MII status *down*, which would be incorrect. Can you confirm that > this snippet of the file really is in the correct place in the reported > sequence - that is, there is already a problem at this step, and that's where > we should look for the problem. > > 3) Could you send me your network scripts for the two slaves and the bonding > interface itself (on RH systems, I think that's > /etc/sysconfig/networking-scripts/ifcfg-*. They should be modeled on the > sample info in <kernel>Documentation/networking/bonding.txt, and I'd like to > check them. > > 4) I'm probably not controlling the slave link state in the same way that you > are, because in the NEC bladeserver that I'm using, I am bringing the e1000e > link-partern ports up & down by using an admins console, using SW I don't > understand. I (so far) have not been able to physically disconnect one of the > (serdes) links connecting the 82571 without also disconnecting the other, as > I have to pull an entire switch module to make the disconnect. Can you give > me mmore information on what your system is, and how you can physically > disconnect on link at a time. Then I might be able to get hold of a similar > setup and see the problem. > > 5) I have been testing on 2.6.29-rc3 and on 2.6.28 kernels, not 2.6.29-rc1 > which is what you reported the problem on. I think its unlikely that the > problem is only on the 2.6.29-rc1 build, but would like to know if you've had > a chance to try any other build, and what the results were. Also please let > me know if you have tested with any other non-INTEL 1GB interfaces, and if > you have *ever* seen bonding work properly on the system you are testing. > > 6) While I can't repro your issue yet, I have made some changes very recently > to the serdes link detect logic in the e1000e driver. They were written to > address a separate issue, and are actually NOT in the kernel that you have > been testing. I also can't see how fixing that problem might fix your > problem. However, because the fixes do concern serdes link detection, and so > does yours, it's probably worth a (long) shot. If you are comfortable trying > them out, I have attached them to this email. They are also being queued for > upstream, but only after some further local testing. > > Thanks > Dave
Konstantin, To get closer to your environment, I reconfigured my network, and same kernel & built-in driver that you used, but channel failover still works in my tests. Because this is without the recent serdes link patches that I referred to earlier, that means I don't expect them to be significant to the problem. But there are some very significant differences in our setups, and I want to align my configuration closer to yours. 1) I am using a different Mezz card, with different EEPROM settings (and so features). Could you please send me "ethtool ethx" and "ethtool -e ethx" settings for the problem interfaces ? I may even spot something incorrect in the programming, but if not, I can probably use all or some of your content to make my card behave more like yours. 2) We have different link parters , and disable link in a different way. I tried to remove the switch modules as you did, but in my bladeserver system, couldn't. There must be some administrative command to allow the latch to unlock, but I am not familiar with it. I'll keep looking. Do you have the same (failing) result if you take the link partners down administratively from the switch console ? 3) I am testing in a different chassis/backplane Let's address the simpler differences first, but if we go another round or two without being able to figure this out, and you are prepared to send us one of the systems with the problem for definite root cause analysis, you can contact me off-line from this bz and we'll work the detail. Dave FYI: here's more info & log that show's how the failover works OK on my system. 2.6.29-rc1 blade in bladeserver Ping from console | +--------+ | bond0 | static address ++------++ | | +---+--+ ++-----+ | eth2 | | eth3 | +---+--+ ++-----+ | | Serdes Backplane | | +---+--+ ++-----+ | 5/4 | | 6/4 | Bladeserver wwitch module/port +---+--+ ++-----+ | | +--+------+----+ | 1GB switch | External to bladeserver +-----+--------+ | +-----+-------+ | ping target | +-------------+ Linux localhost.localdomain 2.6.29-rc1 #3 SMP Fri Feb 13 21:31:17 EST 2009 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# ethtool -i eth2 driver: e1000e version: 0.3.3.3-k6 firmware-version: 5.6-2 bus-info: 0000:07:00.0 [root@localhost ~]# 07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) Subsystem: NEC Corporation Unknown device 834c Flags: bus master, fast devsel, latency 0, IRQ 59 Memory at dee20000 (32-bit, non-prefetchable) [size=128K] Memory at dee00000 (32-bit, non-prefetchable) [size=128K] I/O ports at 4000 [size=32] [virtual] Expansion ROM at 50200000 [disabled] [size=128K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Capabilities: [e0] Express Endpoint IRQ 0 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number d8-25-62-ff-ff-97-16-00 00: 86 80 60 10 47 05 10 00 06 00 00 02 08 00 80 00 10: 00 00 e2 de 00 00 e0 de 01 40 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 33 10 4c 83 30: 00 00 00 00 c8 00 00 00 00 00 00 00 04 01 00 00 07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) Subsystem: NEC Corporation Unknown device 834c Flags: bus master, fast devsel, latency 0, IRQ 60 Memory at dee60000 (32-bit, non-prefetchable) [size=128K] Memory at dee40000 (32-bit, non-prefetchable) [size=128K] I/O ports at 4020 [size=32] [virtual] Expansion ROM at 50220000 [disabled] [size=128K] Capabilities: [c8] Power Management version 2 Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+ Capabilities: [e0] Express Endpoint IRQ 0 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number d8-25-62-ff-ff-97-16-00 00: 86 80 60 10 47 05 10 00 06 00 00 02 08 00 80 00 10: 00 00 e6 de 00 00 e4 de 21 40 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 33 10 4c 83 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0c 02 00 00 Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:16:97:62:25:d8 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:16:97:62:25:d9 // Started a ping to 10.0.0.7, all OK, & TX goes out // physical eth2 // And then I took eth2's link partner out of service from the admin console at the connected switch port 5/4 , [command = oper/port 4/dis] Feb 17 21:36:15 localhost kernel: e1000e: eth2 NIC Link is Down Feb 17 21:36:15 localhost kernel: bonding: bond0: link status definitely down for interface eth2, disabling it Feb 17 21:36:15 localhost kernel: bonding: bond0: making interface eth3 the new active one. (END) Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d8 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:16:97:62:25:d9 // Restored the link from switch 5/4 (eth2) // Feb 17 21:38:29 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Feb 17 21:38:29 localhost kernel: bonding: bond0: link status definitely up for interface eth2. Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d8 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:16:97:62:25:d9 // All still OK, with pings TX out of eth3. Now take down the link at eth3, // switch module 6 port 4 Feb 17 21:41:41 localhost kernel: e1000e: eth3 NIC Link is Down Feb 17 21:41:41 localhost kernel: bonding: bond0: link status definitely down for interface eth3, disabling it Feb 17 21:41:41 localhost kernel: bonding: bond0: making interface eth2 the new active one. Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d8 Slave Interface: eth3 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d9 // Now restore link 3 Feb 17 21:42:56 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Feb 17 21:42:56 localhost kernel: bonding: bond0: link status definitely up for interface eth3. Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d8 Slave Interface: eth3 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:16:97:62:25:d9 // The ping continued in the background throughout.
constantin any reply?
Hello David, sorry for the huge delay, i'll try to answer your questions below. On 02/17/2009 10:00 PM, Graham, David wrote: > To get closer to your environment, I reconfigured my network, and same kernel > & built-in driver that you used, but channel failover still works in my > tests. Because this is without the recent serdes link patches that I referred > to earlier, that means I don't expect them to be significant to the problem. Unfortunately i don't have a direct access to the problematic node thus this takes so long time. Yesterday at last the kernel 2.6.29-rc4 + patches SerdesSM.patch, disable_dmaclkgating.patch and RemoveRXSEQ.patch was tested and it works fine for the failback! [root@hostname ~]# uname -a Linux hostname 2.6.29-rc4.e1000e.ver1 #1 SMP Tue Feb 10 20:47:26 MSK 2009 x86_64 x86_64 x86_64 GNU/Linux [root@hostname ~]# ##Took down the icbay5 uplink [root@hostname ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1e ##Enable icbay5 uplink [root@hostname ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:00:1e ## disable icbay6 (this is still working!!! It used to die right here. [root@hostname ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1c Slave Interface: eth3 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:00:1e > But there are some very significant differences in our setups, and I want to > align my configuration closer to yours. > 1) I am using a different Mezz card, with different EEPROM settings (and so > features). Could you please send me "ethtool ethx" and "ethtool -e ethx" > settings for the problem interfaces ? I may even spot something incorrect in > the programming, but if not, I can probably use all or some of your content > to make my card behave more like yours. i've attached ethtool.info.gz file with the following commands output: # ethtool eth2 # ethtool eth3 # ethtool -e eth2 # ethtool -e eth3 # ethtool -i eth2 # ethtool -i eth3 > 2) We have different link parters , and disable link in a different way. > I tried to remove the switch modules as you did, but in my bladeserver > system, couldn't. There must be some administrative command to allow the > latch to unlock, but I am not familiar with it. I'll keep looking. Do you > have the same (failing) result if you take the link partners down > administratively from the switch console ? Yes. The same failure occurs if we admin down the switch from the virtual connect. But we have to do it for the whole switch. Virtual Connect doesn't allow us to disable just one single port. > FYI: here's more info & log that show's how the failover works OK on my > system. > > 2.6.29-rc1 blade in bladeserver > Ping from console > | > +--------+ > | bond0 | static address > ++------++ > | | > +---+--+ ++-----+ > | eth2 | | eth3 | > +---+--+ ++-----+ > | | Serdes Backplane > | | > +---+--+ ++-----+ > | 5/4 | | 6/4 | Bladeserver wwitch module/port > +---+--+ ++-----+ > | | > +--+------+----+ > | 1GB switch | External to bladeserver > +-----+--------+ > | > +-----+-------+ > | ping target | > +-------------+ This is the same topology of our set up. We use the HP C7000 server chassis with the Virtual Connect enet-F module. > 3) I am testing in a different chassis/backplane > Let's address the simpler differences first, but if we go another round or > two without being able to figure this out, and you are prepared to send us > one of the systems with the problem for definite root cause analysis, you can > contact me off-line from this bz and we'll work the detail. Well, unfortunately sending the system for reproduction does not seem as an option, but if you need/want something to check, we can arrange a WebEx session. Please, let me know if this is needed. Conclusions: well, the latest kernel with your patches does work, thank you very much, David! Now i have to solve my original problem - to make RHEL5-based (2.6.18-x) kernel working. At the moment RHEL5 kernel is affected by 2 issues: 1) that one which seems to be fixed by updating the testkernel from 2.6.29-r1 up to rc4 + 3 your patches. 2) when we break a link, mii status is still reported as "up" in /proc/net/bonding/bond1. (at the same time bonding changes the active slave to the working one correctly). i understand there were a lot of changes since 2.6.18, but i still want to try not to replace the e1000e driver completely from the latest mainstream kernel, but to backport the set of patches to fix this exact issue. Could you please help me pointing the patches that are essential to fix this issue (and probably issue 2)) from your point of view? Thank you very much!
On Tue, Mar 17, 2009 at 10:25 AM, Konstantin Khorenko <khorenko@parallels.com> wrote: > > Conclusions: well, the latest kernel with your patches does work, thank you > very much, David! > Now i have to solve my original problem - to make RHEL5-based (2.6.18-x) > kernel working. > At the moment RHEL5 kernel is affected by 2 issues: > 1) that one which seems to be fixed by updating the testkernel from 2.6.29-r1 > up to rc4 + 3 your patches. > 2) when we break a link, mii status is still reported as "up" in > /proc/net/bonding/bond1. >
Hello Andy, On 03/17/2009 06:04 PM, Andy Gospodarek wrote: > I'm sure if you opened a new bug at bugzilla.redhat.com to address > this issue someone would be willing to make sure this was fixed. Andy, thank you for the care, we don't forget about you. :) In fact, i've already opened the bug, here you are: https://bugzilla.redhat.com/show_bug.cgi?id=483034 So, you saved us one iteration - you already know about the problem, so looking forward for the fix. :)
> At the moment RHEL5 kernel is affected by 2 issues: > 1) that one which seems to be fixed by updating the testkernel from 2.6.29-r1 > up to rc4 + 3 your patches. OK, so it seems that these patches do the trick. Good. We'll use the RH bugzilla to address any backport. But it looks like we should keep this one going until we get these patches in the mainstream. > 2) when we break a link, mii status is still reported as "up" in > /proc/net/bonding/bond1. > � (at the same time bonding changes the active slave to the working one > correctly). I do see an issue in your Comment #6 , but its not exactly the one you mention. I see that when you "Enable icbay5 uplink", eth2 still shows "MII Status: down". Is this yet another issue do you think ? I'd like to know what "ethtool eth2" reports for the link state in the same situation.
> I do see an issue in your Comment #6 , but its not exactly the one you > mention. > I see that when you "Enable icbay5 uplink", eth2 still shows "MII Status: > down". Is this yet another issue do you think ? I'd like to know what > "ethtool > eth2" reports for the link state in the same situation. Yes, this is not that issue that i was talking about. i was talking about the following: This test was run under 2.6.18-028stab057.4 PVC kernel (Parallels Virtuozzo Containers) which was based on 2.6.18-92.1.1.el5 RHEL5 kernel. The test was done on another physical node, but the configuration was identical. [root@hostname1 ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:0c:64 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:0c:66 ### SHUT OFF Virtual Connect Switch for the eth2 uplink [root@hostname1 ~]# date Tue Dec 23 09:10:42 PST 2008 [root@hostname1 ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:0c:64 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:0c:66 [root@hostname1 network-scripts]# ping 10.58.64.1 PING 10.58.64.1 (10.58.64.1) 56(84) bytes of data. 64 bytes from 10.58.64.1: icmp_seq=1 ttl=255 time=1.58 ms 64 bytes from 10.58.64.1: icmp_seq=2 ttl=255 time=0.380 ms --- 10.58.64.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.380/0.980/1.581/0.601 ms ### Turned ON Virtual Connect Switch for the eth2 uplink [root@hostname1 network-scripts]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:0c:64 Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:17:a4:77:0c:66 [root@hostname1 network-scripts]# ping 10.58.64.1 PING 10.58.64.1 (10.58.64.1) 56(84) bytes of data. 64 bytes from 10.58.64.1: icmp_seq=1 ttl=255 time=0.443 ms 64 bytes from 10.58.64.1: icmp_seq=2 ttl=255 time=0.436 ms --- 10.58.64.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.436/0.439/0.443/0.021 ms ### SHUT OFF Virtual Connect Switch for the eth3 uplink [root@hostname1 network-scripts]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth2 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:0c:64 Slave Interface: eth3 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:17:a4:77:0c:66 [root@hostname1 network-scripts]# ping 10.58.64.1 PING 10.58.64.1 (10.58.64.1) 56(84) bytes of data. >From 10.58.64.251 icmp_seq=9 Destination Host Unreachable >From 10.58.64.251 icmp_seq=10 Destination Host Unreachable >From 10.58.64.251 icmp_seq=11 Destination Host Unreachable >From 10.58.64.251 icmp_seq=13 Destination Host Unreachable >From 10.58.64.251 icmp_seq=14 Destination Host Unreachable >From 10.58.64.251 icmp_seq=15 Destination Host Unreachable You see, MII always reports status as UP (even when "Link Failure Count" inc-ed). That is the problem i was talking about. But in my comment #6 you've found the issue that we did not notice: when we "Enable icbay5 uplink", eth2 still shows "MII Status: down". Meanwhile a bit later when we "disable icbay6", eth2 shows "MII Status: up", so this can be just a delay reporting correct state. i'll ask for the details and let you know.
>> 2) when we break a link, mii status is still reported as "up" in >> /proc/net/bonding/bond1. >> � (at the same time bonding changes the active slave to the working one >> correctly). > I do see an issue in your Comment #6 , but its not exactly the one you > mention. > I see that when you "Enable icbay5 uplink", eth2 still shows "MII Status: > down". Is this yet another issue do you think ? I'd like to know what > "ethtool > eth2" reports for the link state in the same situation. David, looks like that NIC does not support MII requests: # mii-tool eth2 SIOCGMIIPHY on 'eth2' failed: Operation not supported Does this mean that we also should not trust ethtool's field "MII status"?
Shall we keep working on this? ethtool's MII status should be correct.
Jesse, i'm sorry, but as i've already said i never had a direct access to affected nodes and thus it took quite long solve failback issue. Nowadays as far as i understand the issue with incorrect MII status report is still there, but it is not a priority for node owners and i'm unable to get feedback / ask for testing / etc. Thus we have no choice other than stall this for some time (indefinite) or may be even close this bug as the original issue with non-working failback was solved, so we'll file another bug for incorrect MII status once we have ability to investigate it. Thank you for the help!