[ 2452.163445] NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 0 timed out [ 2452.163467] ------------[ cut here ]------------ [ 2452.163475] WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0xc5/0x122 [ 2452.163476] Modules linked in: ipmi_si x86_pkg_temp_thermal [ 2452.163482] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G I 4.12.8 #1 [ 2452.163483] task: ffff88086d8fe200 task.stack: ffffc90003214000 [ 2452.163485] RIP: 0010:dev_watchdog+0xc5/0x122 [ 2452.163485] RSP: 0018:ffff88087f843ea0 EFLAGS: 00010286 [ 2452.163486] RAX: 0000000000000039 RBX: ffff88046ca78000 RCX: 0000000000000000 [ 2452.163487] RDX: ffff88087f853310 RSI: ffff88087f84caa8 RDI: ffff88087f84caa8 [ 2452.163487] RBP: ffff88087f843eb0 R08: 0000000000000000 R09: 0000000000000000 [ 2452.163488] R10: 00000000003d0900 R11: ffff88087ff9194c R12: 0000000000000000 [ 2452.163489] R13: ffffffff815a37a2 R14: ffff88046ca78438 R15: ffff88046ca78000 [ 2452.163490] FS: 0000000000000000(0000) GS:ffff88087f840000(0000) knlGS:0000000000000000 [ 2452.163490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2452.163491] CR2: 00000000006b2008 CR3: 0000000001c09000 CR4: 00000000001406e0 [ 2452.163492] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2452.163492] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 2452.163493] Call Trace: [ 2452.163494] <IRQ> [ 2452.163500] call_timer_fn+0x5b/0x123 [ 2452.163501] run_timer_softirq+0x137/0x15a [ 2452.163503] ? tk_clock_read+0xc/0xe [ 2452.163505] ? timekeeping_get_ns+0x1d/0x31 [ 2452.163506] ? ktime_get+0x2d/0x3e [ 2452.163508] __do_softirq+0xe4/0x23a [ 2452.163512] ? tick_program_event+0x5d/0x64 [ 2452.163513] irq_exit+0x4d/0x5b [ 2452.163518] smp_apic_timer_interrupt+0x29/0x34 [ 2452.163520] apic_timer_interrupt+0x86/0x90 [ 2452.163524] RIP: 0010:cpuidle_enter_state+0x12f/0x184 [ 2452.163525] RSP: 0018:ffffc90003217ea0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 2452.163526] RAX: 0000023af054988b RBX: 0000000000000002 RCX: 000000000000001f [ 2452.163527] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000000 [ 2452.163527] RBP: ffffc90003217ed0 R08: 00000000ffffffe3 R09: 000000003ff003ff [ 2452.163528] R10: ffffc90003217e70 R11: 0000000000000004 R12: ffff88086c10d400 [ 2452.163528] R13: 0000023af054988b R14: 0000000000000002 R15: 0000023af0528d14 [ 2452.163529] </IRQ> [ 2452.163531] cpuidle_enter+0x12/0x14 [ 2452.163534] do_idle+0x10e/0x166 [ 2452.163535] cpu_startup_entry+0x1a/0x1c [ 2452.163537] start_secondary+0xd0/0xd3 [ 2452.163540] secondary_startup_64+0x9f/0x9f [ 2452.163541] Code: 70 15 73 00 00 75 38 48 89 df c6 05 64 15 73 00 01 e8 1d 4a fe ff 44 89 e1 48 89 de 48 c7 c7 eb 8f a9 81 48 89 c2 e8 b4 ab b3 ff <0f> ff eb 10 41 ff c4 48 05 40 01 00 00 41 39 f4 75 9a eb 0d 48 [ 2452.163559] ---[ end trace e35aaedfdb6bf6da ]--- [ 2452.163562] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout [ 2452.163579] ixgbe 0000:02:00.0 eth0: Reset adapter [ 2473.173038] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [ 2473.325081] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [ 2473.450701] ixgbe 0000:02:00.0 eth0: NIC Link is Down [ 2473.450784] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [ 2473.537116] ixgbe 0000:02:00.0 eth0: Reset adapter [ 2473.821091] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [ 2473.969178] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [ 2474.112745] ixgbe 0000:02:00.0 eth0: NIC Link is Down [ 2474.112830] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [ 2474.177126] ixgbe 0000:02:00.0 eth0: Reset adapter [ 2474.461154] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [ 2474.613208] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
Server is acting as a router. After NETDEV WATCHDOG - network card is reset / link down / up Server is performing ipv4/ipv6 routing with hfsc attached to vlan interfaces qdisc used in hfsc class is pfifo.
flow controll on eth0 is disabled - and below ethtool -k ethtool -k eth0 Features for eth0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: on receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: on tx-gre-csum-segmentation: on tx-ipxip4-segmentation: on tx-ipxip6-segmentation: on tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off hw-tc-offload: off esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] ethtool -a eth0 Pause parameters for eth0: Autonegotiate: off RX: off TX: off
Latest working kernel with same configuration - but without such a problems was: linux-4.7-rc4
for more information - now one ethernet cart from eight is flapping [15467.174551] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout [15470.758632] ixgbe 0000:02:00.0 eth0: Reset adapter [15471.042547] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [15471.190556] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [15471.325786] ixgbe 0000:02:00.0 eth0: NIC Link is Down [15471.325874] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [15471.394573] ixgbe 0000:02:00.0 eth0: Reset adapter [15471.674592] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [15471.818623] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [15472.255439] ixgbe 0000:02:00.0 eth0: NIC Link is Down [15472.255527] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [15474.050770] ixgbe 0000:02:00.0 eth0: Reset adapter [15474.330804] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [15474.482819] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [15474.884169] ixgbe 0000:02:00.0 eth0: NIC Link is Down [15474.884260] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [15476.710974] ixgbe 0000:02:00.0 eth0: Reset adapter [15476.990989] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [15477.247034] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [15477.668676] ixgbe 0000:02:00.0 eth0: NIC Link is Down [15479.459197] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None - some stats from ethtool shows tx_timeout_count: 20 tx_restart_queue: 173 Full stats for flapping nic: ethtool -S eth0 NIC statistics: rx_packets: 7767843441 tx_packets: 8025263723 rx_bytes: 6399828526281 tx_bytes: 6555440321515 rx_pkts_nic: 7766101929 tx_pkts_nic: 8021027400 rx_bytes_nic: 6430435493532 tx_bytes_nic: 6588439785661 lsc_int: 69 tx_busy: 0 non_eop_descs: 0 rx_errors: 0 tx_errors: 0 rx_dropped: 89 tx_dropped: 0 multicast: 68302 broadcast: 311731 rx_no_buffer_count: 0 collisions: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 hw_rsc_aggregated: 0 hw_rsc_flushed: 0 fdir_match: 54 fdir_miss: 7806649663 fdir_overflow: 0 rx_fifo_errors: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_timeout_count: 20 tx_restart_queue: 173 rx_long_length_errors: 82 rx_short_length_errors: 0 tx_flow_control_xon: 0 rx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_flow_control_xoff: 0 rx_csum_offload_errors: 12603 alloc_rx_page_failed: 0 alloc_rx_buff_failed: 0 rx_no_dma_resources: 1728853 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_queue_0_packets: 1608023617 tx_queue_0_bytes: 1318855627459 tx_queue_1_packets: 1583071804 tx_queue_1_bytes: 1288775532888 tx_queue_2_packets: 1612868585 tx_queue_2_bytes: 1318475306052 tx_queue_3_packets: 1612908620 tx_queue_3_bytes: 1317436359489 tx_queue_4_packets: 1608391097 tx_queue_4_bytes: 1311897495627 tx_queue_5_packets: 0 tx_queue_5_bytes: 0 tx_queue_6_packets: 0 tx_queue_6_bytes: 0 tx_queue_7_packets: 0 tx_queue_7_bytes: 0 tx_queue_8_packets: 0 tx_queue_8_bytes: 0 tx_queue_9_packets: 0 tx_queue_9_bytes: 0 tx_queue_10_packets: 0 tx_queue_10_bytes: 0 tx_queue_11_packets: 0 tx_queue_11_bytes: 0 tx_queue_12_packets: 0 tx_queue_12_bytes: 0 tx_queue_13_packets: 0 tx_queue_13_bytes: 0 tx_queue_14_packets: 0 tx_queue_14_bytes: 0 tx_queue_15_packets: 0 tx_queue_15_bytes: 0 tx_queue_16_packets: 0 tx_queue_16_bytes: 0 tx_queue_17_packets: 0 tx_queue_17_bytes: 0 tx_queue_18_packets: 0 tx_queue_18_bytes: 0 tx_queue_19_packets: 0 tx_queue_19_bytes: 0 tx_queue_20_packets: 0 tx_queue_20_bytes: 0 tx_queue_21_packets: 0 tx_queue_21_bytes: 0 tx_queue_22_packets: 0 tx_queue_22_bytes: 0 tx_queue_23_packets: 0 tx_queue_23_bytes: 0 tx_queue_24_packets: 0 tx_queue_24_bytes: 0 tx_queue_25_packets: 0 tx_queue_25_bytes: 0 tx_queue_26_packets: 0 tx_queue_26_bytes: 0 tx_queue_27_packets: 0 tx_queue_27_bytes: 0 tx_queue_28_packets: 0 tx_queue_28_bytes: 0 tx_queue_29_packets: 0 tx_queue_29_bytes: 0 tx_queue_30_packets: 0 tx_queue_30_bytes: 0 tx_queue_31_packets: 0 tx_queue_31_bytes: 0 tx_queue_32_packets: 0 tx_queue_32_bytes: 0 tx_queue_33_packets: 0 tx_queue_33_bytes: 0 tx_queue_34_packets: 0 tx_queue_34_bytes: 0 tx_queue_35_packets: 0 tx_queue_35_bytes: 0 tx_queue_36_packets: 0 tx_queue_36_bytes: 0 tx_queue_37_packets: 0 tx_queue_37_bytes: 0 tx_queue_38_packets: 0 tx_queue_38_bytes: 0 tx_queue_39_packets: 0 tx_queue_39_bytes: 0 tx_queue_40_packets: 0 tx_queue_40_bytes: 0 tx_queue_41_packets: 0 tx_queue_41_bytes: 0 tx_queue_42_packets: 0 tx_queue_42_bytes: 0 tx_queue_43_packets: 0 tx_queue_43_bytes: 0 tx_queue_44_packets: 0 tx_queue_44_bytes: 0 tx_queue_45_packets: 0 tx_queue_45_bytes: 0 tx_queue_46_packets: 0 tx_queue_46_bytes: 0 tx_queue_47_packets: 0 tx_queue_47_bytes: 0 tx_queue_48_packets: 0 tx_queue_48_bytes: 0 tx_queue_49_packets: 0 tx_queue_49_bytes: 0 tx_queue_50_packets: 0 tx_queue_50_bytes: 0 tx_queue_51_packets: 0 tx_queue_51_bytes: 0 tx_queue_52_packets: 0 tx_queue_52_bytes: 0 tx_queue_53_packets: 0 tx_queue_53_bytes: 0 tx_queue_54_packets: 0 tx_queue_54_bytes: 0 tx_queue_55_packets: 0 tx_queue_55_bytes: 0 tx_queue_56_packets: 0 tx_queue_56_bytes: 0 tx_queue_57_packets: 0 tx_queue_57_bytes: 0 tx_queue_58_packets: 0 tx_queue_58_bytes: 0 tx_queue_59_packets: 0 tx_queue_59_bytes: 0 tx_queue_60_packets: 0 tx_queue_60_bytes: 0 tx_queue_61_packets: 0 tx_queue_61_bytes: 0 tx_queue_62_packets: 0 tx_queue_62_bytes: 0 tx_queue_63_packets: 0 tx_queue_63_bytes: 0 rx_queue_0_packets: 1571561078 rx_queue_0_bytes: 1282564409376 rx_queue_1_packets: 1521956074 rx_queue_1_bytes: 1246814972926 rx_queue_2_packets: 1599206777 rx_queue_2_bytes: 1329637736301 rx_queue_3_packets: 1550789189 rx_queue_3_bytes: 1293201812756 rx_queue_4_packets: 1524330353 rx_queue_4_bytes: 1247609613454 rx_queue_5_packets: 0 rx_queue_5_bytes: 0 rx_queue_6_packets: 0 rx_queue_6_bytes: 0 rx_queue_7_packets: 0 rx_queue_7_bytes: 0 rx_queue_8_packets: 0 rx_queue_8_bytes: 0 rx_queue_9_packets: 0 rx_queue_9_bytes: 0 rx_queue_10_packets: 0 rx_queue_10_bytes: 0 rx_queue_11_packets: 0 rx_queue_11_bytes: 0 rx_queue_12_packets: 0 rx_queue_12_bytes: 0 rx_queue_13_packets: 0 rx_queue_13_bytes: 0 rx_queue_14_packets: 0 rx_queue_14_bytes: 0 rx_queue_15_packets: 0 rx_queue_15_bytes: 0 rx_queue_16_packets: 0 rx_queue_16_bytes: 0 rx_queue_17_packets: 0 rx_queue_17_bytes: 0 rx_queue_18_packets: 0 rx_queue_18_bytes: 0 rx_queue_19_packets: 0 rx_queue_19_bytes: 0 rx_queue_20_packets: 0 rx_queue_20_bytes: 0 rx_queue_21_packets: 0 rx_queue_21_bytes: 0 rx_queue_22_packets: 0 rx_queue_22_bytes: 0 rx_queue_23_packets: 0 rx_queue_23_bytes: 0 rx_queue_24_packets: 0 rx_queue_24_bytes: 0 rx_queue_25_packets: 0 rx_queue_25_bytes: 0 rx_queue_26_packets: 0 rx_queue_26_bytes: 0 rx_queue_27_packets: 0 rx_queue_27_bytes: 0 rx_queue_28_packets: 0 rx_queue_28_bytes: 0 rx_queue_29_packets: 0 rx_queue_29_bytes: 0 rx_queue_30_packets: 0 rx_queue_30_bytes: 0 rx_queue_31_packets: 0 rx_queue_31_bytes: 0 rx_queue_32_packets: 0 rx_queue_32_bytes: 0 rx_queue_33_packets: 0 rx_queue_33_bytes: 0 rx_queue_34_packets: 0 rx_queue_34_bytes: 0 rx_queue_35_packets: 0 rx_queue_35_bytes: 0 rx_queue_36_packets: 0 rx_queue_36_bytes: 0 rx_queue_37_packets: 0 rx_queue_37_bytes: 0 rx_queue_38_packets: 0 rx_queue_38_bytes: 0 rx_queue_39_packets: 0 rx_queue_39_bytes: 0 rx_queue_40_packets: 0 rx_queue_40_bytes: 0 rx_queue_41_packets: 0 rx_queue_41_bytes: 0 rx_queue_42_packets: 0 rx_queue_42_bytes: 0 rx_queue_43_packets: 0 rx_queue_43_bytes: 0 rx_queue_44_packets: 0 rx_queue_44_bytes: 0 rx_queue_45_packets: 0 rx_queue_45_bytes: 0 rx_queue_46_packets: 0 rx_queue_46_bytes: 0 rx_queue_47_packets: 0 rx_queue_47_bytes: 0 rx_queue_48_packets: 0 rx_queue_48_bytes: 0 rx_queue_49_packets: 0 rx_queue_49_bytes: 0 rx_queue_50_packets: 0 rx_queue_50_bytes: 0 rx_queue_51_packets: 0 rx_queue_51_bytes: 0 rx_queue_52_packets: 0 rx_queue_52_bytes: 0 rx_queue_53_packets: 0 rx_queue_53_bytes: 0 rx_queue_54_packets: 0 rx_queue_54_bytes: 0 rx_queue_55_packets: 0 rx_queue_55_bytes: 0 rx_queue_56_packets: 0 rx_queue_56_bytes: 0 rx_queue_57_packets: 0 rx_queue_57_bytes: 0 rx_queue_58_packets: 0 rx_queue_58_bytes: 0 rx_queue_59_packets: 0 rx_queue_59_bytes: 0 rx_queue_60_packets: 0 rx_queue_60_bytes: 0 rx_queue_61_packets: 0 rx_queue_61_bytes: 0 rx_queue_62_packets: 0 rx_queue_62_bytes: 0 rx_queue_63_packets: 0 rx_queue_63_bytes: 0 tx_pb_0_pxon: 0 tx_pb_0_pxoff: 0 tx_pb_1_pxon: 0 tx_pb_1_pxoff: 0 tx_pb_2_pxon: 0 tx_pb_2_pxoff: 0 tx_pb_3_pxon: 0 tx_pb_3_pxoff: 0 tx_pb_4_pxon: 0 tx_pb_4_pxoff: 0 tx_pb_5_pxon: 0 tx_pb_5_pxoff: 0 tx_pb_6_pxon: 0 tx_pb_6_pxoff: 0 tx_pb_7_pxon: 0 tx_pb_7_pxoff: 0 rx_pb_0_pxon: 0 rx_pb_0_pxoff: 0 rx_pb_1_pxon: 0 rx_pb_1_pxoff: 0 rx_pb_2_pxon: 0 rx_pb_2_pxoff: 0 rx_pb_3_pxon: 0 rx_pb_3_pxoff: 0 rx_pb_4_pxon: 0 rx_pb_4_pxoff: 0 rx_pb_5_pxon: 0 rx_pb_5_pxoff: 0 rx_pb_6_pxon: 0 rx_pb_6_pxoff: 0 rx_pb_7_pxon: 0 rx_pb_7_pxoff: 0
Could you provide information about the network device - device name and id (lspci/-n)?
net-next kernel and tested devices: 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02) 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02) 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02) 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02) and: d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > emil.s.tantilov@intel.com changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |emil.s.tantilov@intel.com > > --- Comment #5 from emil.s.tantilov@intel.com --- > Could you provide information about the network device - device name and id > (lspci/-n)? >
same problem on both devices. But currently there left only one flapping port from one nic d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ethtool -S eth0 | grep tx tx_packets: 26166150912 tx_bytes: 21237240220551 tx_pkts_nic: 26159988271 tx_bytes_nic: 21419124752842 tx_busy: 0 tx_errors: 0 tx_dropped: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_timeout_count: 76 tx_restart_queue: 639 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 flow controll is disabled on both nic and switch. Also - this flapping is less frequent when server is more busy (dont know why but meybee it has something to do with cpu scalling/cstates/pstates when cpu is locked on highest frequency and is bussy all the time there is no problem - when traffic is going down there is more nic resets) [36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down [36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter [36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down [36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter [36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down [36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter [36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down [36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout [41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter [41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout [42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter [42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None [42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down [42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss [42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter [42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 [42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) --- > net-next kernel and tested devices: > > 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 01) > 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 01) > 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 01) > 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 01) > 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 02) > 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 02) > 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 02) > 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 > for 10GbE SFP+ (rev 02) > > > and: > > > d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > SFI/SFP+ Network Connection (rev 01) > d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > SFI/SFP+ Network Connection (rev 01) > > > W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze: >> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >> >> emil.s.tantilov@intel.com changed: >> >> What |Removed |Added >> ---------------------------------------------------------------------------- >> CC| |emil.s.tantilov@intel.com >> >> --- Comment #5 from emil.s.tantilov@intel.com --- >> Could you provide information about the network device - device name and id >> (lspci/-n)? >>
I just catch what is happening when nic reset occures perf top -C0 This network card is binded by affinity to cpus 0-4 When nic is resetting there is 100% cpu load on cpu 0 where this nic is bonded also. 20.29% [kernel] [k] _raw_spin_lock_irqsave 19.35% [kernel] [k] md_check_recovery 12.35% [kernel] [k] _raw_spin_unlock_irqrestore 9.14% [kernel] [k] do_raw_spin_lock 5.95% [kernel] [k] atomic64_cmpxchg 3.98% [kernel] [k] md_thread 3.52% [kernel] [k] __wake_up_common 3.44% [kernel] [k] __wake_up 3.05% [kernel] [k] raid1d 2.98% [kernel] [k] md_wakeup_thread 1.91% [kernel] [k] mddev_unlock 1.61% [kernel] [k] to_kthread 1.53% [kernel] [k] blk_flush_plug_list 1.53% [kernel] [k] blk_finish_plug 1.51% [kernel] [k] blk_start_plug 1.28% [kernel] [k] kthread_should_stop 1.18% [kernel] [k] flush_pending_writes 0.99% [kernel] [k] __mutex_trylock_or_owner 0.69% [kernel] [k] mutex_unlock 0.69% [kernel] [k] _raw_spin_lock_irq 0.67% [kernel] [k] _raw_spin_lock 0.48% [kernel] [k] mutex_trylock 0.44% [kernel] [k] mddev_is_clustered 0.40% [kernel] [k] __mutex_trylock 0.36% [kernel] [k] kthread_should_park 0.16% [kernel] [k] acpi_idle_do_entry 0.07% [kernel] [k] skb_release_data 0.04% [kernel] [k] __rtnl_unlock 0.03% [kernel] [k] __slab_free 0.03% [kernel] [k] compound_head 0.03% [kernel] [k] page_frag_free 0.03% [kernel] [k] __free_one_page W dniu 2017-08-22 o 20:02, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #7 from Pawel Staszewski (pstaszewski@itcare.pl) --- > same problem on both devices. > > But currently there left only one flapping port from one nic > > > d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > SFI/SFP+ Network Connection (rev 01) > > > ethtool -S eth0 | grep tx > tx_packets: 26166150912 > tx_bytes: 21237240220551 > tx_pkts_nic: 26159988271 > tx_bytes_nic: 21419124752842 > tx_busy: 0 > tx_errors: 0 > tx_dropped: 0 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > tx_timeout_count: 76 > tx_restart_queue: 639 > tx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > > > flow controll is disabled on both nic and switch. > > Also - this flapping is less frequent when server is more busy (dont > know why but meybee it has something to do with cpu > scalling/cstates/pstates when cpu is locked on highest frequency and is > bussy all the time there is no problem - when traffic is going down > there is more nic resets) > > [36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down > [36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx > work after link loss > [36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter > [36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down > [36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx > work after link loss > [36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter > [36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down > [36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx > work after link loss > [36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter > [36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down > [36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout > [41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter > [41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout > [42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter > [42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > [42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down > [42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx > work after link loss > [42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter > [42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 > [42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow > Control: None > > > W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze: >> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >> >> --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) --- >> net-next kernel and tested devices: >> >> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 01) >> 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 01) >> 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 01) >> 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 01) >> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 02) >> 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 02) >> 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 02) >> 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 >> for 10GbE SFP+ (rev 02) >> >> >> and: >> >> >> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ Network Connection (rev 01) >> d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ Network Connection (rev 01) >> >> >> W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze: >>> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >>> >>> emil.s.tantilov@intel.com changed: >>> >>> What |Removed |Added >>> >>> ---------------------------------------------------------------------------- >>> CC| >>> |emil.s.tantilov@intel.com >>> >>> --- Comment #5 from emil.s.tantilov@intel.com --- >>> Could you provide information about the network device - device name and id >>> (lspci/-n)? >>>
cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 0: 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 2-edge timer 8: 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 8-edge rtc0 9: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi 18: 0 141 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 18-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, i801_smbus 38: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 65536-edge ioat-msix 40: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67584-edge ioat-msix 41: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 69632-edge ioat-msix 42: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 71680-edge ioat-msix 43: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 73728-edge ioat-msix 44: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 75776-edge ioat-msix 45: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 77824-edge ioat-msix 46: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 79872-edge ioat-msix 48: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67174400-edge ioat-msix 50: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67176448-edge ioat-msix 51: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67178496-edge ioat-msix 52: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67180544-edge ioat-msix 53: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67182592-edge ioat-msix 54: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67184640-edge ioat-msix 55: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67186688-edge ioat-msix 56: 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 67188736-edge ioat-msix 57: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 286720-edge ahci[0000:00:11.4] 58: 379370 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2] 59: 300474267 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048576-edge eth0-TxRx-0 60: 1 301005834 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048577-edge eth0-TxRx-1 61: 1 0 301444215 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048578-edge eth0-TxRx-2 62: 1 0 0 302020538 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048579-edge eth0-TxRx-3 63: 1 0 0 0 301997942 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048580-edge eth0-TxRx-4 64: 908 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI 1048581-edge eth0 W dniu 2017-08-23 o 16:43, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #8 from Pawel Staszewski (pstaszewski@itcare.pl) --- > I just catch what is happening when nic reset occures > > perf top -C0 > > This network card is binded by affinity to cpus 0-4 > > When nic is resetting there is 100% cpu load on cpu 0 where this nic is > bonded also. > > > 20.29% [kernel] [k] _raw_spin_lock_irqsave > 19.35% [kernel] [k] md_check_recovery > 12.35% [kernel] [k] _raw_spin_unlock_irqrestore > 9.14% [kernel] [k] do_raw_spin_lock > 5.95% [kernel] [k] atomic64_cmpxchg > 3.98% [kernel] [k] md_thread > 3.52% [kernel] [k] __wake_up_common > 3.44% [kernel] [k] __wake_up > 3.05% [kernel] [k] raid1d > 2.98% [kernel] [k] md_wakeup_thread > 1.91% [kernel] [k] mddev_unlock > 1.61% [kernel] [k] to_kthread > 1.53% [kernel] [k] blk_flush_plug_list > 1.53% [kernel] [k] blk_finish_plug > 1.51% [kernel] [k] blk_start_plug > 1.28% [kernel] [k] kthread_should_stop > 1.18% [kernel] [k] flush_pending_writes > 0.99% [kernel] [k] __mutex_trylock_or_owner > 0.69% [kernel] [k] mutex_unlock > 0.69% [kernel] [k] _raw_spin_lock_irq > 0.67% [kernel] [k] _raw_spin_lock > 0.48% [kernel] [k] mutex_trylock > 0.44% [kernel] [k] mddev_is_clustered > 0.40% [kernel] [k] __mutex_trylock > 0.36% [kernel] [k] kthread_should_park > 0.16% [kernel] [k] acpi_idle_do_entry > 0.07% [kernel] [k] skb_release_data > 0.04% [kernel] [k] __rtnl_unlock > 0.03% [kernel] [k] __slab_free > 0.03% [kernel] [k] compound_head > 0.03% [kernel] [k] page_frag_free > 0.03% [kernel] [k] __free_one_page > > > > W dniu 2017-08-22 o 20:02, bugzilla-daemon@bugzilla.kernel.org pisze: >> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >> >> --- Comment #7 from Pawel Staszewski (pstaszewski@itcare.pl) --- >> same problem on both devices. >> >> But currently there left only one flapping port from one nic >> >> >> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ Network Connection (rev 01) >> >> >> ethtool -S eth0 | grep tx >> tx_packets: 26166150912 >> tx_bytes: 21237240220551 >> tx_pkts_nic: 26159988271 >> tx_bytes_nic: 21419124752842 >> tx_busy: 0 >> tx_errors: 0 >> tx_dropped: 0 >> tx_aborted_errors: 0 >> tx_carrier_errors: 0 >> tx_fifo_errors: 0 >> tx_heartbeat_errors: 0 >> tx_timeout_count: 76 >> tx_restart_queue: 639 >> tx_flow_control_xon: 0 >> tx_flow_control_xoff: 0 >> >> >> flow controll is disabled on both nic and switch. >> >> Also - this flapping is less frequent when server is more busy (dont >> know why but meybee it has something to do with cpu >> scalling/cstates/pstates when cpu is locked on highest frequency and is >> bussy all the time there is no problem - when traffic is going down >> there is more nic resets) >> >> [36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down >> [36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >> work after link loss >> [36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter >> [36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down >> [36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >> work after link loss >> [36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter >> [36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down >> [36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >> work after link loss >> [36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter >> [36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down >> [36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout >> [41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter >> [41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout >> [42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter >> [42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> [42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down >> [42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >> work after link loss >> [42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter >> [42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5 >> [42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow >> Control: None >> >> >> W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze: >>> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >>> >>> --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) --- >>> net-next kernel and tested devices: >>> >>> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 01) >>> 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 01) >>> 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 01) >>> 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 01) >>> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 02) >>> 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 02) >>> 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 02) >>> 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 >>> for 10GbE SFP+ (rev 02) >>> >>> >>> and: >>> >>> >>> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >>> SFI/SFP+ Network Connection (rev 01) >>> d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >>> SFI/SFP+ Network Connection (rev 01) >>> >>> >>> W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze: >>>> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >>>> >>>> emil.s.tantilov@intel.com changed: >>>> >>>> What |Removed |Added >>>> >>>> >>>> ---------------------------------------------------------------------------- >>>> CC| >>>> |emil.s.tantilov@intel.com >>>> >>>> --- Comment #5 from emil.s.tantilov@intel.com --- >>>> Could you provide information about the network device - device name and >>>> id >>>> (lspci/-n)? >>>>
Do you see flaps (link down/up events) even without the reset associated with the Tx hangs? We started a test in our labs attempting to reproduce this issue, but have not had much luck so far. Could you provide more information about the way the interfaces are configured (any settings that differ from default, kernel config may be useful as well). Since you have a kernel that did not show this problem you may be able to bisect it to the patch that introduced this issue. If you are willing and have the time that is.
nic reset are always initiated by [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss there is no flaps without this info About configuration This host have 8x 10Gbit ethernet in bond cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth6 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:65:78:94 Slave queue ID: 0 Slave Interface: eth7 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:65:78:95 Slave queue ID: 0 Slave Interface: eth8 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:61:13:bc Slave queue ID: 0 Slave Interface: eth9 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:61:13:bd Slave queue ID: 0 Slave Interface: eth2 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:e5:b7:aa Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:e5:b7:ab Slave queue ID: 0 Slave Interface: eth0 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:c1:9c:f4 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:25:90:c1:9c:f5 Slave queue ID: 0 ip link ls dev bond0 2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff Traffic that is going thru this bond is ~60Gbit/s / ~9Mpps - for all interfaces. Traffic type is mixed TCP/UDP - all kind of normal IP and IPv6 traffic. and there is not possible to bisect any change / patch from the kernel and make many reboots :) Will try to get some lab test hw and reproduce this .... W dniu 2017-08-24 o 00:28, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #10 from emil.s.tantilov@intel.com --- > Do you see flaps (link down/up events) even without the reset associated with > the Tx hangs? > > We started a test in our labs attempting to reproduce this issue, but have > not > had much luck so far. Could you provide more information about the way the > interfaces are configured (any settings that differ from default, kernel > config > may be useful as well). > > Since you have a kernel that did not show this problem you may be able to > bisect it to the patch that introduced this issue. If you are willing and > have > the time that is. >
(In reply to Pawel Staszewski from comment #11) > nic reset are always initiated by > > [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx > work after link loss > > there is no flaps without this info > > > About configuration > > This host have 8x 10Gbit ethernet in bond From the devices you listed in your previous comment you have 8 X710 devices which should be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with ixgbe, I am guessing you have 2 ixgbe and 6 i40e devices in this bond? Also you mentioned that you are using hfsc, can you provide more information about it and also how is the traffic routed through your interfaces. I am trying to get a better picture of your setup to be able to better replicate it in our labs.
Sorry Previously i send information where i have similar problems - on hosts with same kernel and 8x X710 - have same problem as with 8x 82599 Current configuration where we are focusing on is this with 8x 82599 (4x (2x 10G ports nic's) About the rest of configuration bond0 have attached about 300 vlans - and each vlan have hfsc qdisc attached with ingress qdisc policer (there is no hfsc or ingress qdisc on physical interface) below ip link show: ip l 256: vlan1673@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 257: vlan1674@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 258: vlan1676@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 259: vlan1677@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 260: vlan1678@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 4: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 261: vlan1679@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 5: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 262: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 6: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 263: vlan1681@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 7: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff 264: vlan1682@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 8: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 00:25:90:f9:5b:17 brd ff:ff:ff:ff:ff:ff 265: vlan1683@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 9: eth6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 266: vlan1684@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 10: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 267: vlan1685@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 11: eth8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 268: vlan1686@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 12: eth9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 269: vlan1687@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 13: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 270: vlan1688@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 14: vlan4091@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 271: vlan1689@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 15: vlan4032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 272: vlan1690@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 16: vlan617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 273: vlan1691@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 17: vlan627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 274: vlan1692@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 18: vlan1450@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 275: vlan1693@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 19: vlan1451@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 276: vlan1694@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 20: vlan1452@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 277: vlan1696@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 21: vlan1453@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 278: vlan1697@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 22: vlan1454@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 279: vlan1698@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 23: vlan1455@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 280: vlan1870@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 24: vlan1456@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 281: vlan1871@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 25: vlan1457@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 282: vlan1872@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 26: vlan1458@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 283: vlan1873@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 27: vlan1459@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 284: vlan1874@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 28: vlan1460@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 285: vlan1875@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 29: vlan1461@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 286: vlan1876@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 30: vlan1462@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 287: vlan1877@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 31: vlan1463@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 288: vlan1878@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 32: vlan1464@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 289: vlan1879@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 33: vlan1465@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 290: vlan1880@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 34: vlan1466@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 291: vlan1890@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 35: vlan1467@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 292: vlan1891@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 36: vlan1468@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 293: vlan1892@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 37: vlan1469@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 294: vlan616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 38: vlan1470@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 295: vlan979@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:c1:9c:f4 brd ff:ff:ff:ff:ff:ff 39: vlan1471@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 296: vlan985@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff 40: vlan1472@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 297: vlan2290@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 41: vlan1473@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 298: vlan4090@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 42: vlan1474@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 43: vlan1475@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 44: vlan1476@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 45: vlan1477@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 46: vlan1478@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 47: vlan1479@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 48: vlan1480@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 49: vlan1481@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 50: vlan1482@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 51: vlan1483@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 52: vlan1484@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 53: vlan1485@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 54: vlan1486@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 55: vlan1487@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 56: vlan1488@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 57: vlan1489@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 58: vlan1490@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 59: vlan1491@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 60: vlan1492@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 61: vlan1493@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 62: vlan1494@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 63: vlan1495@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 64: vlan1496@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 65: vlan1497@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 66: vlan1498@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 67: vlan1499@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 68: vlan1500@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 69: vlan1501@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 70: vlan1502@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 71: vlan1503@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 72: vlan1504@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 73: vlan1505@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 74: vlan1506@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 75: vlan1507@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 76: vlan1508@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 77: vlan1509@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 78: vlan1510@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 79: vlan1511@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 80: vlan1512@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 81: vlan1513@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 82: vlan1514@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 83: vlan1515@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 84: vlan1516@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 85: vlan1517@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 86: vlan1518@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 87: vlan1519@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 88: vlan1520@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 89: vlan1521@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 90: vlan1522@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 91: vlan1523@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 92: vlan1524@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 93: vlan1525@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 94: vlan1526@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 95: vlan1527@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 96: vlan1528@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 97: vlan1529@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 98: vlan1530@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 99: vlan1531@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 100: vlan1532@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 101: vlan1534@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 102: vlan1535@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 103: vlan1394@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 104: vlan1393@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 105: vlan1550@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 106: vlan1536@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 107: vlan1537@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 108: vlan1538@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 109: vlan1539@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 110: vlan1540@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 111: vlan1542@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 112: vlan1541@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 113: vlan1543@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 114: vlan1544@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 115: vlan1801@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 116: vlan1546@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 117: vlan1547@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 118: vlan1548@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 119: vlan1549@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 120: vlan3132@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 121: vlan3143@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 122: vlan3104@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 123: vlan3125@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 124: vlan3103@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 125: vlan3115@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 126: vlan3134@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 127: vlan3105@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 128: vlan3113@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 129: vlan3141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 130: vlan4009@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 131: vlan3144@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 132: vlan3130@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 133: vlan1803@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 134: vlan3146@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 135: vlan3148@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 136: vlan3109@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 137: vlan1551@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 138: vlan1552@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 139: vlan1553@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 140: vlan1554@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 141: vlan1555@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 142: vlan1556@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 143: vlan1558@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 144: vlan1559@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 145: vlan1560@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 146: vlan1561@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 147: vlan1562@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 148: vlan1563@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 149: vlan1564@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 150: vlan1565@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 151: vlan1566@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 152: vlan1567@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 153: vlan1568@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 154: vlan1569@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 155: vlan1570@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 156: vlan1571@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 157: vlan1572@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 158: vlan1573@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 159: vlan1574@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 160: vlan1575@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 161: vlan1576@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 162: vlan1577@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 163: vlan1578@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 164: vlan1579@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 165: vlan1580@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 166: vlan1581@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 167: vlan1582@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 168: vlan1583@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 169: vlan1584@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 170: vlan1585@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 171: vlan1586@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 172: vlan1587@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 173: vlan1588@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 174: vlan1589@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 175: vlan1591@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 176: vlan1592@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 177: vlan1593@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 178: vlan1594@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 179: vlan1595@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 180: vlan1596@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 181: vlan1597@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 182: vlan1598@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 183: vlan1599@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 184: vlan1557@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 185: vlan1545@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 186: vlan2001@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 187: vlan250@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 188: vlan4043@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 189: vlan1806@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 190: vlan1600@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 191: vlan1602@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 192: vlan1603@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 193: vlan1604@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 194: vlan1605@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 195: vlan1606@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 196: vlan1607@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 197: vlan1608@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 198: vlan1609@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 199: vlan1610@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 200: vlan1611@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 201: vlan1612@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 202: vlan1613@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 203: vlan1614@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 204: vlan1615@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 205: vlan1616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 206: vlan1617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 207: vlan1618@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 208: vlan1619@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 209: vlan1620@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 210: vlan1621@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 211: vlan1625@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 212: vlan1626@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 213: vlan1627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 214: vlan1628@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 215: vlan1629@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 216: vlan1630@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 217: vlan1631@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 218: vlan1632@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 219: vlan1634@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 220: vlan1635@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 221: vlan1636@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 222: vlan1640@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 223: vlan1641@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 224: vlan1642@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 225: vlan1643@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 226: vlan1644@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 227: vlan1645@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 228: vlan1646@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 229: vlan1647@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 230: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 231: vlan1649@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 232: vlan1650@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 233: vlan1651@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 234: vlan1652@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 235: vlan1653@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 236: vlan1654@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 237: vlan1655@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 238: vlan1656@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 239: vlan1657@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 240: vlan1658@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 241: vlan1659@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 242: vlan1660@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 243: vlan1661@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 244: vlan1662@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 245: vlan1663@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 246: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 247: vlan1665@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 248: vlan1601@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 249: vlan1666@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 250: vlan1667@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 251: vlan1668@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 252: vlan1669@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 253: vlan1670@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 254: vlan1671@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff 255: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc hfsc state UP mode DEFAULT group default qlen 1000 link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff W dniu 2017-08-24 o 01:07, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #12 from emil.s.tantilov@intel.com --- > (In reply to Pawel Staszewski from comment #11) >> nic reset are always initiated by >> >> [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >> work after link loss >> >> there is no flaps without this info >> >> >> About configuration >> >> This host have 8x 10Gbit ethernet in bond > From the devices you listed in your previous comment you have 8 X710 devices > which should > be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with > ixgbe, I am guessing > you have 2 ixgbe and 6 i40e devices in this bond? > > Also you mentioned that you are using hfsc, can you provide more information > about it and also > how is the traffic routed through your interfaces. I am trying to get a > better > picture of your setup > to be able to better replicate it in our labs. >
Today another interface start to flap: [175947.099230] ixgbe 0000:82:00.1 eth9: initiating reset due to tx timeout [175947.099260] ixgbe 0000:82:00.1 eth9: Reset adapter [176000.515273] INFO: rcu_sched self-detected stall on CPU [176000.515285] 39-...: (14999 ticks this GP) idle=b12/140000000000001/0 softirq=59250004/59250004 fqs=6371 [176000.515285] (t=15000 jiffies g=7584203 c=7584202 q=8002) [176000.515293] NMI backtrace for cpu 39 [176000.515296] CPU: 39 PID: 3700 Comm: md2_raid1 Tainted: G W I 4.12.8 #1 [176000.515298] Call Trace: [176000.515300] <IRQ> [176000.515308] dump_stack+0x4d/0x64 [176000.515311] nmi_cpu_backtrace+0x76/0x85 [176000.515317] ? irq_force_complete_move+0xdf/0xdf [176000.515319] nmi_trigger_cpumask_backtrace+0x51/0xc6 [176000.515321] arch_trigger_cpumask_backtrace+0x14/0x17 [176000.515325] rcu_dump_cpu_stacks+0x86/0xb3 [176000.515330] rcu_check_callbacks+0x232/0x5e8 [176000.515333] ? raise_softirq_irqoff+0x9/0x1e [176000.515337] update_process_times+0x2a/0x52 [176000.515343] tick_sched_timer+0xf9/0x12b [176000.515345] __hrtimer_run_queues+0xba/0x182 [176000.515347] hrtimer_interrupt+0x67/0x105 [176000.515349] local_apic_timer_interrupt+0x46/0x49 [176000.515351] smp_apic_timer_interrupt+0x24/0x34 [176000.515354] apic_timer_interrupt+0x86/0x90 [176000.515359] RIP: 0010:atomic64_cmpxchg+0xc/0xe [176000.515360] RSP: 0018:ffffc90009fa7d78 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [176000.515361] RAX: 0000000000000000 RBX: ffff88046b8d8000 RCX: 0000000000000000 [176000.515362] RDX: ffff88086b024980 RSI: 0000000000000000 RDI: ffff88046b8d81e8 [176000.515363] RBP: ffffc90009fa7d78 R08: ffff88086b024980 R09: ffff88086b024980 [176000.515364] R10: ffff88046b8d81e8 R11: 000000000000032d R12: ffff88086d24af00 [176000.515365] R13: ffffc90009fa7e30 R14: ffff88086d24af28 R15: 0000000000000000 [176000.515365] </IRQ> [176000.515368] __mutex_trylock_or_owner+0x44/0x5a [176000.515369] __mutex_trylock+0x9/0x11 [176000.515372] mutex_trylock+0x9/0xe [176000.515375] md_check_recovery+0xdc/0x3cc [176000.515378] raid1d+0x44/0x6e5 [176000.515380] md_thread+0x133/0x149 [176000.515381] ? md_thread+0x133/0x149 [176000.515382] ? wake_up_atomic_t+0x27/0x27 [176000.515383] ? md_do_sync+0xca0/0xca0 [176000.515386] kthread+0xf7/0xfc [176000.515388] ? init_completion+0x24/0x24 [176000.515390] ret_from_fork+0x22/0x30 [176005.487661] ixgbe 0000:82:00.1 eth9: detected SFP+: 6 [176005.635699] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow Control: None [176006.103192] ixgbe 0000:82:00.1 eth9: NIC Link is Down [176006.103359] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx work after link loss [176007.875865] ixgbe 0000:82:00.1 eth9: Reset adapter [176008.151873] ixgbe 0000:82:00.1 eth9: detected SFP+: 6 [176008.303891] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow Control: None [176008.682820] ixgbe 0000:82:00.1 eth9: NIC Link is Down [176008.682906] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx work after link loss [176010.532062] ixgbe 0000:82:00.1 eth9: Reset adapter [176010.812079] ixgbe 0000:82:00.1 eth9: detected SFP+: 6 [176010.968096] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow Control: None [176011.110638] ixgbe 0000:82:00.1 eth9: NIC Link is Down [176011.110730] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx work after link loss [176011.172105] ixgbe 0000:82:00.1 eth9: Reset adapter [176011.456124] ixgbe 0000:82:00.1 eth9: detected SFP+: 6 [176011.600153] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow Control: None W dniu 2017-08-24 o 01:21, bugzilla-daemon@bugzilla.kernel.org pisze: > https://bugzilla.kernel.org/show_bug.cgi?id=196727 > > --- Comment #13 from Pawel Staszewski (pstaszewski@itcare.pl) --- > Sorry > > Previously i send information where i have similar problems - on hosts > with same kernel and > > 8x X710 - have same problem as with > > 8x 82599 > > > Current configuration where we are focusing on is this with 8x 82599 (4x > (2x 10G ports nic's) > > > About the rest of configuration > > bond0 have attached about 300 vlans - and each vlan have hfsc qdisc > attached with ingress qdisc policer (there is no hfsc or ingress qdisc > on physical interface) > > > below ip link show: > ip l > 256: vlan1673@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 257: vlan1674@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode > DEFAULT group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > 258: vlan1676@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 259: vlan1677@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 260: vlan1678@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 4: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 261: vlan1679@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 5: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 262: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 6: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 263: vlan1681@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 7: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > DEFAULT group default qlen 1000 > link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff > 264: vlan1682@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 8: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > DEFAULT group default qlen 1000 > link/ether 00:25:90:f9:5b:17 brd ff:ff:ff:ff:ff:ff > 265: vlan1683@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 9: eth6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 266: vlan1684@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 10: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 267: vlan1685@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 11: eth8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 268: vlan1686@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 12: eth9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 269: vlan1687@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 13: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group > default qlen 1000 > link/sit 0.0.0.0 brd 0.0.0.0 > 270: vlan1688@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 14: vlan4091@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 271: vlan1689@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 15: vlan4032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 272: vlan1690@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 16: vlan617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 273: vlan1691@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 17: vlan627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 274: vlan1692@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 18: vlan1450@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 275: vlan1693@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 19: vlan1451@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 276: vlan1694@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 20: vlan1452@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 277: vlan1696@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 21: vlan1453@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 278: vlan1697@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 22: vlan1454@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 279: vlan1698@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 23: vlan1455@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 280: vlan1870@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 24: vlan1456@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 281: vlan1871@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 25: vlan1457@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 282: vlan1872@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 26: vlan1458@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 283: vlan1873@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 27: vlan1459@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 284: vlan1874@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 28: vlan1460@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 285: vlan1875@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 29: vlan1461@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 286: vlan1876@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 30: vlan1462@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 287: vlan1877@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 31: vlan1463@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 288: vlan1878@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 32: vlan1464@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 289: vlan1879@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 33: vlan1465@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 290: vlan1880@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 34: vlan1466@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 291: vlan1890@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 35: vlan1467@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 292: vlan1891@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 36: vlan1468@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 293: vlan1892@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 37: vlan1469@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 294: vlan616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 38: vlan1470@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 295: vlan979@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:c1:9c:f4 brd ff:ff:ff:ff:ff:ff > 39: vlan1471@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 296: vlan985@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff > 40: vlan1472@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 297: vlan2290@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 41: vlan1473@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 298: vlan4090@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 42: vlan1474@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 43: vlan1475@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 44: vlan1476@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 45: vlan1477@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 46: vlan1478@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 47: vlan1479@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 48: vlan1480@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 49: vlan1481@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 50: vlan1482@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 51: vlan1483@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 52: vlan1484@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 53: vlan1485@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 54: vlan1486@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 55: vlan1487@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 56: vlan1488@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 57: vlan1489@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 58: vlan1490@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 59: vlan1491@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 60: vlan1492@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 61: vlan1493@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 62: vlan1494@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 63: vlan1495@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 64: vlan1496@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 65: vlan1497@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 66: vlan1498@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 67: vlan1499@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 68: vlan1500@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 69: vlan1501@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 70: vlan1502@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 71: vlan1503@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 72: vlan1504@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 73: vlan1505@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 74: vlan1506@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 75: vlan1507@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 76: vlan1508@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 77: vlan1509@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 78: vlan1510@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 79: vlan1511@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 80: vlan1512@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 81: vlan1513@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 82: vlan1514@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 83: vlan1515@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 84: vlan1516@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 85: vlan1517@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 86: vlan1518@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 87: vlan1519@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 88: vlan1520@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 89: vlan1521@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 90: vlan1522@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 91: vlan1523@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 92: vlan1524@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 93: vlan1525@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 94: vlan1526@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 95: vlan1527@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 96: vlan1528@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 97: vlan1529@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 98: vlan1530@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 99: vlan1531@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 100: vlan1532@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 101: vlan1534@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 102: vlan1535@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 103: vlan1394@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 104: vlan1393@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 105: vlan1550@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 106: vlan1536@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 107: vlan1537@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 108: vlan1538@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 109: vlan1539@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 110: vlan1540@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 111: vlan1542@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 112: vlan1541@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 113: vlan1543@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 114: vlan1544@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 115: vlan1801@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 116: vlan1546@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 117: vlan1547@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 118: vlan1548@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 119: vlan1549@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 120: vlan3132@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 121: vlan3143@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 122: vlan3104@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 123: vlan3125@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 124: vlan3103@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 125: vlan3115@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 126: vlan3134@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 127: vlan3105@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 128: vlan3113@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 129: vlan3141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 130: vlan4009@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 131: vlan3144@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 132: vlan3130@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 133: vlan1803@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 134: vlan3146@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 135: vlan3148@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 136: vlan3109@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 137: vlan1551@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 138: vlan1552@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 139: vlan1553@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 140: vlan1554@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 141: vlan1555@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 142: vlan1556@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 143: vlan1558@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 144: vlan1559@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 145: vlan1560@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 146: vlan1561@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 147: vlan1562@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 148: vlan1563@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 149: vlan1564@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 150: vlan1565@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 151: vlan1566@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 152: vlan1567@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 153: vlan1568@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 154: vlan1569@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 155: vlan1570@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 156: vlan1571@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 157: vlan1572@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 158: vlan1573@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 159: vlan1574@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 160: vlan1575@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 161: vlan1576@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 162: vlan1577@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 163: vlan1578@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 164: vlan1579@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 165: vlan1580@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 166: vlan1581@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 167: vlan1582@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 168: vlan1583@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 169: vlan1584@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 170: vlan1585@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 171: vlan1586@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 172: vlan1587@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 173: vlan1588@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 174: vlan1589@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 175: vlan1591@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 176: vlan1592@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 177: vlan1593@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 178: vlan1594@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 179: vlan1595@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 180: vlan1596@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 181: vlan1597@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 182: vlan1598@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 183: vlan1599@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 184: vlan1557@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 185: vlan1545@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 186: vlan2001@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 187: vlan250@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 188: vlan4043@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 189: vlan1806@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 190: vlan1600@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 191: vlan1602@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 192: vlan1603@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 193: vlan1604@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 194: vlan1605@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 195: vlan1606@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 196: vlan1607@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 197: vlan1608@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 198: vlan1609@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 199: vlan1610@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 200: vlan1611@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 201: vlan1612@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 202: vlan1613@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 203: vlan1614@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 204: vlan1615@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 205: vlan1616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 206: vlan1617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 207: vlan1618@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 208: vlan1619@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 209: vlan1620@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 210: vlan1621@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 211: vlan1625@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 212: vlan1626@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 213: vlan1627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 214: vlan1628@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 215: vlan1629@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 216: vlan1630@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 217: vlan1631@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 218: vlan1632@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 219: vlan1634@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 220: vlan1635@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 221: vlan1636@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 222: vlan1640@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 223: vlan1641@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 224: vlan1642@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 225: vlan1643@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 226: vlan1644@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 227: vlan1645@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 228: vlan1646@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 229: vlan1647@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 230: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 231: vlan1649@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 232: vlan1650@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 233: vlan1651@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 234: vlan1652@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 235: vlan1653@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 236: vlan1654@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 237: vlan1655@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 238: vlan1656@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 239: vlan1657@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 240: vlan1658@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 241: vlan1659@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 242: vlan1660@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 243: vlan1661@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 244: vlan1662@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 245: vlan1663@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 246: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 247: vlan1665@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 248: vlan1601@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 249: vlan1666@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 250: vlan1667@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 251: vlan1668@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 252: vlan1669@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 253: vlan1670@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 254: vlan1671@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > 255: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > hfsc state UP mode DEFAULT group default qlen 1000 > link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff > > > W dniu 2017-08-24 o 01:07, bugzilla-daemon@bugzilla.kernel.org pisze: >> https://bugzilla.kernel.org/show_bug.cgi?id=196727 >> >> --- Comment #12 from emil.s.tantilov@intel.com --- >> (In reply to Pawel Staszewski from comment #11) >>> nic reset are always initiated by >>> >>> [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx >>> work after link loss >>> >>> there is no flaps without this info >>> >>> >>> About configuration >>> >>> This host have 8x 10Gbit ethernet in bond >> From the devices you listed in your previous comment you have 8 X710 >> devices >> which should >> be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with >> ixgbe, I am guessing >> you have 2 ixgbe and 6 i40e devices in this bond? >> >> Also you mentioned that you are using hfsc, can you provide more information >> about it and also >> how is the traffic routed through your interfaces. I am trying to get a >> better >> picture of your setup >> to be able to better replicate it in our labs. >>