Bug 196727 - WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0xc5/0x122
Summary: WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0xc5/0x122
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: Other (show other bugs)
Hardware: Intel Linux
: P1 high
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-08-22 06:51 UTC by Pawel Staszewski
Modified: 2017-08-24 09:29 UTC (History)
1 user (show)

See Also:
Kernel Version: 4.12.8
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Pawel Staszewski 2017-08-22 06:51:58 UTC
[ 2452.163445] NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 0 timed out
[ 2452.163467] ------------[ cut here ]------------
[ 2452.163475] WARNING: CPU: 11 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0xc5/0x122
[ 2452.163476] Modules linked in: ipmi_si x86_pkg_temp_thermal
[ 2452.163482] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G          I     4.12.8 #1
[ 2452.163483] task: ffff88086d8fe200 task.stack: ffffc90003214000
[ 2452.163485] RIP: 0010:dev_watchdog+0xc5/0x122
[ 2452.163485] RSP: 0018:ffff88087f843ea0 EFLAGS: 00010286
[ 2452.163486] RAX: 0000000000000039 RBX: ffff88046ca78000 RCX: 0000000000000000
[ 2452.163487] RDX: ffff88087f853310 RSI: ffff88087f84caa8 RDI: ffff88087f84caa8
[ 2452.163487] RBP: ffff88087f843eb0 R08: 0000000000000000 R09: 0000000000000000
[ 2452.163488] R10: 00000000003d0900 R11: ffff88087ff9194c R12: 0000000000000000
[ 2452.163489] R13: ffffffff815a37a2 R14: ffff88046ca78438 R15: ffff88046ca78000
[ 2452.163490] FS:  0000000000000000(0000) GS:ffff88087f840000(0000) knlGS:0000000000000000
[ 2452.163490] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2452.163491] CR2: 00000000006b2008 CR3: 0000000001c09000 CR4: 00000000001406e0
[ 2452.163492] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2452.163492] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2452.163493] Call Trace:
[ 2452.163494]  <IRQ>
[ 2452.163500]  call_timer_fn+0x5b/0x123
[ 2452.163501]  run_timer_softirq+0x137/0x15a
[ 2452.163503]  ? tk_clock_read+0xc/0xe
[ 2452.163505]  ? timekeeping_get_ns+0x1d/0x31
[ 2452.163506]  ? ktime_get+0x2d/0x3e
[ 2452.163508]  __do_softirq+0xe4/0x23a
[ 2452.163512]  ? tick_program_event+0x5d/0x64
[ 2452.163513]  irq_exit+0x4d/0x5b
[ 2452.163518]  smp_apic_timer_interrupt+0x29/0x34
[ 2452.163520]  apic_timer_interrupt+0x86/0x90
[ 2452.163524] RIP: 0010:cpuidle_enter_state+0x12f/0x184
[ 2452.163525] RSP: 0018:ffffc90003217ea0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 2452.163526] RAX: 0000023af054988b RBX: 0000000000000002 RCX: 000000000000001f
[ 2452.163527] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000000
[ 2452.163527] RBP: ffffc90003217ed0 R08: 00000000ffffffe3 R09: 000000003ff003ff
[ 2452.163528] R10: ffffc90003217e70 R11: 0000000000000004 R12: ffff88086c10d400
[ 2452.163528] R13: 0000023af054988b R14: 0000000000000002 R15: 0000023af0528d14
[ 2452.163529]  </IRQ>
[ 2452.163531]  cpuidle_enter+0x12/0x14
[ 2452.163534]  do_idle+0x10e/0x166
[ 2452.163535]  cpu_startup_entry+0x1a/0x1c
[ 2452.163537]  start_secondary+0xd0/0xd3
[ 2452.163540]  secondary_startup_64+0x9f/0x9f
[ 2452.163541] Code: 70 15 73 00 00 75 38 48 89 df c6 05 64 15 73 00 01 e8 1d 4a fe ff 44 89 e1 48 89 de 48 c7 c7 eb 8f a9 81 48 89 c2 e8 b4 ab b3 ff <0f> ff eb 10 41 ff c4 48 05 40 01 00 00 41 39 f4 75 9a eb 0d 48
[ 2452.163559] ---[ end trace e35aaedfdb6bf6da ]---
[ 2452.163562] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[ 2452.163579] ixgbe 0000:02:00.0 eth0: Reset adapter
[ 2473.173038] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[ 2473.325081] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[ 2473.450701] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[ 2473.450784] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss
[ 2473.537116] ixgbe 0000:02:00.0 eth0: Reset adapter
[ 2473.821091] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[ 2473.969178] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[ 2474.112745] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[ 2474.112830] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss
[ 2474.177126] ixgbe 0000:02:00.0 eth0: Reset adapter
[ 2474.461154] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[ 2474.613208] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
Comment 1 Pawel Staszewski 2017-08-22 06:55:19 UTC
Server is acting as a router.
After NETDEV WATCHDOG - network card is reset / link down / up
Server is performing ipv4/ipv6 routing with hfsc attached to vlan interfaces
qdisc used in hfsc class is pfifo.
Comment 2 Pawel Staszewski 2017-08-22 06:57:45 UTC
flow controll on eth0 is disabled - and below ethtool -k
ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]

ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  off
RX:             off
TX:             off
Comment 3 Pawel Staszewski 2017-08-22 07:09:26 UTC
Latest working kernel with same configuration - but without such a problems was:
linux-4.7-rc4
Comment 4 Pawel Staszewski 2017-08-22 10:35:26 UTC
for more information - now one ethernet cart from eight is flapping
[15467.174551] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[15470.758632] ixgbe 0000:02:00.0 eth0: Reset adapter
[15471.042547] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[15471.190556] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[15471.325786] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[15471.325874] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss
[15471.394573] ixgbe 0000:02:00.0 eth0: Reset adapter
[15471.674592] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[15471.818623] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[15472.255439] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[15472.255527] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss
[15474.050770] ixgbe 0000:02:00.0 eth0: Reset adapter
[15474.330804] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[15474.482819] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[15474.884169] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[15474.884260] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx work after link loss
[15476.710974] ixgbe 0000:02:00.0 eth0: Reset adapter
[15476.990989] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[15477.247034] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[15477.668676] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[15479.459197] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None



- some stats from ethtool shows 
    tx_timeout_count: 20
    tx_restart_queue: 173

Full stats for flapping nic:
 ethtool -S eth0
NIC statistics:
     rx_packets: 7767843441
     tx_packets: 8025263723
     rx_bytes: 6399828526281
     tx_bytes: 6555440321515
     rx_pkts_nic: 7766101929
     tx_pkts_nic: 8021027400
     rx_bytes_nic: 6430435493532
     tx_bytes_nic: 6588439785661
     lsc_int: 69
     tx_busy: 0
     non_eop_descs: 0
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 89
     tx_dropped: 0
     multicast: 68302
     broadcast: 311731
     rx_no_buffer_count: 0
     collisions: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     hw_rsc_aggregated: 0
     hw_rsc_flushed: 0
     fdir_match: 54
     fdir_miss: 7806649663
     fdir_overflow: 0
     rx_fifo_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_timeout_count: 20
     tx_restart_queue: 173
     rx_long_length_errors: 82
     rx_short_length_errors: 0
     tx_flow_control_xon: 0
     rx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_flow_control_xoff: 0
     rx_csum_offload_errors: 12603
     alloc_rx_page_failed: 0
     alloc_rx_buff_failed: 0
     rx_no_dma_resources: 1728853
     os2bmc_rx_by_bmc: 0
     os2bmc_tx_by_bmc: 0
     os2bmc_tx_by_host: 0
     os2bmc_rx_by_host: 0
     tx_queue_0_packets: 1608023617
     tx_queue_0_bytes: 1318855627459
     tx_queue_1_packets: 1583071804
     tx_queue_1_bytes: 1288775532888
     tx_queue_2_packets: 1612868585
     tx_queue_2_bytes: 1318475306052
     tx_queue_3_packets: 1612908620
     tx_queue_3_bytes: 1317436359489
     tx_queue_4_packets: 1608391097
     tx_queue_4_bytes: 1311897495627
     tx_queue_5_packets: 0
     tx_queue_5_bytes: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_8_packets: 0
     tx_queue_8_bytes: 0
     tx_queue_9_packets: 0
     tx_queue_9_bytes: 0
     tx_queue_10_packets: 0
     tx_queue_10_bytes: 0
     tx_queue_11_packets: 0
     tx_queue_11_bytes: 0
     tx_queue_12_packets: 0
     tx_queue_12_bytes: 0
     tx_queue_13_packets: 0
     tx_queue_13_bytes: 0
     tx_queue_14_packets: 0
     tx_queue_14_bytes: 0
     tx_queue_15_packets: 0
     tx_queue_15_bytes: 0
     tx_queue_16_packets: 0
     tx_queue_16_bytes: 0
     tx_queue_17_packets: 0
     tx_queue_17_bytes: 0
     tx_queue_18_packets: 0
     tx_queue_18_bytes: 0
     tx_queue_19_packets: 0
     tx_queue_19_bytes: 0
     tx_queue_20_packets: 0
     tx_queue_20_bytes: 0
     tx_queue_21_packets: 0
     tx_queue_21_bytes: 0
     tx_queue_22_packets: 0
     tx_queue_22_bytes: 0
     tx_queue_23_packets: 0
     tx_queue_23_bytes: 0
     tx_queue_24_packets: 0
     tx_queue_24_bytes: 0
     tx_queue_25_packets: 0
     tx_queue_25_bytes: 0
     tx_queue_26_packets: 0
     tx_queue_26_bytes: 0
     tx_queue_27_packets: 0
     tx_queue_27_bytes: 0
     tx_queue_28_packets: 0
     tx_queue_28_bytes: 0
     tx_queue_29_packets: 0
     tx_queue_29_bytes: 0
     tx_queue_30_packets: 0
     tx_queue_30_bytes: 0
     tx_queue_31_packets: 0
     tx_queue_31_bytes: 0
     tx_queue_32_packets: 0
     tx_queue_32_bytes: 0
     tx_queue_33_packets: 0
     tx_queue_33_bytes: 0
     tx_queue_34_packets: 0
     tx_queue_34_bytes: 0
     tx_queue_35_packets: 0
     tx_queue_35_bytes: 0
     tx_queue_36_packets: 0
     tx_queue_36_bytes: 0
     tx_queue_37_packets: 0
     tx_queue_37_bytes: 0
     tx_queue_38_packets: 0
     tx_queue_38_bytes: 0
     tx_queue_39_packets: 0
     tx_queue_39_bytes: 0
     tx_queue_40_packets: 0
     tx_queue_40_bytes: 0
     tx_queue_41_packets: 0
     tx_queue_41_bytes: 0
     tx_queue_42_packets: 0
     tx_queue_42_bytes: 0
     tx_queue_43_packets: 0
     tx_queue_43_bytes: 0
     tx_queue_44_packets: 0
     tx_queue_44_bytes: 0
     tx_queue_45_packets: 0
     tx_queue_45_bytes: 0
     tx_queue_46_packets: 0
     tx_queue_46_bytes: 0
     tx_queue_47_packets: 0
     tx_queue_47_bytes: 0
     tx_queue_48_packets: 0
     tx_queue_48_bytes: 0
     tx_queue_49_packets: 0
     tx_queue_49_bytes: 0
     tx_queue_50_packets: 0
     tx_queue_50_bytes: 0
     tx_queue_51_packets: 0
     tx_queue_51_bytes: 0
     tx_queue_52_packets: 0
     tx_queue_52_bytes: 0
     tx_queue_53_packets: 0
     tx_queue_53_bytes: 0
     tx_queue_54_packets: 0
     tx_queue_54_bytes: 0
     tx_queue_55_packets: 0
     tx_queue_55_bytes: 0
     tx_queue_56_packets: 0
     tx_queue_56_bytes: 0
     tx_queue_57_packets: 0
     tx_queue_57_bytes: 0
     tx_queue_58_packets: 0
     tx_queue_58_bytes: 0
     tx_queue_59_packets: 0
     tx_queue_59_bytes: 0
     tx_queue_60_packets: 0
     tx_queue_60_bytes: 0
     tx_queue_61_packets: 0
     tx_queue_61_bytes: 0
     tx_queue_62_packets: 0
     tx_queue_62_bytes: 0
     tx_queue_63_packets: 0
     tx_queue_63_bytes: 0
     rx_queue_0_packets: 1571561078
     rx_queue_0_bytes: 1282564409376
     rx_queue_1_packets: 1521956074
     rx_queue_1_bytes: 1246814972926
     rx_queue_2_packets: 1599206777
     rx_queue_2_bytes: 1329637736301
     rx_queue_3_packets: 1550789189
     rx_queue_3_bytes: 1293201812756
     rx_queue_4_packets: 1524330353
     rx_queue_4_bytes: 1247609613454
     rx_queue_5_packets: 0
     rx_queue_5_bytes: 0
     rx_queue_6_packets: 0
     rx_queue_6_bytes: 0
     rx_queue_7_packets: 0
     rx_queue_7_bytes: 0
     rx_queue_8_packets: 0
     rx_queue_8_bytes: 0
     rx_queue_9_packets: 0
     rx_queue_9_bytes: 0
     rx_queue_10_packets: 0
     rx_queue_10_bytes: 0
     rx_queue_11_packets: 0
     rx_queue_11_bytes: 0
     rx_queue_12_packets: 0
     rx_queue_12_bytes: 0
     rx_queue_13_packets: 0
     rx_queue_13_bytes: 0
     rx_queue_14_packets: 0
     rx_queue_14_bytes: 0
     rx_queue_15_packets: 0
     rx_queue_15_bytes: 0
     rx_queue_16_packets: 0
     rx_queue_16_bytes: 0
     rx_queue_17_packets: 0
     rx_queue_17_bytes: 0
     rx_queue_18_packets: 0
     rx_queue_18_bytes: 0
     rx_queue_19_packets: 0
     rx_queue_19_bytes: 0
     rx_queue_20_packets: 0
     rx_queue_20_bytes: 0
     rx_queue_21_packets: 0
     rx_queue_21_bytes: 0
     rx_queue_22_packets: 0
     rx_queue_22_bytes: 0
     rx_queue_23_packets: 0
     rx_queue_23_bytes: 0
     rx_queue_24_packets: 0
     rx_queue_24_bytes: 0
     rx_queue_25_packets: 0
     rx_queue_25_bytes: 0
     rx_queue_26_packets: 0
     rx_queue_26_bytes: 0
     rx_queue_27_packets: 0
     rx_queue_27_bytes: 0
     rx_queue_28_packets: 0
     rx_queue_28_bytes: 0
     rx_queue_29_packets: 0
     rx_queue_29_bytes: 0
     rx_queue_30_packets: 0
     rx_queue_30_bytes: 0
     rx_queue_31_packets: 0
     rx_queue_31_bytes: 0
     rx_queue_32_packets: 0
     rx_queue_32_bytes: 0
     rx_queue_33_packets: 0
     rx_queue_33_bytes: 0
     rx_queue_34_packets: 0
     rx_queue_34_bytes: 0
     rx_queue_35_packets: 0
     rx_queue_35_bytes: 0
     rx_queue_36_packets: 0
     rx_queue_36_bytes: 0
     rx_queue_37_packets: 0
     rx_queue_37_bytes: 0
     rx_queue_38_packets: 0
     rx_queue_38_bytes: 0
     rx_queue_39_packets: 0
     rx_queue_39_bytes: 0
     rx_queue_40_packets: 0
     rx_queue_40_bytes: 0
     rx_queue_41_packets: 0
     rx_queue_41_bytes: 0
     rx_queue_42_packets: 0
     rx_queue_42_bytes: 0
     rx_queue_43_packets: 0
     rx_queue_43_bytes: 0
     rx_queue_44_packets: 0
     rx_queue_44_bytes: 0
     rx_queue_45_packets: 0
     rx_queue_45_bytes: 0
     rx_queue_46_packets: 0
     rx_queue_46_bytes: 0
     rx_queue_47_packets: 0
     rx_queue_47_bytes: 0
     rx_queue_48_packets: 0
     rx_queue_48_bytes: 0
     rx_queue_49_packets: 0
     rx_queue_49_bytes: 0
     rx_queue_50_packets: 0
     rx_queue_50_bytes: 0
     rx_queue_51_packets: 0
     rx_queue_51_bytes: 0
     rx_queue_52_packets: 0
     rx_queue_52_bytes: 0
     rx_queue_53_packets: 0
     rx_queue_53_bytes: 0
     rx_queue_54_packets: 0
     rx_queue_54_bytes: 0
     rx_queue_55_packets: 0
     rx_queue_55_bytes: 0
     rx_queue_56_packets: 0
     rx_queue_56_bytes: 0
     rx_queue_57_packets: 0
     rx_queue_57_bytes: 0
     rx_queue_58_packets: 0
     rx_queue_58_bytes: 0
     rx_queue_59_packets: 0
     rx_queue_59_bytes: 0
     rx_queue_60_packets: 0
     rx_queue_60_bytes: 0
     rx_queue_61_packets: 0
     rx_queue_61_bytes: 0
     rx_queue_62_packets: 0
     rx_queue_62_bytes: 0
     rx_queue_63_packets: 0
     rx_queue_63_bytes: 0
     tx_pb_0_pxon: 0
     tx_pb_0_pxoff: 0
     tx_pb_1_pxon: 0
     tx_pb_1_pxoff: 0
     tx_pb_2_pxon: 0
     tx_pb_2_pxoff: 0
     tx_pb_3_pxon: 0
     tx_pb_3_pxoff: 0
     tx_pb_4_pxon: 0
     tx_pb_4_pxoff: 0
     tx_pb_5_pxon: 0
     tx_pb_5_pxoff: 0
     tx_pb_6_pxon: 0
     tx_pb_6_pxoff: 0
     tx_pb_7_pxon: 0
     tx_pb_7_pxoff: 0
     rx_pb_0_pxon: 0
     rx_pb_0_pxoff: 0
     rx_pb_1_pxon: 0
     rx_pb_1_pxoff: 0
     rx_pb_2_pxon: 0
     rx_pb_2_pxoff: 0
     rx_pb_3_pxon: 0
     rx_pb_3_pxoff: 0
     rx_pb_4_pxon: 0
     rx_pb_4_pxoff: 0
     rx_pb_5_pxon: 0
     rx_pb_5_pxoff: 0
     rx_pb_6_pxon: 0
     rx_pb_6_pxoff: 0
     rx_pb_7_pxon: 0
     rx_pb_7_pxoff: 0
Comment 5 emil.s.tantilov 2017-08-22 17:55:11 UTC
Could you provide information about the network device - device name and id (lspci/-n)?
Comment 6 Pawel Staszewski 2017-08-22 17:57:51 UTC
net-next kernel and tested devices:

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 01)
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 01)
02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 01)
02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 01)
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 02)
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 02)
03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 02)
03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 
for 10GbE SFP+ (rev 02)


and:


d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)


W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> emil.s.tantilov@intel.com changed:
>
>             What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                   CC|                            |emil.s.tantilov@intel.com
>
> --- Comment #5 from emil.s.tantilov@intel.com ---
> Could you provide information about the network device - device name and id
> (lspci/-n)?
>
Comment 7 Pawel Staszewski 2017-08-22 18:02:08 UTC
same problem on both devices.

But currently there left only one flapping port from one nic


d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection (rev 01)


  ethtool -S eth0 | grep tx
      tx_packets: 26166150912
      tx_bytes: 21237240220551
      tx_pkts_nic: 26159988271
      tx_bytes_nic: 21419124752842
      tx_busy: 0
      tx_errors: 0
      tx_dropped: 0
      tx_aborted_errors: 0
      tx_carrier_errors: 0
      tx_fifo_errors: 0
      tx_heartbeat_errors: 0
      tx_timeout_count: 76
      tx_restart_queue: 639
      tx_flow_control_xon: 0
      tx_flow_control_xoff: 0


flow controll is disabled on both nic and switch.

Also - this flapping is less frequent when server is more busy (dont 
know why but meybee it has something to do with cpu 
scalling/cstates/pstates when cpu is locked on highest frequency and is 
bussy all the time there is no problem - when traffic is going down 
there is more nic resets)

[36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
work after link loss
[36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter
[36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
work after link loss
[36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter
[36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
work after link loss
[36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter
[36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter
[41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter
[42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None
[42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down
[42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
work after link loss
[42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter
[42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
[42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow 
Control: None


W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) ---
> net-next kernel and tested devices:
>
> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 01)
> 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 01)
> 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 01)
> 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 01)
> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 02)
> 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 02)
> 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 02)
> 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
> for 10GbE SFP+ (rev 02)
>
>
> and:
>
>
> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection (rev 01)
> d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection (rev 01)
>
>
> W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze:
>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>
>> emil.s.tantilov@intel.com changed:
>>
>>              What    |Removed                     |Added
>> ----------------------------------------------------------------------------
>>                    CC|                            |emil.s.tantilov@intel.com
>>
>> --- Comment #5 from emil.s.tantilov@intel.com ---
>> Could you provide information about the network device - device name and id
>> (lspci/-n)?
>>
Comment 8 Pawel Staszewski 2017-08-23 14:43:40 UTC
I just catch what is happening when nic reset occures

perf top -C0

This network card is binded by affinity to cpus 0-4

When nic is resetting there is 100% cpu load on cpu 0 where this nic is 
bonded also.


   20.29%  [kernel]  [k] _raw_spin_lock_irqsave
     19.35%  [kernel]  [k] md_check_recovery
     12.35%  [kernel]  [k] _raw_spin_unlock_irqrestore
      9.14%  [kernel]  [k] do_raw_spin_lock
      5.95%  [kernel]  [k] atomic64_cmpxchg
      3.98%  [kernel]  [k] md_thread
      3.52%  [kernel]  [k] __wake_up_common
      3.44%  [kernel]  [k] __wake_up
      3.05%  [kernel]  [k] raid1d
      2.98%  [kernel]  [k] md_wakeup_thread
      1.91%  [kernel]  [k] mddev_unlock
      1.61%  [kernel]  [k] to_kthread
      1.53%  [kernel]  [k] blk_flush_plug_list
      1.53%  [kernel]  [k] blk_finish_plug
      1.51%  [kernel]  [k] blk_start_plug
      1.28%  [kernel]  [k] kthread_should_stop
      1.18%  [kernel]  [k] flush_pending_writes
      0.99%  [kernel]  [k] __mutex_trylock_or_owner
      0.69%  [kernel]  [k] mutex_unlock
      0.69%  [kernel]  [k] _raw_spin_lock_irq
      0.67%  [kernel]  [k] _raw_spin_lock
      0.48%  [kernel]  [k] mutex_trylock
      0.44%  [kernel]  [k] mddev_is_clustered
      0.40%  [kernel]  [k] __mutex_trylock
      0.36%  [kernel]  [k] kthread_should_park
      0.16%  [kernel]  [k] acpi_idle_do_entry
      0.07%  [kernel]  [k] skb_release_data
      0.04%  [kernel]  [k] __rtnl_unlock
      0.03%  [kernel]  [k] __slab_free
      0.03%  [kernel]  [k] compound_head
      0.03%  [kernel]  [k] page_frag_free
      0.03%  [kernel]  [k] __free_one_page



W dniu 2017-08-22 o 20:02, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #7 from Pawel Staszewski (pstaszewski@itcare.pl) ---
> same problem on both devices.
>
> But currently there left only one flapping port from one nic
>
>
> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection (rev 01)
>
>
>    ethtool -S eth0 | grep tx
>        tx_packets: 26166150912
>        tx_bytes: 21237240220551
>        tx_pkts_nic: 26159988271
>        tx_bytes_nic: 21419124752842
>        tx_busy: 0
>        tx_errors: 0
>        tx_dropped: 0
>        tx_aborted_errors: 0
>        tx_carrier_errors: 0
>        tx_fifo_errors: 0
>        tx_heartbeat_errors: 0
>        tx_timeout_count: 76
>        tx_restart_queue: 639
>        tx_flow_control_xon: 0
>        tx_flow_control_xoff: 0
>
>
> flow controll is disabled on both nic and switch.
>
> Also - this flapping is less frequent when server is more busy (dont
> know why but meybee it has something to do with cpu
> scalling/cstates/pstates when cpu is locked on highest frequency and is
> bussy all the time there is no problem - when traffic is going down
> there is more nic resets)
>
> [36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down
> [36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
> work after link loss
> [36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter
> [36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down
> [36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
> work after link loss
> [36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter
> [36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down
> [36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
> work after link loss
> [36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter
> [36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down
> [36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
> [41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter
> [41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
> [42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter
> [42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
> [42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down
> [42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
> work after link loss
> [42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter
> [42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
> [42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
> Control: None
>
>
> W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze:
>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>
>> --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) ---
>> net-next kernel and tested devices:
>>
>> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 01)
>> 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 01)
>> 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 01)
>> 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 01)
>> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>> 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>> 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>> 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
>> for 10GbE SFP+ (rev 02)
>>
>>
>> and:
>>
>>
>> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> SFI/SFP+ Network Connection (rev 01)
>> d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> SFI/SFP+ Network Connection (rev 01)
>>
>>
>> W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>>
>>> emil.s.tantilov@intel.com changed:
>>>
>>>               What    |Removed                     |Added
>>>
>>> ----------------------------------------------------------------------------
>>>                     CC|                           
>>>                     |emil.s.tantilov@intel.com
>>>
>>> --- Comment #5 from emil.s.tantilov@intel.com ---
>>> Could you provide information about the network device - device name and id
>>> (lspci/-n)?
>>>
Comment 9 Pawel Staszewski 2017-08-23 14:47:21 UTC
cat /proc/interrupts

             CPU0       CPU1       CPU2       CPU3 CPU4       CPU5       
CPU6       CPU7       CPU8       CPU9 CPU10      CPU11      CPU12      
CPU13      CPU14      CPU15 CPU16      CPU17      CPU18      CPU19      
CPU20      CPU21 CPU22      CPU23      CPU24      CPU25      CPU26      
CPU27 CPU28      CPU29      CPU30      CPU31      CPU32      CPU33 
CPU34      CPU35      CPU36      CPU37      CPU38      CPU39
    0:         23          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0 IO-APIC    2-edge      timer
    8:         18          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0 IO-APIC    8-edge      rtc0
    9:          1          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0 IO-APIC    9-fasteoi   acpi
   18:          0        141          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0 IO-APIC   18-fasteoi   
ehci_hcd:usb1, ehci_hcd:usb2, i801_smbus
   38:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 65536-edge      
ioat-msix
   40:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 67584-edge      
ioat-msix
   41:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 69632-edge      
ioat-msix
   42:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 71680-edge      
ioat-msix
   43:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 73728-edge      
ioat-msix
   44:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 75776-edge      
ioat-msix
   45:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 77824-edge      
ioat-msix
   46:          2          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 79872-edge      
ioat-msix
   48:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67174400-edge      ioat-msix
   50:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67176448-edge      ioat-msix
   51:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67178496-edge      ioat-msix
   52:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67180544-edge      ioat-msix
   53:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67182592-edge      ioat-msix
   54:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67184640-edge      ioat-msix
   55:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67186688-edge      ioat-msix
   56:          0          0          0          0 0          0          
0          0          0          0 2          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
67188736-edge      ioat-msix
   57:          0          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 286720-edge      
ahci[0000:00:11.4]
   58:     379370          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 512000-edge      
ahci[0000:00:1f.2]
   59:  300474267          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
1048576-edge      eth0-TxRx-0
   60:          1  301005834          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
1048577-edge      eth0-TxRx-1
   61:          1          0  301444215          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
1048578-edge      eth0-TxRx-2
   62:          1          0          0  302020538 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
1048579-edge      eth0-TxRx-3
   63:          1          0          0          0 301997942          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0 0   PCI-MSI 
1048580-edge      eth0-TxRx-4
   64:        908          0          0          0 0          0          
0          0          0          0 0          0          0          
0          0          0 0          0          0          0          
0          0 0          0          0          0          0          0 
0          0          0          0          0          0 0          
0          0          0          0          0   PCI-MSI 
1048581-edge      eth0



W dniu 2017-08-23 o 16:43, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #8 from Pawel Staszewski (pstaszewski@itcare.pl) ---
> I just catch what is happening when nic reset occures
>
> perf top -C0
>
> This network card is binded by affinity to cpus 0-4
>
> When nic is resetting there is 100% cpu load on cpu 0 where this nic is
> bonded also.
>
>
>     20.29%  [kernel]  [k] _raw_spin_lock_irqsave
>       19.35%  [kernel]  [k] md_check_recovery
>       12.35%  [kernel]  [k] _raw_spin_unlock_irqrestore
>        9.14%  [kernel]  [k] do_raw_spin_lock
>        5.95%  [kernel]  [k] atomic64_cmpxchg
>        3.98%  [kernel]  [k] md_thread
>        3.52%  [kernel]  [k] __wake_up_common
>        3.44%  [kernel]  [k] __wake_up
>        3.05%  [kernel]  [k] raid1d
>        2.98%  [kernel]  [k] md_wakeup_thread
>        1.91%  [kernel]  [k] mddev_unlock
>        1.61%  [kernel]  [k] to_kthread
>        1.53%  [kernel]  [k] blk_flush_plug_list
>        1.53%  [kernel]  [k] blk_finish_plug
>        1.51%  [kernel]  [k] blk_start_plug
>        1.28%  [kernel]  [k] kthread_should_stop
>        1.18%  [kernel]  [k] flush_pending_writes
>        0.99%  [kernel]  [k] __mutex_trylock_or_owner
>        0.69%  [kernel]  [k] mutex_unlock
>        0.69%  [kernel]  [k] _raw_spin_lock_irq
>        0.67%  [kernel]  [k] _raw_spin_lock
>        0.48%  [kernel]  [k] mutex_trylock
>        0.44%  [kernel]  [k] mddev_is_clustered
>        0.40%  [kernel]  [k] __mutex_trylock
>        0.36%  [kernel]  [k] kthread_should_park
>        0.16%  [kernel]  [k] acpi_idle_do_entry
>        0.07%  [kernel]  [k] skb_release_data
>        0.04%  [kernel]  [k] __rtnl_unlock
>        0.03%  [kernel]  [k] __slab_free
>        0.03%  [kernel]  [k] compound_head
>        0.03%  [kernel]  [k] page_frag_free
>        0.03%  [kernel]  [k] __free_one_page
>
>
>
> W dniu 2017-08-22 o 20:02, bugzilla-daemon@bugzilla.kernel.org pisze:
>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>
>> --- Comment #7 from Pawel Staszewski (pstaszewski@itcare.pl) ---
>> same problem on both devices.
>>
>> But currently there left only one flapping port from one nic
>>
>>
>> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> SFI/SFP+ Network Connection (rev 01)
>>
>>
>>     ethtool -S eth0 | grep tx
>>         tx_packets: 26166150912
>>         tx_bytes: 21237240220551
>>         tx_pkts_nic: 26159988271
>>         tx_bytes_nic: 21419124752842
>>         tx_busy: 0
>>         tx_errors: 0
>>         tx_dropped: 0
>>         tx_aborted_errors: 0
>>         tx_carrier_errors: 0
>>         tx_fifo_errors: 0
>>         tx_heartbeat_errors: 0
>>         tx_timeout_count: 76
>>         tx_restart_queue: 639
>>         tx_flow_control_xon: 0
>>         tx_flow_control_xoff: 0
>>
>>
>> flow controll is disabled on both nic and switch.
>>
>> Also - this flapping is less frequent when server is more busy (dont
>> know why but meybee it has something to do with cpu
>> scalling/cstates/pstates when cpu is locked on highest frequency and is
>> bussy all the time there is no problem - when traffic is going down
>> there is more nic resets)
>>
>> [36460.952474] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [36461.158657] ixgbe 0000:02:00.0 eth0: NIC Link is Down
>> [36461.158746] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>> work after link loss
>> [36461.160471] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [36461.444494] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [36461.592517] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [36462.009063] ixgbe 0000:02:00.0 eth0: NIC Link is Down
>> [36462.009172] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>> work after link loss
>> [36463.812677] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [36464.092692] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [36464.244720] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [36464.362284] ixgbe 0000:02:00.0 eth0: NIC Link is Down
>> [36464.362377] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>> work after link loss
>> [36464.456725] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [36464.736739] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [36464.888774] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [36465.201737] ixgbe 0000:02:00.0 eth0: NIC Link is Down
>> [36467.104942] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [41692.270705] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
>> [41692.270728] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [41697.671115] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [41697.923152] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [42054.026250] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
>> [42054.026269] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [42059.426652] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [42059.570683] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>> [42059.674678] ixgbe 0000:02:00.0 eth0: NIC Link is Down
>> [42059.674774] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>> work after link loss
>> [42059.782684] ixgbe 0000:02:00.0 eth0: Reset adapter
>> [42060.062702] ixgbe 0000:02:00.0 eth0: detected SFP+: 5
>> [42060.310735] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow
>> Control: None
>>
>>
>> W dniu 2017-08-22 o 19:57, bugzilla-daemon@bugzilla.kernel.org pisze:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>>
>>> --- Comment #6 from Pawel Staszewski (pstaszewski@itcare.pl) ---
>>> net-next kernel and tested devices:
>>>
>>> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 01)
>>> 02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 01)
>>> 02:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 01)
>>> 02:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 01)
>>> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 02)
>>> 03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 02)
>>> 03:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 02)
>>> 03:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710
>>> for 10GbE SFP+ (rev 02)
>>>
>>>
>>> and:
>>>
>>>
>>> d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>>> SFI/SFP+ Network Connection (rev 01)
>>> d8:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>>> SFI/SFP+ Network Connection (rev 01)
>>>
>>>
>>> W dniu 2017-08-22 o 19:55, bugzilla-daemon@bugzilla.kernel.org pisze:
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>>>
>>>> emil.s.tantilov@intel.com changed:
>>>>
>>>>                What    |Removed                     |Added
>>>>
>>>>
>>>> ----------------------------------------------------------------------------
>>>>                      CC|
>>>>                      |emil.s.tantilov@intel.com
>>>>
>>>> --- Comment #5 from emil.s.tantilov@intel.com ---
>>>> Could you provide information about the network device - device name and
>>>> id
>>>> (lspci/-n)?
>>>>
Comment 10 emil.s.tantilov 2017-08-23 22:28:10 UTC
Do you see flaps (link down/up events) even without the reset associated with the Tx hangs?

We started a test in our labs attempting to reproduce this issue, but have not had much luck so far. Could you provide more information about the way the interfaces are configured (any settings that differ from default, kernel config may be useful as well).

Since you have a kernel that did not show this problem you may be able to bisect it to the patch that introduced this issue. If you are willing and have the time that is.
Comment 11 Pawel Staszewski 2017-08-23 22:37:31 UTC
nic reset are always initiated by

[144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
work after link loss

there is no flaps without this info


About configuration

This host have 8x 10Gbit ethernet in bond

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth6
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:65:78:94
Slave queue ID: 0

Slave Interface: eth7
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:65:78:95
Slave queue ID: 0

Slave Interface: eth8
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:61:13:bc
Slave queue ID: 0

Slave Interface: eth9
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:61:13:bd
Slave queue ID: 0

Slave Interface: eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:e5:b7:aa
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:e5:b7:ab
Slave queue ID: 0

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:c1:9c:f4
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:c1:9c:f5
Slave queue ID: 0


ip link ls dev bond0
2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff


Traffic that is going thru this bond is ~60Gbit/s  / ~9Mpps - for all 
interfaces.


Traffic type is mixed TCP/UDP - all kind of normal  IP and IPv6 traffic.

and there is not possible to bisect any change / patch from the kernel 
and make many reboots :)

Will try to get some lab test hw and reproduce this ....




W dniu 2017-08-24 o 00:28, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #10 from emil.s.tantilov@intel.com ---
> Do you see flaps (link down/up events) even without the reset associated with
> the Tx hangs?
>
> We started a test in our labs attempting to reproduce this issue, but have
> not
> had much luck so far. Could you provide more information about the way the
> interfaces are configured (any settings that differ from default, kernel
> config
> may be useful as well).
>
> Since you have a kernel that did not show this problem you may be able to
> bisect it to the patch that introduced this issue. If you are willing and
> have
> the time that is.
>
Comment 12 emil.s.tantilov 2017-08-23 23:07:15 UTC
(In reply to Pawel Staszewski from comment #11)
> nic reset are always initiated by
> 
> [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx 
> work after link loss
> 
> there is no flaps without this info
> 
> 
> About configuration
> 
> This host have 8x 10Gbit ethernet in bond

From the devices you listed in your previous comment you have 8 X710 devices which should
be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with ixgbe, I am guessing
you have 2 ixgbe and 6 i40e devices in this bond?

Also you mentioned that you are using hfsc, can you provide more information about it and also
how is the traffic routed through your interfaces. I am trying to get a better picture of your setup
to be able to better replicate it in our labs.
Comment 13 Pawel Staszewski 2017-08-23 23:21:31 UTC
Sorry

Previously i send information where i have similar problems - on hosts 
with same kernel and

8x X710 - have same problem as with

8x 82599


Current configuration where we are focusing on is this with 8x 82599 (4x 
(2x 10G ports nic's)


About the rest of configuration

bond0 have attached about 300 vlans - and each vlan have hfsc qdisc 
attached with ingress qdisc policer (there is no hfsc or ingress qdisc 
on physical interface)


below ip link show:
ip l
256: vlan1673@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
257: vlan1674@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode 
DEFAULT group default qlen 1000
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
258: vlan1676@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
259: vlan1677@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
260: vlan1678@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
261: vlan1679@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
262: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
6: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
263: vlan1681@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
7: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff
264: vlan1682@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
8: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode 
DEFAULT group default qlen 1000
     link/ether 00:25:90:f9:5b:17 brd ff:ff:ff:ff:ff:ff
265: vlan1683@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
9: eth6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
266: vlan1684@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
10: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
267: vlan1685@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
11: eth8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
268: vlan1686@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
12: eth9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq 
master bond0 state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
269: vlan1687@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
13: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group 
default qlen 1000
     link/sit 0.0.0.0 brd 0.0.0.0
270: vlan1688@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
14: vlan4091@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
271: vlan1689@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
15: vlan4032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
272: vlan1690@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
16: vlan617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
273: vlan1691@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
17: vlan627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
274: vlan1692@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
18: vlan1450@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
275: vlan1693@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
19: vlan1451@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
276: vlan1694@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
20: vlan1452@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
277: vlan1696@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
21: vlan1453@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
278: vlan1697@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
22: vlan1454@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
279: vlan1698@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
23: vlan1455@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
280: vlan1870@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
24: vlan1456@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
281: vlan1871@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
25: vlan1457@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
282: vlan1872@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
26: vlan1458@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
283: vlan1873@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
27: vlan1459@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
284: vlan1874@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
28: vlan1460@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
285: vlan1875@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
29: vlan1461@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
286: vlan1876@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
30: vlan1462@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
287: vlan1877@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
31: vlan1463@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
288: vlan1878@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
32: vlan1464@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
289: vlan1879@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
33: vlan1465@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
290: vlan1880@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
34: vlan1466@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
291: vlan1890@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
35: vlan1467@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
292: vlan1891@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
36: vlan1468@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
293: vlan1892@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
37: vlan1469@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
294: vlan616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
38: vlan1470@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
295: vlan979@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:c1:9c:f4 brd ff:ff:ff:ff:ff:ff
39: vlan1471@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
296: vlan985@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff
40: vlan1472@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
297: vlan2290@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
41: vlan1473@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
298: vlan4090@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
42: vlan1474@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
43: vlan1475@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
44: vlan1476@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
45: vlan1477@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
46: vlan1478@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
47: vlan1479@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
48: vlan1480@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
49: vlan1481@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
50: vlan1482@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
51: vlan1483@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
52: vlan1484@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
53: vlan1485@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
54: vlan1486@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
55: vlan1487@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
56: vlan1488@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
57: vlan1489@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
58: vlan1490@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
59: vlan1491@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
60: vlan1492@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
61: vlan1493@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
62: vlan1494@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
63: vlan1495@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
64: vlan1496@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
65: vlan1497@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
66: vlan1498@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
67: vlan1499@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
68: vlan1500@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
69: vlan1501@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
70: vlan1502@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
71: vlan1503@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
72: vlan1504@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
73: vlan1505@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
74: vlan1506@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
75: vlan1507@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
76: vlan1508@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
77: vlan1509@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
78: vlan1510@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
79: vlan1511@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
80: vlan1512@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
81: vlan1513@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
82: vlan1514@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
83: vlan1515@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
84: vlan1516@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
85: vlan1517@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
86: vlan1518@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
87: vlan1519@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
88: vlan1520@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
89: vlan1521@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
90: vlan1522@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
91: vlan1523@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
92: vlan1524@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
93: vlan1525@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
94: vlan1526@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
95: vlan1527@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
96: vlan1528@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
97: vlan1529@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
98: vlan1530@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
99: vlan1531@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
100: vlan1532@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
101: vlan1534@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
102: vlan1535@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
103: vlan1394@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
104: vlan1393@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
105: vlan1550@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
106: vlan1536@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
107: vlan1537@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
108: vlan1538@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
109: vlan1539@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
110: vlan1540@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
111: vlan1542@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
112: vlan1541@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
113: vlan1543@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
114: vlan1544@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
115: vlan1801@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
116: vlan1546@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
117: vlan1547@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
118: vlan1548@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
119: vlan1549@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
120: vlan3132@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
121: vlan3143@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
122: vlan3104@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
123: vlan3125@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
124: vlan3103@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
125: vlan3115@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
126: vlan3134@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
127: vlan3105@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
128: vlan3113@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
129: vlan3141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
130: vlan4009@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
131: vlan3144@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
132: vlan3130@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
133: vlan1803@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
134: vlan3146@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
135: vlan3148@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
136: vlan3109@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
137: vlan1551@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
138: vlan1552@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
139: vlan1553@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
140: vlan1554@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
141: vlan1555@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
142: vlan1556@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
143: vlan1558@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
144: vlan1559@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
145: vlan1560@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
146: vlan1561@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
147: vlan1562@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
148: vlan1563@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
149: vlan1564@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
150: vlan1565@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
151: vlan1566@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
152: vlan1567@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
153: vlan1568@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
154: vlan1569@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
155: vlan1570@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
156: vlan1571@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
157: vlan1572@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
158: vlan1573@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
159: vlan1574@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
160: vlan1575@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
161: vlan1576@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
162: vlan1577@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
163: vlan1578@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
164: vlan1579@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
165: vlan1580@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
166: vlan1581@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
167: vlan1582@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
168: vlan1583@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
169: vlan1584@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
170: vlan1585@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
171: vlan1586@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
172: vlan1587@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
173: vlan1588@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
174: vlan1589@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
175: vlan1591@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
176: vlan1592@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
177: vlan1593@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
178: vlan1594@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
179: vlan1595@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
180: vlan1596@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
181: vlan1597@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
182: vlan1598@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
183: vlan1599@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
184: vlan1557@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
185: vlan1545@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
186: vlan2001@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
187: vlan250@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
188: vlan4043@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
189: vlan1806@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
190: vlan1600@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
191: vlan1602@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
192: vlan1603@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
193: vlan1604@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
194: vlan1605@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
195: vlan1606@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
196: vlan1607@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
197: vlan1608@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
198: vlan1609@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
199: vlan1610@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
200: vlan1611@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
201: vlan1612@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
202: vlan1613@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
203: vlan1614@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
204: vlan1615@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
205: vlan1616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
206: vlan1617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
207: vlan1618@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
208: vlan1619@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
209: vlan1620@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
210: vlan1621@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
211: vlan1625@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
212: vlan1626@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
213: vlan1627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
214: vlan1628@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
215: vlan1629@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
216: vlan1630@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
217: vlan1631@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
218: vlan1632@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
219: vlan1634@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
220: vlan1635@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
221: vlan1636@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
222: vlan1640@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
223: vlan1641@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
224: vlan1642@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
225: vlan1643@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
226: vlan1644@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
227: vlan1645@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
228: vlan1646@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
229: vlan1647@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
230: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
231: vlan1649@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
232: vlan1650@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
233: vlan1651@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
234: vlan1652@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
235: vlan1653@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
236: vlan1654@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
237: vlan1655@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
238: vlan1656@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
239: vlan1657@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
240: vlan1658@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
241: vlan1659@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
242: vlan1660@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
243: vlan1661@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
244: vlan1662@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
245: vlan1663@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
246: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
247: vlan1665@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
248: vlan1601@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
249: vlan1666@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
250: vlan1667@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
251: vlan1668@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
252: vlan1669@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
253: vlan1670@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
254: vlan1671@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
255: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
hfsc state UP mode DEFAULT group default qlen 1000
     link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff


W dniu 2017-08-24 o 01:07, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #12 from emil.s.tantilov@intel.com ---
> (In reply to Pawel Staszewski from comment #11)
>> nic reset are always initiated by
>>
>> [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>> work after link loss
>>
>> there is no flaps without this info
>>
>>
>> About configuration
>>
>> This host have 8x 10Gbit ethernet in bond
>  From the devices you listed in your previous comment you have 8 X710 devices
> which should
> be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with
> ixgbe, I am guessing
> you have 2 ixgbe and 6 i40e devices in this bond?
>
> Also you mentioned that you are using hfsc, can you provide more information
> about it and also
> how is the traffic routed through your interfaces. I am trying to get a
> better
> picture of your setup
> to be able to better replicate it in our labs.
>
Comment 14 Pawel Staszewski 2017-08-24 09:29:59 UTC
Today another interface start to flap:

[175947.099230] ixgbe 0000:82:00.1 eth9: initiating reset due to tx timeout
[175947.099260] ixgbe 0000:82:00.1 eth9: Reset adapter
[176000.515273] INFO: rcu_sched self-detected stall on CPU
[176000.515285]         39-...: (14999 ticks this GP) 
idle=b12/140000000000001/0 softirq=59250004/59250004 fqs=6371
[176000.515285]          (t=15000 jiffies g=7584203 c=7584202 q=8002)
[176000.515293] NMI backtrace for cpu 39
[176000.515296] CPU: 39 PID: 3700 Comm: md2_raid1 Tainted: G        W 
I     4.12.8 #1
[176000.515298] Call Trace:
[176000.515300]  <IRQ>
[176000.515308]  dump_stack+0x4d/0x64
[176000.515311]  nmi_cpu_backtrace+0x76/0x85
[176000.515317]  ? irq_force_complete_move+0xdf/0xdf
[176000.515319]  nmi_trigger_cpumask_backtrace+0x51/0xc6
[176000.515321]  arch_trigger_cpumask_backtrace+0x14/0x17
[176000.515325]  rcu_dump_cpu_stacks+0x86/0xb3
[176000.515330]  rcu_check_callbacks+0x232/0x5e8
[176000.515333]  ? raise_softirq_irqoff+0x9/0x1e
[176000.515337]  update_process_times+0x2a/0x52
[176000.515343]  tick_sched_timer+0xf9/0x12b
[176000.515345]  __hrtimer_run_queues+0xba/0x182
[176000.515347]  hrtimer_interrupt+0x67/0x105
[176000.515349]  local_apic_timer_interrupt+0x46/0x49
[176000.515351]  smp_apic_timer_interrupt+0x24/0x34
[176000.515354]  apic_timer_interrupt+0x86/0x90
[176000.515359] RIP: 0010:atomic64_cmpxchg+0xc/0xe
[176000.515360] RSP: 0018:ffffc90009fa7d78 EFLAGS: 00000246 ORIG_RAX: 
ffffffffffffff10
[176000.515361] RAX: 0000000000000000 RBX: ffff88046b8d8000 RCX: 
0000000000000000
[176000.515362] RDX: ffff88086b024980 RSI: 0000000000000000 RDI: 
ffff88046b8d81e8
[176000.515363] RBP: ffffc90009fa7d78 R08: ffff88086b024980 R09: 
ffff88086b024980
[176000.515364] R10: ffff88046b8d81e8 R11: 000000000000032d R12: 
ffff88086d24af00
[176000.515365] R13: ffffc90009fa7e30 R14: ffff88086d24af28 R15: 
0000000000000000
[176000.515365]  </IRQ>
[176000.515368]  __mutex_trylock_or_owner+0x44/0x5a
[176000.515369]  __mutex_trylock+0x9/0x11
[176000.515372]  mutex_trylock+0x9/0xe
[176000.515375]  md_check_recovery+0xdc/0x3cc
[176000.515378]  raid1d+0x44/0x6e5
[176000.515380]  md_thread+0x133/0x149
[176000.515381]  ? md_thread+0x133/0x149
[176000.515382]  ? wake_up_atomic_t+0x27/0x27
[176000.515383]  ? md_do_sync+0xca0/0xca0
[176000.515386]  kthread+0xf7/0xfc
[176000.515388]  ? init_completion+0x24/0x24
[176000.515390]  ret_from_fork+0x22/0x30
[176005.487661] ixgbe 0000:82:00.1 eth9: detected SFP+: 6
[176005.635699] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow 
Control: None
[176006.103192] ixgbe 0000:82:00.1 eth9: NIC Link is Down
[176006.103359] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx 
work after link loss
[176007.875865] ixgbe 0000:82:00.1 eth9: Reset adapter
[176008.151873] ixgbe 0000:82:00.1 eth9: detected SFP+: 6
[176008.303891] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow 
Control: None
[176008.682820] ixgbe 0000:82:00.1 eth9: NIC Link is Down
[176008.682906] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx 
work after link loss
[176010.532062] ixgbe 0000:82:00.1 eth9: Reset adapter
[176010.812079] ixgbe 0000:82:00.1 eth9: detected SFP+: 6
[176010.968096] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow 
Control: None
[176011.110638] ixgbe 0000:82:00.1 eth9: NIC Link is Down
[176011.110730] ixgbe 0000:82:00.1 eth9: initiating reset to clear Tx 
work after link loss
[176011.172105] ixgbe 0000:82:00.1 eth9: Reset adapter
[176011.456124] ixgbe 0000:82:00.1 eth9: detected SFP+: 6
[176011.600153] ixgbe 0000:82:00.1 eth9: NIC Link is Up 10 Gbps, Flow 
Control: None




W dniu 2017-08-24 o 01:21, bugzilla-daemon@bugzilla.kernel.org pisze:
> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>
> --- Comment #13 from Pawel Staszewski (pstaszewski@itcare.pl) ---
> Sorry
>
> Previously i send information where i have similar problems - on hosts
> with same kernel and
>
> 8x X710 - have same problem as with
>
> 8x 82599
>
>
> Current configuration where we are focusing on is this with 8x 82599 (4x
> (2x 10G ports nic's)
>
>
> About the rest of configuration
>
> bond0 have attached about 300 vlans - and each vlan have hfsc qdisc
> attached with ingress qdisc policer (there is no hfsc or ingress qdisc
> on physical interface)
>
>
> below ip link show:
> ip l
> 256: vlan1673@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 257: vlan1674@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
> DEFAULT group default qlen 1000
>       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 258: vlan1676@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 259: vlan1677@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 260: vlan1678@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 4: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 261: vlan1679@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 5: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 262: vlan1680@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 6: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 263: vlan1681@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 7: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
> DEFAULT group default qlen 1000
>       link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff
> 264: vlan1682@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 8: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
> DEFAULT group default qlen 1000
>       link/ether 00:25:90:f9:5b:17 brd ff:ff:ff:ff:ff:ff
> 265: vlan1683@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 9: eth6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 266: vlan1684@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 10: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 267: vlan1685@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 11: eth8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 268: vlan1686@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 12: eth9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond0 state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 269: vlan1687@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 13: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group
> default qlen 1000
>       link/sit 0.0.0.0 brd 0.0.0.0
> 270: vlan1688@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 14: vlan4091@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 271: vlan1689@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 15: vlan4032@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 272: vlan1690@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 16: vlan617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 273: vlan1691@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 17: vlan627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 274: vlan1692@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 18: vlan1450@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 275: vlan1693@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 19: vlan1451@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 276: vlan1694@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 20: vlan1452@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 277: vlan1696@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 21: vlan1453@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 278: vlan1697@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 22: vlan1454@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 279: vlan1698@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 23: vlan1455@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 280: vlan1870@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 24: vlan1456@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 281: vlan1871@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 25: vlan1457@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 282: vlan1872@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 26: vlan1458@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 283: vlan1873@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 27: vlan1459@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 284: vlan1874@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 28: vlan1460@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 285: vlan1875@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 29: vlan1461@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 286: vlan1876@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 30: vlan1462@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 287: vlan1877@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 31: vlan1463@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 288: vlan1878@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 32: vlan1464@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 289: vlan1879@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 33: vlan1465@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 290: vlan1880@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 34: vlan1466@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 291: vlan1890@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 35: vlan1467@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 292: vlan1891@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 36: vlan1468@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 293: vlan1892@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 37: vlan1469@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 294: vlan616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 38: vlan1470@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 295: vlan979@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:c1:9c:f4 brd ff:ff:ff:ff:ff:ff
> 39: vlan1471@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 296: vlan985@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:f9:5b:16 brd ff:ff:ff:ff:ff:ff
> 40: vlan1472@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 297: vlan2290@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 41: vlan1473@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 298: vlan4090@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 42: vlan1474@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 43: vlan1475@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 44: vlan1476@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 45: vlan1477@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 46: vlan1478@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 47: vlan1479@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 48: vlan1480@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 49: vlan1481@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 50: vlan1482@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 51: vlan1483@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 52: vlan1484@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 53: vlan1485@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 54: vlan1486@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 55: vlan1487@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 56: vlan1488@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 57: vlan1489@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 58: vlan1490@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 59: vlan1491@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 60: vlan1492@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 61: vlan1493@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 62: vlan1494@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 63: vlan1495@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 64: vlan1496@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 65: vlan1497@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 66: vlan1498@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 67: vlan1499@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 68: vlan1500@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 69: vlan1501@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 70: vlan1502@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 71: vlan1503@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 72: vlan1504@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 73: vlan1505@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 74: vlan1506@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 75: vlan1507@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 76: vlan1508@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 77: vlan1509@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 78: vlan1510@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 79: vlan1511@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 80: vlan1512@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 81: vlan1513@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 82: vlan1514@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 83: vlan1515@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 84: vlan1516@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 85: vlan1517@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 86: vlan1518@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 87: vlan1519@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 88: vlan1520@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 89: vlan1521@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 90: vlan1522@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 91: vlan1523@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 92: vlan1524@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 93: vlan1525@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 94: vlan1526@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 95: vlan1527@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 96: vlan1528@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 97: vlan1529@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 98: vlan1530@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 99: vlan1531@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 100: vlan1532@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 101: vlan1534@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 102: vlan1535@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 103: vlan1394@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 104: vlan1393@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 105: vlan1550@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 106: vlan1536@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 107: vlan1537@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 108: vlan1538@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 109: vlan1539@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 110: vlan1540@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 111: vlan1542@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 112: vlan1541@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 113: vlan1543@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 114: vlan1544@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 115: vlan1801@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 116: vlan1546@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 117: vlan1547@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 118: vlan1548@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 119: vlan1549@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 120: vlan3132@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 121: vlan3143@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 122: vlan3104@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 123: vlan3125@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 124: vlan3103@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 125: vlan3115@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 126: vlan3134@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 127: vlan3105@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 128: vlan3113@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 129: vlan3141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 130: vlan4009@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 131: vlan3144@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 132: vlan3130@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 133: vlan1803@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 134: vlan3146@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 135: vlan3148@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 136: vlan3109@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 137: vlan1551@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 138: vlan1552@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 139: vlan1553@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 140: vlan1554@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 141: vlan1555@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 142: vlan1556@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 143: vlan1558@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 144: vlan1559@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 145: vlan1560@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 146: vlan1561@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 147: vlan1562@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 148: vlan1563@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 149: vlan1564@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 150: vlan1565@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 151: vlan1566@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 152: vlan1567@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 153: vlan1568@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 154: vlan1569@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 155: vlan1570@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 156: vlan1571@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 157: vlan1572@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 158: vlan1573@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 159: vlan1574@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 160: vlan1575@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 161: vlan1576@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 162: vlan1577@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 163: vlan1578@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 164: vlan1579@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 165: vlan1580@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 166: vlan1581@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 167: vlan1582@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 168: vlan1583@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 169: vlan1584@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 170: vlan1585@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 171: vlan1586@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 172: vlan1587@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 173: vlan1588@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 174: vlan1589@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 175: vlan1591@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 176: vlan1592@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 177: vlan1593@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 178: vlan1594@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 179: vlan1595@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 180: vlan1596@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 181: vlan1597@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 182: vlan1598@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 183: vlan1599@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 184: vlan1557@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 185: vlan1545@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 186: vlan2001@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 187: vlan250@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 188: vlan4043@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 189: vlan1806@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 190: vlan1600@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 191: vlan1602@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 192: vlan1603@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 193: vlan1604@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 194: vlan1605@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 195: vlan1606@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 196: vlan1607@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 197: vlan1608@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 198: vlan1609@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 199: vlan1610@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 200: vlan1611@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 201: vlan1612@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 202: vlan1613@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 203: vlan1614@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 204: vlan1615@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 205: vlan1616@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 206: vlan1617@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 207: vlan1618@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 208: vlan1619@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 209: vlan1620@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 210: vlan1621@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 211: vlan1625@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 212: vlan1626@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 213: vlan1627@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 214: vlan1628@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 215: vlan1629@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 216: vlan1630@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 217: vlan1631@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 218: vlan1632@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 219: vlan1634@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 220: vlan1635@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 221: vlan1636@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 222: vlan1640@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 223: vlan1641@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 224: vlan1642@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 225: vlan1643@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 226: vlan1644@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 227: vlan1645@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 228: vlan1646@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 229: vlan1647@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 230: vlan1648@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 231: vlan1649@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 232: vlan1650@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 233: vlan1651@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 234: vlan1652@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 235: vlan1653@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 236: vlan1654@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 237: vlan1655@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 238: vlan1656@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 239: vlan1657@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 240: vlan1658@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 241: vlan1659@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 242: vlan1660@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 243: vlan1661@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 244: vlan1662@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 245: vlan1663@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 246: vlan1664@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 247: vlan1665@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 248: vlan1601@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 249: vlan1666@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 250: vlan1667@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 251: vlan1668@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 252: vlan1669@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 253: vlan1670@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 254: vlan1671@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
> 255: vlan1672@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> hfsc state UP mode DEFAULT group default qlen 1000
>       link/ether 00:25:90:65:78:94 brd ff:ff:ff:ff:ff:ff
>
>
> W dniu 2017-08-24 o 01:07, bugzilla-daemon@bugzilla.kernel.org pisze:
>> https://bugzilla.kernel.org/show_bug.cgi?id=196727
>>
>> --- Comment #12 from emil.s.tantilov@intel.com ---
>> (In reply to Pawel Staszewski from comment #11)
>>> nic reset are always initiated by
>>>
>>> [144970.418219] ixgbe 0000:02:00.0 eth0: initiating reset to clear Tx
>>> work after link loss
>>>
>>> there is no flaps without this info
>>>
>>>
>>> About configuration
>>>
>>> This host have 8x 10Gbit ethernet in bond
>>   From the devices you listed in your previous comment you have 8 X710
>>   devices
>> which should
>> be i40e and 2 82599 (ixgbe). From the dmesg logs showing eth0 resetting with
>> ixgbe, I am guessing
>> you have 2 ixgbe and 6 i40e devices in this bond?
>>
>> Also you mentioned that you are using hfsc, can you provide more information
>> about it and also
>> how is the traffic routed through your interfaces. I am trying to get a
>> better
>> picture of your setup
>> to be able to better replicate it in our labs.
>>

Note You need to log in before you can comment on or make changes to this bug.