Bug 205427 - Qdisc running seqcount may cause performance issues
Summary: Qdisc running seqcount may cause performance issues
Status: NEW
Alias: None
Product: Networking
Classification: Unclassified
Component: IPV4 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-05 05:27 UTC by Wen Yang
Modified: 2020-03-15 11:08 UTC (History)
5 users (show)

See Also:
Kernel Version: 4.9
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Wen Yang 2019-11-05 05:27:57 UTC
[ 1261.949409] NMI watchdog: BUG: soft lockup - CPU#80 stuck for 23s! [tc:12076]
[ 1261.956565] Modules linked in: kpatch_D871570(O) kpatch(OE) intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel lrw glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr iohub_sriov(O) mei_me ioatdma i2c_i801 lpc_ich ipmi_si mei dca shpchp mfd_core i2c_smbus wmi ipmi_msghandler acpi_power_meter acpi_pad ip_tables ext4 jbd2 mbcache virtio_net virtio_blk i2c_algo_bit drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci virtio_pci drm virtio_ring(E) libahci virtio libata nvme nvme_core i2c_core
[ 1262.014116] CPU: 80 PID: 12076 Comm: tc Tainted: G           OE K 4.9.151-015.ali3000.alios7.x86_64 #1
[ 1262.023437] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.23.34 02/14/2019
[ 1262.032154] task: ffff887f48dc8000 task.stack: ffffc90037128000
[ 1262.038091] RIP: 0010:[<ffffffff8160fbee>]  [<ffffffff8160fbee>] __gnet_stats_copy_basic+0x8e/0x90
[ 1262.047093] RSP: 0018:ffffc9003712b930  EFLAGS: 00000202
[ 1262.052418] RAX: 000000000011845b RBX: ffffc9003712b9a0 RCX: ffff887f4e39b2b0
[ 1262.059569] RDX: 0000000000000000 RSI: ffffc9003712b93c RDI: ffff887f4e3950bc
[ 1262.066871] RBP: ffffc9003712b968 R08: 0000000000000004 R09: ffff887f5252bf7c
[ 1262.074166] R10: ffff887f58718100 R11: 0000000000000008 R12: 0000000000000000
[ 1262.081465] R13: ffff887f5252bf4c R14: ffff887f4e39b200 R15: ffff887f4e39b2b0
[ 1262.088765] FS:  00007f0a974cd740(0000) GS:ffff887f7f000000(0000) knlGS:0000000000000000
[ 1262.097157] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1262.103067] CR2: 00000000006474c0 CR3: 0000007f50006000 CR4: 00000000007606f0
[ 1262.110368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1262.117665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1262.124962] PKRU: 55555554
[ 1262.127822] Stack:
[ 1262.129985]  ffffffff8160fce8 0000000000000009 0000000000000000 ffffffff8160ff24
[ 1262.137779]  ffff887f58718100 d0e0e5e734dba31b ffff887f58718100 ffffc9003712ba30
[ 1262.145574]  ffffffff8164d383 0000000000000009 0000000000000000 0000000000000000
[ 1262.153363] Call Trace:
[ 1262.155968]  [<ffffffff8160fce8>] ? gnet_stats_copy_basic+0x38/0xe0
[ 1262.162397]  [<ffffffff8160ff24>] ? gnet_stats_start_copy_compat+0x94/0x100
[ 1262.169527]  [<ffffffff8164d383>] tc_fill_qdisc+0x283/0x400
[ 1262.175258]  [<ffffffff8164d635>] tc_dump_qdisc_root+0x135/0x1a0
[ 1262.181430]  [<ffffffff8164d744>] tc_dump_qdisc+0xa4/0x150
[ 1262.187077]  [<ffffffff81654eb1>] netlink_dump+0x231/0x2c0
[ 1262.192721]  [<ffffffff81655d68>] __netlink_dump_start+0x168/0x1a0
[ 1262.199067]  [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0
[ 1262.205413]  [<ffffffff81634093>] rtnetlink_rcv_msg+0x1c3/0x230
[ 1262.211494]  [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0
[ 1262.217835]  [<ffffffff81633ed0>] ? rtnl_newlink+0x860/0x860
[ 1262.223651]  [<ffffffff81657b64>] netlink_rcv_skb+0xa4/0xc0
[ 1262.229387]  [<ffffffff8162e785>] rtnetlink_rcv+0x15/0x20
[ 1262.234942]  [<ffffffff8165754c>] netlink_unicast+0x18c/0x220
[ 1262.240844]  [<ffffffff8165793b>] netlink_sendmsg+0x35b/0x3b0
[ 1262.246758]  [<ffffffff815fca58>] sock_sendmsg+0x38/0x50
[ 1262.252233]  [<ffffffff815fd4ad>] ___sys_sendmsg+0x29d/0x2b0
[ 1262.258060]  [<ffffffff811ed21f>] ? do_wp_page+0x39f/0x850
[ 1262.263708]  [<ffffffff811f04ea>] ? handle_mm_fault+0x6da/0xd50
[ 1262.269791]  [<ffffffff815fde04>] __sys_sendmsg+0x54/0x90
[ 1262.275359]  [<ffffffff815fde52>] SyS_sendmsg+0x12/0x20
[ 1262.280744]  [<ffffffff81003c04>] do_syscall_64+0x74/0x180
[ 1262.286387]  [<ffffffff81741c8e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
[ 1262.293337] Code: 85 ff 74 18 8b 07 a8 01 75 1f 48 8b 11 48 89 16 8b 51 08 89 56 08 39 07 75 ea f3 c3 48 8b 01 48 89 06 8b 41 08 89 46 08 c3 f3 90 <eb> d7 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 8b 47 1c 48 89 
[ 1262.314295] Kernel panic - not syncing: softlockup: hung tasks
[ 1262.320291] CPU: 80 PID: 12076 Comm: tc Tainted: G           OELK 4.9.151-015.ali3000.alios7.x86_64 #1
[ 1262.329904] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.23.34 02/14/2019
[ 1262.338904]  ffff887f7f003e50 ffffffff8139f342 0000000000000000 ffffffff81a4f7f1
[ 1262.346674]  ffff887f7f003ed8 ffffffff811b2245 ffffc90000000008 ffff887f7f003ee8
[ 1262.354456]  ffff887f7f003e80 d0e0e5e734dba31b ffff887f7f003ea7 0000000000000000
[ 1262.362249] Call Trace:
[ 1262.364846]  <IRQ> 
[ 1262.366785]  [<ffffffff8139f342>] dump_stack+0x63/0x81
[ 1262.372095]  [<ffffffff811b2245>] panic+0xf8/0x244
[ 1262.377049]  [<ffffffff81158296>] watchdog_timer_fn+0x226/0x230
[ 1262.383126]  [<ffffffff81158070>] ? watchdog_park_threads+0x70/0x70
[ 1262.389555]  [<ffffffff811085e3>] __hrtimer_run_queues+0xf3/0x270
[ 1262.395807]  [<ffffffff81108d8a>] hrtimer_interrupt+0x9a/0x180
[ 1262.401796]  [<ffffffff810580f8>] local_apic_timer_interrupt+0x38/0x60
[ 1262.408491]  [<ffffffff81745405>] smp_apic_timer_interrupt+0x45/0x60
[ 1262.415006]  [<ffffffff81743b90>] apic_timer_interrupt+0xa0/0xb0
[ 1262.421177]  <EOI> 
[ 1262.423118]  [<ffffffff8160fbee>] ? __gnet_stats_copy_basic+0x8e/0x90
[ 1262.429871]  [<ffffffff8160fce8>] ? gnet_stats_copy_basic+0x38/0xe0
[ 1262.436301]  [<ffffffff8160ff24>] ? gnet_stats_start_copy_compat+0x94/0x100
[ 1262.443428]  [<ffffffff8164d383>] tc_fill_qdisc+0x283/0x400
[ 1262.449164]  [<ffffffff8164d635>] tc_dump_qdisc_root+0x135/0x1a0
[ 1262.455329]  [<ffffffff8164d744>] tc_dump_qdisc+0xa4/0x150
[ 1262.460978]  [<ffffffff81654eb1>] netlink_dump+0x231/0x2c0
[ 1262.466620]  [<ffffffff81655d68>] __netlink_dump_start+0x168/0x1a0
[ 1262.472965]  [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0
[ 1262.479311]  [<ffffffff81634093>] rtnetlink_rcv_msg+0x1c3/0x230
[ 1262.485394]  [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0
[ 1262.491737]  [<ffffffff81633ed0>] ? rtnl_newlink+0x860/0x860
[ 1262.497560]  [<ffffffff81657b64>] netlink_rcv_skb+0xa4/0xc0
[ 1262.503295]  [<ffffffff8162e785>] rtnetlink_rcv+0x15/0x20
[ 1262.508854]  [<ffffffff8165754c>] netlink_unicast+0x18c/0x220
[ 1262.514760]  [<ffffffff8165793b>] netlink_sendmsg+0x35b/0x3b0
[ 1262.520666]  [<ffffffff815fca58>] sock_sendmsg+0x38/0x50
[ 1262.526140]  [<ffffffff815fd4ad>] ___sys_sendmsg+0x29d/0x2b0
[ 1262.531960]  [<ffffffff811ed21f>] ? do_wp_page+0x39f/0x850
[ 1262.537607]  [<ffffffff811f04ea>] ? handle_mm_fault+0x6da/0xd50
[ 1262.543691]  [<ffffffff815fde04>] __sys_sendmsg+0x54/0x90
[ 1262.549360]  [<ffffffff815fde52>] SyS_sendmsg+0x12/0x20
[ 1262.554750]  [<ffffffff81003c04>] do_syscall_64+0x74/0x180
[ 1262.560402]  [<ffffffff81741c8e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6


CPU 80                                                                  CPU 1
execute : tc qd show dev bond0                                 netperf

rtnetlink_rcv (acquired rtnl_mutex)
...
...                                                            __dev_xmit_skb
__gnet_stats_copy_basic       




If netperf on CPU 1 runs for a long time, the tc program may repeatedly acquire seqcount for a long time, resulting in softlockup. At this time, the tc program holds the large lock of rtnl_mutex, which also affects other programs in the system.
Comment 1 Wen Yang 2019-11-05 05:42:22 UTC
edb09eb17ed8: "net: sched: do not acquire qdisc spinlock in qdisc/class stats dump" may require further optimization.

 void
-__gnet_stats_copy_basic(struct gnet_stats_basic_packed *bstats,
+__gnet_stats_copy_basic(const seqcount_t *running,
+			struct gnet_stats_basic_packed *bstats,
 			struct gnet_stats_basic_cpu __percpu *cpu,
 			struct gnet_stats_basic_packed *b)
 {
+	unsigned int seq;
+
 	if (cpu) {
 		__gnet_stats_copy_basic_cpu(bstats, cpu);
-	} else {
+		return;
+	}
+	do {
+		if (running)
+			seq = read_seqcount_begin(running);
 		bstats->bytes = b->bytes;
 		bstats->packets = b->packets;
-	}
+	} while (running && read_seqcount_retry(running, seq));
 }
Comment 2 Wen Yang 2019-11-05 06:04:21 UTC
(In reply to Wen Yang from comment #1)
> edb09eb17ed8: "net: sched: do not acquire qdisc spinlock in qdisc/class
> stats dump" may require further optimization.
> 
>  void
> -__gnet_stats_copy_basic(struct gnet_stats_basic_packed *bstats,
> +__gnet_stats_copy_basic(const seqcount_t *running,
> +                     struct gnet_stats_basic_packed *bstats,
>                       struct gnet_stats_basic_cpu __percpu *cpu,
>                       struct gnet_stats_basic_packed *b)
>  {
> +     unsigned int seq;
> +
>       if (cpu) {
>               __gnet_stats_copy_basic_cpu(bstats, cpu);
> -     } else {
> +             return;
> +     }
> +     do {
> +             if (running)
> +                     seq = read_seqcount_begin(running);
>               bstats->bytes = b->bytes;
>               bstats->packets = b->packets;
> -     }
> +     } while (running && read_seqcount_retry(running, seq));
>  }

Just to get the statistics of bstats, it is repeated here.
It may not be necessary.
Comment 3 Eric Dumazet 2019-11-05 06:07:05 UTC
What kind of cpus is used on this platform ?

The 'running' sequence should eventually be 'released' by the netperf.

It looks like the CPU 80 never sees low order bit of sequence being 0.
Comment 4 Cong Wang 2019-11-05 19:46:39 UTC
Your kernel warning is Tainted, was there anything happened before this soft lockup?

And, 4.9 is too old for upstream, please try to reproduce this on a more recent kernel release, ideally the latest.
Comment 5 Wen Yang 2019-11-08 17:29:42 UTC
(In reply to Eric Dumazet from comment #3)
> What kind of cpus is used on this platform ?
> 
> The 'running' sequence should eventually be 'released' by the netperf.
> 
> It looks like the CPU 80 never sees low order bit of sequence being 0.

Thank you very much for your comments. By analyzing several vmcores, we found that the lowest bit of Qdisc->running is indeed 1.
crash> Qdisc  ffff887da5990c00
struct Qdisc {
  enqueue = 0xffffffff81651890 <pfifo_fast_enqueue>,
  dequeue = 0xffffffff81651770 <pfifo_fast_dequeue>,
  flags = 372,
  limit = 0,
  ops = 0xffffffff81d86040 <pfifo_fast_ops>,
  stab = 0x0,
  hash = {
    next = 0x0,
    pprev = 0xffff887da5991028
  },
  handle = 0,
  parent = 1,
  u32_node = 0x0,
  dev_queue = 0xffff887e24580000,
  rate_est = {
    bps = 0,
    pps = 0
  },
  cpu_bstats = 0x60818043e8d0,
  cpu_qstats = 0x60818043e930,
  gso_skb = {
    next = 0xffff887da5990c80,
    prev = 0xffff887da5990c80,
    qlen = 0,
    lock = {
      {
        rlock = {
          raw_lock = {
            {
              val = {
                counter = 0
              },
              {
                locked = 0 '\000',
                pending = 0 '\000'
              },
              {
                locked_pending = 0,
                tail = 0
              }
            }
          }
        }
      }
    }
  },
  q = {
    head = 0x0,
    tail = 0x0,
    qlen = 0,
    lock = {
      {
        rlock = {
          raw_lock = {
            {
              val = {
                counter = 0
              },
              {
                locked = 0 '\000',
                pending = 0 '\000'
              },
              {
                locked_pending = 0,
                tail = 0
              }
            }
          }
        }
      }
    }
  },
  bstats = {
    bytes = 0,
    packets = 0
  },
  running = {
    sequence = 4624263
  },
  qstats = {
    qlen = 0,
    backlog = 0,
    drops = 0,
    requeues = 0,
    overlimits = 0
  },
  state = 0,
  next_sched = 0xffff887cba53e000,
  skb_bad_txq = {
    next = 0xffff887da5990ce8,
    prev = 0xffff887da5990ce8,
    qlen = 0,
    lock = {
      {
        rlock = {
          raw_lock = {
            {
              val = {
                counter = 0
              },
              {
                locked = 0 '\000',
                pending = 0 '\000'
              },
              {
                locked_pending = 0,
                tail = 0
              }
            }
          }
        }
      }
    }
  },
  padded = 0,
  refcnt = {
    counter = 1
  },
  busylock = {
    {
      rlock = {
        raw_lock = {
          {
            val = {
              counter = 0
            },
            {
              locked = 0 '\000',
              pending = 0 '\000'
            },
            {
              locked_pending = 0,
              tail = 0
            }
          }
        }
      }
    }
  }
}
crash>


It may be that the packet traffic at that moment is too large, or it may be a bug in a third-party kernel module, causing this soft lockup.

We may avoid this problem by replacing read_seqcount_begin() with raw_read_seqcount_latch() here.

diff --git a/net/core/gen_stats.c b/net/core/gen_stats.c
index 1d653fbfcf52..8dad81e4be26 100644
--- a/net/core/gen_stats.c
+++ b/net/core/gen_stats.c
@@ -150,7 +150,7 @@ __gnet_stats_copy_basic(const seqcount_t *running,
        }
        do {
                if (running)
-                       seq = read_seqcount_begin(running);
+                       seq = raw_read_seqcount_latch(running);
                bstats->bytes = b->bytes;
                bstats->packets = b->packets;
        } while (running && read_seqcount_retry(running, seq));

I would greatly appreciate it if you kindly give me some suggestions。
Comment 6 Eric Dumazet 2019-11-08 18:09:54 UTC
I wonder if there is a bug then... about one Qdisc->running change being lost
and low order bit always at the wrong value.

raw_read_seqcount_latch() can not be used in the Qdisc context.

raw_read_seqcount_latch() is only used in some classes having dual set
of data structures like in kernel/time/timekeeping.c

Qdisc->running seems to be stuck on your host, right ?

What qdisc are you using ?

If this is pfifo_fast, could you try another one like fq ?
Comment 7 Wen Yang 2019-11-09 14:46:23 UTC
(In reply to Eric Dumazet from comment #6)
> I wonder if there is a bug then... about one Qdisc->running change being lost
> and low order bit always at the wrong value.
> 
> raw_read_seqcount_latch() can not be used in the Qdisc context.
> 
> raw_read_seqcount_latch() is only used in some classes having dual set
> of data structures like in kernel/time/timekeeping.c
> 
> Qdisc->running seems to be stuck on your host, right ?
> 
> What qdisc are you using ?
> 
> If this is pfifo_fast, could you try another one like fq ?

Yes, disc->running is stuck.

I read the code over and over and found that there might be a problem in this place:

3159 static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
3160                                  struct net_device *dev,
3161                                  struct netdev_queue *txq)
3162 {
...
...
3198         } else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) &&
3199                    qdisc_run_begin(q)) {     

*** 
The statements between these && conditions are not necessarily executed sequentially. If the qdisc_run_begin(q) statement is executed first and the other conditions are not satisfied, the lowest bit of q->running will always be 1.
***
...
3206                 qdisc_bstats_update(q, skb);
...
3215
3216                 qdisc_run_end(q);
3217                 rc = NET_XMIT_SUCCESS;
3218         }
Comment 8 Wen Yang 2019-11-09 15:15:47 UTC
This patch may fix it, please help me to check it.
Thanks.


diff --git a/net/core/dev.c b/net/core/dev.c
index 20c7a67..0ee8161 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3602,27 +3602,27 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 	spinlock_t *root_lock = qdisc_lock(q);
 	struct sk_buff *to_free = NULL;
 	bool contended;
-	int rc;
+	int rc = NET_XMIT_SUCCESS;

 	qdisc_calculate_pkt_len(skb, q);

 	if (q->flags & TCQ_F_NOLOCK) {
-		if ((q->flags & TCQ_F_CAN_BYPASS) && q->empty &&
-		    qdisc_run_begin(q)) {
-			if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED,
-					      &q->state))) {
-				__qdisc_drop(skb, &to_free);
-				rc = NET_XMIT_DROP;
-				goto end_run;
-			}
-			qdisc_bstats_cpu_update(q, skb);
+		if ((q->flags & TCQ_F_CAN_BYPASS) && q->empty) {
+			if (qdisc_run_begin(q)) {
+				if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED,
+								&q->state))) {
+					__qdisc_drop(skb, &to_free);
+					rc = NET_XMIT_DROP;
+					goto end_run;
+				}
+				qdisc_bstats_cpu_update(q, skb);

-			rc = NET_XMIT_SUCCESS;
-			if (sch_direct_xmit(skb, q, dev, txq, NULL, true))
-				__qdisc_run(q);
+				if (sch_direct_xmit(skb, q, dev, txq, NULL, true))
+					__qdisc_run(q);

 end_run:
-			qdisc_run_end(q);
+				qdisc_run_end(q);
+			}
 		} else {
 			rc = q->enqueue(skb, q, &to_free) & NET_XMIT_MASK;
 			qdisc_run(q);
@@ -3647,26 +3647,27 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 	if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
 		__qdisc_drop(skb, &to_free);
 		rc = NET_XMIT_DROP;
-	} else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) &&
-		   qdisc_run_begin(q)) {
-		/*
-		 * This is a work-conserving queue; there are no old skbs
-		 * waiting to be sent out; and the qdisc is not running -
-		 * xmit the skb directly.
-		 */
+	} else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q)) {
+		if (qdisc_run_begin(q)) {
+			/*
+			 * This is a work-conserving queue; there are no old skbs
+			 * waiting to be sent out; and the qdisc is not running -
+			 * xmit the skb directly.
+			 */

-		qdisc_bstats_update(q, skb);
+			qdisc_bstats_update(q, skb);

-		if (sch_direct_xmit(skb, q, dev, txq, root_lock, true)) {
-			if (unlikely(contended)) {
-				spin_unlock(&q->busylock);
-				contended = false;
+			if (sch_direct_xmit(skb, q, dev, txq, root_lock, true)) {
+				if (unlikely(contended)) {
+					spin_unlock(&q->busylock);
+					contended = false;
+				}
+				__qdisc_run(q);
 			}
-			__qdisc_run(q);
-		}

-		qdisc_run_end(q);
-		rc = NET_XMIT_SUCCESS;
+			qdisc_run_end(q);
+			rc = NET_XMIT_SUCCESS;
+		}
 	} else {
 		rc = q->enqueue(skb, q, &to_free) & NET_XMIT_MASK;
 		if (qdisc_run_begin(q)) {
Comment 9 Eric Dumazet 2019-11-10 23:40:52 UTC
First of all, you have not tried a recent kernel.

Hint : The problem might have been solved already.
Comment 10 Wen Yang 2019-11-11 06:38:20 UTC
(In reply to Eric Dumazet from comment #9)
> First of all, you have not tried a recent kernel.
> 
> Hint : The problem might have been solved already.


Thanks for your comments, 
we have analyzed the qdisc code and the vmcore again.

1, The flags of the stuck Qdisc:
crash> Qdisc.flags  -x ffff887da5990c00
  flags = 0x174

Corresponding code:
#define TCQ_F_CAN_BYPASS        4
…
#define TCQ_F_NOLOCK           0x100 /* qdisc does not require locking */

This is a NOLOCK qdisc.

2, The stats of the stuck Qdisc:
  q = {
    head = 0x0,
    tail = 0x0,
    qlen = 0x0,
…
  bstats = {
    bytes = 0x0,
    packets = 0x0
  },
…
  qstats = {
    qlen = 0x0,
    backlog = 0x0,
    drops = 0x0,
    requeues = 0x0,
    overlimits = 0x0
  },
  

bstats , qstats and q.qlen are all 0, so we may speculate that qdisc_reset() has been executed.


3,  This patch is very likely to fix this problem, we will try to test it.

d518d2ed8640
net/sched: fix race between deactivation and dequeue for NOLOCK qdisc.

--
Regards,
Comment 11 Eric Dumazet 2019-11-11 16:02:36 UTC
I asked you if you tried another qdisc than pfifo_fast.

You are looking at stats of the qdisc, but pfifo_fast has per-cpu stats which are not seen in the 'struct Qdisc'  storage.

Note You need to log in before you can comment on or make changes to this bug.