Bug 217586
Summary: | High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later | ||
---|---|---|---|
Product: | Linux | Reporter: | Vivek Anand (vivekanand754) |
Component: | Kernel | Assignee: | Virtual assignee for kernel bugs (linux-kernel) |
Status: | NEEDINFO --- | ||
Severity: | high | CC: | bagasdotme, mjguzik |
Priority: | P3 | ||
Hardware: | Other | ||
OS: | Linux | ||
Kernel Version: | Subsystem: | ||
Regression: | No | Bisected commit-id: | |
Attachments: | check empty slots before locking |
Description
Vivek Anand
2023-06-22 10:30:01 UTC
> Is there any known issue/resolution for the same? None, that I'm aware of but you could bisect: https://docs.kernel.org/admin-guide/bug-bisect.html There is a lot of version between 5.17.3 to 5.19.17 It will take time to check exact version from where the issue is coming. Meanwhile can you help me with troubleshooting steps why it is consuming more cpu? Observation from perf report: # Children Self Command Shared Object Symbol # ........ ........ ............... .......................... ............................................................................................................................. # 46.20% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] worker_thread | ---worker_thread | --46.20%--process_one_work | --46.19%--htable_gc | --46.19%--htable_selective_cleanup | |--20.76%--_raw_spin_lock_bh | | | --15.30%--preempt_count_add | | | --4.78%--in_lock_functions | |--16.30%--__local_bh_enable_ip | | | |--6.12%--preempt_count_sub | | | |--2.45%--check_preemption_disabled | | | --1.29%--__this_cpu_preempt_check | |--3.25%--__cond_resched | --2.37%--_raw_spin_unlock_bh 46.20% 0.00% kworker/0:3-eve [kernel.kallsyms] [k] kthread | ---kthread worker_thread | --46.20%--process_one_work | --46.19%--htable_gc | --46.19%--htable_selective_cleanup | |--20.76%--_raw_spin_lock_bh | | | --15.30%--preempt_count_add | | | --4.78%--in_lock_functions | |--16.30%--__local_bh_enable_ip | | | |--6.12%--preempt_count_sub | | | |--2.45%--check_preemption_disabled | | | --1.29%--__this_cpu_preempt_check | |--3.25%--__cond_resched | --2.37%--_raw_spin_unlock_bh (In reply to Vivek Anand from comment #2) > There is a lot of version between 5.17.3 to 5.19.17 > It will take time to check exact version from where the issue is coming. > To help narrowing version range: does v5.15 have this regression? (In reply to Bagas Sanjaya from comment #3) > (In reply to Vivek Anand from comment #2) > > There is a lot of version between 5.17.3 to 5.19.17 > > It will take time to check exact version from where the issue is coming. > > > > To help narrowing version range: does v5.15 have this regression? Oops, I mean please try v5.18 first, and then try latest mainline. As per analysis it seems that the RCA for this issue is the enablement of IBRS mitigation for spectre_v2 vulnerability which was added in Kernel 5.18.14 by commit 6ad0ad2bf8a6 ("x86/bugs: Report Intel retbleed vulnerability"). I have tried reverting the code changes which were added as part of above commit in 5.19.17 and could see significant improvement in CPU consumption. Also , I tried with different Kernel versions and I could not see the issue till Kernel 5.18.13 and from Kernel 5.18.14 onwards I can see the issue. Below are the links : https://lkml.org/lkml/2022/7/23/205 https://lkml.iu.edu/hypermail/linux/kernel/2209.1/02248.html https://www.bleepingcomputer.com/news/linux/vmware-70-percent-drop-in-linux-esxi-vm-performance-with-retbleed-fixes/ Can you try latest mainline (v6.4) to see if it fixes your regression? Issue observed with latest kernel (v6.4) also. ~ # uname -r 6.4.0 ~ # top | grep events_power_efficient 8 root 20 0 0 0 0 R 100.0 0.0 11:00.58 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 100.0 0.0 11:03.58 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.3 0.0 11:06.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.7 0.0 11:09.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.7 0.0 11:12.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.7 0.0 11:15.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.7 0.0 11:18.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 100.0 0.0 11:21.57 kworker/0:1+events_power_efficient 8 root 20 0 0 0 0 R 99.7 0.0 11:24.57 kworker/0:1+events_power_efficient Unsetting config "CONFIG_CPU_IBRS_ENTRY" also improved CPU (CPU consumption by events_power_efficient process dropped from ~100% to ~7%). Any suggestions if I can go with above change or not. What could be the impact if I disable "CONFIG_CPU_IBRS_ENTRY" config. Created attachment 304973 [details]
check empty slots before locking
It may be the routine is get called more than it used to (or is getting called way too often). By any chance have you increased the hash size for netfilter? Regardless of any of the above, the cleaning routine is incredibly inefficient. Can you try out the patch I attached? I'm probably going to redo it later, but it would be nice to know if it sorts it out as is. The gcinterval/expire entry was set too low (30ms/15ms instead of 30s/15s), which was causing the Linux kworker thread events_power_efficient to be called excessively. Setting it to appropriate value fixed the issue. Issue Scenario: --hashlimit-htable-gcinterval 30 --hashlimit-htable-expire 15 Fixed Scenario: --hashlimit-htable-gcinterval 30000 --hashlimit-htable-expire 15000 |