Bug 218109 - [AMD] [RYZEN] Binned down CPUs are reported as offline
Summary: [AMD] [RYZEN] Binned down CPUs are reported as offline
Status: NEW
Alias: None
Product: Linux
Classification: Unclassified
Component: Kernel (show other bugs)
Hardware: AMD Linux
: P3 normal
Assignee: Virtual assignee for kernel bugs
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-06 19:34 UTC by Peter Jung
Modified: 2024-03-13 18:16 UTC (History)
3 users (show)

See Also:
Kernel Version: 6.6
Subsystem:
Regression: No
Bisected commit-id:


Attachments
acpidump (823.67 KB, text/plain)
2023-11-06 21:09 UTC, Peter Jung
Details

Description Peter Jung 2023-11-06 19:34:26 UTC
Currently AMD Ryzen CPUs which are not the maximum core state and probably binned down CPUs are reported as offline.
I don't think this is intended to be like this. This is reproduceable on 5900X and 5800X (and probably others).

This issue was for a short time fixed in 6.2 or 6.3 (im not sure)

This can be checked with following command:
```
cat /sys/devices/system/cpu/online
0-23
cat /sys/devices/system/cpu/offline 
24-31
```

Example cpupower frequency set -g:

```
sudo cpupower frequency-set -g performance
[sudo] password for ptr1337: 
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
Setting cpu: 8
Setting cpu: 9
Setting cpu: 10
Setting cpu: 11
Setting cpu: 12
Setting cpu: 13
Setting cpu: 14
Setting cpu: 15
Setting cpu: 16
Setting cpu: 17
Setting cpu: 18
Setting cpu: 19
Setting cpu: 20
Setting cpu: 21
Setting cpu: 22
Setting cpu: 23
Following CPUs are offline:
24-31
cpupower set operation was not performed on them
```

RCU does also the restrict the CPUs to 32, instead of 24:

```
[    0.095488] rcu: Preemptible hierarchical RCU implementation.
[    0.095488] rcu:     RCU restricting CPUs from NR_CPUS=320 to nr_cpu_ids=32.
[    0.095489] rcu:     RCU priority boosting: priority 1 delay 500 ms.
[    0.095489]  Trampoline variant of Tasks RCU enabled.
[    0.095490]  Rude variant of Tasks RCU enabled.
[    0.095490]  Tracing variant of Tasks RCU enabled.
[    0.095490] rcu: RCU calculated value of scheduler-enlistment delay is 50 jiffies.
[    0.095491] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=32
```

Same counts for pcpu-alloc:
```
[    0.042056] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15 
[    0.042060] pcpu-alloc: [0] 16 17 18 19 20 21 22 23 [0] 24 25 26 27 28 29 30 31 
```

If this could be an regression at the RCU Subsystem or others needs to be checked, im not sure how this correctly works.

System Information:

Operating System: CachyOS Linux 
KDE Plasma Version: 5.27.9
KDE Frameworks Version: 5.111.0
Qt Version: 5.15.11
Kernel Version: 6.6.0-4-cachyos-sched-ext-lto (64-bit)
Graphics Platform: Wayland
Processors: 24 × AMD Ryzen 9 5900X 12-Core Processor
Memory: 31,2 GiB of RAM
Graphics Processor: NVIDIA GeForce GTX 1070 Ti/PCIe/SSE2
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: B550 AORUS ELITE AX V2
Comment 1 Mario Limonciello (AMD) 2023-11-06 20:49:53 UTC
Would you mind sharing an acpidump for your system?  I think this is actually a BIOS bug not a kernel bug.
Comment 2 Peter Jung 2023-11-06 21:09:13 UTC
Created attachment 305373 [details]
acpidump

acpidump
Comment 3 Peter Jung 2023-11-06 21:09:57 UTC
Hi Mario,

I have attached you one.
Since we know a bunch of people, which have this issue on their Zen3 CPU it could then also affect several BIOS.
Comment 4 Mario Limonciello (AMD) 2023-11-06 21:18:36 UTC
> [008h 0008   1]                     Revision : 04

It appears that the MADT in this BIOS conforms to FADT Major version 04, no minor version specified.

https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#fixed-acpi-description-table-fadt

> $ grep "Runtime Online Capable : 0" apic.dsl    | wc -l
32

None of the processors are declared as runtime online capable (hotpluggable)

> $ grep "Enabled : 1" apic.dsl    | wc -l
24
> $ grep "Enabled : 0" apic.dsl    | wc -l
8

You can see that your /proc/cpuinfo matches the 12 cores, but the BIOS has entries for another 4 cores that makes Linux advertise this incorrectly.

The way this stuff all works is that there is a function acpi_is_processor_usable() which looks at the FADT version to decide whether to trust the values in the MADT.  This is because the bit for "Runtime Online Capable" wasn't introduced until FADT major 6 minor 3

https://github.com/torvalds/linux/blob/d2f51b3516dade79269ff45eae2a7668ae711b25/arch/x86/kernel/acpi/boot.c#L200

Here is the ACPI spec location for it:
https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/05_ACPI_Software_Programming_Model/ACPI_Software_Programming_Model.html#processor-local-apic-structure

> I have attached you one.
Since we know a bunch of people, which have this issue on their Zen3 CPU it could then also affect several BIOS.

Is this just a cosmetic reporting issue, or is there a functional Linux problem because of this BIOS bug?
Comment 5 Mario Limonciello (AMD) 2023-11-06 21:23:22 UTC
> This issue was for a short time fixed in 6.2 or 6.3 (im not sure)

The reason that this changed after kernel 6.3 is because there was a regression reported by forcing this to be ACPI 6.3 or later for hotplug CPUs to work.
This is the commit that changed the behavior.

https://github.com/torvalds/linux/commit/fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c
Comment 6 Mario Limonciello (AMD) 2023-11-06 21:25:39 UTC
Sorry I made one mistake.  This is the field that has FADT major/minor.  You have 6.2 not 4.0.

[008h 0008   1]                     Revision : 06
[083h 0131   1]          FADT Minor Revision : 02
Comment 7 Peter Jung 2023-11-07 16:13:57 UTC
Hi Mario,

Thank you very much for your explanation and going into the details.
I am not not sure, how much this could affect the general linux system.

The only regression, which I found, which happened due of this issue was in sched-ext - but this is nothing in the offical kernel. You can follow the issue here:

https://github.com/sched-ext/sched_ext/issues/69

Setting nr_cpus=24 fixed this issue and hetjun also has impleneted a fix for this for "scx_rusty".

But the question is, how much this false reporting does affect the RCU Subsystem, Watchdog and percpualloc. (There could be maybe other also, these are which I know and report the threads as 32 instead of 24)
Comment 8 Mario Limonciello (AMD) 2023-11-07 17:50:12 UTC
> Thank you very much for your explanation and going into the details.

One more thing - 

You might want to read this original bug report to better understand how we got where we are.

https://lore.kernel.org/all/20230327191026.3454-1-eric.devolder@oracle.com/
> The only regression, which I found, which happened due of this issue was in
> sched-ext - but this is nothing in the offical kernel. You can follow the
> issue here:

I'll leave a comment.

> But the question is, how much this false reporting does affect the RCU
> Subsystem, Watchdog and percpualloc. (There could be maybe other also, these
> are which I know and report the threads as 32 instead of 24)

My expectation is that it shouldn't break other things and it's just cosmetic, but if you find otherwise please mention.
Comment 9 David Vernet 2024-03-13 18:16:25 UTC
Hi Mario,

I don't think I'd say this is just cosmetic. The kernel looks at the possible cpumap in a bunch of places, and does all sorts of allocations, work, etc based on what it thinks are CPUs that could someday come online. Wouldn't this affect any place where cpu_possible_mask is referenced?

Note You need to log in before you can comment on or make changes to this bug.