Bug 57471 - 12 core system configured for flat interrupt mode
Summary: 12 core system configured for flat interrupt mode
Status: CLOSED INVALID
Alias: None
Product: ACPI
Classification: Unclassified
Component: Config-Processors (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Aaron Lu
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-05-03 02:00 UTC by erbenton
Modified: 2013-07-08 00:41 UTC (History)
1 user (show)

See Also:
Kernel Version: 3.8.2
Subsystem:
Regression: No
Bisected commit-id:


Attachments
dmesg log with apic=debug added to kernel cmd line (63.96 KB, text/plain)
2013-06-29 02:30 UTC, erbenton
Details
dmar acpi table (7.27 KB, text/plain)
2013-07-05 17:39 UTC, erbenton
Details

Description erbenton 2013-05-03 02:00:05 UTC
Sandy Bridge-E 6 core/HT (12 processors) is being configured in logical flat mode.
From dmesg:
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x00] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x00] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x00] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x00] disabled)

Once booted i checked the local apic's:

cpu  0 logical 01000000 physical 00000000
cpu  1 logical 02000000 physical 01000000
cpu  2 logical 04000000 physical 02000000
cpu  3 logical 08000000 physical 03000000
cpu  4 logical 10000000 physical 04000000
cpu  5 logical 20000000 physical 05000000
cpu  6 logical 40000000 physical 06000000
cpu  7 logical 80000000 physical 07000000
cpu  8 logical 00000000 physical 08000000
cpu  9 logical 00000000 physical 09000000
cpu 10 logical 00000000 physical 0a000000
cpu 11 logical 00000000 physical 0b000000

I would have expected cluster mode with logical id's (in hex):
   01,02,04,08,11,12,14,18,21,22,24,28
x0 isnt a valid cluster id as the lower nibble is a mask meaning only 1,2,4,8 are selectable.
Flat mode is only capable of supporting 8 processors or less.
Comment 1 Aaron Lu 2013-06-28 09:13:40 UTC
Can you please attach dmesg with apic=debug added to the kernel command line? Thanks.
Comment 2 erbenton 2013-06-29 02:30:51 UTC
Created attachment 106271 [details]
dmesg log with apic=debug added to kernel cmd line

I have attached the requested info.
Comment 3 Aaron Lu 2013-07-01 02:15:11 UTC
From your dmesg, the apic is in physical flat mode, which supports more than 8 cpus:
[    0.020000] Switched APIC routing to physical flat.

Hi Len,
Do you know if this is the expected setup here? Thanks.
Comment 4 Aaron Lu 2013-07-01 03:08:45 UTC
I saw other systems will use x2apic_cluster mode, perhaps your CPU doesn't support that?

[    0.033861] Enabled IRQ remapping in x2apic mode
[    0.033864] Enabling x2apic
[    0.033867] Enabled x2apic
[    0.033877] Switched APIC routing to cluster x2apic.
Comment 5 erbenton 2013-07-03 03:28:11 UTC
I agree with your assessment of physical flat mode. I guess as long as you only use physical destination IDs its ok. The logical id's ought to be filled in properly tho.

I have BIOS (intel bios) enabled for Virtualization and VTd (why 2 opts? don't know) there is no specific option there to enable X2Apic. The kernel is configured to support x2apic ( CONFIG_X86_X2APIC=y ). After boot I checked for x2apic support, "cpuid 1" returns ecx[21]=1 supporting x2apic, and msr 0x1b bit 10 = 0 which shows x2 mode is disabled. The DMAR table is there in acpi  and has 2 DRHD structures defined that look right, I can "talk" to either apic defined in these structures.   So, whats keeping x2mode from being setup? I see no mention of it in the logs at all. I'll see if I can determine what the kernel is thinking, I have the source.
Comment 6 erbenton 2013-07-03 03:57:41 UTC
Browsing the kernel source I found that if x2apic isn't enabled by BIOS then the kernel won't enable it. So x2apic not working is due to BIOS not handing off to the kernel in x2apic mode. I wish i could file a "bios bug/missing feature" on this. It would be nice to have it working
Comment 7 Aaron Lu 2013-07-03 04:45:30 UTC
(In reply to comment #6)
> Browsing the kernel source I found that if x2apic isn't enabled by BIOS then
> the kernel won't enable it. So x2apic not working is due to BIOS not handing
> off to the kernel in x2apic mode. I wish i could file a "bios bug/missing
> feature" on this. It would be nice to have it working

It may be due to BIOS doesn't think this system should support x2apic. I'm not sure of the optimal setting of APIC for your system, I think you can ask such questions in linux kernel mailing list, people may answer your questions there.
Comment 8 Aaron Lu 2013-07-05 07:09:08 UTC
Most likely, the DMAR table in your system doesn't claim support of interrupt remapping. Can you please attach acpidump here? Thanks.
Comment 9 erbenton 2013-07-05 17:39:37 UTC
Created attachment 106815 [details]
dmar acpi table

attached requested dmar acpi table
Comment 10 Aaron Lu 2013-07-06 05:19:59 UTC
The DMAR table doesn't claim support of interrupt remapping(flags is 0x00 instead of 0x01) so Linux will not force enable x2apic mode. I don't see Linux does anything wrong with APIC config for your system, what do you think?
Comment 11 erbenton 2013-07-06 18:45:53 UTC
I agree, this isn't a linux issue. This issue is due to the bios not setting up acpi correctly. I doubt it will get fixed.
This can be closed.
Comment 12 Aaron Lu 2013-07-08 00:41:02 UTC
BIOS doesn't set interrupt remapping flag in DMAR table.

Note You need to log in before you can comment on or make changes to this bug.