Distribution: 2.6.9 Hardware Environment: ES7000 ia-32 Software Environment: System initialization Problem Description: New generation of ES7000 x86-64 and ia32 servers will have up to 144 I/O APICs, and it is time to re-consider the low value of 8 for possible I/O APICs for SMP and 32 for NUMA systems in i386 and 32 in x86-64. Other platforms are probably planning on many more I/O APICs in the near future. This is a proposed patch to make the MAX_IO_APICS a configurable value (as Len recommended for this case, to assure flexibility): diff -puN arch/i386/Kconfig~max_ioapics arch/i386/Kconfig --- linux-2.6.9/arch/i386/Kconfig~max_ioapics 2004-10-28 02:15:40.362929560 - 0700 +++ linux-2.6.9-natalie/arch/i386/Kconfig 2004-10-28 02:17:46.479756888 - 0700 @@ -485,6 +485,18 @@ config NR_CPUS This is purely to save memory - each supported CPU adds approximately eight kilobytes to the kernel image. +config MAX_IO_APICS + int "Maximum number of IO-APICs (1-256)" + range 1 256 + depends on X86_IO_APIC + default "32" if X86_NUMAQ || X86_SUMMIT || X86_BIGSMP + default "256" if X86_ES7000 || X86_GENERICARCH + default "8" + help + This option allows to specify the maximum number of I/O interrupt + controllers (IO-APICs). The maximim supported value is 256 and + the minimum is 1. + config SCHED_SMT bool "SMT (Hyperthreading) scheduler support" depends on SMP diff -puN include/asm-i386/apicdef.h~max_ioapics include/asm-i386/apicdef.h --- linux-2.6.9/include/asm-i386/apicdef.h~max_ioapics 2004-10-28 02:15:40.441917552 -0700 +++ linux-2.6.9-natalie/include/asm-i386/apicdef.h 2004-10-28 02:18:18.571878144 -0700 @@ -108,11 +108,7 @@ #define APIC_BASE (fix_to_virt(FIX_APIC_BASE)) -#ifdef CONFIG_NUMA - #define MAX_IO_APICS 32 -#else - #define MAX_IO_APICS 8 -#endif +#define MAX_IO_APICS CONFIG_MAX_IO_APICS /* * the local APIC register structure, memory mapped. Not terribly well diff -puN include/asm-x86_64/apicdef.h~max_ioapics include/asm-x86_64/apicdef.h --- linux-2.6.9/include/asm-x86_64/apicdef.h~max_ioapics 2004-10-28 02:15:40.536903112 -0700 +++ linux-2.6.9-natalie/include/asm-x86_64/apicdef.h 2004-10-28 02:18:48.129384720 -0700 @@ -108,7 +108,7 @@ #define APIC_BASE (fix_to_virt(FIX_APIC_BASE)) -#define MAX_IO_APICS 32 +#define MAX_IO_APICS CONFIG_MAX_IO_APICS /* * the local APIC register structure, memory mapped. Not terribly well diff -puN include/asm-x86_64/apicdef.h~max_ioapics include/asm-x86_64/apicdef.h --- linux-2.6.9/include/asm-x86_64/apicdef.h~max_ioapics 2004-10-28 02:15:40.000000000 -0700 +++ linux-2.6.9-root/include/asm-x86_64/apicdef.h 2004-10-28 02:18:48.000000000 -0700 @@ -108,7 +108,7 @@ #define APIC_BASE (fix_to_virt(FIX_APIC_BASE)) -#define MAX_IO_APICS 32 +#define MAX_IO_APICS CONFIG_MAX_IO_APICS /* * the local APIC register structure, memory mapped. Not terribly well diff -puN arch/x86_64/Kconfig~max_ioapics arch/x86_64/Kconfig --- linux-2.6.9/arch/x86_64/Kconfig~max_ioapics 2004-11-16 11:24:38.256336128 - 0800 +++ linux-2.6.9-root/arch/x86_64/Kconfig 2004-11-16 11:32:34.785892624 - 0800 @@ -294,6 +294,17 @@ config NR_CPUS This is purely to save memory - each supported CPU requires memory in the static kernel configuration. +config MAX_IO_APICS + int "Maximum number of IO-APICs (1-256)" + range 1 256 + depends on X86_IO_APIC + default "256" if GENERIC_CPU + default "32" + help + This option allows to specify the maximum number of I/O interrupt + controllers (IO-APICs). The maximim supported value is 256 and + the minimum is 1. + config GART_IOMMU bool "IOMMU support" help Steps to reproduce:
spelling fix in that patch in arch/i386/Kconfig in second line of help text: should be "maximum"
Thanks, Dominik. Looks like I copied the typo to the other Kconfig (for x86- 64) also...
Len, please close this "bug report". Natalie, please bring patches to LKML instead of bugzilla.
This bug will be marked closed when a fix has shipped in the upstream kernel.
Created attachment 4539 [details] Updated MAX_IO_APICS patch Further testing has revealed that increasing the # of IO-APICs for i386 to 256 causes the kmap and pkmap memory areas to overlap, which prevented the system from booting because the kernel panics early in the boot sequence. Next generation ES7000 requirements for maximum IO-APIC limits have been finalized for both i386 and x86_64 architectures. The new limits are: i386 - 65 IOAPICs x86_64 - 129 IOAPICs
Created attachment 4541 [details] Updated MAX_IO_APICS patch Fixed up typos in the help descriptions of Kconfig's in previous updated patch.
Created attachment 4558 [details] Kernel boot trace containing pkmap and kmap overlap panic As per Len Brown's request, this is a 32-bit SMP kernel boot trace containing pkmap and kmap area overlap panic when MAX_IO_APICS is increased to 256.
Created attachment 5013 [details] tweaked version of the Unisys patch above I've applied this patch to the acpi-test tree.
May 17th Andi Kleen already hard-coded x86_64 to MAX_IO_APICS 128, up from 32. We'll increase i386 to 64 the same way -- no config option.
MAX_IO_APICS 64 for i386 shipped in 2.6.13-rc3
I'll be out of the office with limited access to email until 07/28.