Distribution: Scientific linux 4.2 Hardware Environment: Dual xeon 3.2GHz Problem Description: When using isolcpus kernel parameter, init task may runs on one of the isolated cpus, which is not someting we want. For example, on my test platform, init task always run on last cpu, whatever isolcpus value is. Since isolated cpus are isolated from scheduler task balancing, all processes will run on only one cpu. Also, one of the use of isolcpus parameter is for isolate a realtime task from all normal tasks. For example, on my system, if I set isolcpus=3, I expect to have all normal tasks on cpu 0, 1 and 2, which let me run a real time task on cpu 3. Good behaviour would be to launch init task on one of the non isolated cpus.
Just FYI, what some folks do to ensure that init doesn't start on an isolated cpu is run init via some init=<executable that execs init> redirection when booting the kernel.
I tried to recreate it on my machine(i386 platform,2.6.17-rc3 kernel),but failed to do so,is this can be reproducable consistently ? Thanks Srinivasa DS
The cause for the problem is,process which executes the /sbin/init can be executed on any processor(cpu isolation code is not yet executed at this point and hence even on isolated cpu) and when it exec's /sbin/init, init has the capability to run on that processor(eventhough it is isolated). init/main.c ================================================================== static int init(void * unused) { lock_kernel(); /* * init can run on any cpu. ========> can be run anywhere, */ ======== cpu isolation code is not yet executed set_cpus_allowed(current, CPU_MASK_ALL); /* * Tell the world that we're going to be the grim * reaper of innocent orphaned children. * * We don't want people to have to make incorrect * assumptions about where in the task array this * can be found. */ child_reaper = current; ........................ ....................... do_pre_smp_initcalls(); smp_init(); sched_init_smp(); ========> cpu isolation code is executed from here cpuset_init_smp(); ................... .................... if (execute_command) { run_init_process(execute_command); printk(KERN_WARNING "Failed to execute %s. Attempting " "defaults...\n", execute_command); } run_init_process("/sbin/init"); ====> exec's /sbin/init run_init_process("/etc/init"); run_init_process("/bin/init"); run_init_process("/bin/sh"); panic("No init found. Try passing init= option to kernel."); } ======================================================= Debugging further .......
Emmanuel Pacaud This is the patch,which makes sures that init runs on non-isolated cpus. I tested this on my i386 machine,it was working.Please test this and give me your feed back. Signed-off-by: Srinivasa DS <srinivasa@in.ibm.com> --- kernel/sched.c | 1 + 1 file changed, 1 insertion(+) Index: linux-2.6.17.1/kernel/sched.c =================================================================== --- linux-2.6.17.1.orig/kernel/sched.c 2006-06-20 15:01:55.000000000 +0530 +++ linux-2.6.17.1/kernel/sched.c 2006-06-29 15:46:28.000000000 +0530 @@ -5961,6 +5961,7 @@ cpus_andnot(cpu_default_map, *cpu_map, cpu_isolated_map); build_sched_domains(&cpu_default_map); + sched_setaffinity(1,cpu_default_map); } static void arch_destroy_sched_domains(const cpumask_t *cpu_map)
For some reason, some mails exchanged regarding this bug weren't archived here. So here's the other proposed patch. Unfortunately, I'll be not able to test them before 3 weeks. ----- Nick Piggin <nickpiggin@yahoo.com.au> wrote: > > Andrew Morton wrote: > > > > Begin forwarded message: > > > > Date: Thu, 22 Jun 2006 05:18:30 -0700 > > From: bugme-daemon@bugzilla.kernel.org > > To: bugme-new@lists.osdl.org > > Subject: [Bugme-new] [Bug 6733] New: init task run on wrong cpu when using isolcpus snip > > Good behaviour would be to launch init task on one of the non isolated cpus. > > Hmm, fair enough I guess. > > I don't know if something like this patch would do the trick? > Hopefully Emmanuel can test it? Index: linux-2.6/kernel/sched.c =================================================================== --- linux-2.6.orig/kernel/sched.c 2006-06-03 16:00:06.000000000 +1000 +++ linux-2.6/kernel/sched.c 2006-06-23 05:39:54.000000000 +1000 @@ -4355,7 +4355,7 @@ void show_state(void) * NOTE: this function does not set the idle thread's NEED_RESCHED * flag, to make booting more robust. */ -void __devinit init_idle(task_t *idle, int cpu) +void __cpuinit init_idle(task_t *idle, int cpu) { runqueue_t *rq = cpu_rq(cpu); unsigned long flags; @@ -4976,7 +4976,7 @@ static void cpu_attach_domain(struct sch } /* cpus with isolated domains */ -static cpumask_t __devinitdata cpu_isolated_map = CPU_MASK_NONE; +cpumask_t __cpuinitdata cpu_isolated_map = CPU_MASK_NONE; /* Setup the mask of cpus configured for isolated domains */ static int __init isolated_cpu_setup(char *str) @@ -6082,11 +6082,20 @@ static int update_sched_domains(struct n void __init sched_init_smp(void) { + cpumask_t non_isolated_cpus; + lock_cpu_hotplug(); arch_init_sched_domains(&cpu_online_map); + cpus_andnot(non_isolated_cpus, cpu_online_map, cpu_isolated_map); + if (cpus_empty(non_isolated_cpus)) + cpu_set(smp_processor_id(), non_isolated_cpus); unlock_cpu_hotplug(); /* XXX: Theoretical race here - CPU may be hotplugged now */ hotcpu_notifier(update_sched_domains, 0); + + /* Move init over to a non-isolated CPU */ + if (set_cpus_allowed(current, non_isolated_cpus) < 0) + BUG(); } #else void __init sched_init_smp(void)
And Ingo Molnar's post: * Andrew Morton <akpm@osdl.org> wrote: > Hopefully Emmanuel can test it? > > Index: linux-2.6/kernel/sched.c > =================================================================== the fix certainly makes sense, so if it fixes the problem it has my ack for 2.6.18: Acked-by: Ingo Molnar <mingo@elte.hu> Ingo
I've just tested both patches, and they seem to work fine for me. Tested on a dual xeon machine, hyperthreading disabled, with current vanilla kernel from git repository (I've applied these patches manually, since they don't apply cleanly anymore).