Bug 15948

Summary:	race condition by setting cpu scaling governor
Product:	Other	Reporter:	Andrej Gelenberg (andrej.gelenberg)
Component:	Other	Assignee:	other_other
Status:	CLOSED OBSOLETE
Severity:	normal	CC:	akpm, alan
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	2.6.35-rc4	Subsystem:
Regression:	Yes	Bisected commit-id:
Attachments:	Logs, before kernel panic achook full dmesg from kdump [PATCH] [CPUFREQ] fix race condition in store_scaling_governor [CPUFREQ] Revert "[CPUFREQ] remove rwsem lock from CPUFREQ_GOV_STOP call (second call site)" test for the bug

Description Andrej Gelenberg 2010-05-09 18:00:42 UTC

Created attachment 26302 [details]
Logs, before kernel panic

Kernel panic at system start.

gdb kernel/build/vmlinux bug/vmcore 
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/andrej/kernel/build/vmlinux...done.

warning: exec file is newer than core file.
[New Thread 2867]
#0  set_wq_data (cwq=0xffff8800af9b0100, work=0xffffffff818738e0) at /home/andrej/kernel/linux/kernel/workqueue.c:225
225             BUG_ON(!work_pending(work));
(gdb) bt
#0  set_wq_data (cwq=0xffff8800af9b0100, work=0xffffffff818738e0) at /home/andrej/kernel/linux/kernel/workqueue.c:225
#1  insert_work (cwq=0xffff8800af9b0100, work=0xffffffff818738e0) at /home/andrej/kernel/linux/kernel/workqueue.c:243
#2  __queue_work (cwq=0xffff8800af9b0100, work=0xffffffff818738e0) at /home/andrej/kernel/linux/kernel/workqueue.c:260
#3  0xffffffff8106591a in delayed_work_timer_fn (__data=<value optimized out>) at /home/andrej/kernel/linux/kernel/workqueue.c:316
#4  0xffffffff8105ea62 in __run_timers (h=<value optimized out>) at /home/andrej/kernel/linux/kernel/timer.c:1026
#5  run_timer_softirq (h=<value optimized out>) at /home/andrej/kernel/linux/kernel/timer.c:1218
#6  0xffffffff81059913 in __do_softirq () at /home/andrej/kernel/linux/kernel/softirq.c:219
#7  0xffffffff810288ea in ?? () at /home/andrej/kernel/linux/arch/x86/kernel/entry_64.S:1226
#8  0xffffffff8181ff58 in ?? ()
#9  0xffffffff8181ff78 in ?? ()
#10 0xffffffff8102a36d in do_softirq () at /home/andrej/kernel/linux/arch/x86/kernel/irq_64.c:80
Backtrace stopped: frame did not save the PC

Comment 1 Andrej Gelenberg 2010-05-09 18:09:05 UTC

Created attachment 26303 [details]
achook

This is my tool, which cause kernel panic, if run from udevd at system start. If i run this tool after system booted, no kernel panic occur.

Comment 2 Andrew Morton 2010-05-10 22:02:24 UTC

How do you know that do_dbs_timer() caused this?

Comment 3 Andrej Gelenberg 2010-05-11 12:05:21 UTC

I have this problem, when udev run achook and no AC there. Udev rule:

SUBSYSTEM!="power_supply",GOTO=achook_end
KERNEL=="AC0",GOTO=achook_run
KERNEL=="ACAD",GOTO=achook_run
GOTO=achook_end
LABEL=achook_run
RUN+="BINDIR/achook"
LABEL=achook_end

Comment 4 Andrej Gelenberg 2010-05-11 12:06:26 UTC

Created attachment 26340 [details]
full dmesg from kdump

Full dmesg from kdump

Comment 5 Andrej Gelenberg 2010-05-11 12:07:33 UTC

back trace from 2.6.34-rc7:
[New Thread 1902]
#0  __list_del (h=<value optimized out>) at /home/andrej/kernel/linux/include/linux/list.h:92
92              next->prev = prev;
(gdb) bt
#0  __list_del (h=<value optimized out>) at /home/andrej/kernel/linux/include/linux/list.h:92
#1  detach_timer (h=<value optimized out>) at /home/andrej/kernel/linux/kernel/timer.c:596
#2  __run_timers (h=<value optimized out>) at /home/andrej/kernel/linux/kernel/timer.c:998
#3  run_timer_softirq (h=<value optimized out>) at /home/andrej/kernel/linux/kernel/timer.c:1218
#4  0xffffffff81059913 in __do_softirq () at /home/andrej/kernel/linux/kernel/softirq.c:219
#5  0xffffffff810288ea in ?? () at /home/andrej/kernel/linux/arch/x86/kernel/entry_64.S:1226
#6  0xffffffff8181ff58 in ?? ()
#7  0xffffffff8181ff78 in ?? ()
#8  0xffffffff8102a36d in do_softirq () at /home/andrej/kernel/linux/arch/x86/kernel/irq_64.c:80
Backtrace stopped: frame did not save the PC

Comment 6 Andrej Gelenberg 2010-05-11 12:08:41 UTC

> How do you know that do_dbs_timer() caused this?
With gdb: print worh->run (something like this)

Comment 7 Andrej Gelenberg 2010-05-11 14:00:45 UTC

Created attachment 26344 [details]
[PATCH] [CPUFREQ] fix race condition in store_scaling_governor

There is a race condition in store_scaling_governor (drivers/cpufreq/cpufreq.c) if switch scaling governor very fast. I wrapped store_scaling_governor with mutex lock cpufreq_governor_mutex.
I will send my patch to the mailing list.

Comment 8 Andrej Gelenberg 2010-05-14 10:58:32 UTC

Created attachment 26377 [details]
[CPUFREQ] Revert "[CPUFREQ] remove rwsem lock from CPUFREQ_GOV_STOP call (second call site)"

Comment 9 Andrej Gelenberg 2010-07-05 18:59:48 UTC

The bug is still there in 2.6.35-rc4 :(

Comment 10 Andrej Gelenberg 2010-07-05 19:32:58 UTC

Created attachment 27026 [details]
test for the bug

run and see, if you had the bug.

Comment 11 Andrej Gelenberg 2010-07-05 19:37:56 UTC

Comment on attachment 27026 [details]
test for the bug

fix mime

Comment 12 Andrew Morton 2010-07-08 00:07:33 UTC

Please dont' run a bugzilla thread and a linux-kernel thread in parallel.  It makes it really hard.

Nobody looks in bugzilla so it's best to stick to the email thread.

I have cpufreq-revert-remove-rwsem-lock-from-cpufreq_gov_stop-call-second-call-site.patch queued under my "cpufreq, needed in 2.6.35" category.  cpufreq maintainership seems kinda dead.  I'll take care of it, somehow.