Bug 76561 - Kernel deadlocks trying to acquire hrtimer base and runqueue spinlocks in different order on different CPUs
Summary: Kernel deadlocks trying to acquire hrtimer base and runqueue spinlocks in dif...
Status: NEW
Alias: None
Product: Other
Classification: Unclassified
Component: Other (show other bugs)
Hardware: IA-32 Linux
: P1 low
Assignee: other_other
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-05-21 08:28 UTC by regmka
Modified: 2014-05-21 08:28 UTC (History)
0 users

See Also:
Kernel Version: 3.13.9
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description regmka 2014-05-21 08:28:03 UTC
The kernel may deadlock while printing the message in the routine clockevents_increase_min_delta()
while other CPUs handle a timer interrupt. CPU backtraces are:

CPU0:
   c165254a (T) _raw_spin_lock+58
   c1082984 (t) try_to_wake_up+292          ; <= acquires a runqueue spinlock (1) @ http://lxr.free-electrons.com/source/kernel/sched/core.c#L1561
   c1082abf (T) wake_up_process+31
   c1651351 (t) __up+43 
   c109750a (T) up+58 
   c10a3715 (T) console_unlock+565 
   c10a3c12 (T) vprintk_emit+722 
   c16473b2 (T) printk+80 
   c10b742d (t) clockevents_program_min_delta+253 
   c10b7d9f (T) clockevents_program_event+175 
   c10b9419 (T) tick_program_event+41
   c1077a89 (t) hrtimer_force_reprogram+137 
   c1077b8d (t) __remove_hrtimer+141 
   c1077c9c (T) hrtimer_try_to_cancel+124   ; <= acquires a timer base spinlock (2) @ http://lxr.free-electrons.com/source/kernel/hrtimer.c#L1082
   c1077cf9 (T) hrtimer_cancel+25 
   c10b9c5e (T) tick_nohz_idle_exit+158 
   c10a52ef (T) cpu_startup_entry+287

CPU1:
   0xc1652607 <_raw_spin_lock_irqsave+71>
   0xc1077bac <lock_hrtimer_base.isra.13+28>
   0xc1078259 <__hrtimer_start_range_ns+41> ; <= acquires a timer base spinlock (3) @ http://lxr.free-electrons.com/source/kernel/hrtimer.c#L979
   0xc1080203 <start_bandwidth_timer+99>
   0xc108f6ec <enqueue_task_rt+316>         ; <= acquires a rt_bandwidth spinlock
   0xc107e1b5 <enqueue_task+69>
   0xc1080ac2 <activate_task+34>
   0xc1080ddc <ttwu_do_activate.constprop.87+44>
   0xc108298d <try_to_wake_up+301>          ; <= acquires a runqueue spinlock @ http://lxr.free-electrons.com/source/kernel/sched/core.c#L1561
   0xc1082abf <wake_up_process+31>
   0xc10e7045 <watchdog_timer_fn+53>
   0xc1077d7f <__run_hrtimer+111>
   0xc10789a5 <hrtimer_interrupt+501>
   0xc1043dc5 <hpet_interrupt_handler+21>
   0xc10a5ba5 <handle_irq_event_percpu+53>
   0xc10a5d41 <handle_irq_event+49>
   0xc10a8036 <handle_edge_irq+102>
   0xc1011884 <handle_irq+116>
   0xc165a26c <do_IRQ+60>
   0xc165a033 <common_interrupt+51>
   0xc1016dcc <default_idle+28>
   0xc1016e8a <amd_e400_idle+58>
   0xc10174f6 <arch_cpu_idle+38>
   0xc10a5399 <cpu_startup_entry+457>
   0xc10388a8 <start_secondary+520>

CPU2:
   0xc1652541 <_raw_spin_lock+49>
   0xc108f6d6 <enqueue_task_rt+294>         ; <= acquires a rt_bandwidth spinlock @ http://lxr.free-electrons.com/source/kernel/sched/rt.c#L57
   0xc107e1b5 <enqueue_task+69>
   0xc1080ac2 <activate_task+34>
   0xc1080ddc <ttwu_do_activate.constprop.87+44>
   0xc108298d <try_to_wake_up+301>          ; <= acquires a runqueue spinlock (4) @ http://lxr.free-electrons.com/source/kernel/sched/core.c#L1561
   0xc1082abf <wake_up_process+31>
   0xc10e7045 <watchdog_timer_fn+53>
   0xc1077d7f <__run_hrtimer+111>
   0xc10789a5 <hrtimer_interrupt+501>
   0xc1043dc5 <hpet_interrupt_handler+21>
   0xc10a5ba5 <handle_irq_event_percpu+53>
   0xc10a5d41 <handle_irq_event+49>
   0xc10a8036 <handle_edge_irq+102>
   0xc1011884 <handle_irq+116>
   0xc165a26c <do_IRQ+60>
   0xc165a033 <common_interrupt+51>
   0xc1016dcc <default_idle+28>
   0xc1016e8a <amd_e400_idle+58>
   0xc10174f6 <arch_cpu_idle+38>
   0xc10a5399 <cpu_startup_entry+457>
   0xc10388a8 <start_secondary+520>

Note CONFIG_RT_GROUP_SCHED = n by default so there's a single rt_bandwidth entity in the kernel.
If runqueues (1) and (4) and timer bases (2) and (3) are the same the following deadlock occurs:

CPU0                                                         CPU1                                                             CPU2
----                                                         ----                                                             ----
lock_hrtimer_base(timer, &flags); @ hrtimer_try_to_cancel()  raw_spin_lock(&rq->lock); @ try_to_wake_up()                     raw_spin_lock(&rq->lock); @ try_to_wake_up()
raw_spin_lock(&rq->lock); @ try_to_wake_up()                 raw_spin_lock(&rt_b->rt_runtime_lock); @ start_rt_bandwidth()
                                                             lock_hrtimer_base(timer, &flags); @  __hrtimer_start_range_ns()  raw_spin_lock(&rt_b->rt_runtime_lock); @ start_rt_bandwidth()

The CPU0 waits for the CPU2 to release the runqueue lock, the CPU2 waits for the CPU1 to release the rt_bandwidth lock,
the CPU1 waits for CPU0 to release the hrtimer base lock => deadlock.

Note You need to log in before you can comment on or make changes to this bug.