Bug 45091

Summary: linux-3.4-rc7: rcu_sched self-detected stall on CPU
Product: Memory Management Reporter: Martin Mokrejs (mmokrejs)
Component: OtherAssignee: Andrew Morton (akpm)
Status: RESOLVED INSUFFICIENT_DATA    
Severity: normal CC: alan
Priority: P1    
Hardware: All   
OS: Linux   
URL: https://lkml.org/lkml/2012/7/19/508
Kernel Version: linux-3.4-rc7 Subsystem:
Regression: No Bisected commit-id:
Attachments: messages

Description Martin Mokrejs 2012-07-23 18:44:22 UTC 
Created attachment 76021 [details]
messages

I am opening a bug here so that I can attach old logs and make them accessible to those interested in the thread from https://lkml.org/lkml/2012/7/19/508 .

Hi,
  first of all I am a pure linux kernel user, not developer. I have no idea
what this bug is about. My impression from your email is that serial driver is
or could be involved ...
  I hit this bug on a a normal laptop (Dell Vostro 3550) with a 3G usb-stick modem connected to it using the serial option driver.
  Originally I thought it was related to file copy from an external HDD drive to itself into another directory, just to stress the usb3.0 driver and the onboard USB3 chip from TexasInstruments.

  Sorry that this bug follows somewhere in the middle of the thread, I just want to place the /var/log/messages somewhere. ;-)



# grep RCU /var/log/messages
May  7 10:52:46 vostro kernel: [    0.000000] Hierarchical RCU implementation.
May  7 10:52:46 vostro kernel: [    0.000000]   RCU dyntick-idle grace-period acceleration is enabled.
...
Jun 14 21:45:44 vostro kernel: [    0.000000] Hierarchical RCU implementation.
Jun 14 21:45:44 vostro kernel: [    0.000000]   CONFIG_RCU_FANOUT set to non-default value of 32
Jun 14 21:45:44 vostro kernel: [    0.000000]   RCU dyntick-idle grace-period acceleration is enabled.
...
until my current 3.4.4 instance.

# grep CONFIG_RCU_CPU_STALL_TIMEOUT /usr/src/linux-3.4.4/.config
CONFIG_RCU_CPU_STALL_TIMEOUT=60
#
# grep CONFIG_RCU_FANOUT /usr/src/linux-3.4.4/.config
CONFIG_RCU_FANOUT=32
# CONFIG_RCU_FANOUT_EXACT is not set
#

I deleted the old kernel but provided I did not fiddle much meanwhile with their .config
then I probably had something like:

# grep CONFIG_RCU_FANOUT /usr/src/linux-3.2.14/.config
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
#


Now, when I look into the /var/log/message for the kernel version when the 'non-default value of 32'
message appeared for the first time, I see:

Jun 14 21:45:44 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #6 SMP Thu Jun 14 21:27:12 MEST 2012
Jun 14 21:45:44 vostro kernel: [    0.000000] Command line: root=/dev/sda5 pciehp.pciehp_debug=1

True, I do not use the serial console anymore ... But it must be related to some .config value because
I did not have this message for all 3.4.2 instances:

# grep 'Linux version' /var/log/messages
May  7 10:52:46 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 10:56:02 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 11:03:32 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 14:04:34 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 14:11:56 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 14:48:32 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 15:26:22 vostro kernel: [    0.000000] Linux version 3.4.0-rc5-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #20 SMP Sat May 5 00:56:44 MEST 2012
May  7 20:30:02 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  7 20:34:59 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  7 20:38:37 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  7 20:55:42 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  7 21:33:24 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  7 22:35:32 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May  8 09:59:24 vostro kernel: [    0.000000] Linux version 3.4.0-rc6-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 7 20:08:38 MEST 2012
May 14 18:37:29 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 14 18:23:41 MEST 2012
May 14 20:36:00 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 14 18:23:41 MEST 2012
May 14 20:52:43 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 14 18:23:41 MEST 2012
May 14 21:05:15 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #1 SMP Mon May 14 18:23:41 MEST 2012
May 15 01:06:17 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue May 15 00:08:38 MEST 2012
May 15 08:43:47 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue May 15 00:08:38 MEST 2012
May 17 19:12:01 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #5 SMP Thu May 17 00:55:06 MEST 2012
May 18 22:01:22 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #5 SMP Thu May 17 00:55:06 MEST 2012
May 23 00:08:48 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #7 SMP Tue May 22 11:26:19 MEST 2012
May 23 10:51:52 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #7 SMP Tue May 22 11:26:19 MEST 2012
May 23 16:47:01 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #7 SMP Tue May 22 11:26:19 MEST 2012
May 23 18:02:24 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #7 SMP Tue May 22 11:26:19 MEST 2012
May 23 18:46:53 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 23 18:51:00 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 25 10:13:53 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 25 10:26:17 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 09:18:25 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 10:06:31 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 10:46:18 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.3.6 (Gentoo 4.3.6-r1 p1.0, pie-10.1.5) ) #9 SMP Wed May 30 10:39:21 MEST 2012
May 30 11:05:41 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.3.6 (Gentoo 4.3.6-r1 p1.0, pie-10.1.5) ) #9 SMP Wed May 30 10:39:21 MEST 2012
May 30 11:20:50 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 11:29:56 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 15:51:41 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
May 30 15:55:50 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.3.6 (Gentoo 4.3.6-r1 p1.0, pie-10.1.5) ) #9 SMP Wed May 30 10:39:21 MEST 2012
Jun  6 20:34:21 vostro kernel: [    0.000000] Linux version 3.4.0-rc7-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Wed May 23 18:36:50 MEST 2012
Jun  6 20:39:27 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #2 SMP Fri Jun 1 10:16:13 MEST 2012
Jun  8 09:34:11 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #2 SMP Fri Jun 1 10:16:13 MEST 2012
Jun  8 21:36:20 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #2 SMP Fri Jun 1 10:16:13 MEST 2012
Jun 10 20:56:25 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #2 SMP Fri Jun 1 10:16:13 MEST 2012
Jun 11 22:06:42 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #3 SMP Mon Jun 11 19:32:03 MEST 2012
Jun 12 13:04:54 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Mon Jun 11 22:43:51 MEST 2012
Jun 12 17:58:41 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Mon Jun 11 22:43:51 MEST 2012
Jun 13 17:07:57 vostro kernel: [    0.000000] Linux version 3.4.0-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #3 SMP Mon Jun 11 19:32:03 MEST 2012
Jun 14 12:46:27 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Mon Jun 11 22:43:51 MEST 2012
Jun 14 17:12:03 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Mon Jun 11 22:43:51 MEST 2012

Jun 14 21:45:44 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #6 SMP Thu Jun 14 21:27:12 MEST 2012
Jun 15 00:35:14 vostro kernel: [    0.000000] Linux version 3.4.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #7 SMP Thu Jun 14 22:07:06 MEST 2012
Jun 19 14:57:10  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #2 SMP Tue Jun 19 13:49:49 MEST 2012
Jun 19 15:58:36  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 20 09:15:55  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 21 22:04:10  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 23 20:51:04  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 24 20:59:19  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 25 09:14:26  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 25 11:53:42  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jun 28 14:30:47  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jul  1 19:49:07  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jul  2 12:11:22  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jul  4 09:00:42  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jul  5 00:01:06  kernel: [    0.000000] Linux version 3.4.3-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #4 SMP Tue Jun 19 15:56:04 MEST 2012
Jul  7 10:25:26  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #1 SMP Thu Jul 5 19:43:01 MEST 2012
Jul  8 17:21:49  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #1 SMP Thu Jul 5 19:43:01 MEST 2012
Jul 10 13:04:42  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #2 SMP Mon Jul 9 01:01:45 MEST 2012
Jul 10 21:17:26  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #2 SMP Mon Jul 9 01:01:45 MEST 2012
Jul 13 10:00:25  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #3 SMP Thu Jul 12 20:46:57 MEST 2012
Jul 13 17:17:48  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #4 SMP Fri Jul 13 14:07:08 MEST 2012
Jul 16 11:32:06  kernel: [    0.000000] Linux version 3.4.4-default (root@vostro) (gcc version 4.6.3 (Gentoo 4.6.3 p1.3, pie-0.5.1) ) #4 SMP Fri Jul 13 14:07:08 MEST 2012



Going back through older /var/log/messages* files I see that a different .config
or behavior I used to have with:

# gzip -dc /var/log/messages.6.gz | grep RCU
Mar 21 09:59:57 vostro kernel: Hierarchical RCU implementation.
Mar 21 10:10:05 vostro kernel: Hierarchical RCU implementation.
Mar 21 21:41:53 vostro kernel: Hierarchical RCU implementation.
Mar 22 13:52:16 vostro kernel: Hierarchical RCU implementation.
Mar 22 15:00:44 vostro kernel: Hierarchical RCU implementation.
Mar 22 16:59:18 vostro kernel: Hierarchical RCU implementation.
# gzip -dc /var/log/messages.6.gz | grep 'Linux version'
Mar 21 09:59:57 vostro kernel: Linux version 3.2.11-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Tue Mar 20 23:40:00 MET 2012
Mar 21 10:10:05 vostro kernel: Linux version 3.2.2-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.0, pie-0.4.7) ) #25 SMP Fri Feb 10 16:21:55 MET 2012
Mar 21 21:41:53 vostro kernel: Linux version 3.2.11-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #8 SMP Tue Mar 20 23:40:00 MET 2012
Mar 22 13:52:16 vostro kernel: Linux version 3.2.11-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #9 SMP Thu Mar 22 13:47:20 MET 2012
Mar 22 15:00:44 vostro kernel: Linux version 3.2.11-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #9 SMP Thu Mar 22 13:47:20 MET 2012
Mar 22 16:59:18 vostro kernel: Linux version 3.2.11-default (root@vostro) (gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) ) #11 SMP Thu Mar 22 15:40:45 MET 2012
# 

Maybe I just had not enabled dynamic ticks?




Back to the rcu_stall logs, I had them only at these times:

# grep 'rcu_sched' /var/log/messages
May 18 20:19:45 vostro kernel: [90345.151040] INFO: rcu_sched self-detected stall on CPU { 2}  (t=6000 jiffies)
May 18 20:22:45 vostro kernel: [90524.911374] INFO: rcu_sched self-detected stall on CPU { 2}  (t=24003 jiffies)
May 18 20:25:45 vostro kernel: [90704.671711] INFO: rcu_sched self-detected stall on CPU { 2}  (t=42006 jiffies)
May 18 20:28:45 vostro kernel: [90884.429174] INFO: rcu_sched self-detected stall on CPU { 2}  (t=60009 jiffies)
May 18 20:31:45 vostro kernel: [91064.189507] INFO: rcu_sched self-detected stall on CPU { 2}  (t=78012 jiffies)
May 18 20:34:45 vostro kernel: [91243.949835] INFO: rcu_sched self-detected stall on CPU { 2}  (t=96015 jiffies)
May 18 20:37:45 vostro kernel: [91423.710165] INFO: rcu_sched self-detected stall on CPU { 2}  (t=114018 jiffies)
May 18 20:40:46 vostro kernel: [91603.470496] INFO: rcu_sched self-detected stall on CPU { 2}  (t=132021 jiffies)
May 18 20:43:46 vostro kernel: [91783.230826] INFO: rcu_sched self-detected stall on CPU { 2}  (t=150024 jiffies)
May 18 20:46:46 vostro kernel: [91962.991156] INFO: rcu_sched self-detected stall on CPU { 2}  (t=168027 jiffies)
May 18 20:49:46 vostro kernel: [92142.751488] INFO: rcu_sched self-detected stall on CPU { 2}  (t=186030 jiffies)
May 18 20:52:46 vostro kernel: [92322.511818] INFO: rcu_sched self-detected stall on CPU { 2}  (t=204033 jiffies)
May 18 20:55:46 vostro kernel: [92502.272148] INFO: rcu_sched self-detected stall on CPU { 2}  (t=222036 jiffies)
May 18 20:58:46 vostro kernel: [92682.032478] INFO: rcu_sched self-detected stall on CPU { 2}  (t=240039 jiffies)
May 18 21:01:46 vostro kernel: [92861.792809] INFO: rcu_sched self-detected stall on CPU { 2}  (t=258042 jiffies)
May 18 21:04:46 vostro kernel: [93041.553139] INFO: rcu_sched self-detected stall on CPU { 2}  (t=276045 jiffies)
May 18 21:07:46 vostro kernel: [93221.313470] INFO: rcu_sched self-detected stall on CPU { 2}  (t=294048 jiffies)
May 18 21:10:46 vostro kernel: [93401.071147] INFO: rcu_sched self-detected stall on CPU { 2}  (t=312051 jiffies)
May 18 21:13:46 vostro kernel: [93580.831473] INFO: rcu_sched self-detected stall on CPU { 2}  (t=330054 jiffies)
May 18 21:16:46 vostro kernel: [93760.591798] INFO: rcu_sched self-detected stall on CPU { 2}  (t=348057 jiffies)
May 18 21:19:46 vostro kernel: [93940.352124] INFO: rcu_sched self-detected stall on CPU { 2}  (t=366060 jiffies)
May 18 21:22:46 vostro kernel: [94120.112449] INFO: rcu_sched self-detected stall on CPU { 2}  (t=384063 jiffies)
May 18 21:25:46 vostro kernel: [94299.872775] INFO: rcu_sched self-detected stall on CPU { 2}  (t=402066 jiffies)
May 18 21:28:46 vostro kernel: [94479.633101] INFO: rcu_sched self-detected stall on CPU { 2}  (t=420069 jiffies)
May 18 21:31:46 vostro kernel: [94659.393426] INFO: rcu_sched self-detected stall on CPU { 2}  (t=438072 jiffies)
May 18 21:34:46 vostro kernel: [94839.153751] INFO: rcu_sched self-detected stall on CPU { 2}  (t=456075 jiffies)
May 18 21:37:46 vostro kernel: [95018.914076] INFO: rcu_sched self-detected stall on CPU { 2}  (t=474078 jiffies)
May 18 21:40:46 vostro kernel: [95198.674404] INFO: rcu_sched self-detected stall on CPU { 2}  (t=492081 jiffies)
May 18 21:43:46 vostro kernel: [95378.434727] INFO: rcu_sched self-detected stall on CPU { 2}  (t=510084 jiffies)
May 18 21:49:18 vostro kernel: [95558.195053] INFO: rcu_sched self-detected stall on CPU { 2}  (t=528087 jiffies)
May 18 21:49:46 vostro kernel: [95737.955380] INFO: rcu_sched self-detected stall on CPU { 2}  (t=546090 jiffies)
May 18 21:52:46 vostro kernel: [95917.712930] INFO: rcu_sched self-detected stall on CPU { 2}  (t=564093 jiffies)


If you found out something of interest in the above information and need some .config
I possibly had at some time please let me know. At the moment I have these
kernels with config.gz in them:

-rw-r--r--  1 root root 4458896 Mar 20 19:51 vmlinuz-2.6.32.56
-rw-r--r--  1 root root 4118560 Mar 26 15:23 vmlinuz-2.6.32.59
-rw-r--r--  1 root root 3806688 Mar 14 02:10 vmlinuz-2.6.39.4
-rw-r--r--  1 root root 3214528 Feb  4 12:53 vmlinuz-2.6.39.4-default
-rw-r--r--  1 root root 3670000 Jun 15 11:45 vmlinuz-3.0.34
-rw-r--r--  1 root root 3642832 Jun 19 13:09 vmlinuz-3.0.35
-rw-r--r--  1 root root 3937776 Mar 22 16:57 vmlinuz-3.2.11
-rw-r--r--  1 root root 3932272 Apr 11 01:37 vmlinuz-3.2.12
-rw-r--r--  1 root root 3921904 Mar 31 19:05 vmlinuz-3.2.12-no-debug
-rw-r--r--  1 root root 3932400 Mar 31 02:16 vmlinuz-3.2.13
-rw-r--r--  1 root root 3940240 May  4 10:41 vmlinuz-3.2.14
-rw-r--r--  1 root root 4116160 Feb 10 16:23 vmlinuz-3.2.2
-rw-r--r--  1 root root 3341056 Feb  4 12:54 vmlinuz-3.2.2-default
-rw-r--r--  1 root root 3769120 Jul 12 22:05 vmlinuz-3.2.23
-rw-r--r--  1 root root 3970912 Mar 20 13:12 vmlinuz-3.2.5
-rw-r--r--  1 root root 3969536 Mar 14 00:54 vmlinuz-3.2.9
-rw-r--r--  1 root root 3602480 May  1 00:38 vmlinuz-3.3
-rw-r--r--  1 root root 3598912 May  1 00:42 vmlinuz-3.3.2
-rw-r--r--  1 root root 3854448 Jun 14 22:08 vmlinuz-3.3.4
-rw-r--r--  1 root root 3861680 Jun 15 11:30 vmlinuz-3.3.8
-rw-r--r--  1 root root 4028032 Apr 11 18:25 vmlinuz-3.4-rc2
-rw-r--r--  1 root root 3654064 Apr 16 11:22 vmlinuz-3.4-rc3
-rw-r--r--  1 root root 3652672 Apr 18 20:13 vmlinuz-3.4-rc3-jeff-garzik
-rw-r--r--  1 root root 3653232 Apr 17 09:12 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d
-rw-r--r--  1 root root 3654528 Apr 18 19:33 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d_with_yinghai
-rw-r--r--  1 root root 3656560 Apr 18 19:35 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d_with_yinghai-jeff-garzik
-rw-r--r--  1 root root 3656464 Apr 19 20:05 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d_with_yinghai-jeff-garzik__SATA-speed
-rw-r--r--  1 root root 3656480 Apr 21 01:33 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d_with_yinghai-jeff-garzik__SATA-speed_extra_pciehp
-rw-r--r--  1 root root 3647360 Apr 19 00:34 vmlinuz-3.4-rc3-reverted-486b10b9f43500741cd63a878d0ef23cd87fc66d_with_yinghai-jeff-garzik__no_CONFIG_HOTPLUG_PCI_PCIE
-rw-r--r--  1 root root 3627568 Apr 26 23:59 vmlinuz-3.4-rc3_no_xhci
-rw-r--r--  1 root root 3835808 May  5 16:14 vmlinuz-3.4-rc5
-rw-r--r--  1 root root 3834592 May  4 14:44 vmlinuz-3.4-rc5_no__CONFIG_HOTPLUG_PCI_ACPI
-rw-r--r--  1 root root 3834288 May  7 20:12 vmlinuz-3.4-rc6
-rw-r--r--  1 root root 3844064 May 14 18:01 vmlinuz-3.4-rc6__with_IOAT_DMA
-rw-r--r--  1 root root 3858480 May 19 18:12 vmlinuz-3.4-rc7__with_IOAT_DMA
-rw-r--r--  1 root root 3853488 May 22 11:27 vmlinuz-3.4-rc7__with_IOAT_DMA_and_touchpad_LED
-rw-r--r--  1 root root 3854784 May 23 18:38 vmlinuz-3.4-rc7__with_IOAT_DMA_and_touchpad_LED_and_rss_patch
-rw-r--r--  1 root root 3828112 May 30 10:44 vmlinuz-3.4-rc7__with_IOAT_DMA_and_touchpad_LED_and_rss_patch2
-rw-r--r--  1 root root 3847952 Jun 11 19:33 vmlinuz-3.4.0
-rw-r--r--  1 root root 3835824 Jun 19 15:21 vmlinuz-3.4.2
-rw-r--r--  1 root root 3870048 Jul  5 19:32 vmlinuz-3.4.3
-rw-r--r--  1 root root 3854784 Jul 13 14:08 vmlinuz-3.4.4


Best,
Martin

Mike Galbraith wrote:
> On Fri, 2012-07-20 at 16:30 +0800, Michael Wang wrote: 
>> On 07/20/2012 03:00 PM, Mike Galbraith wrote:
>>> On Fri, 2012-07-20 at 11:09 +0800, Michael Wang wrote: 
>>>> Hi, Mike, Martin, Dan
>>>>
>>>> I'm currently taking an eye on the rcu stall issue which was reported by
>>>> you in the mail:
>>>>
>>>> rcu: endless stalls
>>>>    From: Mike Galbraith
>>>> linux-3.4-rc7: rcu_sched self-detected stall on CPU
>>>>    From: Martin Mokrejs
>>>> RCU stalls in linux-next
>>>>    From: Dan Carpenter
>>>>
>>>> I try to reproduce the issue on my X86 server with 12 cpu
>>>
>>> The 'endless stalls' box was 341.33333 times larger.  Dunno if you can
>>> even set a serial port slow enough to approximate all cores trying to
>>> gripe through a single pinhole simultaneously.
>>
>> Hi, Mike
>>
>> Thanks for your reply.
>>
>> So you mean this issue is still existing on you box and you can see it
>> without doing any special things?
>
> It's not my box (thank god).  It was initially triggered by tasks
> exiting simultaneously on all cores.  They jammed up, endless stall
> followed.
>
>> I just want to try to reproduce it but it's impossible for me to get
>> some hardware as yours...
>>
>> So is there any idea on how to reproduce it on normal hardware?
>
> No, AFAIK this problem is restricted to size XXL boxen, with all the
> joys that come along with having way too many CPUs.
>
> -Mike