Bug 41442

Should be t=6000, t=24030, t=42060. On Debian shiped kernel, I have call trace dumps every 60 seconds. I do not understand why on my custom optimized kernel, I have this INFO not every 60 (I still have RCU_CPU_STALL_TIMEOUT=60 in kernel config).

Looks that this topic have been seen on linux-kernel o xen-devel mailing list few weeks ago.

http://lists.xensource.com/archives/html/xen-devel/2011-07/msg00331.html
http://osdir.com/ml/linux-kernel/2011-07/msg00036.html

Few interesting things. System is uniprocess. But both kernels are built with SMP support (to have DOM0 support). Also I have disabled CONFIG_RCU_FAST_NO_HZ, however have CONFIG_NO_HZ=1. I also boot with "threadirqs" kernel boot option.

Created attachment 69512 [details]
Prevent boot-time RCU grace periods

Is this the guest OS getting the stalls?  If so, is the guest running on a single emulated CPU or more than one?

Either way, does this patch help?

No, this is DOM0 running on single CPU. (evne if guest, it couldn't be more than 1, as I have only one core in system).

Help doesn't helped. :(

Oops. I wanted to say that patch doesn't helped with my problem. I still have RCU stall, and no progress in booting.

Any other idea? Is this is happening to other people, or is this just me? If so why?

This bug have somehow high priority for me, as it completely prevents me from booting system on new kernels.

Thanks.

There are a couple of additional things to try:

1. Try disabling the rcu_cpu_stall_suppress module parameter at boot time via the kernel boot arguments.  If your kernel boots just fine, then we have a problem with RCU (see #2 below), otherwise RCU is correctly pointing us at a problem (see #3 below).

2. Pull the rcu/next commit series from the -rcu git tree at git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-2.6-rcu.git -- I gave you the patch that seemed to me to be the most likely to address the problem if RCU is at fault, but perhaps one of the remaining patches will help.

3. Examine the code called from __reserver_region_with_split() and see if there is an infinite loop.  This sort of thing is what the RCU stall warnings were designed to detect.  See Documentation/RCU/stallwarn.txt for more information.  Sorry I cannot help more here, as I am not a Xen expert.

I was trying booting with rcu_cpu_stall_suppress=1 and rcu_cpu_stall_suppress=0, but non helps when booting on Xen. I did not wait, so do not know if message is printed, but kernel looked to be stalled anyway.

I do not see reason why RCU stall detection here was buggy. From the beginning, I was thinking, that it is infinite loop in reserve_region_with_split (basing on call trace), just wanted someone more aware of this code to look into it, especially that bug only appears under Xen.


I will disable memory balooning support in kernel, to see if this can be a problem.  I also have some problem with locating reserver_region_with_split function, I see where it is called from, but I cannot find its definition in kernel tree. :(

Witold,

Can you:
 1). Attach a serial console to your laptop and on the Xen command line add:
"console=com1 com1=115200,8n1 loglvl=all" and on the Linux command line:
"console=hvc0 debug loglevel=8 initcall_debug" and attach the whole serial console to this bug?
 2). Bug says "regression" - what version of Xen and Linux kernel worked on your machine before this?

1) Unfortunately, this laptop doesn't have serial console port (at least without port replicator, which I do not have). I will try booting same xen and kernel on desktop machine.

2) Yes, this is regression, however I'm probably comparing here to the Debian kernel, which isn't based on pv_ops, but older xen sources.

I was trying to do test multiboot booting using qemu, however I do not know to pass additional modules (there is only kernel/initrd options).

(In reply to comment #11)
> 1) Unfortunately, this laptop doesn't have serial console port (at least
> without port replicator, which I do not have). I will try booting same xen
> and
> kernel on desktop machine.

They are pretty cheap on eBay. But yes, please try the same environment on a desktop.

> 
> 2) Yes, this is regression, however I'm probably comparing here to the Debian
> kernel, which isn't based on pv_ops, but older xen sources.

That is not a regression. A regression on this bugzilla system is against stock (ftp.kernel.org) Linux kernels.

> 
> I was trying to do test multiboot booting using qemu, however I do not know
> to
> pass additional modules (there is only kernel/initrd options).

Can you make an disk image that has your normal partition?

> Can you make an disk image that has your normal partition?

Nice trick, it was easy!

However, in qemu bug doesn't appear.

I cannot reproduce this bug with 3.1.0-rc4-00131-g9e79e3e or with Debian's 3.0.0-1-686-pae. Both boots without problem.

I guess, will need serial port. Eventually delay each line of debug output somehow to write it down. :)

Well, buying will be simpler. There are two port replicators on allegro. 12.90 zł :) Will report back in few days.

There is a kernel commandline to delay message printing. Also if you can upload it to youtube (readable) or take a couple of pictures, writing it down will not be necessary.

I'm aware of kernel message delaying (boot_delay=MSECS kernel command line option). I even have modified printk.c to have some other usefull delaying mechanisms. 

However I do not know way to delay xen hypervisor console messages.

netconsol could work too - you would have to make sure to build your e1000 in the kernel.

> However I do not know way to delay xen hypervisor console messages.

You can do this: "console=vga vga=text-80x50,keep console_to_ring sync_console loglvl=all" on the Xen hypervisor line.

And naturally on your Linux "console=hvc0 debug loglevel=8 earlyprintk=xen" and add that slow-down config option - that should make the whole output be much much slower.

Well, I have some photos http://smp.if.uj.edu.pl/~baryluk/xen-debuging-2011-09-02/ but movie was of very bad quality. Unfortunetly not everything is delayed, so I still do not have time to take photo. Photos show multiple boots actually, and one of them end actually in different error. I can also reproduce it (it is probably bug due to the console and message delaying enabled). Will have serial console in Tuesday. However can try taking more photos for better coverage of boot process :)

> netconsol could work too - you would have to make sure to build your e1000 in
the kernel.
netconsole will not work. my computer is using tg3 module, and it needs firmware from userspace (udev) to be functional.

Looking at your pictures, I think the rcu_sched_state is printing its message b/c of:
http://smp.if.uj.edu.pl/~baryluk/xen-debuging-2011-09-02/DSC01919.JPG
which has:
BUG: soft lockup - CPU#0 stuck for 23s!

But before we got that route to figure out what went wrong there, lets see what the serial console with all the fancy debug options (http://wiki.xensource.com/xenwiki/XenParavirtOps) tells us.

In regards to the firmware - you can make the kernel have the firmware built in. Not sure exactly how - but there is a thread on LKML about radeon and firmware - and it is a similar concept.

I have now working serial console (it was disabled in BIOS, which prevented early boot from using it, and I forgot password to BIOS, so I couldn't change enable it back), but actually I cannot reproduce problem anymore. I tested at 3.3-rc2+ or 3.2.4. Both boots, and works on Xen.

Now with working serial console, I could try to bisect problem, but because it is working now, it is of small priority (I have other testing and error fixing to do first).
 
@19, Konrad, as of building in a firmware into kernel, I already discovered this. I will remember this in future, when trying to use netconsole!

Thx for reporting back. I will close it as unreproducible.